Disaster Recovery Planning: Building Organizational Resilience

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

BUSINESS IMPACT ANALYSIS WORKSHEET
═══════════════════════════════════════════════════════════════════
 
BUSINESS FUNCTION: Online Order Processing
DEPARTMENT: E-Commerce Operations
OWNER: VP of Digital Commerce
 
DESCRIPTION:
Customer-facing order placement, payment processing, and order 
confirmation for all e-commerce channels (web, mobile, marketplace).
 
SUPPORTING IT SYSTEMS:
├── Order Management System (OMS)
├── Payment Gateway Integration
├── Product Catalog Database
├── Inventory Management System
├── Customer Database
└── Email/Notification Services
 
IMPACT ANALYSIS:
┌────────────────┬─────────────────────────────────────────────────┐
│ Duration       │ Impact                                          │
├────────────────┼─────────────────────────────────────────────────┤
│ 0-1 hour       │ Minor: Some failed transactions, customer       │
│                │ frustration, ~$25K revenue loss                 │
├────────────────┼─────────────────────────────────────────────────┤
│ 1-4 hours      │ Moderate: Significant revenue loss (~$100K),    │
│                │ social media complaints, competitor capture     │
├────────────────┼─────────────────────────────────────────────────┤
│ 4-24 hours     │ Severe: Major revenue loss (~$600K), press      │
│                │ coverage, customer defection, SLA penalties     │
├────────────────┼─────────────────────────────────────────────────┤
│ 1-7 days       │ Critical: Existential impact (~$4M), executive  │
│                │ escalation, regulatory scrutiny, brand damage   │
├────────────────┼─────────────────────────────────────────────────┤
│ 1+ month       │ Catastrophic: Business viability threatened     │
└────────────────┴─────────────────────────────────────────────────┘
 
MAXIMUM TOLERABLE DOWNTIME (MTD): 4 hours
RECOVERY TIME OBJECTIVE (RTO): 2 hours (50% safety margin to MTD)
RECOVERY POINT OBJECTIVE (RPO): 15 minutes
 
DATA LOSS IMPACT:
├── Lost orders require manual re-entry from payment processor
├── Customer trust impact if order confirmations not sent
├── Inventory sync issues if gap exceeds 15 minutes
└── Financial reconciliation complexity
 
DEPENDENCIES:
├── CRITICAL: Payment Gateway (external SaaS - degraded mode possible)
├── CRITICAL: Database Cluster
├── HIGH: Inventory System (can operate briefly without)
├── MEDIUM: Email Service (can queue for later)
└── LOW: Analytics (can delay indefinitely)
 
RECOVERY PRIORITY: TIER 1 (First to recover)
REVIEWED BY: [Name, Title] DATE: [Date]
APPROVED BY: [Executive Sponsor] DATE: [Date]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

DR STRATEGY FRAMEWORK
═══════════════════════════════════════════════════════════════════
 
TIER 1: MISSION CRITICAL (RTO < 1 hour, RPO < 15 min)
────────────────────────────────────────────────────
SYSTEMS: Order Processing, Payment, Core Database
 
STRATEGY: Active-Active Multi-Region
├── Primary Region: US-East (Virginia)
├── Secondary Region: US-West (Oregon)
├── Architecture:
│   ├── Global load balancer (Route53/Cloud DNS)
│   ├── Application layer in both regions
│   ├── Database: Aurora Global Database (sync replication)
│   ├── Cache: Replicated Redis clusters
│   └── File Storage: S3 Cross-Region Replication
│
├── Failover Mechanism: Automated health-check triggered
├── Failback: Manual with validation
├── Data Sync: Continuous, sub-second lag monitored
└── Cost: $X,XXX/month for standby infrastructure
 
═══════════════════════════════════════════════════════════════════
 
TIER 2: BUSINESS CRITICAL (RTO < 4 hours, RPO < 1 hour)
───────────────────────────────────────────────────────
SYSTEMS: CRM, Inventory, Analytics, Internal Tools
 
STRATEGY: Warm Standby with Continuous Replication
├── Primary Region: US-East
├── DR Region: US-West
├── Architecture:
│   ├── Reduced-capacity application instances (scaled on activation)
│   ├── Database replicas (async, monitored lag)
│   ├── Hourly configuration backups
│   └── Pre-staged AMIs/container images
│
├── Failover Mechanism: Semi-automated (human approval, scripted execution)
├── Failback: Scheduled maintenance window
├── Data Sync: Hourly snapshots + continuous log shipping
└── Cost: $X,XXX/month for standby infrastructure
 
═══════════════════════════════════════════════════════════════════
 
TIER 3: STANDARD (RTO < 24 hours, RPO < 4 hours)
────────────────────────────────────────────────
SYSTEMS: Development, Staging, Non-critical Internal Apps
 
STRATEGY: Cold Standby with Daily Backup
├── Primary Region: US-East
├── DR Region: US-West (backup storage only)
├── Architecture:
│   ├── No running infrastructure in DR region
│   ├── Daily full backups copied cross-region
│   ├── Infrastructure-as-Code for rapid provisioning
│   └── Documented manual procedures
│
├── Failover Mechanism: Manual provisioning from code + restore
├── Failback: Rebuild in primary when available
├── Data Sync: Daily scheduled backup
└── Cost: Storage only (~$XXX/month)
 
═══════════════════════════════════════════════════════════════════
 
TIER 4: NON-CRITICAL (RTO > 1 week, RPO > 24 hours)
───────────────────────────────────────────────────
SYSTEMS: Archives, Historical Data, Legacy Systems
 
STRATEGY: Backup and Restore
├── Backup: Weekly full, daily incremental
├── Storage: Cross-region cold storage (Glacier, Archive)
├── Recovery: Manual on-demand
└── Cost: Minimal storage costs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117

DISASTER RECOVERY RUNBOOK
═══════════════════════════════════════════════════════════════════
 
DOCUMENT METADATA:
├── Document ID: DR-RUNBOOK-001
├── Last Updated: 2024-01-15
├── Approved By: [Name, Title]
├── Next Review: 2024-04-15
└── Version: 3.2
 
───────────────────────────────────────────────────────────────────
SECTION 1: OVERVIEW
───────────────────────────────────────────────────────────────────
 
PURPOSE:
This runbook provides step-by-step procedures for recovering the 
Order Processing System following a disaster affecting the US-East 
primary region.
 
SCOPE:
├── Order Management Application
├── Order Database (PostgreSQL)
├── Redis Cache Layer
├── Integration endpoints (Payment, Inventory)
└── Supporting configuration
 
OUT OF SCOPE:
├── Payment Gateway (separate runbook: DR-RUNBOOK-003)
├── Inventory System (separate runbook: DR-RUNBOOK-004)
└── Customer Authentication (separate runbook: DR-RUNBOOK-002)
 
RECOVERY OBJECTIVES:
├── RTO Target: 60 minutes
├── RPO Target: 15 minutes
└── Dependencies must be recovered first (see Prerequisites)
 
───────────────────────────────────────────────────────────────────
SECTION 2: PREREQUISITES
───────────────────────────────────────────────────────────────────
 
BEFORE EXECUTING THIS RUNBOOK, CONFIRM:
□ Incident Commander has authorized DR activation
□ Payment Gateway DR complete (or degraded mode acceptable)
□ VPN access to DR region established
□ Required credentials available (see Appendix A for vault paths)
□ Communication channels established (Slack #incident-room)
 
REQUIRED ACCESS:
├── AWS Console access (DR account)
├── SSH keys for bastion hosts (stored in: [location])
├── Database admin credentials (stored in: [vault path])
└── DNS management access (Route53)
 
───────────────────────────────────────────────────────────────────
SECTION 3: RECOVERY PROCEDURES
───────────────────────────────────────────────────────────────────
 
PHASE 1: DATABASE RECOVERY (Estimated: 15 minutes)
─────────────────────────────────────────────────
 
STEP 1.1: Verify DR Database Replica Status
EXECUTOR: Database Team Lead
COMMAND:
    $ aws rds describe-db-instances --db-instance-identifier orders-dr
    
EXPECTED OUTPUT: "DBInstanceStatus": "available"
 
IF STATUS IS NOT "available":
    → See Appendix B: Database Troubleshooting
    → STOP and escalate to Database DBA on-call
 
STEP 1.2: Promote DR Database to Primary
EXECUTOR: Database Team Lead
COMMAND:
    $ aws rds promote-read-replica \
        --db-instance-identifier orders-dr
 
EXPECTED OUTPUT: Promotion initiated message
 
WAIT: 5-10 minutes for promotion to complete
VERIFICATION:
    $ aws rds describe-db-instances --db-instance-identifier orders-dr
    Look for: "DBInstanceStatus": "available" 
              "ReadReplicaSourceDBInstanceIdentifier": null
 
□ CHECKPOINT: Database promoted and available
  TIME: ___:___ (target: T+15 min)
 
... [Additional phases continue with same detail level]
 
───────────────────────────────────────────────────────────────────
SECTION 4: VALIDATION
───────────────────────────────────────────────────────────────────
 
POST-RECOVERY VALIDATION CHECKLIST:
□ Database queries returning expected data
□ Application health endpoints returning 200
□ Sample order can be placed end-to-end
□ Payment integration functional
□ Order confirmation emails sending
□ Monitoring and alerting active
□ No error spikes in logging
 
───────────────────────────────────────────────────────────────────
SECTION 5: FAILBACK PROCEDURES
───────────────────────────────────────────────────────────────────
 
[Documented procedures for returning to primary region]
 
───────────────────────────────────────────────────────────────────
APPENDICES
───────────────────────────────────────────────────────────────────
 
APPENDIX A: Credential Locations
APPENDIX B: Common Troubleshooting
APPENDIX C: Escalation Contacts
APPENDIX D: Communication Templates

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

COMMUNICATION MATRIX DURING DR
═══════════════════════════════════════════════════════════════════
 
INTERNAL COMMUNICATIONS:
┌──────────────────────────────────────────────────────────────────┐
│ Audience          │ Channel         │ Frequency    │ Owner       │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ DR Team           │ War Room (Zoom) │ Continuous   │ IC          │
│                   │ Slack #dr-ops   │              │             │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Leadership        │ Email/Slack     │ Every 30 min │ Comms Lead  │
│                   │ Executive brief │              │             │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ All Employees     │ Status page     │ Hourly       │ Comms Lead  │
│                   │ All-hands Slack │              │             │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Support Team      │ Dedicated chan  │ Real-time    │ Ops Lead    │
│                   │ Case updates    │              │             │
└──────────────────────────────────────────────────────────────────┘
 
EXTERNAL COMMUNICATIONS:
┌──────────────────────────────────────────────────────────────────┐
│ Audience          │ Channel         │ Frequency    │ Owner       │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Customers         │ Status Page     │ Every 30 min │ Comms Lead  │
│                   │ Email (major)   │              │             │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Partners/Vendors  │ Direct email    │ As needed    │ Account Mgr │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Media/Press       │ PR Statement    │ If needed    │ PR Team     │
│                   │ Spokesperson    │              │             │
├───────────────────┼─────────────────┼──────────────┼─────────────┤
│ Regulators        │ Formal notice   │ Per require- │ Legal/Compl │
│                   │                 │ ments        │             │
└──────────────────────────────────────────────────────────────────┘
 
ESCALATION THRESHOLDS:
├── 15 min: Initial assessment to leadership
├── 30 min: Customer-facing status page update
├── 1 hour: Executive briefing if not resolved
├── 4 hours: Board notification for major incident
├── 24+ hours: External communications review with PR
└── Any data breach: Immediate legal/compliance involvement

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

QUARTERLY DR PLAN REVIEW CHECKLIST
═══════════════════════════════════════════════════════════════════
 
DOCUMENTATION CURRENCY:
□ All system names match current production naming
□ IP addresses, endpoints, connection strings are current
□ Commands and scripts work without modification
□ Screenshots and diagrams reflect current UI/architecture
□ Referenced tools and access methods are current
 
PERSONNEL AND CONTACTS:
□ All named personnel still in same roles
□ Contact information verified (phone, email, Slack)
□ Alternates/backups identified for each key role
□ New team members added and trained
□ Departed personnel removed
 
INFRASTRUCTURE ALIGNMENT:
□ DR infrastructure matches documented configuration
□ Backup schedules match documented frequencies
□ Replication lag within documented parameters
□ Credentials and access still valid
□ Network connectivity verified
 
TEST RESULTS:
□ Last drill results reviewed
□ Remediation items from last drill completed
□ RTO/RPO achievements documented
□ Any new risks or gaps identified
 
BUSINESS ALIGNMENT:
□ Business criticality tiers still accurate
□ RTO/RPO requirements still appropriate
□ New systems added to appropriate tier
□ Decommissioned systems removed
 
SIGN-OFF:
□ Technical owner approval: _____________ Date: _______
□ Business owner approval: _____________ Date: _______
□ Compliance review: __________________ Date: _______
 
NEXT REVIEW SCHEDULED: _______________

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

CLOUD-NATIVE DR ARCHITECTURE
═══════════════════════════════════════════════════════════════════
 
                        ┌───────────────────────────┐
                        │     Global DNS (Route53)  │
                        │     Health-check based    │
                        │     failover              │
                        └──────────┬────────────────┘
                                   │
               ┌───────────────────┴───────────────────┐
               ▼                                       ▼
   PRIMARY REGION (US-East)              DR REGION (US-West)
   ─────────────────────────             ─────────────────────
   
   ┌─────────────────────┐              ┌─────────────────────┐
   │  Kubernetes Cluster │              │  Kubernetes Cluster │
   │  (EKS/GKE/AKS)      │              │  (Standby or Active)│
   │                     │              │                     │
   │  ┌─────────────────┐│              │  ┌─────────────────┐│
   │  │ App Deployments ││              │  │ App Deployments ││
   │  │ (Running)       ││              │  │ (Scaled down or ││
   │  └─────────────────┘│              │  │  running)       ││
   │                     │              │  └─────────────────┘│
   │  ┌─────────────────┐│              │  ┌─────────────────┐│
   │  │ Ingress/LB      ││              │  │ Ingress/LB      ││
   │  └─────────────────┘│              │  └─────────────────┘│
   └─────────────────────┘              └─────────────────────┘
              │                                    │
              ▼                                    ▼
   ┌─────────────────────┐              ┌─────────────────────┐
   │  Managed Database   │◄────sync────►│  Database Replica  │
   │  (RDS/CloudSQL)     │              │  (Read replica or   │
   │                     │              │   promoted primary) │
   └─────────────────────┘              └─────────────────────┘
              │                                    │
              ▼                                    ▼
   ┌─────────────────────┐              ┌─────────────────────┐
   │  Object Storage     │◄────CRR─────►│  Object Storage    │
   │  (S3/GCS)           │              │  (Replicated)       │
   └─────────────────────┘              └─────────────────────┘
 
INFRASTRUCTURE STORED IN:
├── Git repository (Terraform/Pulumi)
├── Container registry (both regions)
├── Secrets manager (cross-region)
└── GitOps controller (ArgoCD/Flux in each region)