Cross-Region Backup: Geographic Data Protection

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

Network Round-Trip Times (Approximate):
═══════════════════════════════════════════════════════════════════
 
WITHIN REGION (Same Cloud Region, Different AZs):
├── 1-5 ms typical latency
├── Synchronous replication viable
└── 200-1000 sync writes/second achievable
 
SAME CONTINENT (e.g., US East to US West):
├── 60-80 ms latency
├── Synchronous severely impacts write performance
├── 12-16 sync writes/second maximum
└── Asynchronous recommended for most workloads
 
INTERCONTINENTAL (e.g., US to Europe):
├── 80-120 ms latency
├── Synchronous impractical for write-heavy workloads
├── 8-12 sync writes/second maximum
└── Asynchronous required for performance
 
GLOBAL (e.g., US to Asia-Pacific):
├── 150-250 ms latency
├── Synchronous only for extremely low-volume critical writes
├── 4-6 sync writes/second maximum
└── Asynchronous essential
 
┌─────────────────────────────────────────────────────────────────┐
│  EXAMPLE CALCULATION:                                          │
│                                                                 │
│  Application needs 500 writes/second                           │
│  Cross-region latency: 70 ms                                   │
│                                                                 │
│  With synchronous replication:                                 │
│  • Each write takes 70 ms for remote confirm                   │
│  • Single thread: 1000/70 = 14 writes/second max               │
│  • Need 36 parallel writers to achieve 500/sec                 │
│  • BUT: all waiting for same network, contention issues        │
│  • RESULT: Doesn't scale, latency compounds                    │
│                                                                 │
│  With asynchronous replication:                                │
│  • Writes complete in <5 ms locally                           │
│  • 500 writes/second easily achieved                           │
│  • RPO = replication lag (seconds to minutes typically)        │
│  • RESULT: Scalable, but potential data loss window           │
└─────────────────────────────────────────────────────────────────┘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

SCHEDULED BACKUP COPY
═════════════════════════════════════════════════════════════
 
Primary Region                    Secondary Region
┌──────────────────┐              ┌──────────────────┐
│                  │              │                  │
│   Production     │              │   Backup Store   │
│   Database       │              │   (Cold)         │
│                  │              │                  │
│      │           │              │        ▲         │
│      ▼           │              │        │         │
│   Local Backup   │──scheduled───│────────┘         │
│   (nightly)      │    copy      │                  │
│                  │              │                  │
└──────────────────┘              └──────────────────┘
 
Characteristics:
• RPO: Hours to days (backup frequency + transfer time)
• RTO: Hours (need to provision infrastructure in DR region)
• Cost: Low (only pay for storage and periodic transfer)
• Complexity: Low (simple scheduled job)
• Best For: Non-critical systems, compliance archival

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

CONTINUOUS REPLICATION TO STANDBY
═════════════════════════════════════════════════════════════
 
Primary Region                    Secondary Region (Warm)
┌──────────────────┐              ┌──────────────────┐
│                  │              │                  │
│   Production  ◄──┼──traffic────┼──X (standby)     │
│   Database       │              │   Replica        │
│       │          │              │       ▲          │
│       │     continuous          │       │          │
│       └─────replication─────────┼───────┘          │
│                  │              │                  │
│   Application    │              │   Application    │
│   Servers        │              │   Servers        │
│   (active)       │              │   (standby)      │
│                  │              │                  │
└──────────────────┘              └──────────────────┘
 
Characteristics:
• RPO: Seconds to minutes (replication lag)
• RTO: Minutes to hours (promote replica, redirect traffic)
• Cost: Medium (running standby infrastructure)
• Complexity: Medium (replication monitoring, failover automation)
• Best For: Business-critical applications with moderate RTOs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

ACTIVE-ACTIVE MULTI-REGION
═════════════════════════════════════════════════════════════
 
        ┌─────────────────────────────────────────────┐
        │          Global Traffic Manager             │
        │     (Route53, Traffic Manager, Cloud LB)    │
        └──────────┬───────────────────┬──────────────┘
                   │                   │
                   ▼                   ▼
Primary Region                    Secondary Region
┌──────────────────┐              ┌──────────────────┐
│                  │◄────sync────►│                  │
│   Production     │   replicate  │   Production     │
│   Database       │              │   Database       │
│                  │              │                  │
│   Application    │              │   Application    │
│   Servers        │              │   Servers        │
│   (active)       │              │   (active)       │
│       ▲          │              │        ▲         │
│       │          │              │        │         │
│    traffic       │              │     traffic      │
│                  │              │                  │
└──────────────────┘              └──────────────────┘
 
Characteristics:
• RPO: Zero to seconds (depends on sync/async)
• RTO: Seconds to minutes (just stop routing to failed region)
• Cost: High (2x infrastructure, fully running in both regions)
• Complexity: High (conflict resolution, split-brain prevention)
• Best For: Mission-critical, zero-downtime requirements

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

PILOT LIGHT PATTERN
═════════════════════════════════════════════════════════════
 
Primary Region                    Secondary Region (Pilot)
┌──────────────────┐              ┌──────────────────┐
│                  │              │                  │
│   Production     │──replicate──►│   Database       │
│   Database       │   (async)    │   Replica        │
│                  │              │   (running)      │
│   Application    │              │                  │
│   Servers        │              │   Application    │
│   (10 instances) │              │   (0 instances)  │
│                  │              │   AMIs ready     │
│   Load Balancer  │              │   LB configured  │
│   (active)       │              │   (inactive)     │
│                  │              │                  │
└──────────────────┘              └──────────────────┘
 
On Disaster:
1. Promote DB replica to primary
2. Launch application instances from AMIs
3. Activate load balancer
4. Update DNS/traffic routing
 
Characteristics:
• RPO: Seconds to minutes (replication lag)
• RTO: 30-60 minutes (instance launch + warmup)
• Cost: Low-Medium (only DB replica running, minimal compute)
• Complexity: Medium (automated scaling scripts needed)
• Best For: Balance of cost and recovery speed

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

SCENARIO: 10 TB database, 3% daily change rate
═══════════════════════════════════════════════════════════════════
 
WITHOUT OPTIMIZATION:
├── Daily transfer: 10 TB (full backup)
├── Transfer time: ~22 hours @ 100 MB/s
├── Daily cost: ~$150
├── Annual cost: ~$55,000
└── PROBLEM: 22 hours doesn't fit in backup window
 
WITH INCREMENTAL:
├── Daily transfer: 300 GB (3% changed)
├── Transfer time: ~40 minutes @ 100 MB/s
├── Daily cost: ~$4.50
└── Annual cost: ~$1,650
 
WITH INCREMENTAL + COMPRESSION (3x ratio):
├── Daily transfer: 100 GB
├── Transfer time: ~13 minutes @ 100 MB/s
├── Daily cost: ~$1.50
├── Annual cost: ~$550
└── RESULT: 99% cost reduction, fits any backup window
 
WITH DEDUPLICATION ACROSS SOURCES (50% shared data):
├── Additional reduction if backing up similar systems
├── Transfer: ~50 GB per system after first
└── Significant savings for fleet-wide backup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

CROSS-REGION DR TEST RUNBOOK
═══════════════════════════════════════════════════════════════════
 
PRE-TEST PREPARATION:
□ Notify stakeholders of test window
□ Confirm DR region infrastructure status
□ Verify replication lag is within acceptable bounds
□ Stage monitoring dashboards for both regions
□ Confirm rollback procedures are documented
 
PHASE 1: DATA VALIDATION (T+0 to T+30 min)
□ Verify last successful replication timestamp
□ Check database replica consistency
□ Validate file storage sync status
□ Compare object counts between regions
□ Run data integrity checksums on sample datasets
 
PHASE 2: FAILOVER EXECUTION (T+30 to T+60 min)
□ Stop traffic to primary region (or simulate failure)
□ Promote DR database replica to primary
□ Start/verify application services in DR region
□ Warm up caches and connection pools
□ Activate load balancer in DR region
□ Execute DNS failover (manual or automated)
 
PHASE 3: VALIDATION (T+60 to T+120 min)
□ Verify DNS propagation (test from multiple locations)
□ Execute functional smoke tests against DR endpoint
□ Validate all integrations (payments, email, APIs)
□ Check monitoring and alerting in DR region
□ Run performance baseline tests
□ Validate data writes are working in DR
 
PHASE 4: EXTENDED OPERATION (T+120 to T+240 min)
□ Operate in DR region for minimum 2 hours
□ Monitor for issues, performance degradation
□ Execute sample business transactions
□ Verify logging and observability
 
PHASE 5: FAILBACK (if testing both directions)
□ Reverse replication direction
□ Return traffic to primary
□ Validate primary operation restored
□ Resume normal replication
 
POST-TEST:
□ Document actual vs expected timings
□ Note any issues or surprises
□ Update runbooks based on learnings
□ Report RTO achieved vs target
□ Schedule remediation for any gaps