Why Database Replication Matters

On February 28, 2017, Amazon Web Services' S3 service in US-East-1 experienced a major outage lasting nearly four hours. The ripple effects were catastrophic: thousands of websites and applications that depended on S3—or on services that depended on S3—became unavailable. Companies lost millions of dollars. Some businesses discovered their "disaster recovery" plans were hosted in the same region that failed.

In March 2021, a fire at OVHcloud's Strasbourg data center destroyed one building entirely and damaged another. Hundreds of thousands of websites went offline. For some companies, the damage was permanent—their data was lost because backups were stored in the same facility.

These aren't hypothetical scenarios—they're recent history. Disaster recovery (DR) is the discipline of planning for, and recovering from, catastrophic failures that affect entire data centers, regions, or even cloud providers. Database replication is the foundational technology that makes DR possible for your most critical asset: your data.

High availability (HA) and disaster recovery (DR) are related but distinct concepts. Understanding the difference is essential for proper planning.

High Availability (HA):

• Protects against component failures (disk, server, network device)
• Typically operates within a single data center or region
• Failover is automatic and fast (seconds to minutes)
• Goal: continuous operation with minimal or no interruption
• Example: Primary database fails; replica in same availability zone takes over

Disaster Recovery (DR):

• Protects against site-wide failures (data center, region, provider)
• Operates across geographically separated locations
• Recovery may be manual or semi-automated (minutes to hours)
• Goal: restore operations after catastrophic failure
• Example: Entire AWS region fails; failover to a different region

Why both are necessary:

A system can have excellent HA within a region but zero DR capability. Consider:

• Primary database in us-east-1a with replica in us-east-1b
• Automatic failover between availability zones (HA: ✓)
• If all of us-east-1 fails (major AWS outage, natural disaster, power grid failure): complete outage (DR: ✗)

Conversely, a system with DR but poor HA experiences frequent minor outages for simple hardware failures.

Mature organizations implement both:

1. HA within each region: Multi-AZ deployment, automatic failover
2. DR across regions: Replicated data, documented procedures, tested recovery

Two metrics define disaster recovery requirements: Recovery Point Objective (RPO) and Recovery Time Objective (RTO). These are the foundation of DR planning and directly influence architecture decisions.

Recovery Point Objective (RPO):

RPO answers: How much data can we afford to lose?

• Measured in time: "We can lose up to 15 minutes of data"
• Determines replication strategy and frequency
• Business-driven: based on cost of lost transactions/data

Recovery Time Objective (RTO):

RTO answers: How long can we be down?

• Measured in time: "We must be operational within 4 hours"
• Determines automation level and infrastructure readiness
• Business-driven: based on cost of downtime (revenue, reputation, SLA penalties)

The cost curve:

Lower RPO and RTO requirements exponentially increase costs:

Setting objectives:

RPO and RTO should be set based on business impact analysis:

1. Identify critical systems — Which databases, if lost, would halt business operations?
2. Quantify downtime cost — Revenue per hour, SLA penalties, customer churn, regulatory fines
3. Quantify data loss cost — Value of transactions in RPO window, ability to reconstruct manually
4. Compare to DR implementation cost — Balance cost of disaster against cost of prevention

Different objectives for different data:

Not all data warrants the same RPO/RTO:

• Payment transactions: RPO = 0, RTO = 5 minutes (cannot lose or delay payments)
• User profiles: RPO = 15 minutes, RTO = 1 hour (annoying but recoverable)
• Analytics data: RPO = 24 hours, RTO = 24 hours (can be regenerated)

Several architectural patterns use database replication to enable disaster recovery, each with different RPO, RTO, and cost characteristics.

Pattern 1: Backup to Remote Storage

Simplest DR approach:

• Regular backups (daily/hourly) stored in a separate region
• Continuous WAL archiving for point-in-time recovery
• On disaster: Provision infrastructure, restore from backup

Characteristics:

• RPO: Backup interval (or WAL archive lag)
• RTO: Hours (infrastructure provisioning + restore time)
• Cost: Low (only storage costs for backups)
• Use when: Cost-sensitive, can tolerate longer recovery

Pattern 2: Warm Standby

Pre-provisioned infrastructure in DR region:

• Infrastructure ready but not actively serving traffic
• Asynchronous replication keeps data relatively current
• On disaster: Promote standby, redirect traffic

Characteristics:

• RPO: Replication lag (seconds to minutes)
• RTO: Minutes to 1 hour (promotion + DNS propagation)
• Cost: Medium (idle infrastructure + replication bandwidth)
• Use when: Moderate RPO/RTO requirements, predictable costs

Pattern 3: Hot Standby (Active-Passive)

Fully operational DR region:

• DR region receives replicated data in near real-time
• DR infrastructure runs continuously (can serve read traffic)
• On disaster: Near-instant promotion

Characteristics:

• RPO: Seconds (async) or zero (sync)
• RTO: Minutes (automated failover achievable)
• Cost: High (full duplicate infrastructure)
• Use when: Stringent RPO/RTO, mission-critical systems

Pattern 4: Active-Active Multi-Region

Both regions actively serve traffic:

• Writes accepted in both regions with conflict resolution
• No "failover" needed—traffic simply routes away from failed region
• Provides both HA and DR simultaneously

Characteristics:

• RPO: Zero for local writes (may lose in-flight cross-region replication)
• RTO: Seconds (just routing change)
• Cost: Very high (full capacity in multiple regions + complexity)
• Use when: Global presence, maximum availability requirements

Implementing cross-region replication for DR requires careful attention to network configuration, monitoring, and operational procedures.

Network considerations:

Replication bandwidth calculation:

For a database with:

• 1,000 write transactions per second
• Average 500 bytes per write
• 50% overhead for replication metadata

Bandwidth = 1000 × 500 × 1.5 = 750,000 bytes/sec = ~6 Mbps

This is baseline; bursts during traffic spikes can be 10-100x higher. Provision accordingly.

Handling replication lag:

Cross-region inherently has higher latency (50-200+ ms depending on distance). This affects:

1. Synchronous replication: Every write waits for cross-region round-trip (often unacceptable)
2. Asynchronous replication: Lag accumulates during high-write periods

Strategies for managing lag:

• Parallel apply: Multiple worker threads applying changes on DR replica
• Batched replication: Aggregate changes for efficiency (increases lag slightly)
• Monitoring and alerting: Alert when lag exceeds RPO threshold
• Write throttling: In extreme cases, slow down primary to let DR catch up

When disaster strikes, having documented, tested procedures is the difference between a controlled recovery and chaos. DR failover is significantly more complex than local HA failover.

Pre-failover checklist:

1. Confirm the disaster — Is primary region truly unavailable, or is this a monitoring false positive?
2. Assess data state — What is the replication lag at the DR site? What data will be lost?
3. Notify stakeholders — Communicate to leadership, customers, support teams
4. Prepare DR infrastructure — Ensure DR application servers, load balancers are ready
5. Document decision — Record who authorized failover, when, and why

Database failover steps:

Post-failover operations:

• Communication: Regular updates to stakeholders on status
• Monitoring: Heightened alerting during DR operation
• Capacity management: DR may have less capacity than primary; watch for overload
• Prevent split-brain: Ensure primary region cannot accept writes if it recovers

Failback planning:

Once the primary region recovers, you need a plan to return to normal operations:

1. Assess primary region: Is it stable? What caused the failure?
2. Resynchronize data: Replicate changes made during DR back to primary
3. Validate primary: Run tests to confirm primary is healthy
4. Gradual traffic shift: Incrementally move traffic back (canary deployment)
5. Full cutover: Return to normal primary/DR configuration
6. Post-mortem: Document what happened, how it was handled, lessons learned

An untested DR plan is not a plan—it's a hope. Real disasters expose gaps that documentation reviews cannot. Regular, realistic DR testing is essential.

Types of DR tests:

What to validate during DR tests:

Game Day methodology:

The most effective DR test is a "Game Day"—a scheduled, announced exercise where the team executes DR procedures against production (during low-traffic periods):

1. Schedule — Pick a low-traffic window (Saturday morning, holiday)
2. Announce — Notify stakeholders; have support ready for user impact
3. Execute — Follow DR runbooks exactly as you would in real disaster
4. Measure — Record timestamps, issues encountered, deviations from plan
5. Operate — Run from DR for a period (1 hour to 24 hours)
6. Return — Fail back to primary region
7. Retrospective — Document lessons learned, update runbooks

Netflix's famous "Chaos Monkey" and broader Chaos Engineering practices extend this concept to continuous, smaller-scale failure injection.

Understanding real-world disasters helps in designing robust DR strategies.

Scenario 1: Cloud Provider Region Outage (AWS US-East-1, 2017)

What happened:

• Human error during S3 maintenance took down core S3 servers
• Cascading failures affected many dependent services
• Duration: ~4 hours

Lessons:

• Avoid single-region dependencies, even for "always on" services
• Dependencies on core cloud services (S3, IAM) can cascade
• DR in a different region would have provided immediate failover

Scenario 2: Data Center Fire (OVHcloud Strasbourg, 2021)

What happened:

• Fire destroyed entire data center building
• Some customers lost data permanently (backups in same facility)
• Duration: Permanent for some, weeks for full restoration

Lessons:

• Physical separation between primary and backups is essential
• "Same provider, different data center" is not DR if they share physical infrastructure
• Off-site, geographically distant backups are mandatory

Scenario 3: Ransomware Attack (Multiple Organizations)

What happens:

• Attackers encrypt production databases and demand payment
• Organizations without offline backups face permanent data loss or payment
• Recovery time: Days to weeks

Lessons:

• Backups must be isolated from production network (air-gapped or immutable)
• Cross-region replication propagates encryption (useless for ransomware DR)
• Offline, immutable backups with tested restores are critical

Scenario 4: Submarine Cable Cut (Multiple Events)

What happens:

• Undersea cable damage (anchor, earthquake) severs connectivity between regions
• Traffic between regions becomes unavailable or severely degraded
• Duration: Days to weeks for cable repair

Lessons:

• Cross-region replication depends on network connectivity
• Multi-path network design (multiple cable routes) improves resilience
• Users in affected regions need local data access

Disaster recovery protects organizations from catastrophic data loss and extended outages. Let's consolidate the key concepts:

Module complete:

You have now explored all four motivations for database replication:

1. Read Scalability — Distributing read load across replicas
2. High Availability — Surviving component failures with automatic failover
3. Geographic Distribution — Serving global users with local latency
4. Disaster Recovery — Protecting against catastrophic site-wide failures

These motivations are not mutually exclusive—a well-designed replication architecture often addresses multiple goals simultaneously. Cross-region replicas, for example, provide read scaling, geographic distribution, AND disaster recovery.

In subsequent modules, we'll explore how to implement replication: leader-follower patterns, leaderless approaches, quorum-based systems, and partitioning strategies that complement replication.