Why Database Replication Matters

At 2:47 AM on a Saturday night, a hard drive in your production database server fails. The RAID controller catches the failure and begins rebuilding—but six minutes later, a second drive in the same array fails (a correlated failure, often caused by drives from the same manufacturing batch reaching end-of-life simultaneously). The database goes offline. Your application returns errors to every user. Revenue stops. On-call engineers scramble.

This scenario—or variations involving power failures, network partitions, kernel panics, or human error—occurs every day across the industry. The question isn't whether your database will experience failure, but when and how your system will respond.

High availability (HA) is the discipline of designing systems that continue operating despite component failures. In the context of databases, this means ensuring data remains accessible even when individual database servers fail. Replication is the foundational mechanism that makes database high availability possible.

Every component in a computing system will eventually fail. Understanding failure modes is the first step toward designing for resilience.

Hardware failure rates:

Commercial hardware experiences predictable failure rates, typically measured as Annual Failure Rate (AFR):

These numbers may seem comfortable for individual components, but the math changes dramatically at scale:

The fleet effect:

With 100 servers, each with 5% annual failure rate:

• Expected server failures per year: 5
• Expected server failures per month: ~0.4
• Probability of at least one failure in 30 days: ~35%

With 1,000 servers:

• Expected failures per month: ~4
• Probability of at least one failure in a week: ~90%

At scale, failure is not an exception—it's a constant reality. Amazon, Google, and other hyperscalers report that hardware failures occur every few minutes across their fleets.

Beyond hardware:

Hardware is only one failure mode. Production systems also contend with:

• Software bugs: Memory leaks, race conditions, resource exhaustion
• Configuration errors: Human mistakes during deployment or maintenance
• Network partitions: Loss of connectivity between components
• Dependency failures: Upstream services becoming unavailable
• Capacity exhaustion: Disk full, memory exhausted, connection limits reached
• Security incidents: Attacks requiring immediate shutdown or isolation

Availability is precisely defined as the proportion of time a system is operational and accessible. It's commonly expressed as a percentage or as "nines."

Availability formula:

Availability = Uptime / (Uptime + Downtime)

Or equivalently:

Availability = MTBF / (MTBF + MTTR)

Where:

• MTBF (Mean Time Between Failures): Average time the system operates before failing
• MTTR (Mean Time To Repair): Average time to restore service after failure

The nines of availability:

What different businesses require:

• 99% (two nines): Acceptable for internal tools, development environments
• 99.9% (three nines): Standard for most SaaS applications, content sites
• 99.99% (four nines): Expected for e-commerce, financial services, enterprise software
• 99.999% (five nines): Required for payment processing, telecommunications, emergency services
• 99.9999% (six nines): Critical infrastructure (nuclear control, aviation, medical life-support)

The cost curve:

Each additional nine is exponentially harder (and more expensive) to achieve. Going from 99% to 99.9% might require adding read replicas and automated failover. Going from 99.99% to 99.999% might require multi-region active-active deployment, custom tooling, and 24/7 dedicated SRE teams.

A single-server database has an inherent availability ceiling: when that server fails, the database is unavailable until repair. Replication fundamentally changes this equation by ensuring data exists on multiple independent servers.

The single-server problem:

Consider a database server with:

• 99.9% availability (realistic for well-maintained hardware)
• MTTR of 1 hour (including detection, diagnosis, and recovery)

This server will experience approximately 8.76 hours of downtime per year—unacceptable for most production applications.

Replication as redundancy:

With one primary and one synchronous replica:

• Both must fail simultaneously for data to be unavailable
• If failures are independent, P(both fail) = P(primary fails) × P(replica fails)
• If each has 99.9% availability: combined availability approaches 99.9999%

In practice, the improvement isn't quite this dramatic due to correlated failures and failover transition time, but replication typically adds 1-2 nines of availability.

Key requirements for HA through replication:

Synchronous vs. asynchronous replication for HA:

Most production systems use asynchronous replication for performance, accepting small potential data loss during rare failover events. Critical systems (banking, healthcare) may use synchronous replication despite the latency cost.

When the primary database fails, the system must detect the failure and transition to a replica. This process—failover—is the critical path in high availability architecture.

Failure detection:

Before failover can occur, the system must determine that the primary is truly unavailable. This is deceptively difficult:

• False positive (declaring primary dead when it's alive): Can cause split-brain scenarios with data corruption
• False negative (failing to detect a dead primary): Prolongs outage

Common detection mechanisms:

1. Heartbeat monitoring: Regular pings between nodes; failure to respond triggers alert
2. Consensus-based detection: Multiple observers must agree the primary is unavailable (reduces false positives)
3. Client-reported failures: Applications report when they can't reach the database
4. Proxy-layer detection: Load balancers or proxies detect connection failures

Types of failover:

Split-brain: The nightmare scenario:

Split-brain occurs when both the old primary and a promoted replica believe they are the primary, accepting writes independently. This creates divergent data that may be impossible to reconcile.

Preventing split-brain:

1. STONITH (Shoot The Other Node In The Head): Before promoting a replica, forcibly terminate the old primary (power off, network isolation)
2. Fencing: Ensure the old primary cannot accept writes (revoke VIP, disable network interface)
3. Quorum requirements: Require majority of cluster members to agree before any node can be primary
4. Epoch/term numbers: Track leadership generations; reject writes from old leaders

Failover steps (typical automatic failover):

1. Failure detection: Monitoring determines primary is unreachable (multiple missed heartbeats)
2. Fencing: If possible, forcibly isolate or shut down old primary
3. Replica selection: Choose the replica with most up-to-date data (lowest replication lag)
4. Promotion: Selected replica stops accepting replication, becomes new primary
5. Reconfiguration: Other replicas point to new primary
6. Client redirection: VIP or DNS updated to point to new primary; connections retried

Several architectural patterns provide high availability for databases, each with different trade-offs in complexity, cost, and availability level.

Active-Passive (Primary-Standby):

The simplest HA architecture:

• One active primary handles all traffic
• One or more passive standbys receive replication but serve no traffic
• On failure, a standby is promoted to primary

Pros: Simple, low resource overhead (standbys can be smaller) Cons: Standby resources underutilized; promotion takes time Availability: Typically 99.9% to 99.99%

Active-Active (Multi-Primary):

More complex, higher availability:

• Multiple nodes all accept writes
• Requires conflict resolution for concurrent writes to same data
• Can use geographic distribution for local performance

Pros: No failover delay (every node is already active); higher throughput Cons: Complex conflict resolution; typically requires application awareness Availability: Can achieve 99.99% to 99.999%

Shared-Nothing Clusters:

Distributed databases (Cassandra, CockroachDB, Spanner):

• Data partitioned and replicated across nodes
• Any node can serve any request (with routing)
• No single point of failure by design

Pros: Inherent HA; scales horizontally; no failover process Cons: More complex data model; eventual consistency (typically) Availability: Can achieve 99.999%+ with proper deployment

Cloud-managed HA:

Major cloud providers offer managed databases with built-in HA:

• AWS RDS Multi-AZ: Synchronous replication to standby in different availability zone; automatic failover in 60-120 seconds
• Google Cloud SQL HA: Regional instances with automatic failover
• Azure SQL Database: Built-in replication and automatic failover
• AWS Aurora: Storage-level replication across 3 AZs; failover in 30 seconds or less

Trade-off: Managed HA reduces operational burden but limits customization and may have higher cost.

Understanding how major platforms implement database HA provides practical insights for your own architectures.

Pattern 1: PostgreSQL with Patroni

Patroni is an open-source tool for PostgreSQL HA:

• Uses etcd, Consul, or ZooKeeper for distributed consensus
• Automatic leader election and failover
• Handles replica promotion, reconfiguration, and client redirection
• Used by GitLab, Zalando, and many others

Architecture:

• 3+ PostgreSQL nodes (1 primary, 2+ replicas)
• 3+ etcd nodes for consensus (can be colocated or separate)
• HAProxy or pgBouncer for connection pooling and routing
• Virtual IP or DNS for client connection

Pattern 2: MySQL with Orchestrator

Orchestrator is a MySQL HA and replication management tool:

• Topology discovery and visualization
• Automated failure detection and recovery
• Intelligent replica promotion (chooses best candidate)
• Used by GitHub, Booking.com, and others

Key features:

• Detects intermediate master failures and restructures topology
• Supports complex replication hierarchies
• Provides both automated and manual failover options

Pattern 3: Amazon Aurora

Aurora separates compute from storage for unique HA characteristics:

• Storage layer automatically replicates data 6 ways across 3 AZs
• Compute (database engine) can fail without data loss
• Read replicas share the same storage layer (instant provisioning)
• Failover promotes read replica to primary in ~30 seconds

Why it's notable:

• Storage-level redundancy beyond what traditional replication provides
• Replicas don't increase storage cost
• Near-zero data loss even with asynchronous replication semantics

High availability isn't just about database architecture—it requires holistic system design that anticipates and gracefully handles failures at every layer.

Application-level considerations:

Connection handling during failover:

During failover, applications experience:

1. Existing connections become invalid (old primary is unreachable)
2. New connection attempts may fail (new primary not yet accepting connections)
3. Brief period of complete unavailability (10-60 seconds)

Best practices:

• Use connection pooling libraries that handle reconnection automatically
• Implement exponential backoff for retry attempts
• Set appropriate connection and query timeouts (not too short, not too long)
• Use read endpoints for read traffic (replicas remain available during primary failover)

Testing resilience:

High availability transforms databases from single points of failure into resilient systems that survive component failures. Let's consolidate the key concepts:

What's next:

High availability keeps your database running when local failures occur. But what if you need to serve users across continents? The next page explores Geographic Distribution—how replication enables low-latency access for globally distributed users and provides protection against regional disasters.