Leader-Follower Replication

Imagine a financial application processes a wire transfer of $1,000,000. The transaction is written to the leader database and the client receives a success response. One second later, the leader's server experiences a total hardware failure.

Question: Is that million-dollar transfer safe?

The answer depends entirely on one configuration choice: synchronous versus asynchronous replication.

• If replication was synchronous, at least one follower received and acknowledged the write before the client was told 'success.' The transfer is safe.
• If replication was asynchronous, the write may have existed only on the failed leader. The transfer may be lost.

This single configuration option—when the leader waits for followers during a commit—is one of the most consequential decisions in database architecture. It directly trades off between durability guarantees and write performance.

To understand synchronous vs. asynchronous replication, we must first understand the timeline of a write commit. The critical question is: at what point do we tell the client their write succeeded?

The Five Stages of a Write:

1. Client Request — Application sends write to the leader.
2. Leader WAL Write — Leader durably logs the change to its own disk.
3. Replication Streaming — Leader sends the log entry to followers.
4. Follower Acknowledgment — Followers confirm receipt (or apply, depending on mode).
5. Client Response — Leader tells the client the write succeeded.

The key decision: Does stage 5 wait for stage 4?

• Synchronous: Yes. The leader waits for at least one follower's acknowledgment.
• Asynchronous: No. The leader responds after stage 2 (its own WAL write).

Synchronous replication ensures that every committed transaction exists on at least two nodes (the leader and one or more followers) before the client is told the transaction succeeded.

This provides a powerful guarantee: no committed data can be lost due to a single node failure.

Synchronous Replication Configurations:

Single Synchronous Follower: The leader designates one follower as synchronous. All commits wait for this one follower. If the synchronous follower fails, the leader either waits forever (availability loss) or falls back to async (durability risk).

Multiple Synchronous Followers: The leader can wait for N out of M followers. PostgreSQL's synchronous_standby_names supports FIRST N (wait for first N to respond) or ANY N (require N responses from the list). This provides fault tolerance: one sync follower can fail without blocking commits.

Quorum-Based Sync: Advanced systems use quorum writes: commit when a majority (e.g., 2 of 3) acknowledge. This balances durability with availability—no single follower is critical.

Asynchronous replication decouples the client response from follower replication. The leader commits as soon as its own WAL write completes, without waiting for followers.

This provides the lowest possible write latency—but with a trade-off in durability guarantees.

The Risk is Real:

Asynchronous data loss isn't just theoretical. Consider these real-world scenarios:

• Hardware failure — The leader's server has a total power loss or storage controller failure.
• Process crash without fsync — The database process crashes and the OS hasn't flushed buffers to disk.
• Network partition during failover — The leader is unreachable, a follower is promoted, and the old leader comes back with 'orphan' transactions.

Quantifying the Risk:

The data loss window equals the replication lag. With typical async setups:

• Same-datacenter: 10-100ms of potential data loss
• Cross-region: 100ms-1000ms of potential data loss

For a system doing 1000 writes/second, 100ms of lag means up to 100 potentially lost transactions per failure.

Pure synchronous and pure asynchronous represent two extremes. Most production systems operate somewhere in between, using semi-synchronous or hybrid modes that balance durability against availability and performance.

Per-Transaction Control:

Advanced databases allow durability level to be set per-transaction, not just globally. This enables intelligent trade-offs:

-- Critical financial transaction: wait for two replicas
SET synchronous_commit = 'remote_apply';
BEGIN;
INSERT INTO transfers (amount, from_account, to_account) VALUES (1000000, 'A', 'B');
COMMIT;

-- Low-priority logging: async is fine
SET synchronous_commit = 'off';
BEGIN;
INSERT INTO audit_log (event, timestamp) VALUES ('user_login', NOW());
COMMIT;

This approach provides the best of both worlds: critical data gets strong guarantees, bulk/logging operations remain fast.

Choosing between sync and async isn't just intuition—we can quantify the trade-offs mathematically. Understanding the numbers helps make informed decisions.

Latency Impact:

Synchronous replication adds at least one round-trip time (RTT) to each transaction. Let's calculate the impact:

For write-heavy workloads, synchronous cross-region replication can reduce throughput by 10-100x.

Data Loss Quantification:

Asynchronous replication risks losing data in the 'replication window.' We can estimate expected data loss:

Expected Data Loss Per Failure = Replication Lag × Write Rate

Annualized Risk:

If you have one leader failure per year with 100ms lag and 1000 writes/sec:

• Expected lost transactions per year: ~100

For financial systems, losing 100 transactions might mean regulatory violations. For analytics, it might be irrelevant.

With a deep understanding of sync and async trade-offs, let's provide concrete guidance for different scenarios.

Hybrid Strategies:

Same-DC Sync + Cross-Region Async: Maintain a synchronous replica in the same data center for HA and a second replica asynchronously in a remote region for DR. This provides fast failover with strong durability locally, plus protection against datacenter-level disasters.

Per-Table Configuration: Some databases allow replication configuration per table. Critical tables (accounts, orders) replicate synchronously; high-volume tables (events, logs) replicate asynchronously.

Application-Level Routing: The application knows which writes are critical. It can specify durability requirements per transaction, routing critical writes through a synchronous path.

Different databases implement synchronous/asynchronous replication with varying configurations and semantics. Let's examine the major implementations.

We've explored one of the most fundamental trade-offs in database replication: when to wait for followers during commit. Let's consolidate the essential insights:

What's Next:

With writes flowing through the leader and replicating (synchronously or asynchronously) to followers, we must address a critical scenario: what happens when the leader fails? The next page explores failover handling—how systems detect leader failure, elect a new leader, and transition without data loss or prolonged downtime.