Replication - Learning Module

Loading content...

0/241

Synchronous Replication

The Promise of Zero Data Loss

Imagine a financial trading system where a transaction confirmation is sent to a client, but the server crashes immediately afterward. If that transaction existed only on the failed server—if replicas hadn't received it yet—committed data has been lost. The client believes the trade occurred; the system has no record of it.\n\nThis nightmare scenario is precisely what synchronous replication prevents. By requiring that writes are confirmed on multiple replicas before acknowledging to the client, synchronous replication provides the strongest durability guarantee: if you receive confirmation, the data survives any single node failure.\n\nBut this guarantee comes at a cost. Every write must wait for network round-trips to replicas. Every replica failure affects write availability. Understanding these trade-offs is essential for designing systems that truly cannot afford data loss.

What You Will Learn

By the end of this page, you will understand the mechanics of synchronous replication, the specific consistency and durability guarantees it provides, its performance characteristics and overhead, failure handling strategies, and when to choose synchronous over asynchronous replication.

Definition and Mechanics

Synchronous replication is a replication mode where the primary node waits for one or more replicas to acknowledge receipt and/or application of a write before confirming the transaction to the client.\n\nThe key characteristic is blocking behavior: the write operation blocks until replicas respond. This ensures that committed writes exist on multiple nodes at the moment of acknowledgment.

Converting Mermaid diagram...

The Synchronous Replication Protocol:\n\n1. Client initiates write: The client sends an INSERT, UPDATE, DELETE, or COMMIT to the primary.\n\n2. Primary writes locally: The primary writes the transaction to its local Write-Ahead Log (WAL) and flushes to disk.\n\n3. Primary forwards to replicas: The primary sends WAL records to configured synchronous replicas in parallel.\n\n4. Replicas acknowledge: Each replica writes the WAL records to its own disk and sends an acknowledgment to the primary.\n\n5. Primary confirms to client: Only after receiving acknowledgment from the required number of replicas does the primary confirm the commit to the client.\n\nThe critical insight is that the client's view of "committed" includes confirmation from replicas—not just the primary.

Levels of Synchronous Confirmation

Different databases offer varying levels of synchronous acknowledgment:\n\n• Remote Write — Replica acknowledges when WAL is written to its OS buffer (fast, some risk)\n• Remote Flush — Replica acknowledges when WAL is flushed to disk (durable)\n• Remote Apply — Replica acknowledges when WAL is applied to data files (visible to reads)\n\nEach level trades latency for stronger guarantees. 'Remote Apply' ensures read-your-writes consistency on replicas but adds significant latency.

Consistency and Durability Guarantees

Synchronous replication provides specific, well-defined guarantees that are essential for understanding its value proposition:

Synchronous Replication Guarantees

•Zero Data Loss on Single Node Failure — If the primary fails immediately after acknowledging a commit, the data exists on at least one synchronous replica. No committed transactions are lost.
•Durability Before Acknowledgment — The client knows that when they receive a commit confirmation, the data is durable on multiple physically independent storage systems.
•Failover Safety — Any synchronous replica can be promoted to primary with complete data integrity. No special recovery procedures or data reconciliation are required.
•Read-Your-Writes (with Remote Apply) — If the replica has applied the WAL before acknowledgment, reads from that replica will see the committed write immediately.

What Synchronous DOES Guarantee

•Committed data survives primary failure
•Replicas have durably received the data
•No data loss during clean failover
•RPO (Recovery Point Objective) = 0

What Synchronous Does NOT Guarantee

•Immediate read visibility on all replicas
•Protection against multi-node failures
•Write availability during replica failures
•Identical performance to asynchronous

The Multi-Node Failure Caveat

Synchronous replication with one synchronous replica protects against any single node failure. If both the primary AND the synchronous replica fail simultaneously (e.g., same rack power failure, correlated failures), data loss is still possible. For protection against correlated failures, you need either more synchronous replicas or geographic distribution.

Performance Characteristics and Overhead

Synchronous replication introduces unavoidable performance overhead. Understanding these characteristics is essential for capacity planning and setting appropriate expectations:

Latency Impact:\n\nEvery write operation incurs additional latency equal to:\n\n\nSync Replication Latency = Network RTT to Replica + Replica Disk Write Time\n\n\nFor same-datacenter replicas, this adds approximately 1-5ms per write. For cross-region replicas, this adds 50-200ms per write—a potentially catastrophic overhead for latency-sensitive applications.\n\nThroughput Impact:\n\nSynchronous replication does not inherently reduce throughput for independent concurrent writes. Multiple transactions can replicate in parallel. However:\n\n- Head-of-line blocking: If replica becomes slow (disk I/O saturation), all writes wait.\n- Connection limits: Each synchronous replication connection consumes resources.\n- Network bandwidth: High write volume requires sustained network capacity to replicas.

Synchronous Replication Latency by Replica Location
Replica Location	Typical Network RTT	Disk Write Time	Total Added Latency	Practical Use
Same rack	< 0.1ms	1-2ms	~2ms	High-frequency trading, real-time
Same datacenter, different rack	0.2-0.5ms	1-2ms	~3ms	Standard production HA
Different datacenter, same region	1-5ms	1-2ms	~5-7ms	Regional disaster recovery
Different region (continental)	30-80ms	1-2ms	~35-85ms	Geographic DR (use carefully)
Cross-continental	100-200ms	1-2ms	~105-205ms	Rarely practical for sync

The N+1 Latency Problem

When configuring N synchronous replicas, your commit latency is bounded by the slowest replica. If you have 2 synchronous replicas and one has disk issues, all your commits wait. This 'weakest link' behavior is why proper monitoring and replica health maintenance is critical.

Measuring Synchronous Replication Overhead (PostgreSQL)
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
-- Check current synchronous commit setting
SHOW synchronous_commit;
 
-- View synchronous standby names configuration
SHOW synchronous_standby_names;
 
-- Measure write latency with sync vs async
-- Step 1: Create test table
CREATE TABLE latency_test (id SERIAL PRIMARY KEY, data TEXT, created_at TIMESTAMP DEFAULT now());
 
-- Step 2: Test with synchronous_commit = on (default)
\timing on
INSERT INTO latency_test (data) VALUES ('sync test');
 
-- Step 3: Compare with synchronous_commit = off (for this session only)
SET synchronous_commit = off;
INSERT INTO latency_test (data) VALUES ('async test');
 
-- Step 4: Monitor replication lag
SELECT 
    client_addr,
    application_name,
    sync_state,
    pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS send_lag_bytes,
    pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_lag_bytes,
    pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

Failure Handling in Synchronous Replication

Synchronous replication's strong guarantees come with a critical trade-off: replica failures directly impact write availability. When a synchronous replica becomes unavailable, the primary cannot acknowledge writes—it waits indefinitely for an acknowledgment that will never come.\n\nHandling this failure mode is one of the most important operational aspects of synchronous replication:

Synchronous Replica Failure Scenarios

•Replica Crash — The replica process terminates. Network connection drops. Primary detects failure after TCP timeout. All pending writes block until reconfiguration.
•Replica Disk Failure — The replica cannot write to disk. WAL acknowledgment never sent. Primary blocks indefinitely on affected transactions.
•Network Partition — The replica is running but unreachable from the primary. Primary cannot distinguish between slow replica and failed replica.
•Replica Overload — The replica is too slow (disk I/O saturation, CPU overload). Technically "alive" but acknowledgment delayed beyond acceptable thresholds.

Mitigation Strategies:\n\n1. Automatic Failover with Priority Lists\n\nConfigure multiple replicas as potential synchronous targets with priority ordering. If the highest-priority replica fails, the next-priority replica becomes synchronous.\n\npostgresql\n-- PostgreSQL: Priority-based synchronous standby selection\nsynchronous_standby_names = 'FIRST 1 (replica1, replica2, replica3)'\n-- replica1 is sync. If it fails, replica2 becomes sync automatically.\n\n\n2. Quorum-Based Synchronous Replication\n\nRequire acknowledgment from any N of M replicas rather than specific replicas. This provides resilience as long as enough replicas are available.\n\npostgresql\n-- PostgreSQL: ANY requires 2 of 3 replicas to acknowledge\nsynchronous_standby_names = 'ANY 2 (replica1, replica2, replica3)'\n-- Commit succeeds if any 2 replicas acknowledge.\n\n\n3. Timeouts and Graceful Degradation\n\nSome systems allow configuring timeouts for synchronous acknowledgment. If exceeded, the write either fails or degrades to asynchronous mode.\n\n4. Manual Intervention\n\nIn cases where automatic recovery is unsafe, the operator manually reconfigures synchronous standby names to exclude the failed replica.

The Availability vs. Consistency Choice

When all synchronous replicas fail, you face a critical decision:\n\nOption A (Consistent): Block all writes until a replica recovers. Prioritizes data durability. Causes downtime.\n\nOption B (Available): Fall back to asynchronous mode. Accepts possible data loss. Maintains availability.\n\nThis is the CAP theorem in action. Most production systems configure automatic fallback with alerts, accepting the brief window of reduced durability for continued availability.

Implementation Patterns Across Databases

Major database systems implement synchronous replication with varying approaches and terminology. Understanding these implementations helps you configure appropriate settings for your workload:

PostgreSQL Synchronous Replication Configuration:\n\nPostgreSQL provides granular control over synchronous replication behavior through several parameters:\n\nKey Configuration Parameters:\n- synchronous_commit: Controls commit acknowledgment level\n- synchronous_standby_names: Defines which replicas are synchronous\n\nsynchronous_commit Options:\n- off — No waiting (asynchronous, possible data loss)\n- local — Wait for local WAL flush only\n- remote_write — Wait for replica to receive in OS buffer\n- on — Wait for replica to flush to disk (default when sync standby exists)\n- remote_apply — Wait for replica to apply to data files

PostgreSQL Synchronous Replication Configuration
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
-- On Primary: Configure synchronous standby (postgresql.conf or ALTER SYSTEM)
ALTER SYSTEM SET synchronous_standby_names = 'FIRST 1 (replica1, replica2)';
ALTER SYSTEM SET synchronous_commit = 'on';
SELECT pg_reload_conf();
 
-- Verify configuration
SHOW synchronous_standby_names;
SHOW synchronous_commit;
 
-- Check which replicas are currently synchronous
SELECT 
    application_name,
    client_addr,
    sync_state,  -- 'sync' = synchronous, 'async' = asynchronous, 'potential' = fallback
    sync_priority
FROM pg_stat_replication
ORDER BY sync_priority;
 
-- Per-transaction control (useful for less critical writes)
BEGIN;
SET LOCAL synchronous_commit = 'local';  -- This transaction doesn't wait for replica
INSERT INTO logs (message) VALUES ('Non-critical log entry');
COMMIT;

When to Use Synchronous Replication

Synchronous replication is not universally appropriate. The latency overhead and availability trade-offs make it suitable for specific use cases while problematic for others:

Ideal Use Cases

•Financial transactions — Zero tolerance for data loss; regulatory requirements for durability
•Order processing — Customer-facing commitments that cannot be rolled back
•Healthcare records — Patient safety requires guaranteed persistence
•Legal documents — Compliance requires provable durability
•Inventory management — Stock levels must be accurate to prevent overselling
•Same-datacenter HA — Low latency makes sync overhead acceptable

Problematic Use Cases

•Logging and analytics — High volume, loss-tolerant data doesn't justify overhead
•Cross-continental replication — 100-200ms latency per write is unacceptable for most apps
•High-frequency writes — Latency accumulates; throughput suffers
•User-generated content — Social posts, comments can tolerate eventual durability
•Caching layers — By definition, cache data is reconstructable
•Temporary/session data — Expires anyway; durability not valuable

Hybrid Approach: Per-Transaction Control

Many production systems use synchronous replication for critical writes and asynchronous for less important data. PostgreSQL's per-transaction synchronous_commit setting and MongoDB's per-operation write concern enable this pattern. For example, use {w: 'majority'} for payment processing but {w: 1} for activity logging—all within the same database.

Decision Framework:\n\nWhen evaluating synchronous replication, answer these questions:\n\n1. What is the business cost of losing one committed transaction? If catastrophic, sync is justified.\n\n2. What is the acceptable write latency? If sub-10ms is required, same-DC sync is viable; cross-region sync is not.\n\n3. What is the acceptable downtime during replica failures? If zero tolerance, you need multiple sync replicas with quorum.\n\n4. Is the data reconstructable? If yes (derived data, caches, logs), async is appropriate.\n\n5. What are regulatory/compliance requirements? Financial services often mandate sync for audit trails.

Operational Best Practices

Running synchronous replication in production requires careful operational practices to balance durability guarantees with availability:

Synchronous Replication Best Practices

•Configure multiple synchronous candidates — Use FIRST or ANY syntax to enable automatic failover between replicas. A single sync replica is a single point of failure for write availability.
•Monitor replication lag obsessively — Even synchronous replication can exhibit lag during transient network issues. Alert on lag > 1 second.
•Place sync replicas strategically — Same-datacenter for low latency; different racks for failure independence. Cross-DC sync only for specific durability requirements.
•Test failover regularly — Simulate replica failures monthly. Verify automatic promotion and data integrity.
•Size replicas identically to primary — A replica with slower disks becomes your bottleneck. Synchronous replicas need identical (or better) I/O capacity.
•Configure appropriate timeouts — Too short causes unnecessary fallback to async; too long causes extended blocking during failures. Start with 30-60 seconds for most workloads.
•Document failure procedures — When all sync replicas fail, operators must know whether to wait, fallback, or intervene manually.

Monitoring Synchronous Replication Health
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
-- PostgreSQL: Comprehensive sync replication monitoring query
SELECT 
    application_name,
    client_addr,
    state,
    sync_state,
    sync_priority,
    -- Lag calculations
    pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS send_lag_bytes,
    pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_lag_bytes,
    pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,
    pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS replay_lag_pretty,
    -- Time-based lag (PostgreSQL 10+)
    replay_lag,
    write_lag,
    flush_lag,
    -- Connection health
    EXTRACT(EPOCH FROM (now() - backend_start)) AS connection_age_seconds,
    EXTRACT(EPOCH FROM (now() - reply_time)) AS seconds_since_last_reply
FROM pg_stat_replication
WHERE sync_state IN ('sync', 'potential')
ORDER BY sync_priority;
 
-- Alert conditions:
-- 1. sync_state != 'sync' for expected replica
-- 2. replay_lag_bytes > 10MB (threshold depends on write rate)
-- 3. seconds_since_last_reply > 30

The Maintenance Window Problem

When you take a synchronous replica offline for maintenance, you must either:\n\n1. Promote another replica to synchronous status first\n2. Temporarily relax to fewer required replicas\n3. Accept that the maintenance window is a write-availability risk\n\nAlways plan maintenance during low-traffic periods and communicate the reduced durability during the window.

Summary: Synchronous Replication

We've explored synchronous replication in depth—its mechanics, guarantees, performance characteristics, failure handling, and operational practices. Let's consolidate the key insights:

Key Takeaways

•Synchronous replication guarantees zero data loss — Committed writes exist on multiple nodes before client acknowledgment. Single-node failure loses no committed data.
•Latency is the cost — Every write waits for network round-trip plus replica disk write. Same-DC adds ~3ms; cross-region adds 50-200ms.
•Replica failure affects write availability — When synchronous replicas are unreachable, writes block. Mitigation requires multiple candidates and quorum-based configuration.
•Different acknowledgment levels exist — Remote write, remote flush, remote apply provide varying trade-offs between latency and guarantee strength.
•Use synchronous selectively — Critical financial, healthcare, and transactional data justifies the overhead. Logging, analytics, and temporary data do not.
•Operational discipline is essential — Monitor obsessively, test failover regularly, size replicas appropriately, and document procedures.

What's Next:\n\nSynchronous replication prioritizes consistency at the cost of latency and availability. In the next page, we'll explore Asynchronous Replication—the approach that prioritizes performance and availability, accepting that replicas may temporarily fall behind. We'll examine its mechanics, replication lag implications, and strategies for handling the consistency trade-offs it introduces.

Page Complete

You now understand synchronous replication's mechanics, guarantees, and trade-offs. It provides the strongest durability guarantee at the cost of write latency and availability during replica failures. For systems where data loss is unacceptable, synchronous replication is the gold standard. Next, we'll explore the alternative: asynchronous replication.