System Design (HLD)Why Replication?

Why Database Replication Matters

LevelIntermediate

Duration90 mins

TopicWhy Replication?

3 / 4

Geographic Distribution: Serving Users Globally

The Physics of Global Latency

Light travels at approximately 300,000 kilometers per second in a vacuum—but through fiber optic cables, this drops to about 200,000 km/s due to the refractive index of glass. A packet traveling from Tokyo to New York must traverse roughly 11,000 km of cable, taking a minimum of 55 milliseconds for a one-way journey—and 110 ms for a round trip. Add routing overhead, TCP handshakes, and application processing, and real-world latencies often exceed 200 ms.

For a user in Tokyo accessing a database in Virginia, every database operation incurs this penalty. A page load requiring 10 sequential queries faces 2+ seconds of network latency alone—a frustrating experience that drives users away. According to Google research, 53% of mobile users abandon sites that take longer than 3 seconds to load.

Geographic distribution addresses this fundamental limitation by placing data closer to users. Rather than forcing all queries to traverse the globe, replicas in regional data centers serve local traffic with single-digit millisecond latency. This isn't optimization—it's the only way to build truly global applications.

What You Will Learn

By the end of this page, you will understand the physics constraints that necessitate geographic distribution, how to design multi-region database architectures, different replication topologies for global systems, strategies for handling write conflicts in distributed data, and how major platforms achieve global scale with acceptable latency.

Understanding Global Latency

Network latency between regions is governed by physical distance and the speed of light—constraints that no amount of hardware improvement can overcome. Understanding these fundamentals is essential for designing global systems.

Speed of light constraints:

The theoretical minimum round-trip time (RTT) between locations can be calculated from distance:

RTT (minimum) = (2 × distance) / (speed of light in fiber)
RTT (minimum) = (2 × distance) / 200,000 km/s

In practice, cables don't follow straight-line paths, and routing adds overhead. Real-world RTT is typically 1.5-2x the theoretical minimum.

Real-World Inter-Region Network Latency
Route	Distance (approx)	Theoretical Min RTT	Typical RTT
US East ↔ US West	4,000 km	40 ms	60-80 ms
US East ↔ Europe	6,000 km	60 ms	80-100 ms
US West ↔ Asia (Pacific)	10,000 km	100 ms	130-180 ms
Europe ↔ Asia	8,000 km	80 ms	150-200 ms
Sydney ↔ São Paulo	13,500 km	135 ms	280-350 ms
Within same region	<100 km	<1 ms	1-5 ms

Impact on user experience:

Humans perceive delays differently at different thresholds:

< 100 ms: Feels instantaneous
100-300 ms: Noticeable but acceptable
300-1000 ms: Feels slow; users become impatient
> 1000 ms: Frustrating; users may abandon the action

The multiplication effect:

Modern web pages often require multiple sequential database operations:

Authenticate user (1 query)
Fetch user preferences (1 query)
Load primary content (1-3 queries)
Fetch related recommendations (1-2 queries)
Check notifications/alerts (1 query)

With 8 sequential queries at 200 ms each = 1.6 seconds of pure network latency, before any database or application processing time.

Parallelization helps, but many queries have dependencies (can't load user data before authenticating). Geographic distribution eliminates this multiplication problem by reducing each query's latency.

CDNs Don't Solve Everything

Content Delivery Networks (CDNs) effectively distribute static assets, but dynamic database queries cannot be cached at the edge (with rare exceptions). User-specific, real-time, or frequently-updated data must come from the database. Geographic distribution of the database itself is the only solution for dynamic content.

Multi-Region Architectures

Several architectural patterns enable geographically distributed databases. Each makes different trade-offs between consistency, latency, and operational complexity.

Pattern 1: Primary in One Region, Read Replicas Globally

The simplest multi-region pattern:

Single primary in one region handles all writes
Read replicas deployed in each region serve local reads
Writes still incur cross-region latency

Trade-offs:

✅ Simple consistency model (single source of truth)
✅ Read latency is local
❌ Write latency is global (users far from primary suffer)
❌ Primary region becomes a single point of failure

Best for: Read-heavy applications where writes are infrequent (news sites, product catalogs, reference data).

Pattern 2: Active-Active Multi-Region (Multi-Primary)

Multiple regions each have a primary that accepts writes:

Writes are processed locally with single-digit latency
Changes replicated asynchronously to other regions
Conflicts must be detected and resolved

Trade-offs:

✅ Write latency is local in every region
✅ No single region is a single point of failure
❌ Conflict resolution is complex
❌ Eventual consistency between regions

Best for: Applications where write latency matters and conflicts are manageable (social media, collaborative tools, gaming).

Pattern 3: Partitioned by Region (Geo-Sharding)

Data is partitioned so each region owns specific data:

User data lives in their home region
All operations for that data are local
Cross-region access is the exception, not the rule

Trade-offs:

✅ Both reads and writes are local for most operations
✅ No conflicts (each region owns its data)
❌ Cross-region access is slow
❌ User mobility (travel, region changes) complicates the model

Best for: Applications with strong regional data affinity (banking, healthcare, compliance-driven data residency).

Multi-Region Architecture Comparison
Pattern	Read Latency	Write Latency	Consistency	Complexity
Read Replicas Globally	Local (low)	Global (high)	Strong	Low
Active-Active Multi-Primary	Local (low)	Local (low)	Eventual	High
Geo-Sharding	Local (low)	Local (low)	Strong (per shard)	Medium

Hybrid Approaches Are Common

Production systems often combine patterns: read replicas globally for catalog data, geo-sharding for user profiles, and active-active for high-write features like messaging. Choose the pattern that fits each data domain rather than forcing one pattern across all data.

Replication Topologies for Global Distribution

How replicas are connected—the replication topology—significantly impacts latency, consistency, and failure handling in global systems.

Star Topology:

        [Europe Replica]
              |
              |
[Asia Replica]----[US Primary]----[South America Replica]
              |
              |
        [Australia Replica]

All replicas connect to a central primary:

Simple to reason about
Primary is bottleneck for all replication
If primary fails, all replication stops until failover

Ring Topology:

[US Primary] --→ [Europe] --→ [Asia] --→ [Australia] --→ [US Primary]

Changes propagate in a circle:

Distributes replication load
Latency accumulates around the ring
Single failure breaks the ring (requires healing)

Mesh Topology:

[US Primary] ←--→ [Europe]
     ↑   ↘    ↗      ↑
     |     ↘↗        |
     ↓    ↗ ↘        ↓
[Australia] ←--→ [Asia]

Every node connects to every other node:

Maximum redundancy
Changes reach all nodes quickly
Complex conflict resolution; n² connections

Hierarchical (Tiered) Topology:

              [Global Primary]
                 /       \
        [US Regional]  [EU Regional]
         /     \          /     \
    [US-W]  [US-E]   [EU-W]  [EU-C]

Regional primaries aggregate local replicas:

Reduces cross-region replication traffic
Regional failures are isolated
More complex failover procedures

Topology Selection Criteria

•Latency sensitivity: Mesh reaches all nodes fastest
•Operational simplicity: Star is easiest to manage
•Failure isolation: Hierarchical contains regional failures
•Network costs: Ring and star minimize connections
•Conflict frequency: Star/hierarchical reduce conflicts

Common Production Choices

•Most RDBMS: Star (single primary)
•CockroachDB: Mesh (all-to-all gossip)
•Cassandra: Mesh with configurable RF
•MySQL Group Replication: Mesh (within group)
•Custom large-scale: Hierarchical

Replication Factor in Distributed Systems

Distributed databases like Cassandra use 'replication factor' (RF) to control how many nodes store each piece of data. RF=3 means 3 copies. Higher RF improves availability and read performance but increases write costs and storage. Geographic placement of replicas (rack-aware, region-aware) ensures copies are distributed for fault tolerance.

Write Conflict Resolution

When multiple regions accept writes, the same data may be modified in different locations before replication propagates. This creates write conflicts that must be detected and resolved.

Types of conflicts:

Update-Update conflict: Same row updated in two regions with different values
Insert-Insert conflict: Same primary key inserted in two regions
Update-Delete conflict: Row updated in one region, deleted in another
Referential integrity conflict: Child row inserted where parent doesn't exist (yet)

Conflict detection:

Conflicts are detected during replication when incoming changes conflict with local state. Detection requires tracking change metadata:

Timestamps: When was each change made?
Version vectors: What version did each change see?
Sequence numbers: What order did changes occur in?

Resolution strategies:

Conflict Resolution Strategies

•Last-Write-Wins (LWW): Highest timestamp wins; simple but may lose valid updates
•First-Write-Wins: Earliest timestamp wins; alternative to LWW with same trade-offs
•Origin-Wins: Changes from designated 'authoritative' region always win
•Merge: Combine changes (e.g., for counters: add both increments)
•Application-callback: Notify application to resolve conflict with business logic
•Conflict-free data types (CRDTs): Data structures designed to merge without conflicts

Last-Write-Wins in detail:

LWW is the most common strategy due to its simplicity:

Region A at T=100: UPDATE users SET name='Alice' WHERE id=1
Region B at T=101: UPDATE users SET name='Alicia' WHERE id=1

With LWW: Final value is 'Alicia' (later timestamp)

Critical requirement: Accurate, synchronized clocks across all regions. Clock skew can cause "earlier" writes to have "later" timestamps, leading to unexpected results.

Limitation: LWW can silently discard valid updates. If two support agents simultaneously update a customer record—one updating phone, one updating email—LWW will keep only one agent's changes.

Merge strategies for specific data types:

For structured data, intelligent merging is possible:

Counters: Add all increments (CRDT: G-Counter)
Sets: Union of all additions, intersection of deletions (CRDT: OR-Set)
Text: Operational Transformation or CRDT (like Yjs) for collaborative editing
JSON documents: Field-level conflict detection; merge non-conflicting fields

Avoid Conflicts by Design

The best conflict resolution is no conflict at all. Design data models and write patterns to minimize conflicts: partition data by user/tenant (each user's data written in one region), use append-only logs instead of updates, or design operations that commute (counter increments, set additions).

Data Residency and Compliance

Geographic distribution isn't only about performance—legal and regulatory requirements often mandate where data can be stored and processed.

Key regulations:

Major Data Residency Regulations
Regulation	Jurisdiction	Key Requirements
GDPR	European Union	Personal data of EU residents may require processing within EU; transfers outside require safeguards
CCPA/CPRA	California, USA	Consumer privacy rights; less strict residency requirements than GDPR
LGPD	Brazil	Similar to GDPR; personal data protections for Brazilian residents
PDPA	Singapore	Data protection requirements; cross-border transfer restrictions
China's DSL/PIPL	China	Critical data must remain in China; strict transfer requirements
Russia's Data Law	Russia	Personal data of Russian citizens must be stored in Russia

Architectural implications:

Compliance requirements directly impact database architecture:

Regional isolation:

Deploy separate database clusters per regulatory region
No replication across compliance boundaries
Simplifies compliance but complicates global operations

Data classification:

Classify data by sensitivity and regulatory scope
Only restricted data remains in-region; non-sensitive data can replicate globally
Requires careful schema and query design

Encryption in transit:

Cross-region replication must use encrypted connections
Some regulations require specific encryption standards

Audit logging:

Track all cross-region data access
Demonstrate compliance during audits

Multi-Tenancy and Data Residency

For multi-tenant SaaS applications, consider tenant-level data residency configuration. Allow enterprise customers to choose their data region (EU tenants in Frankfurt, US tenants in Virginia). This often requires sharding by tenant with region as a shard key factor.

Example: GDPR-compliant global architecture:

EU cluster: Stores all EU user personal data; deployed in Frankfurt or Dublin
US cluster: Stores US user data; deployed in Virginia or Oregon
Global metadata cluster: Non-personal data (product catalog, configuration) replicated globally
Access control: API gateway routes requests to appropriate cluster based on user residency
Audit: All cross-region access logged for compliance reporting

This architecture trades some operational simplicity for regulatory compliance. Users experience low latency (their data is local), and the system meets data residency requirements.

Global Traffic Routing

Getting users to the right database region requires intelligent traffic routing at multiple layers.

DNS-based routing:

Global DNS services (AWS Route 53, Cloudflare, NS1) route users to the nearest region based on:

Geographic location: User's IP mapped to approximate location
Latency measurement: Active probing to determine actual latency
Health checks: Route away from unhealthy regions

Example Route 53 configuration:

api.example.com → Latency-based routing
- If user in US: Return US load balancer IP
- If user in EU: Return EU load balancer IP
- If user in APAC: Return Singapore load balancer IP

Application-layer routing:

For more complex routing logic:

API Gateway: Determines user's home region from authentication token
Service mesh: Routes requests to appropriate backend based on headers
Database proxy: Routes queries to correct database cluster

Handling user mobility:

Users travel. A user registered in the US may access the application from Tokyo:

Option 1: Always route to home region — Higher latency but consistent data
Option 2: Global read replicas — Read from local replica, write to home region
Option 3: Migrate user data — Moving users between regions (complex, rare)

geo-routing.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
/**
 * Geographic routing example for database connections.
 * Routes users to appropriate database based on home region.
 */
 
interface User {
  id: string;
  homeRegion: 'us-east' | 'eu-west' | 'ap-south';
}
 
interface DatabaseConfig {
  primary: string;
  readReplica: string;
}
 
const REGION_DATABASES: Record<string, DatabaseConfig> = {
  'us-east': {
    primary: 'postgres://primary.us-east.db.example.com:5432/app',
    readReplica: 'postgres://replica.us-east.db.example.com:5432/app',
  },
  'eu-west': {
    primary: 'postgres://primary.eu-west.db.example.com:5432/app',
    readReplica: 'postgres://replica.eu-west.db.example.com:5432/app',
  },
  'ap-south': {
    primary: 'postgres://primary.ap-south.db.example.com:5432/app',
    readReplica: 'postgres://replica.ap-south.db.example.com:5432/app',
  },
};
 
// Mapping from user's current location to nearest read replica
const NEAREST_READ_REPLICA: Record<string, string> = {
  'us-east': 'us-east',
  'us-west': 'us-east',  // Closest read replica
  'eu-west': 'eu-west',
  'eu-central': 'eu-west',
  'ap-south': 'ap-south',
  'ap-northeast': 'ap-south',
};
 
class GeoRouter {
  /**
   * Get database connection for write operations.
   * Writes always go to user's home region primary.
   */
  getWriteConnection(user: User): string {
    const regionConfig = REGION_DATABASES[user.homeRegion];
    if (!regionConfig) {
      throw new Error(`Unknown home region: ${user.homeRegion}`);
    }
    return regionConfig.primary;
  }
 
  /**
   * Get database connection for read operations.
   * Reads go to nearest replica (local to user's current location).
   */
  getReadConnection(user: User, currentLocation: string): string {
    const nearestRegion = NEAREST_READ_REPLICA[currentLocation] || user.homeRegion;
    const regionConfig = REGION_DATABASES[nearestRegion];
    
    if (!regionConfig) {
      // Fallback to home region
      return REGION_DATABASES[user.homeRegion].readReplica;
    }
    
    return regionConfig.readReplica;
  }
 
  /**
   * Get database for consistency-critical reads.
   * Returns home region replica to ensure user sees their own writes.
   */
  getConsistentReadConnection(user: User): string {
    return REGION_DATABASES[user.homeRegion].readReplica;
  }
}
 
// Usage example
const router = new GeoRouter();
const user: User = { id: 'user-123', homeRegion: 'us-east' };
 
// User is currently in Tokyo
const currentLocation = 'ap-northeast';
 
// Write: goes to US (home region) - higher latency but correct region
const writeConn = router.getWriteConnection(user);
 
// Read: goes to Asia replica - low latency
const readConn = router.getReadConnection(user, currentLocation);
 
// Consistent read after write: goes to US replica
const consistentReadConn = router.getConsistentReadConnection(user);

Real-World Global Database Systems

Major platforms have solved global distribution in various ways. Understanding their approaches provides practical patterns for your own systems.

Google Spanner:

Spanner is a globally distributed database that provides strong consistency across regions using TrueTime (GPS and atomic clock synchronized timestamps).

Key features:

Horizontal scaling across regions
Synchronous replication for strong consistency
SQL interface with full ACID transactions
Automatic sharding and rebalancing

Trade-off: Write latency is higher due to synchronous replication (still typically < 200ms). Read latency is local with bounded staleness reads.

CockroachDB:

Inspired by Spanner, open-source implementation with similar goals:

Key features:

PostgreSQL-compatible SQL
Automatic data placement based on table localities
Configurable replication zones for compliance
Follower reads for lower-latency eventual reads

Trade-off: Without specialized hardware (TrueTime), uses hybrid logical clocks with slightly weaker guarantees.

Amazon Aurora Global Database:

Aurora extends traditional MySQL/PostgreSQL to global scale:

Key features:

Primary region for writes
Up to 5 secondary regions for reads
Storage-level replication (< 1 second typical lag)
Cross-region failover in < 1 minute

Trade-off: Not full multi-primary; writes must go to primary region.

YouTube/Google:

YouTube uses a combination approach:

Key features:

Vitess for MySQL sharding and horizontal scaling
Regional clusters for user data
Global CDN for video content
Eventually consistent view counts (famous design choice)

Trade-off: Accepts eventual consistency for high-write data (views, likes) to achieve scale.

Global Database Systems Comparison
System	Consistency Model	Write Location	Read Latency	Complexity
Google Spanner	Strong (TrueTime)	Any region	Local (bounded)	High (specialized HW)
CockroachDB	Serializable	Any region	Local (follower)	Medium-High
Aurora Global	Strong (in region)	Primary region only	Local replicas	Low-Medium
Cassandra	Tunable (eventual)	Any region	Local	Medium
Custom (Vitess)	Varies	Varies	Varies	High (custom build)

Start With Managed Services

Unless you have specific requirements that managed services can't meet, start with Aurora Global Database, Spanner, or CockroachDB Cloud. Building a custom global database infrastructure is a multi-year investment that only the largest organizations can justify.

Summary: Geographic Distribution Through Replication

Geographic distribution enables global applications to provide fast, compliant, and resilient data access. Let's consolidate the key concepts:

Key Takeaways

•Physics constrains latency — Speed of light sets hard limits on cross-region round trips. No optimization overcomes 100+ ms for intercontinental traffic.
•Local data means local latency — Placing replicas near users reduces query latency from hundreds of milliseconds to single digits.
•Architecture patterns vary — Read replicas globally, active-active multi-primary, and geo-sharding each fit different use cases.
•Conflicts are inevitable with multi-primary — Last-write-wins, merge strategies, and CRDTs provide resolution options. Design to minimize conflicts.
•Compliance drives architecture — GDPR, data sovereignty laws, and industry regulations often mandate geographic placement of data.
•Routing complexity increases — DNS-based routing, application-layer decisions, and user mobility handling require careful design.
•Proven systems exist — Spanner, CockroachDB, Aurora Global, and Cassandra provide battle-tested foundations for global distribution.

What's next:

Geographic distribution addresses latency and compliance. But what happens when an entire region fails—a natural disaster, major cloud outage, or geopolitical event? The next page explores Disaster Recovery—how replication protects against catastrophic, region-wide failures.

Page Complete

You now understand geographic distribution as a motivation for database replication. You can design multi-region architectures, choose appropriate replication topologies, implement conflict resolution strategies, and address data residency compliance. Next, we explore disaster recovery for protecting against catastrophic failures.

3 / 4

Loading learning content...

System Design (HLD)Why Replication?

Why Database Replication Matters

LevelIntermediate

Duration90 mins

TopicWhy Replication?

3 / 4

Geographic Distribution: Serving Users Globally

The Physics of Global Latency

What You Will Learn

Understanding Global Latency

Speed of light constraints:

The theoretical minimum round-trip time (RTT) between locations can be calculated from distance:

RTT (minimum) = (2 × distance) / (speed of light in fiber)
RTT (minimum) = (2 × distance) / 200,000 km/s

In practice, cables don't follow straight-line paths, and routing adds overhead. Real-world RTT is typically 1.5-2x the theoretical minimum.

Real-World Inter-Region Network Latency
Route	Distance (approx)	Theoretical Min RTT	Typical RTT
US East ↔ US West	4,000 km	40 ms	60-80 ms
US East ↔ Europe	6,000 km	60 ms	80-100 ms
US West ↔ Asia (Pacific)	10,000 km	100 ms	130-180 ms
Europe ↔ Asia	8,000 km	80 ms	150-200 ms
Sydney ↔ São Paulo	13,500 km	135 ms	280-350 ms
Within same region	<100 km	<1 ms	1-5 ms

Impact on user experience:

Humans perceive delays differently at different thresholds:

< 100 ms: Feels instantaneous
100-300 ms: Noticeable but acceptable
300-1000 ms: Feels slow; users become impatient
> 1000 ms: Frustrating; users may abandon the action

The multiplication effect:

Modern web pages often require multiple sequential database operations:

Authenticate user (1 query)
Fetch user preferences (1 query)
Load primary content (1-3 queries)
Fetch related recommendations (1-2 queries)
Check notifications/alerts (1 query)

With 8 sequential queries at 200 ms each = 1.6 seconds of pure network latency, before any database or application processing time.

CDNs Don't Solve Everything

Multi-Region Architectures

Several architectural patterns enable geographically distributed databases. Each makes different trade-offs between consistency, latency, and operational complexity.

Pattern 1: Primary in One Region, Read Replicas Globally

The simplest multi-region pattern:

Single primary in one region handles all writes
Read replicas deployed in each region serve local reads
Writes still incur cross-region latency

Trade-offs:

✅ Simple consistency model (single source of truth)
✅ Read latency is local
❌ Write latency is global (users far from primary suffer)
❌ Primary region becomes a single point of failure

Best for: Read-heavy applications where writes are infrequent (news sites, product catalogs, reference data).

Pattern 2: Active-Active Multi-Region (Multi-Primary)

Multiple regions each have a primary that accepts writes:

Writes are processed locally with single-digit latency
Changes replicated asynchronously to other regions
Conflicts must be detected and resolved

Trade-offs:

✅ Write latency is local in every region
✅ No single region is a single point of failure
❌ Conflict resolution is complex
❌ Eventual consistency between regions

Best for: Applications where write latency matters and conflicts are manageable (social media, collaborative tools, gaming).

Pattern 3: Partitioned by Region (Geo-Sharding)

Data is partitioned so each region owns specific data:

User data lives in their home region
All operations for that data are local
Cross-region access is the exception, not the rule

Trade-offs:

✅ Both reads and writes are local for most operations
✅ No conflicts (each region owns its data)
❌ Cross-region access is slow
❌ User mobility (travel, region changes) complicates the model

Best for: Applications with strong regional data affinity (banking, healthcare, compliance-driven data residency).

Multi-Region Architecture Comparison
Pattern	Read Latency	Write Latency	Consistency	Complexity
Read Replicas Globally	Local (low)	Global (high)	Strong	Low
Active-Active Multi-Primary	Local (low)	Local (low)	Eventual	High
Geo-Sharding	Local (low)	Local (low)	Strong (per shard)	Medium

Hybrid Approaches Are Common

Replication Topologies for Global Distribution

How replicas are connected—the replication topology—significantly impacts latency, consistency, and failure handling in global systems.

Star Topology:

        [Europe Replica]
              |
              |
[Asia Replica]----[US Primary]----[South America Replica]
              |
              |
        [Australia Replica]

All replicas connect to a central primary:

Simple to reason about
Primary is bottleneck for all replication
If primary fails, all replication stops until failover

Ring Topology:

[US Primary] --→ [Europe] --→ [Asia] --→ [Australia] --→ [US Primary]

Changes propagate in a circle:

Distributes replication load
Latency accumulates around the ring
Single failure breaks the ring (requires healing)

Mesh Topology:

[US Primary] ←--→ [Europe]
     ↑   ↘    ↗      ↑
     |     ↘↗        |
     ↓    ↗ ↘        ↓
[Australia] ←--→ [Asia]

Every node connects to every other node:

Maximum redundancy
Changes reach all nodes quickly
Complex conflict resolution; n² connections

Hierarchical (Tiered) Topology:

              [Global Primary]
                 /       \
        [US Regional]  [EU Regional]
         /     \          /     \
    [US-W]  [US-E]   [EU-W]  [EU-C]

Regional primaries aggregate local replicas:

Reduces cross-region replication traffic
Regional failures are isolated
More complex failover procedures

Topology Selection Criteria

•Latency sensitivity: Mesh reaches all nodes fastest
•Operational simplicity: Star is easiest to manage
•Failure isolation: Hierarchical contains regional failures
•Network costs: Ring and star minimize connections
•Conflict frequency: Star/hierarchical reduce conflicts

Common Production Choices

•Most RDBMS: Star (single primary)
•CockroachDB: Mesh (all-to-all gossip)
•Cassandra: Mesh with configurable RF
•MySQL Group Replication: Mesh (within group)
•Custom large-scale: Hierarchical

Replication Factor in Distributed Systems

Write Conflict Resolution

When multiple regions accept writes, the same data may be modified in different locations before replication propagates. This creates write conflicts that must be detected and resolved.

Types of conflicts:

Update-Update conflict: Same row updated in two regions with different values
Insert-Insert conflict: Same primary key inserted in two regions
Update-Delete conflict: Row updated in one region, deleted in another
Referential integrity conflict: Child row inserted where parent doesn't exist (yet)

Conflict detection:

Conflicts are detected during replication when incoming changes conflict with local state. Detection requires tracking change metadata:

Timestamps: When was each change made?
Version vectors: What version did each change see?
Sequence numbers: What order did changes occur in?

Resolution strategies:

Conflict Resolution Strategies

•Last-Write-Wins (LWW): Highest timestamp wins; simple but may lose valid updates
•First-Write-Wins: Earliest timestamp wins; alternative to LWW with same trade-offs
•Origin-Wins: Changes from designated 'authoritative' region always win
•Merge: Combine changes (e.g., for counters: add both increments)
•Application-callback: Notify application to resolve conflict with business logic
•Conflict-free data types (CRDTs): Data structures designed to merge without conflicts

Last-Write-Wins in detail:

LWW is the most common strategy due to its simplicity:

Region A at T=100: UPDATE users SET name='Alice' WHERE id=1
Region B at T=101: UPDATE users SET name='Alicia' WHERE id=1

With LWW: Final value is 'Alicia' (later timestamp)

Critical requirement: Accurate, synchronized clocks across all regions. Clock skew can cause "earlier" writes to have "later" timestamps, leading to unexpected results.

Limitation: LWW can silently discard valid updates. If two support agents simultaneously update a customer record—one updating phone, one updating email—LWW will keep only one agent's changes.

Merge strategies for specific data types:

For structured data, intelligent merging is possible:

Counters: Add all increments (CRDT: G-Counter)
Sets: Union of all additions, intersection of deletions (CRDT: OR-Set)
Text: Operational Transformation or CRDT (like Yjs) for collaborative editing
JSON documents: Field-level conflict detection; merge non-conflicting fields

Avoid Conflicts by Design

Data Residency and Compliance

Geographic distribution isn't only about performance—legal and regulatory requirements often mandate where data can be stored and processed.

Key regulations:

Major Data Residency Regulations
Regulation	Jurisdiction	Key Requirements
GDPR	European Union	Personal data of EU residents may require processing within EU; transfers outside require safeguards
CCPA/CPRA	California, USA	Consumer privacy rights; less strict residency requirements than GDPR
LGPD	Brazil	Similar to GDPR; personal data protections for Brazilian residents
PDPA	Singapore	Data protection requirements; cross-border transfer restrictions
China's DSL/PIPL	China	Critical data must remain in China; strict transfer requirements
Russia's Data Law	Russia	Personal data of Russian citizens must be stored in Russia

Architectural implications:

Compliance requirements directly impact database architecture:

Regional isolation:

Deploy separate database clusters per regulatory region
No replication across compliance boundaries
Simplifies compliance but complicates global operations

Data classification:

Classify data by sensitivity and regulatory scope
Only restricted data remains in-region; non-sensitive data can replicate globally
Requires careful schema and query design

Encryption in transit:

Cross-region replication must use encrypted connections
Some regulations require specific encryption standards

Audit logging:

Track all cross-region data access
Demonstrate compliance during audits

Multi-Tenancy and Data Residency

Example: GDPR-compliant global architecture:

EU cluster: Stores all EU user personal data; deployed in Frankfurt or Dublin
US cluster: Stores US user data; deployed in Virginia or Oregon
Global metadata cluster: Non-personal data (product catalog, configuration) replicated globally
Access control: API gateway routes requests to appropriate cluster based on user residency
Audit: All cross-region access logged for compliance reporting

This architecture trades some operational simplicity for regulatory compliance. Users experience low latency (their data is local), and the system meets data residency requirements.

Global Traffic Routing

Getting users to the right database region requires intelligent traffic routing at multiple layers.

DNS-based routing:

Global DNS services (AWS Route 53, Cloudflare, NS1) route users to the nearest region based on:

Geographic location: User's IP mapped to approximate location
Latency measurement: Active probing to determine actual latency
Health checks: Route away from unhealthy regions

Example Route 53 configuration:

api.example.com → Latency-based routing
- If user in US: Return US load balancer IP
- If user in EU: Return EU load balancer IP
- If user in APAC: Return Singapore load balancer IP

Application-layer routing:

For more complex routing logic:

API Gateway: Determines user's home region from authentication token
Service mesh: Routes requests to appropriate backend based on headers
Database proxy: Routes queries to correct database cluster

Handling user mobility:

Users travel. A user registered in the US may access the application from Tokyo:

Option 1: Always route to home region — Higher latency but consistent data
Option 2: Global read replicas — Read from local replica, write to home region
Option 3: Migrate user data — Moving users between regions (complex, rare)

geo-routing.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
/**
 * Geographic routing example for database connections.
 * Routes users to appropriate database based on home region.
 */
 
interface User {
  id: string;
  homeRegion: 'us-east' | 'eu-west' | 'ap-south';
}
 
interface DatabaseConfig {
  primary: string;
  readReplica: string;
}
 
const REGION_DATABASES: Record<string, DatabaseConfig> = {
  'us-east': {
    primary: 'postgres://primary.us-east.db.example.com:5432/app',
    readReplica: 'postgres://replica.us-east.db.example.com:5432/app',
  },
  'eu-west': {
    primary: 'postgres://primary.eu-west.db.example.com:5432/app',
    readReplica: 'postgres://replica.eu-west.db.example.com:5432/app',
  },
  'ap-south': {
    primary: 'postgres://primary.ap-south.db.example.com:5432/app',
    readReplica: 'postgres://replica.ap-south.db.example.com:5432/app',
  },
};
 
// Mapping from user's current location to nearest read replica
const NEAREST_READ_REPLICA: Record<string, string> = {
  'us-east': 'us-east',
  'us-west': 'us-east',  // Closest read replica
  'eu-west': 'eu-west',
  'eu-central': 'eu-west',
  'ap-south': 'ap-south',
  'ap-northeast': 'ap-south',
};
 
class GeoRouter {
  /**
   * Get database connection for write operations.
   * Writes always go to user's home region primary.
   */
  getWriteConnection(user: User): string {
    const regionConfig = REGION_DATABASES[user.homeRegion];
    if (!regionConfig) {
      throw new Error(`Unknown home region: ${user.homeRegion}`);
    }
    return regionConfig.primary;
  }
 
  /**
   * Get database connection for read operations.
   * Reads go to nearest replica (local to user's current location).
   */
  getReadConnection(user: User, currentLocation: string): string {
    const nearestRegion = NEAREST_READ_REPLICA[currentLocation] || user.homeRegion;
    const regionConfig = REGION_DATABASES[nearestRegion];
    
    if (!regionConfig) {
      // Fallback to home region
      return REGION_DATABASES[user.homeRegion].readReplica;
    }
    
    return regionConfig.readReplica;
  }
 
  /**
   * Get database for consistency-critical reads.
   * Returns home region replica to ensure user sees their own writes.
   */
  getConsistentReadConnection(user: User): string {
    return REGION_DATABASES[user.homeRegion].readReplica;
  }
}
 
// Usage example
const router = new GeoRouter();
const user: User = { id: 'user-123', homeRegion: 'us-east' };
 
// User is currently in Tokyo
const currentLocation = 'ap-northeast';
 
// Write: goes to US (home region) - higher latency but correct region
const writeConn = router.getWriteConnection(user);
 
// Read: goes to Asia replica - low latency
const readConn = router.getReadConnection(user, currentLocation);
 
// Consistent read after write: goes to US replica
const consistentReadConn = router.getConsistentReadConnection(user);

Real-World Global Database Systems

Major platforms have solved global distribution in various ways. Understanding their approaches provides practical patterns for your own systems.

Google Spanner:

Spanner is a globally distributed database that provides strong consistency across regions using TrueTime (GPS and atomic clock synchronized timestamps).

Key features:

Horizontal scaling across regions
Synchronous replication for strong consistency
SQL interface with full ACID transactions
Automatic sharding and rebalancing

Trade-off: Write latency is higher due to synchronous replication (still typically < 200ms). Read latency is local with bounded staleness reads.

CockroachDB:

Inspired by Spanner, open-source implementation with similar goals:

Key features:

PostgreSQL-compatible SQL
Automatic data placement based on table localities
Configurable replication zones for compliance
Follower reads for lower-latency eventual reads

Trade-off: Without specialized hardware (TrueTime), uses hybrid logical clocks with slightly weaker guarantees.

Amazon Aurora Global Database:

Aurora extends traditional MySQL/PostgreSQL to global scale:

Key features:

Primary region for writes
Up to 5 secondary regions for reads
Storage-level replication (< 1 second typical lag)
Cross-region failover in < 1 minute

Trade-off: Not full multi-primary; writes must go to primary region.

YouTube/Google:

YouTube uses a combination approach:

Key features:

Vitess for MySQL sharding and horizontal scaling
Regional clusters for user data
Global CDN for video content
Eventually consistent view counts (famous design choice)

Trade-off: Accepts eventual consistency for high-write data (views, likes) to achieve scale.

Global Database Systems Comparison
System	Consistency Model	Write Location	Read Latency	Complexity
Google Spanner	Strong (TrueTime)	Any region	Local (bounded)	High (specialized HW)
CockroachDB	Serializable	Any region	Local (follower)	Medium-High
Aurora Global	Strong (in region)	Primary region only	Local replicas	Low-Medium
Cassandra	Tunable (eventual)	Any region	Local	Medium
Custom (Vitess)	Varies	Varies	Varies	High (custom build)

Start With Managed Services

Summary: Geographic Distribution Through Replication

Geographic distribution enables global applications to provide fast, compliant, and resilient data access. Let's consolidate the key concepts:

Key Takeaways

•Physics constrains latency — Speed of light sets hard limits on cross-region round trips. No optimization overcomes 100+ ms for intercontinental traffic.
•Local data means local latency — Placing replicas near users reduces query latency from hundreds of milliseconds to single digits.
•Architecture patterns vary — Read replicas globally, active-active multi-primary, and geo-sharding each fit different use cases.
•Conflicts are inevitable with multi-primary — Last-write-wins, merge strategies, and CRDTs provide resolution options. Design to minimize conflicts.
•Compliance drives architecture — GDPR, data sovereignty laws, and industry regulations often mandate geographic placement of data.
•Routing complexity increases — DNS-based routing, application-layer decisions, and user mobility handling require careful design.
•Proven systems exist — Spanner, CockroachDB, Aurora Global, and Cassandra provide battle-tested foundations for global distribution.

What's next:

Page Complete

3 / 4