Geo-Distributed Architecture

Once you've decided to deploy across multiple regions, you face a fundamental architectural choice: should all regions actively serve traffic (active-active), or should some regions wait in reserve for failures (active-passive)?

This decision profoundly impacts your system's complexity, consistency model, operational characteristics, and cost structure. There's no universally correct answer—the right choice depends on your specific requirements, organizational capabilities, and acceptable trade-offs.

In this page, we'll deeply examine both patterns, understanding not just what they are but when each is appropriate and how to implement them effectively.

In an active-passive architecture (also called primary-secondary, master-slave, or standby), one region handles all production traffic while one or more secondary regions maintain synchronized copies of data and infrastructure, ready to take over if the primary fails.

Anatomy of Active-Passive

Primary Region:

• Receives 100% of production traffic
• Hosts the authoritative database
• All write operations occur here
• Full production infrastructure running at scale

Secondary Region(s):

• No production traffic under normal operation
• Maintains replicated copy of data (typically asynchronous)
• Infrastructure may be running at reduced capacity (hot standby) or not running at all (cold standby)
• Regular testing ensures failover capability

Standby Types

The secondary region's readiness level significantly affects both cost and recovery time:

Hot Standby:

• Full infrastructure running in passive region
• Data replicated with minimal lag
• Application servers running but not serving traffic
• Failover possible in minutes
• Most expensive passive option

Warm Standby:

• Core infrastructure running (databases, critical services)
• Non-critical services scaled down or stopped
• Data replication active with acceptable lag
• Failover requires some scaling/startup (10-30 minutes)
• Moderate cost

Cold Standby:

• Minimal infrastructure running
• Data replicated to storage (may have significant lag)
• Infrastructure must be provisioned during failover
• Failover time measured in hours
• Lowest passive cost

Advantages of Active-Passive

Simplicity:

• Single source of truth for all data
• No conflict resolution required
• Transactions work normally in primary
• Simpler mental model for developers

Consistency:

• Strong consistency by default
• No replication lag affects user experience (in primary)
• No split-brain scenarios during normal operation

Cost Efficiency:

• Passive region can run at reduced capacity
• No cross-region traffic during normal operation
• Data transfer costs limited to replication

Easier Operations:

• Clear ownership model (primary is authoritative)
• Simpler monitoring (focus on primary, verification of secondary)
• Straightforward capacity planning

Disadvantages of Active-Passive

Wasted Capacity:

• Secondary infrastructure sits idle most of the time
• Still paying for capacity that's rarely used
• Cold standby addresses this but increases RTO

Latency for Distant Users:

• All users connect to primary region
• Users far from primary experience geography-based latency
• CDN can help with static content but not dynamic operations

Failover Complexity:

• Failover is by definition an exceptional event
• Exceptional events are harder to test and validate
• Staff may be unfamiliar with failover procedures when needed

Data Loss Risk:

• Asynchronous replication means potential data loss during failover (RPO > 0)
• Synchronous replication is expensive and adds latency
• Trade-off between RPO and performance

Failover is the critical process of promoting the passive region to active when the primary fails. Getting this right is fundamental to the value proposition of multi-region deployment.

Failover Triggers

Automatic Failover:

• Health checks detect primary unavailability
• Automated systems initiate failover sequence
• Fastest response time but highest risk of false positives

Manual Failover:

• Operators detect issue and decide to failover
• Human validation prevents false positive triggering
• Slower response but more considered

Hybrid:

• Automated detection with human approval gate
• Balance between speed and safety
• Common for critical systems

The Failover Sequence

A typical active-passive failover follows this sequence:

Critical Failover Considerations

DNS TTL: DNS-based failover is limited by TTL (Time To Live) settings. If DNS TTL is 1 hour, clients will continue attempting to reach the failed primary for up to an hour after failover. Common mitigations:

• Keep TTL low (60-300 seconds) for production endpoints
• Use anycast or other non-DNS routing for faster failover
• Implement client-side retry with alternate endpoints

Split Brain: If the primary is network-partitioned (not actually down), both regions might believe they're primary:

• Old primary still processing requests from cached DNS
• New primary accepting requests after failover
• Both writing to their local databases
• Severe data inconsistency results

Mitigation requires fencing: ensuring the old primary cannot process writes after failover:

• STONITH (Shoot The Other Node In The Head): Forcibly power off old primary
• Lease-based: Primary requires actively renewed lease to process writes
• Quorum-based: Write requires acknowledgment from majority of replicas

Data Loss Window: Asynchronous replication means some committed transactions in primary may not have replicated to secondary:

• Identify unreplicated transactions during failover
• Communicate potential data loss to affected users
• Have process to reconcile if old primary recovers

Failback Considerations

After failover, you eventually want to return to the original topology (failback):

1. Recover original primary: Bring original primary back online
2. Replicate new data: Sync changes made during failover to original primary
3. Resolve conflicts: Handle any transactions from split-brain period
4. Reverse failover: Promote original primary, demote temporary primary

Failback is often more complex than failover because there's now divergent data to reconcile.

In an active-active architecture, all regions simultaneously serve production traffic. Each region can process both read and write operations, maintaining independent but synchronized copies of data.

Anatomy of Active-Active

All Regions:

• Receive production traffic (typically routed by proximity)
• Maintain local copy of data
• Process both reads and writes
• Replicate changes to other regions (typically asynchronous)
• Operate independently if network partitioned from other regions

Active-Active Variants

Fully Symmetric Active-Active:

• All regions handle all operations equally
• Data replicated bidirectionally between all regions
• Most complex but most resilient

Regional Active-Active:

• Users assigned to "home" region
• Home region is authoritative for that user's data
• Cross-region operations route to home region
• Simpler consistency model at cost of some latency

Active-Active with Leader:

• All regions active for read/write
• One region designated as coordination point for certain operations
• Hybrid of active-active and active-passive benefits

Why Active-Active Is Harder

Active-active introduces fundamental distributed systems challenges:

Advantages of Active-Active

Low Latency Globally:

• Users connect to nearby region
• Latency is regional, not cross-continental
• Better user experience everywhere

No Wasted Capacity:

• All infrastructure is serving production traffic
• Better ROI on multi-region investment
• Natural load distribution

Seamless Failover:

• No failover process needed (regions are already handling traffic)
• Failed region's traffic simply shifts to other regions
• RTO approaches zero for properly implemented active-active

Blast Radius Isolation:

• Regional issues affect only that region's users
• Bad deployments can be rolled back before global exposure
• Natural A/B testing infrastructure

Disadvantages of Active-Active

Complexity:

• Dramatically more complex than active-passive
• Conflict resolution logic pervades application
• Distributed systems expertise required

Eventual Consistency:

• Users may see stale data (typically seconds, but potentially longer)
• Application must handle consistency anomalies
• Some features may be difficult to implement correctly

Data Integrity Challenges:

• Unique constraints, foreign keys, and invariants are harder to maintain
• Application-level enforcement often required
• Risk of subtle bugs that manifest rarely

Higher Cost:

• Full infrastructure in all regions
• Cross-region data transfer costs
• Significant engineering investment

When the same data is modified in multiple regions before replication occurs, a conflict exists. Resolving these conflicts correctly is fundamental to active-active integrity.

Why Conflicts Occur

Conflicts are inevitable when:

1. Two users modify the same record in different regions
2. Replication lag exceeds modification frequency
3. Network partition separates regions temporarily

Conflict Resolution Strategies

Last-Write-Wins (LWW):

• Attach timestamp to each write
• Latest timestamp wins during conflict
• Simple but loses data (earlier writes are discarded)
• Requires synchronized clocks (NTP can drift)

Highest-Source-Wins:

• Assign precedence to regions (e.g., Region A > Region B)
• Use precedence when timestamps equal
• Simple but creates implicit primary

Custom Merge:

• Application-specific logic merges conflicting values
• Can preserve information from both writes
• Requires careful design per data type
• Example: Shopping cart - merge by taking union of items

CRDTs (Conflict-free Replicated Data Types):

• Data structures designed to merge automatically
• No conflict resolution needed—merges are deterministic
• Limited to specific data types (counters, sets, maps, etc.)
• Growing ecosystem of CRDT implementations

User Resolution:

• Present conflicts to users for manual resolution
• Preserves all information and human judgment
• Poor user experience for frequent conflicts
• Appropriate for document editing, rarely for real-time systems

Preventing Conflicts

Often the best conflict resolution is conflict prevention:

Data Partitioning: Assign data to specific regions based on user or tenant:

• User's home region owns their data
• Cross-region operations route to home region
• Conflicts impossible for home-region data
• Trade-off: latency for cross-region access

Coordination Services: Use distributed locks or consensus for critical operations:

• Acquire lock before write
• Prevents concurrent modification
• Trade-off: performance and availability impact

Append-Only Data Models: Design data as append-only logs:

• Conflicts become concurrent events in log
• Resolution happens at read time
• Natural for event sourcing architectures

Single-Regional Operations: Route certain operation types to single region:

• Writes to region A, reads from any region
• Simpler consistency at cost of write latency
• Hybrid of active-active and active-passive

Real-World Conflict Examples

E-commerce Inventory:

• Conflict: Two regions sell last item simultaneously
• Resolution: Oversell and compensate (common in e-commerce)
• Better: Reserve inventory through coordination

Social Media Counter:

• Conflict: Like count updated in multiple regions
• Resolution: CRDT counter (grow-only counter)
• Result: Counts converge without data loss

User Profile:

• Conflict: User updates profile in two regions
• Resolution: Last-write-wins (usually acceptable)
• Better: Field-level LWW to preserve more data

Let's systematically compare active-passive and active-active across key dimensions:

Availability

Latency

Consistency

Cost

Operational Complexity

With the detailed analysis above, let's develop a decision framework:

Choose Active-Passive When:

Strong consistency is critical:

• Financial transactions requiring ACID guarantees
• Systems where conflicts could cause compliance issues
• Applications where eventual consistency would confuse users

Team expertise is limited:

• Team lacks distributed systems experience
• Cannot invest in training/hiring for active-active complexity
• Operational maturity isn't sufficient for active-active

Cost is constrained:

• Budget doesn't support full multi-region capacity
• Primary goal is disaster recovery, not latency
• Return on active-active doesn't justify investment

RTO requirements are relaxed:

• Minutes to hours of recovery time is acceptable
• Business can tolerate brief outages during failover
• SLA requirements are 99.9% or lower

Choose Active-Active When:

Low latency is required globally:

• Real-time applications (gaming, communication, trading)
• User experience directly impacts business metrics
• Competing with geo-distributed alternatives

Near-zero RTO is required:

• SLA requirements of 99.99% or higher
• Cost of any downtime exceeds multi-region investment
• Continuous availability is core to value proposition

Eventual consistency is acceptable:

• Application can be designed for eventual consistency
• Conflicts can be resolved sensibly
• Users won't be confused by consistency delays

Team can handle complexity:

• Strong distributed systems expertise
• Operational maturity to manage complexity
• Engineering culture values thorough testing and observability

Hybrid Approaches

Read Active-Active, Write Active-Passive:

• Reads served from any region
• Writes routed to primary region
• Simpler consistency for writes, low latency for reads
• Works well when reads dominate traffic (common)

Active-Active with Leader Election:

• Most operations are active-active
• Coordinatedoperations (ID generation, critical transactions) go through leader
• Balance of latency and consistency

Feature-Based Patterns:

• Different features use different patterns
• Real-time features: active-active
• Transactional features: active-passive
• Complexity: only where needed

These hybrid approaches often provide better tradeoff profiles than pure active-passive or pure active-active.

We've deeply examined the two fundamental multi-region patterns. Let's consolidate the key insights:

What's next:

Both patterns depend on data replication across regions. The next page dives deep into data replication: synchronous vs asynchronous replication, consistency models, replication lag management, and strategies for keeping data synchronized across geographic distances.