System Design (HLD)High Availability

Availability vs Consistency Trade-offs

LevelAdvanced

Duration75 mins

TopicHigh Availability

5 / 5

Practical Compromises

Practical Engineering: Navigating the Real World

The CAP theorem describes constraints, but engineers build systems. Real-world systems don't simply "choose CP" or "choose AP"—they employ creative compromises that deliver practical outcomes better than either extreme.

This final page of the module is about engineering pragmatism—the patterns, techniques, and operational practices that allow systems to:

Provide strong consistency most of the time while remaining available when it matters.
Accept eventual consistency for most operations while protecting critical paths.
Recover gracefully from temporary inconsistency states.
Balance theoretical purity with operational reality.

These are the lessons learned from building and operating systems at scale.

What You Will Master

By the end of this page, you will have a toolkit of practical compromises used by real systems—patterns you can apply directly to your own designs, with concrete implementation guidance and operational considerations.

Graceful Degradation Patterns

The best systems don't fail completely—they degrade gracefully, maintaining partial functionality while clearly communicating their reduced capabilities.

Pattern 1: Consistency Tiers

Not all data is equally critical. Segment your data into consistency tiers and handle each appropriately during degradation.

Consistency Tier Framework
Tier	Data Examples	Normal Mode	Degraded Mode
Tier 1: Critical	Account balances, inventory, credentials	Strong consistency	Reject operations (fail closed)
Tier 2: Important	Orders, user profiles, preferences	Strong consistency	Accept with warning; queue for verification
Tier 3: Nice-to-have	Analytics, recommendations, feeds	Eventual consistency	Continue with stale data
Tier 4: Disposable	Caches, derived data, previews	Best effort	Skip entirely or serve stale

Pattern 2: Progressive Unavailability

Rather than all-or-nothing availability, progressively shed load and features as conditions worsen.

Stage 1 (Healthy): All features available, full consistency.

Stage 2 (Stressed): Disable Tier 4 features (analytics collection, real-time recommendations). Reduce Tier 3 refresh rates.

Stage 3 (Degraded): Accept Tier 2 data with async verification. Alert operators. Show degradation indicators to users.

Stage 4 (Critical): Only Tier 1 operations proceed. Non-critical pages show "service maintenance" message.

Stage 5 (Failure): Full outage. Incident response activated.

Pattern 3: Read Path vs Write Path Separation

Reads and writes have different consistency requirements. Handle them independently.

graceful-degradation.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
// Graceful degradation implementation
 
enum SystemHealth {
  HEALTHY = 'healthy',
  STRESSED = 'stressed',
  DEGRADED = 'degraded',
  CRITICAL = 'critical',
}
 
interface DegradationConfig {
  tier1Features: string[]; // Always available until critical
  tier2Features: string[]; // Reduced in degraded
  tier3Features: string[]; // Disabled when stressed
  tier4Features: string[]; // First to go
}
 
class GracefulController {
  private currentHealth: SystemHealth = SystemHealth.HEALTHY;
 
  async handleRequest(feature: string, operation: () => Promise<any>): Promise<any> {
    const tier = this.getFeatureTier(feature);
    
    switch (this.currentHealth) {
      case SystemHealth.HEALTHY:
        // Normal operation for all tiers
        return await operation();
 
      case SystemHealth.STRESSED:
        if (tier === 4) {
          // Skip Tier 4 entirely
          return { status: 'skipped', reason: 'system_stressed' };
        }
        if (tier === 3) {
          // Tier 3 with reduced quality
          return await this.withReducedQuality(operation);
        }
        return await operation();
 
      case SystemHealth.DEGRADED:
        if (tier >= 3) {
          return { status: 'skipped', reason: 'system_degraded' };
        }
        if (tier === 2) {
          // Accept but queue for verification
          const result = await operation();
          await this.queueForVerification(feature, result);
          return { ...result, _warning: 'pending_verification' };
        }
        return await operation();
 
      case SystemHealth.CRITICAL:
        if (tier > 1) {
          return { status: 'unavailable', reason: 'system_critical' };
        }
        // Only Tier 1 proceeds
        return await operation();
    }
  }
 
  // Reduce quality for reads: use stale cache, skip enrichments
  private async withReducedQuality(operation: () => Promise<any>): Promise<any> {
    return await operation({ 
      skipEnrichments: true,
      cacheOnly: true,
      maxStaleSeconds: 300,
    });
  }
 
  // Queue operation for later verification
  private async queueForVerification(feature: string, result: any): Promise<void> {
    await this.verificationQueue.enqueue({
      feature,
      result,
      timestamp: Date.now(),
    });
  }
}
 
// Usage in request handler
const controller = new GracefulController();
 
app.post('/api/orders', async (req, res) => {
  const result = await controller.handleRequest('order.create', async () => {
    return await orderService.create(req.body);
  });
 
  if (result._warning) {
    res.setHeader('X-Service-Warning', result._warning);
  }
  
  res.json(result);
});

Fail Loud, Not Silent

When operating in degraded mode, make it visible. Return headers, show UI indicators, and alert operators. Silent degradation leads to incorrect assumptions about data quality. Users and systems should know when they're operating with reduced guarantees.

Hybrid Architectures: The Best of Both Worlds

Hybrid architectures combine CP and AP components strategically, applying each where appropriate within the same system.

Pattern: CP Core with AP Edge

The "source of truth" maintains strong consistency, while edge caches and read replicas provide high availability for reads.

hybrid-architecture.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
HYBRID ARCHITECTURE: CP CORE + AP EDGE
═══════════════════════════════════════════════════════════════════
 
                         ┌─────────────────────────────────┐
    Users ─────────────▶ │     Edge/CDN (AP Layer)         │
                         │  • Cached reads (stale OK)      │
                         │  • Static content               │
                         │  • Write-through to core        │
                         └───────────────┬─────────────────┘
                                         │
                         ┌───────────────▼─────────────────┐
    Internal ──────────▶ │    Application Layer            │
    Services             │  • Routes by operation type     │
                         │  • Enforces consistency rules   │
                         └───────────────┬─────────────────┘
                                         │
              ┌──────────────────────────┼──────────────────────────┐
              │                          │                          │
              ▼                          ▼                          ▼
    ┌─────────────────┐        ┌─────────────────┐        ┌─────────────────┐
    │  Read Replicas  │        │   CP Core DB    │        │  Event Stream   │
    │  (Eventually    │◀───────│  (Source of     │───────▶│  (Eventually    │
    │   Consistent)   │  async │   Truth)        │  async │   Consistent)   │
    └─────────────────┘        └─────────────────┘        └─────────────────┘
         │                            │                          │
         ▼                            ▼                          ▼
    Catalog browsing           Write operations            Analytics
    Product pages              Transactions                Notifications
    Search results             Inventory updates           Audit logs

Pattern: Saga for Distributed Transactions

When a transaction spans multiple services, use the Saga pattern: a sequence of local transactions with compensating actions if any step fails.

Example: E-commerce Order Saga

Reserve Inventory → If fails, stop.
Process Payment → If fails, release inventory.
Create Shipment → If fails, refund payment, release inventory.
Confirm Order → Success!

Each step is a local transaction (CP within that service). The saga provides eventual consistency across services without distributed locking.

Pattern: CQRS (Command Query Responsibility Segregation)

Separate the write model (commands) from the read model (queries). Each can have different consistency characteristics.

CQRS Consistency Split
Aspect	Command Side (Write)	Query Side (Read)
Consistency	Strong (CP)	Eventual (AP)
Optimized For	Correctness	Performance
Data Model	Normalized, transactional	Denormalized, cached
Scaling	Single leader or consensus	Unlimited read replicas
Latency	Higher (consensus overhead)	Lower (local reads)

cqrs-implementation.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
// CQRS Implementation Example
 
// Command Service: Strong Consistency
class OrderCommandService {
  private writeDb: PostgresClient; // Single CP database
 
  async createOrder(command: CreateOrderCommand): Promise<OrderId> {
    return await this.writeDb.transaction(async (tx) => {
      // Validate inventory (strong read)
      const inventory = await tx.query(
        'SELECT quantity FROM inventory WHERE sku = $1 FOR UPDATE',
        [command.sku]
      );
      
      if (inventory.quantity < command.quantity) {
        throw new InsufficientInventoryError();
      }
 
      // Atomically update inventory and create order
      await tx.query(
        'UPDATE inventory SET quantity = quantity - $1 WHERE sku = $2',
        [command.quantity, command.sku]
      );
 
      const order = await tx.query(
        'INSERT INTO orders (sku, quantity, user_id) VALUES ($1, $2, $3) RETURNING id',
        [command.sku, command.quantity, command.userId]
      );
 
      // Publish event for query side (async)
      await this.eventBus.publish(new OrderCreatedEvent(order.id, command));
 
      return order.id;
    });
  }
}
 
// Query Service: Eventual Consistency
class OrderQueryService {
  private readReplicas: ReadReplica[];
  private cache: RedisClient;
 
  constructor() {
    // Subscribe to events to update read models
    this.eventBus.subscribe('OrderCreated', this.handleOrderCreated.bind(this));
    this.eventBus.subscribe('OrderShipped', this.handleOrderShipped.bind(this));
  }
 
  async getOrder(orderId: OrderId): Promise<OrderView | null> {
    // Try cache first
    const cached = await this.cache.get(`order:${orderId}`);
    if (cached) return JSON.parse(cached);
 
    // Load from read-optimized view
    const order = await this.readReplicas.random().query(
      'SELECT * FROM order_views WHERE id = $1',
      [orderId]
    );
 
    if (order) {
      await this.cache.setex(`order:${orderId}`, 300, JSON.stringify(order));
    }
 
    return order;
  }
 
  async getOrdersForUser(userId: UserId): Promise<OrderView[]> {
    // Always serve from cache/replica (eventual consistency OK for list view)
    return await this.readReplicas.random().query(
      'SELECT * FROM order_views WHERE user_id = $1 ORDER BY created_at DESC',
      [userId]
    );
  }
 
  // Event handlers update read models
  private async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    await this.readDb.query(
      'INSERT INTO order_views (...) VALUES (...)',
      [/* denormalized order data */]
    );
    await this.cache.del(`orders:user:${event.userId}`);
  }
}

CQRS Complexity

CQRS adds significant complexity: event sourcing, eventual consistency between models, and potential for query side to lag. Use it when the performance benefits of separated read/write scaling justify the operational overhead—not as a default pattern.

Conflict Resolution Strategies

When you choose availability, conflicts will occur. Having a robust conflict resolution strategy is essential for system integrity.

Strategy 1: Last-Write-Wins (LWW)

The simplest strategy: the most recent write (by timestamp) wins.

Pros: Simple, deterministic, no manual intervention. Cons: Can silently lose data; clock skew can cause "wrong" winner. Best for: Non-critical data, profiles with single-user ownership, caches.

conflict-resolution-strategies.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
// Conflict resolution strategy implementations
 
interface ConflictingVersions<T> {
  versions: Array<{
    value: T;
    timestamp: number;
    nodeId: string;
    vectorClock?: VectorClock;
  }>;
}
 
// Strategy 1: Last-Write-Wins
function resolveByLWW<T>(conflict: ConflictingVersions<T>): T {
  return conflict.versions.reduce((latest, current) => 
    current.timestamp > latest.timestamp ? current : latest
  ).value;
}
 
// Strategy 2: Merge with Custom Logic
function resolveShoppingCart(conflict: ConflictingVersions<ShoppingCart>): ShoppingCart {
  const allItems = new Map<string, CartItem>();
 
  // Union all items across versions
  for (const version of conflict.versions) {
    for (const item of version.value.items) {
      const existing = allItems.get(item.sku);
      if (!existing || item.addedAt > existing.addedAt) {
        allItems.set(item.sku, item);
      }
    }
  }
 
  // For removed items, use most recent action
  const removals = new Map<string, { removedAt: number }>();
  for (const version of conflict.versions) {
    for (const removal of version.value.removedItems || []) {
      const existing = removals.get(removal.sku);
      if (!existing || removal.removedAt > existing.removedAt) {
        removals.set(removal.sku, removal);
      }
    }
  }
 
  // Apply removals
  for (const [sku, removal] of removals) {
    const item = allItems.get(sku);
    if (item && removal.removedAt > item.addedAt) {
      allItems.delete(sku);
    }
  }
 
  return { items: Array.from(allItems.values()) };
}
 
// Strategy 3: Conflict to Human
interface ConflictRecord<T> {
  key: string;
  versions: ConflictingVersions<T>;
  detectedAt: number;
  status: 'pending' | 'resolved';
  resolvedBy?: string;
  resolution?: T;
}
 
async function escalateToHuman<T>(
  key: string,
  conflict: ConflictingVersions<T>
): Promise<void> {
  const record: ConflictRecord<T> = {
    key,
    versions: conflict,
    detectedAt: Date.now(),
    status: 'pending',
  };
 
  await conflictQueue.enqueue(record);
  await alerting.notifyConflictReview(record);
  
  // System may continue with a default (e.g., LWW) until human resolves
}
 
// Strategy 4: Vector Clock with Conflict Detection
function detectConflictWithVectorClock<T>(
  conflict: ConflictingVersions<T>
): { isConflict: boolean; winner?: T } {
  // Sort by vector clock
  const sorted = [...conflict.versions].sort((a, b) => 
    compareVectorClocks(a.vectorClock!, b.vectorClock!)
  );
 
  // If all versions are causally ordered, no conflict
  for (let i = 0; i < sorted.length - 1; i++) {
    if (!happensBefore(sorted[i].vectorClock!, sorted[i + 1].vectorClock!)) {
      // Concurrent writes detected - true conflict
      return { isConflict: true };
    }
  }
 
  // Causally ordered - last one wins
  return { isConflict: false, winner: sorted[sorted.length - 1].value };
}

Strategy Selection Guide

Conflict Resolution Strategy Selection
Strategy	Data Type	Conflict Frequency	Acceptable Data Loss
Last-Write-Wins	Single-owner, low-cardinality	Low	Yes (loses loser's write)
Merge/Union	Sets, counters, accumulators	Any	No (preserves all)
Application Logic	Complex domain objects	Medium	Depends on logic
Human Review	Critical business data	Low (should be!)	No (requires resolution)
CRDTs	Collaborative, real-time	High	No (by design)

Monitor Conflict Rates

Track how often conflicts occur and how they're resolved. A spike in conflicts may indicate a design problem (too much contention on hot keys) or an operational issue (replica lag). If human-resolved conflicts are backing up, your AP choice may not be sustainable.

Operational Practices for Consistency

Architecture is only half the story. Operational practices determine whether your consistency choices hold up in production.

Practice 1: Chaos Engineering for Partitions

Regularly inject network partitions in non-production (and carefully in production) to verify behavior.

Partition Testing Checklist

•Does CP behavior actually reject requests during partition? Or does it silently serve stale data?
•Does AP behavior properly merge conflicts when partition heals?
•Do alerts fire appropriately during partition events?
•Can operators manually intervene if needed?
•Does the system automatically recover when partition heals?
•Are there any edge cases where the system enters an unrecoverable state?

Practice 2: Runbooks for Consistency Incidents

Create detailed runbooks for consistency-related incidents:

Partition Detected:

Verify partition with network diagnostics.
Identify which zones/replicas are affected.
Confirm degraded mode activated correctly.
Monitor for unexpected behavior.
Prepare for post-partition reconciliation.

Conflict Storm (many conflicts occurring):

Identify source of contention.
Consider temporary strong consistency for affected keys.
Increase reconciliation workers if queue is backing up.
Investigate root cause (hot key? failed node returning?).

Stale Data Reported:

Verify replication lag.
Check for stuck transactions or locks.
Force refresh if necessary.
Investigate why monitoring didn't alert.

consistency-monitoring-dashboard.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Grafana dashboard configuration for consistency monitoring
 
dashboard:
  title: "Consistency Health"
  
panels:
  - title: "Replication Lag (P99)"
    query: |
      histogram_quantile(0.99, 
        rate(replication_lag_seconds_bucket[5m])
      )
    alert:
      - name: "High Replication Lag"
        condition: "> 10s"
        severity: "warning"
      - name: "Critical Replication Lag"
        condition: "> 60s"
        severity: "critical"
 
  - title: "Consistency Level Distribution"
    query: |
      rate(operations_total[5m]) by (consistency_level)
    description: "Track what consistency levels are actually being used"
 
  - title: "Conflict Rate"
    query: |
      rate(conflicts_detected_total[5m])
    alert:
      - name: "High Conflict Rate"
        condition: "> 10/minute"
        severity: "warning"
 
  - title: "Partition Events"
    query: |
      increase(partition_events_total[1h])
    description: "Track how often partitions are detected"
 
  - title: "Degraded Mode Activations"
    query: |
      changes(system_health_state[1d])
    description: "How often is the system entering degraded mode?"
 
  - title: "Pending Conflict Reviews"
    query: |
      conflict_queue_size
    alert:
      - name: "Conflict Backlog"
        condition: "> 100"
        severity: "warning"
 
  - title: "Reconciliation Progress"
    query: |
      rate(reconciliation_operations_total[5m])
    description: "Post-partition reconciliation throughput"

Practice 3: Regular Consistency Audits

Periodically verify that data across replicas is actually consistent:

Sample-based verification: Read the same records from multiple replicas; compare values.
Checksum comparison: Compute checksums of data ranges across replicas.
Reconciliation reports: Review how many conflicts occurred, how they were resolved.
Consistency SLI: Measure what percentage of reads return the latest value (test with known writes).

Practice 4: Incident Reviews

After any consistency incident:

What was the root cause?
Did the system behave as designed?
Were alerts timely and actionable?
Did runbooks work?
What changes would prevent recurrence?

Practice Makes Perfect

Run consistency drills regularly. Like fire drills, they ensure the team knows what to do when real incidents occur. Simulate partition scenarios, practice failovers, and verify that automated systems work as expected.

Real-World Compromise Patterns from Production

Let's examine specific patterns used by major systems to balance consistency and availability in practice.

Facebook's TAO: Read-Your-Writes at Scale

•Problem: 2+ billion users; strong consistency would be too slow.
•Solution: Read-your-writes per user; eventual consistency globally.
•How: After write, user's session reads from master for a short window. Other reads go to nearest replica.
•Result: Users see their own actions immediately; friends see updates within seconds. No user notices the lag.

Uber's Ringpop: Consistent Hashing with Ownership

•Problem: Rides are time-sensitive; can't have double-dispatching or lost rides.
•Solution: Each key (ride ID) has a single owner node at any time.
•How: Consistent hashing assigns ownership; SWIM protocol detects failures and reassigns.
•Result: Strong consistency for each ride (only owner writes); high availability through fast failover.

LinkedIn's Espresso: Caught Reads

•Problem: High read volume; strong consistency for all reads too expensive.
•Solution: 'Caught reads'—include timestamp of minimum acceptable freshness.
•How: Client specifies 'I need data from at least timestamp T'. Replica only serves if it has replicated past T.
•Result: Read-your-writes without routing all reads to master. Reads can go to any caught-up replica.

Slack's Message Ordering

•Problem: Messages must appear in order; but availability is critical for collaboration.
•Solution: Per-channel ordering, not global ordering.
•How: Each channel has a sequence number. Messages within a channel are strictly ordered. Messages across channels can be interleaved.
•Result: Strong consistency where it matters (conversation flow). High availability because channels are independent.

Common Themes in Successful Compromises

Scope consistency narrowly: Per-user, per-channel, per-entity—not global.
Make freshness visible: Clients specify or understand staleness tolerance.
Use ownership patterns: Single writer eliminates write conflicts.
Fast failure detection: The faster you detect problems, the shorter the compromise period.
Automatic recovery: Manual recovery is too slow for modern availability expectations.

Study Production Systems

Read engineering blogs from companies operating at scale: Facebook, Google, Uber, Netflix, LinkedIn. Their published architecture papers are a goldmine of practical compromise patterns refined through years of production experience.

Future Directions in Consistency

The field of distributed consistency continues to evolve. Being aware of emerging approaches helps you prepare for future options.

Emerging Technologies

Future Consistency Approaches

•Programmable Consistency: Languages and frameworks that let developers specify consistency requirements declaratively, with the system choosing the optimal implementation.
•Hardware-Assisted Ordering: Intel's Optane, persistent memory, and hardware timestamps reducing the software overhead of consistency protocols.
•Global Time Services: Wider availability of precise time (TrueTime-like services) enabling Spanner-style consistency without Google's infrastructure.
•CRDT Standardization: Growing libraries and frameworks for CRDTs making conflict-free designs more accessible.
•Formal Verification: Increasing use of formal methods (TLA+, Alloy) to prove consistency properties of distributed systems before deployment.
•Edge Computing Consistency: New models for consistency across edge locations with intermittent connectivity—beyond traditional datacenter assumptions.

The Trend: Toward Tunability

The industry is moving away from single-consistency-model databases toward systems that offer:

Per-query consistency selection.
Automatic consistency adaptation based on conditions.
Clear visibility into consistency guarantees being provided.
Better tools for understanding and debugging consistency issues.

This trend reflects the reality that consistency requirements are application-specific and even operation-specific—not something that should be baked into infrastructure once and forgotten.

Stay Current

Follow research from systems conferences (OSDI, SOSP, NSDI) and industry blogs. The state of the art in consistency is advancing, and techniques considered impractical today may become standard tomorrow.

Module Summary: Availability vs Consistency Mastery

We've completed a comprehensive journey through the availability vs consistency trade-off—one of the most fundamental concepts in distributed systems. Let's consolidate the wisdom from this entire module.

Module Key Takeaways

•CAP is a constraint, not a limitation — Understanding the theorem enables you to make informed trade-offs rather than fighting impossible battles.
•Consistency exists on a spectrum — Linearizability, causal consistency, session guarantees, eventual consistency—choose the weakest model that satisfies requirements.
•Per-operation, not per-system — Different operations warrant different consistency models. Hybrid architectures are the norm, not the exception.
•Quantify the trade-offs — Cost of inconsistency vs cost of unavailability must be concrete numbers, not abstract preferences.
•Communicate in business terms — Translate technical constraints into business impact for stakeholder alignment.
•Graceful degradation over binary failure — Tier your features; progressively reduce quality rather than failing completely.
•Conflict resolution is required, not optional — If you choose availability, plan for conflicts before they occur.
•Operational practices make the difference — Chaos engineering, runbooks, monitoring, and drills ensure your design works in production.

What You've Mastered

Through this module, you've developed the expertise to:

Understand deeply: The CAP theorem, PACELC, and consistency models at a level that allows you to critique and improve system designs.
Decide wisely: Apply systematic frameworks to CP vs AP decisions, considering business requirements, regulatory constraints, and user expectations.
Tune effectively: Configure quorum levels, session guarantees, and adaptive consistency to optimize for your specific requirements.
Align with business: Translate between technical and business language, quantify trade-offs, and document decisions for organizational memory.
Implement practically: Design hybrid architectures, implement conflict resolution, and operate systems with robust consistency practices.

This is the knowledge that principal engineers bring to distributed systems design—not just understanding the theory, but applying it pragmatically to build systems that actually work.

Module Complete

You've completed the Availability vs Consistency Trade-offs module. You now possess a principal-engineer-level understanding of one of distributed systems' most fundamental challenges. Apply this knowledge thoughtfully—the best system is not the most consistent or the most available, but the one that makes the right trade-offs for its users and business.

5 / 5

Loading learning content...

System Design (HLD)High Availability

Availability vs Consistency Trade-offs

LevelAdvanced

Duration75 mins

TopicHigh Availability

5 / 5

Practical Compromises

Practical Engineering: Navigating the Real World

This final page of the module is about engineering pragmatism—the patterns, techniques, and operational practices that allow systems to:

Provide strong consistency most of the time while remaining available when it matters.
Accept eventual consistency for most operations while protecting critical paths.
Recover gracefully from temporary inconsistency states.
Balance theoretical purity with operational reality.

These are the lessons learned from building and operating systems at scale.

What You Will Master

Graceful Degradation Patterns

The best systems don't fail completely—they degrade gracefully, maintaining partial functionality while clearly communicating their reduced capabilities.

Pattern 1: Consistency Tiers

Not all data is equally critical. Segment your data into consistency tiers and handle each appropriately during degradation.

Consistency Tier Framework
Tier	Data Examples	Normal Mode	Degraded Mode
Tier 1: Critical	Account balances, inventory, credentials	Strong consistency	Reject operations (fail closed)
Tier 2: Important	Orders, user profiles, preferences	Strong consistency	Accept with warning; queue for verification
Tier 3: Nice-to-have	Analytics, recommendations, feeds	Eventual consistency	Continue with stale data
Tier 4: Disposable	Caches, derived data, previews	Best effort	Skip entirely or serve stale

Pattern 2: Progressive Unavailability

Rather than all-or-nothing availability, progressively shed load and features as conditions worsen.

Stage 1 (Healthy): All features available, full consistency.

Stage 2 (Stressed): Disable Tier 4 features (analytics collection, real-time recommendations). Reduce Tier 3 refresh rates.

Stage 3 (Degraded): Accept Tier 2 data with async verification. Alert operators. Show degradation indicators to users.

Stage 4 (Critical): Only Tier 1 operations proceed. Non-critical pages show "service maintenance" message.

Stage 5 (Failure): Full outage. Incident response activated.

Pattern 3: Read Path vs Write Path Separation

Reads and writes have different consistency requirements. Handle them independently.

graceful-degradation.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
// Graceful degradation implementation
 
enum SystemHealth {
  HEALTHY = 'healthy',
  STRESSED = 'stressed',
  DEGRADED = 'degraded',
  CRITICAL = 'critical',
}
 
interface DegradationConfig {
  tier1Features: string[]; // Always available until critical
  tier2Features: string[]; // Reduced in degraded
  tier3Features: string[]; // Disabled when stressed
  tier4Features: string[]; // First to go
}
 
class GracefulController {
  private currentHealth: SystemHealth = SystemHealth.HEALTHY;
 
  async handleRequest(feature: string, operation: () => Promise<any>): Promise<any> {
    const tier = this.getFeatureTier(feature);
    
    switch (this.currentHealth) {
      case SystemHealth.HEALTHY:
        // Normal operation for all tiers
        return await operation();
 
      case SystemHealth.STRESSED:
        if (tier === 4) {
          // Skip Tier 4 entirely
          return { status: 'skipped', reason: 'system_stressed' };
        }
        if (tier === 3) {
          // Tier 3 with reduced quality
          return await this.withReducedQuality(operation);
        }
        return await operation();
 
      case SystemHealth.DEGRADED:
        if (tier >= 3) {
          return { status: 'skipped', reason: 'system_degraded' };
        }
        if (tier === 2) {
          // Accept but queue for verification
          const result = await operation();
          await this.queueForVerification(feature, result);
          return { ...result, _warning: 'pending_verification' };
        }
        return await operation();
 
      case SystemHealth.CRITICAL:
        if (tier > 1) {
          return { status: 'unavailable', reason: 'system_critical' };
        }
        // Only Tier 1 proceeds
        return await operation();
    }
  }
 
  // Reduce quality for reads: use stale cache, skip enrichments
  private async withReducedQuality(operation: () => Promise<any>): Promise<any> {
    return await operation({ 
      skipEnrichments: true,
      cacheOnly: true,
      maxStaleSeconds: 300,
    });
  }
 
  // Queue operation for later verification
  private async queueForVerification(feature: string, result: any): Promise<void> {
    await this.verificationQueue.enqueue({
      feature,
      result,
      timestamp: Date.now(),
    });
  }
}
 
// Usage in request handler
const controller = new GracefulController();
 
app.post('/api/orders', async (req, res) => {
  const result = await controller.handleRequest('order.create', async () => {
    return await orderService.create(req.body);
  });
 
  if (result._warning) {
    res.setHeader('X-Service-Warning', result._warning);
  }
  
  res.json(result);
});

Fail Loud, Not Silent

Hybrid Architectures: The Best of Both Worlds

Hybrid architectures combine CP and AP components strategically, applying each where appropriate within the same system.

Pattern: CP Core with AP Edge

The "source of truth" maintains strong consistency, while edge caches and read replicas provide high availability for reads.

hybrid-architecture.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
HYBRID ARCHITECTURE: CP CORE + AP EDGE
═══════════════════════════════════════════════════════════════════
 
                         ┌─────────────────────────────────┐
    Users ─────────────▶ │     Edge/CDN (AP Layer)         │
                         │  • Cached reads (stale OK)      │
                         │  • Static content               │
                         │  • Write-through to core        │
                         └───────────────┬─────────────────┘
                                         │
                         ┌───────────────▼─────────────────┐
    Internal ──────────▶ │    Application Layer            │
    Services             │  • Routes by operation type     │
                         │  • Enforces consistency rules   │
                         └───────────────┬─────────────────┘
                                         │
              ┌──────────────────────────┼──────────────────────────┐
              │                          │                          │
              ▼                          ▼                          ▼
    ┌─────────────────┐        ┌─────────────────┐        ┌─────────────────┐
    │  Read Replicas  │        │   CP Core DB    │        │  Event Stream   │
    │  (Eventually    │◀───────│  (Source of     │───────▶│  (Eventually    │
    │   Consistent)   │  async │   Truth)        │  async │   Consistent)   │
    └─────────────────┘        └─────────────────┘        └─────────────────┘
         │                            │                          │
         ▼                            ▼                          ▼
    Catalog browsing           Write operations            Analytics
    Product pages              Transactions                Notifications
    Search results             Inventory updates           Audit logs

Pattern: Saga for Distributed Transactions

When a transaction spans multiple services, use the Saga pattern: a sequence of local transactions with compensating actions if any step fails.

Example: E-commerce Order Saga

Reserve Inventory → If fails, stop.
Process Payment → If fails, release inventory.
Create Shipment → If fails, refund payment, release inventory.
Confirm Order → Success!

Each step is a local transaction (CP within that service). The saga provides eventual consistency across services without distributed locking.

Pattern: CQRS (Command Query Responsibility Segregation)

Separate the write model (commands) from the read model (queries). Each can have different consistency characteristics.

CQRS Consistency Split
Aspect	Command Side (Write)	Query Side (Read)
Consistency	Strong (CP)	Eventual (AP)
Optimized For	Correctness	Performance
Data Model	Normalized, transactional	Denormalized, cached
Scaling	Single leader or consensus	Unlimited read replicas
Latency	Higher (consensus overhead)	Lower (local reads)

cqrs-implementation.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
// CQRS Implementation Example
 
// Command Service: Strong Consistency
class OrderCommandService {
  private writeDb: PostgresClient; // Single CP database
 
  async createOrder(command: CreateOrderCommand): Promise<OrderId> {
    return await this.writeDb.transaction(async (tx) => {
      // Validate inventory (strong read)
      const inventory = await tx.query(
        'SELECT quantity FROM inventory WHERE sku = $1 FOR UPDATE',
        [command.sku]
      );
      
      if (inventory.quantity < command.quantity) {
        throw new InsufficientInventoryError();
      }
 
      // Atomically update inventory and create order
      await tx.query(
        'UPDATE inventory SET quantity = quantity - $1 WHERE sku = $2',
        [command.quantity, command.sku]
      );
 
      const order = await tx.query(
        'INSERT INTO orders (sku, quantity, user_id) VALUES ($1, $2, $3) RETURNING id',
        [command.sku, command.quantity, command.userId]
      );
 
      // Publish event for query side (async)
      await this.eventBus.publish(new OrderCreatedEvent(order.id, command));
 
      return order.id;
    });
  }
}
 
// Query Service: Eventual Consistency
class OrderQueryService {
  private readReplicas: ReadReplica[];
  private cache: RedisClient;
 
  constructor() {
    // Subscribe to events to update read models
    this.eventBus.subscribe('OrderCreated', this.handleOrderCreated.bind(this));
    this.eventBus.subscribe('OrderShipped', this.handleOrderShipped.bind(this));
  }
 
  async getOrder(orderId: OrderId): Promise<OrderView | null> {
    // Try cache first
    const cached = await this.cache.get(`order:${orderId}`);
    if (cached) return JSON.parse(cached);
 
    // Load from read-optimized view
    const order = await this.readReplicas.random().query(
      'SELECT * FROM order_views WHERE id = $1',
      [orderId]
    );
 
    if (order) {
      await this.cache.setex(`order:${orderId}`, 300, JSON.stringify(order));
    }
 
    return order;
  }
 
  async getOrdersForUser(userId: UserId): Promise<OrderView[]> {
    // Always serve from cache/replica (eventual consistency OK for list view)
    return await this.readReplicas.random().query(
      'SELECT * FROM order_views WHERE user_id = $1 ORDER BY created_at DESC',
      [userId]
    );
  }
 
  // Event handlers update read models
  private async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    await this.readDb.query(
      'INSERT INTO order_views (...) VALUES (...)',
      [/* denormalized order data */]
    );
    await this.cache.del(`orders:user:${event.userId}`);
  }
}

CQRS Complexity

Conflict Resolution Strategies

When you choose availability, conflicts will occur. Having a robust conflict resolution strategy is essential for system integrity.

Strategy 1: Last-Write-Wins (LWW)

The simplest strategy: the most recent write (by timestamp) wins.

conflict-resolution-strategies.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
// Conflict resolution strategy implementations
 
interface ConflictingVersions<T> {
  versions: Array<{
    value: T;
    timestamp: number;
    nodeId: string;
    vectorClock?: VectorClock;
  }>;
}
 
// Strategy 1: Last-Write-Wins
function resolveByLWW<T>(conflict: ConflictingVersions<T>): T {
  return conflict.versions.reduce((latest, current) => 
    current.timestamp > latest.timestamp ? current : latest
  ).value;
}
 
// Strategy 2: Merge with Custom Logic
function resolveShoppingCart(conflict: ConflictingVersions<ShoppingCart>): ShoppingCart {
  const allItems = new Map<string, CartItem>();
 
  // Union all items across versions
  for (const version of conflict.versions) {
    for (const item of version.value.items) {
      const existing = allItems.get(item.sku);
      if (!existing || item.addedAt > existing.addedAt) {
        allItems.set(item.sku, item);
      }
    }
  }
 
  // For removed items, use most recent action
  const removals = new Map<string, { removedAt: number }>();
  for (const version of conflict.versions) {
    for (const removal of version.value.removedItems || []) {
      const existing = removals.get(removal.sku);
      if (!existing || removal.removedAt > existing.removedAt) {
        removals.set(removal.sku, removal);
      }
    }
  }
 
  // Apply removals
  for (const [sku, removal] of removals) {
    const item = allItems.get(sku);
    if (item && removal.removedAt > item.addedAt) {
      allItems.delete(sku);
    }
  }
 
  return { items: Array.from(allItems.values()) };
}
 
// Strategy 3: Conflict to Human
interface ConflictRecord<T> {
  key: string;
  versions: ConflictingVersions<T>;
  detectedAt: number;
  status: 'pending' | 'resolved';
  resolvedBy?: string;
  resolution?: T;
}
 
async function escalateToHuman<T>(
  key: string,
  conflict: ConflictingVersions<T>
): Promise<void> {
  const record: ConflictRecord<T> = {
    key,
    versions: conflict,
    detectedAt: Date.now(),
    status: 'pending',
  };
 
  await conflictQueue.enqueue(record);
  await alerting.notifyConflictReview(record);
  
  // System may continue with a default (e.g., LWW) until human resolves
}
 
// Strategy 4: Vector Clock with Conflict Detection
function detectConflictWithVectorClock<T>(
  conflict: ConflictingVersions<T>
): { isConflict: boolean; winner?: T } {
  // Sort by vector clock
  const sorted = [...conflict.versions].sort((a, b) => 
    compareVectorClocks(a.vectorClock!, b.vectorClock!)
  );
 
  // If all versions are causally ordered, no conflict
  for (let i = 0; i < sorted.length - 1; i++) {
    if (!happensBefore(sorted[i].vectorClock!, sorted[i + 1].vectorClock!)) {
      // Concurrent writes detected - true conflict
      return { isConflict: true };
    }
  }
 
  // Causally ordered - last one wins
  return { isConflict: false, winner: sorted[sorted.length - 1].value };
}

Strategy Selection Guide

Conflict Resolution Strategy Selection
Strategy	Data Type	Conflict Frequency	Acceptable Data Loss
Last-Write-Wins	Single-owner, low-cardinality	Low	Yes (loses loser's write)
Merge/Union	Sets, counters, accumulators	Any	No (preserves all)
Application Logic	Complex domain objects	Medium	Depends on logic
Human Review	Critical business data	Low (should be!)	No (requires resolution)
CRDTs	Collaborative, real-time	High	No (by design)

Monitor Conflict Rates

Operational Practices for Consistency

Architecture is only half the story. Operational practices determine whether your consistency choices hold up in production.

Practice 1: Chaos Engineering for Partitions

Regularly inject network partitions in non-production (and carefully in production) to verify behavior.

Partition Testing Checklist

•Does CP behavior actually reject requests during partition? Or does it silently serve stale data?
•Does AP behavior properly merge conflicts when partition heals?
•Do alerts fire appropriately during partition events?
•Can operators manually intervene if needed?
•Does the system automatically recover when partition heals?
•Are there any edge cases where the system enters an unrecoverable state?

Practice 2: Runbooks for Consistency Incidents

Create detailed runbooks for consistency-related incidents:

Partition Detected:

Verify partition with network diagnostics.
Identify which zones/replicas are affected.
Confirm degraded mode activated correctly.
Monitor for unexpected behavior.
Prepare for post-partition reconciliation.

Conflict Storm (many conflicts occurring):

Identify source of contention.
Consider temporary strong consistency for affected keys.
Increase reconciliation workers if queue is backing up.
Investigate root cause (hot key? failed node returning?).

Stale Data Reported:

Verify replication lag.
Check for stuck transactions or locks.
Force refresh if necessary.
Investigate why monitoring didn't alert.

consistency-monitoring-dashboard.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Grafana dashboard configuration for consistency monitoring
 
dashboard:
  title: "Consistency Health"
  
panels:
  - title: "Replication Lag (P99)"
    query: |
      histogram_quantile(0.99, 
        rate(replication_lag_seconds_bucket[5m])
      )
    alert:
      - name: "High Replication Lag"
        condition: "> 10s"
        severity: "warning"
      - name: "Critical Replication Lag"
        condition: "> 60s"
        severity: "critical"
 
  - title: "Consistency Level Distribution"
    query: |
      rate(operations_total[5m]) by (consistency_level)
    description: "Track what consistency levels are actually being used"
 
  - title: "Conflict Rate"
    query: |
      rate(conflicts_detected_total[5m])
    alert:
      - name: "High Conflict Rate"
        condition: "> 10/minute"
        severity: "warning"
 
  - title: "Partition Events"
    query: |
      increase(partition_events_total[1h])
    description: "Track how often partitions are detected"
 
  - title: "Degraded Mode Activations"
    query: |
      changes(system_health_state[1d])
    description: "How often is the system entering degraded mode?"
 
  - title: "Pending Conflict Reviews"
    query: |
      conflict_queue_size
    alert:
      - name: "Conflict Backlog"
        condition: "> 100"
        severity: "warning"
 
  - title: "Reconciliation Progress"
    query: |
      rate(reconciliation_operations_total[5m])
    description: "Post-partition reconciliation throughput"

Practice 3: Regular Consistency Audits

Periodically verify that data across replicas is actually consistent:

Sample-based verification: Read the same records from multiple replicas; compare values.
Checksum comparison: Compute checksums of data ranges across replicas.
Reconciliation reports: Review how many conflicts occurred, how they were resolved.
Consistency SLI: Measure what percentage of reads return the latest value (test with known writes).

Practice 4: Incident Reviews

After any consistency incident:

What was the root cause?
Did the system behave as designed?
Were alerts timely and actionable?
Did runbooks work?
What changes would prevent recurrence?

Practice Makes Perfect

Real-World Compromise Patterns from Production

Let's examine specific patterns used by major systems to balance consistency and availability in practice.

Facebook's TAO: Read-Your-Writes at Scale

•Problem: 2+ billion users; strong consistency would be too slow.
•Solution: Read-your-writes per user; eventual consistency globally.
•How: After write, user's session reads from master for a short window. Other reads go to nearest replica.
•Result: Users see their own actions immediately; friends see updates within seconds. No user notices the lag.

Uber's Ringpop: Consistent Hashing with Ownership

•Problem: Rides are time-sensitive; can't have double-dispatching or lost rides.
•Solution: Each key (ride ID) has a single owner node at any time.
•How: Consistent hashing assigns ownership; SWIM protocol detects failures and reassigns.
•Result: Strong consistency for each ride (only owner writes); high availability through fast failover.

LinkedIn's Espresso: Caught Reads

•Problem: High read volume; strong consistency for all reads too expensive.
•Solution: 'Caught reads'—include timestamp of minimum acceptable freshness.
•How: Client specifies 'I need data from at least timestamp T'. Replica only serves if it has replicated past T.
•Result: Read-your-writes without routing all reads to master. Reads can go to any caught-up replica.

Slack's Message Ordering

•Problem: Messages must appear in order; but availability is critical for collaboration.
•Solution: Per-channel ordering, not global ordering.
•How: Each channel has a sequence number. Messages within a channel are strictly ordered. Messages across channels can be interleaved.
•Result: Strong consistency where it matters (conversation flow). High availability because channels are independent.

Common Themes in Successful Compromises

Scope consistency narrowly: Per-user, per-channel, per-entity—not global.
Make freshness visible: Clients specify or understand staleness tolerance.
Use ownership patterns: Single writer eliminates write conflicts.
Fast failure detection: The faster you detect problems, the shorter the compromise period.
Automatic recovery: Manual recovery is too slow for modern availability expectations.

Study Production Systems

Future Directions in Consistency

The field of distributed consistency continues to evolve. Being aware of emerging approaches helps you prepare for future options.

Emerging Technologies

Future Consistency Approaches

•Programmable Consistency: Languages and frameworks that let developers specify consistency requirements declaratively, with the system choosing the optimal implementation.
•Hardware-Assisted Ordering: Intel's Optane, persistent memory, and hardware timestamps reducing the software overhead of consistency protocols.
•Global Time Services: Wider availability of precise time (TrueTime-like services) enabling Spanner-style consistency without Google's infrastructure.
•CRDT Standardization: Growing libraries and frameworks for CRDTs making conflict-free designs more accessible.
•Formal Verification: Increasing use of formal methods (TLA+, Alloy) to prove consistency properties of distributed systems before deployment.
•Edge Computing Consistency: New models for consistency across edge locations with intermittent connectivity—beyond traditional datacenter assumptions.

The Trend: Toward Tunability

The industry is moving away from single-consistency-model databases toward systems that offer:

Per-query consistency selection.
Automatic consistency adaptation based on conditions.
Clear visibility into consistency guarantees being provided.
Better tools for understanding and debugging consistency issues.

This trend reflects the reality that consistency requirements are application-specific and even operation-specific—not something that should be baked into infrastructure once and forgotten.

Stay Current

Module Summary: Availability vs Consistency Mastery

Module Key Takeaways

•CAP is a constraint, not a limitation — Understanding the theorem enables you to make informed trade-offs rather than fighting impossible battles.
•Consistency exists on a spectrum — Linearizability, causal consistency, session guarantees, eventual consistency—choose the weakest model that satisfies requirements.
•Per-operation, not per-system — Different operations warrant different consistency models. Hybrid architectures are the norm, not the exception.
•Quantify the trade-offs — Cost of inconsistency vs cost of unavailability must be concrete numbers, not abstract preferences.
•Communicate in business terms — Translate technical constraints into business impact for stakeholder alignment.
•Graceful degradation over binary failure — Tier your features; progressively reduce quality rather than failing completely.
•Conflict resolution is required, not optional — If you choose availability, plan for conflicts before they occur.
•Operational practices make the difference — Chaos engineering, runbooks, monitoring, and drills ensure your design works in production.

What You've Mastered

Through this module, you've developed the expertise to:

Understand deeply: The CAP theorem, PACELC, and consistency models at a level that allows you to critique and improve system designs.
Decide wisely: Apply systematic frameworks to CP vs AP decisions, considering business requirements, regulatory constraints, and user expectations.
Tune effectively: Configure quorum levels, session guarantees, and adaptive consistency to optimize for your specific requirements.
Align with business: Translate between technical and business language, quantify trade-offs, and document decisions for organizational memory.
Implement practically: Design hybrid architectures, implement conflict resolution, and operate systems with robust consistency practices.

This is the knowledge that principal engineers bring to distributed systems design—not just understanding the theory, but applying it pragmatically to build systems that actually work.

Module Complete

5 / 5