Eventual Consistency - Learning Module

Loading content...

0/273

Application-Level Handling

Building Robust Applications on Eventually Consistent Foundations

The infrastructure provides eventual consistency—your application must provide correctness. This is the fundamental reality of building on eventually consistent systems. The database won't prevent your application from reading stale data, overwriting concurrent updates, or making decisions based on outdated information. Your application logic must handle these scenarios explicitly.

This page bridges the gap between theoretical consistency models and practical application development. We'll explore battle-tested patterns for common challenges: managing inventory without overselling, enabling collaborative editing without data loss, handling financial operations safely, and designing user experiences that embrace eventual consistency rather than fighting it.

What You Will Learn

By the end of this page, you will understand how to design application logic that works correctly with eventual consistency, handle domain-specific challenges like inventory and payments, build user interfaces that provide a good experience despite staleness, and implement compensating actions for when optimistic assumptions fail.

The key insight is that eventual consistency shifts responsibility. Instead of relying on the database to guarantee consistency, applications must:

Detect potential inconsistencies
Handle inconsistencies gracefully when they occur
Recover when optimistic assumptions prove wrong
Communicate uncertainty to users appropriately

This isn't a weakness—it's a design pattern that enables the scalability and availability that strong consistency cannot provide.

Domain Modeling for Eventual Consistency

How you model your domain directly impacts how well it handles eventual consistency. Certain modeling approaches are naturally more resilient to temporary inconsistencies than others.

Principles for EC-Friendly Domain Models:

Domain Modeling Principles

•Prefer Append-Only Models: Instead of updating counters, append events. Counters can be derived. Events are immutable and conflict-free.
•Use Natural Idempotency Keys: Design entities with natural unique identifiers (order IDs, request IDs) that prevent duplicate operations.
•Embrace State Machines: Model entities with explicit states and transitions. States can be reconciled; arbitrary updates cannot.
•Aggregate Boundaries: Keep consistency requirements within aggregate boundaries. Cross-aggregate consistency is eventually consistent.
•Compensating Actions: Design every action with a corresponding undo action. This enables recovery from optimistic failures.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// ❌ BRITTLE MODEL: Mutable state that can conflict
interface BrittleOrder {
    id: string;
    items: OrderItem[];
    status: string;           // Direct mutation
    totalAmount: number;      // Derived but stored
    lastModified: Date;       // Race conditions
}
 
// Each update mutates the order directly
// Conflicts when multiple services update simultaneously
// No history, hard to debug issues
 
// ✅ RESILIENT MODEL: Event-sourced with clear state machine
interface OrderEvent {
    orderId: string;
    eventId: string;          // Unique, idempotent
    eventType: 'CREATED' | 'ITEM_ADDED' | 'ITEM_REMOVED' | 
               'SUBMITTED' | 'PAYMENT_CONFIRMED' | 'SHIPPED' | 'CANCELLED';
    timestamp: number;
    payload: any;
}
 
// State machine defines valid transitions
const ORDER_STATE_MACHINE = {
    DRAFT: ['SUBMITTED', 'CANCELLED'],
    SUBMITTED: ['PAYMENT_CONFIRMED', 'CANCELLED'],
    PAYMENT_CONFIRMED: ['SHIPPED', 'REFUNDING'],
    SHIPPED: ['DELIVERED', 'RETURN_REQUESTED'],
    CANCELLED: [],  // Terminal state
    DELIVERED: ['RETURN_REQUESTED'],
    // ... etc
};
 
class Order {
    private events: OrderEvent[] = [];
    
    // Derive current state from events
    get state(): OrderState {
        return this.events.reduce(
            (state, event) => this.applyEvent(state, event),
            initialOrderState
        );
    }
    
    // Commands validate against current state
    submit(): OrderEvent[] {
        const current = this.state;
        if (!ORDER_STATE_MACHINE[current.status].includes('SUBMITTED')) {
            throw new Error(`Cannot submit order in state ${current.status}`);
        }
        
        return [{
            orderId: this.id,
            eventId: generateUniqueId(),
            eventType: 'SUBMITTED',
            timestamp: Date.now(),
            payload: { submittedAt: Date.now() }
        }];
    }
    
    // Events are append-only, no conflicts
    addEvent(event: OrderEvent): void {
        // Check idempotency
        if (this.events.find(e => e.eventId === event.eventId)) {
            return;  // Already applied, skip
        }
        this.events.push(event);
    }
}

Event Sourcing and Eventual Consistency

Event sourcing is particularly powerful with eventual consistency. Events are immutable facts that can be replicated and replayed. Different replicas might have different events at any moment, but they'll converge to the same state once all events propagate. The derived state is eventually consistent, but the events themselves are always consistent.

Case Study: Inventory Management

Inventory management is one of the most challenging domains for eventual consistency. The core problem: How do you prevent overselling when inventory reads might be stale?

Consider a flash sale where 100 units are available and 1,000 customers try to buy simultaneously. With eventual consistency, multiple replicas might all show "100 units available" and accept 200+ orders.

Solution Approaches:

Inventory Management Strategies
Strategy	How It Works	Trade-offs
Pessimistic (Strong Consistency)	Lock inventory row, decrement, unlock	Serialized access, bottleneck under load
Optimistic with CAS	Read version, decrement, conditional write	Retries under contention, scalable
Reservation Pattern	Reserve first, confirm later within timeout	Requires cleanup of expired reservations
Oversell + Compensate	Accept all orders, cancel excess later	Better UX, requires compensation logic
Probabilistic Limiting	Use approximate counts, reject when uncertain	May reject valid orders

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
interface InventoryReservation {
    reservationId: string;
    productId: string;
    quantity: number;
    expiresAt: number;
    status: 'PENDING' | 'CONFIRMED' | 'EXPIRED' | 'RELEASED';
}
 
class InventoryService {
    private readonly RESERVATION_TTL = 10 * 60 * 1000; // 10 minutes
    
    // Step 1: Create a reservation (quick, can be eventually consistent)
    async reserve(productId: string, quantity: number): Promise<ReservationResult> {
        const reservationId = generateUniqueId();
        const expiresAt = Date.now() + this.RESERVATION_TTL;
        
        // Optimistically create reservation
        const reservation: InventoryReservation = {
            reservationId,
            productId,
            quantity,
            expiresAt,
            status: 'PENDING'
        };
        
        await this.db.write(`reservation:${reservationId}`, reservation);
        
        // Trigger async validation
        this.validateReservationAsync(reservationId, productId, quantity);
        
        return { 
            reservationId, 
            expiresAt,
            status: 'PENDING' // Will be confirmed or rejected async
        };
    }
    
    // Step 2: Validate reservation against actual inventory
    private async validateReservationAsync(
        reservationId: string,
        productId: string,
        quantity: number
    ): Promise<void> {
        try {
            // Use stronger consistency for the actual inventory check
            const inventory = await this.db.read(
                `inventory:${productId}`,
                { consistency: 'QUORUM' }
            );
            
            // Calculate available = total - confirmed reservations - pending reservations
            const confirmedReserved = await this.getConfirmedReservations(productId);
            const pendingReserved = await this.getPendingReservations(productId);
            const available = inventory.quantity - confirmedReserved - pendingReserved;
            
            if (available >= quantity) {
                // Confirm the reservation
                await this.confirmReservation(reservationId);
            } else {
                // Not enough inventory, reject
                await this.rejectReservation(reservationId);
            }
        } catch (error) {
            // On error, leave as PENDING - will expire if not confirmed
            console.error(`Reservation validation failed: ${error}`);
        }
    }
    
    // Step 3: Confirm reservation when order is complete
    async confirmOrder(reservationId: string): Promise<ConfirmResult> {
        const reservation = await this.db.read(`reservation:${reservationId}`);
        
        if (reservation.status !== 'PENDING' && reservation.status !== 'CONFIRMED') {
            return { success: false, reason: 'Reservation expired or invalid' };
        }
        
        // Decrement actual inventory with conditional write
        const result = await this.decrementInventory(
            reservation.productId,
            reservation.quantity
        );
        
        if (result.success) {
            await this.updateReservation(reservationId, { status: 'CONFIRMED' });
            return { success: true };
        } else {
            // Inventory changed - reservation no longer valid
            await this.updateReservation(reservationId, { status: 'RELEASED' });
            return { success: false, reason: 'Inventory no longer available' };
        }
    }
    
    // Background job: Expire old reservations
    async cleanupExpiredReservations(): Promise<void> {
        const now = Date.now();
        const expired = await this.db.query(
            'reservations',
            { status: 'PENDING', expiresAt: { $lt: now } }
        );
        
        for (const reservation of expired) {
            await this.updateReservation(reservation.reservationId, { status: 'EXPIRED' });
        }
    }
}

The Oversell Strategy

Many successful e-commerce systems intentionally allow overselling by a small percentage (e.g., 5%). It's often better to accept an order optimistically and cancel the rare oversold order (with an apology and discount) than to reject orders during a sale. This is a business decision enabled by eventual consistency. Amazon's shopping cart famously allows adding items that are out of stock—availability is later verified at checkout.

Case Study: Financial Operations

Financial operations seem incompatible with eventual consistency—money should never be lost or duplicated. But many financial systems do use eventual consistency successfully, employing specific patterns to maintain correctness.

Key Insight: The goal isn't to make every read consistent, but to ensure that modifications are atomic, idempotent, and recoverable.

Essential Patterns for Financial Operations:

Financial EC Patterns

•Transaction Logs: Record every operation as an immutable event. Balance is derived from the log, not stored directly.
•Idempotency Keys: Every financial request includes a unique key. Retries are safe because the same key produces the same result.
•Double-Entry Bookkeeping: Every debit has a corresponding credit. Total across all accounts is always zero (invariant check).
•Saga Pattern: Complex operations are broken into compensatable steps. If step 5 fails, steps 1-4 are reversed.
•Reconciliation: Periodic background jobs verify invariants (sum of all accounts = 0) and flag discrepancies.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
interface TransferSaga {
    sagaId: string;
    fromAccount: string;
    toAccount: string;
    amount: number;
    steps: SagaStep[];
    status: 'RUNNING' | 'COMPLETED' | 'COMPENSATING' | 'COMPENSATED' | 'FAILED';
}
 
interface SagaStep {
    name: string;
    status: 'PENDING' | 'COMPLETED' | 'FAILED' | 'COMPENSATED';
    action: () => Promise<void>;
    compensation: () => Promise<void>;
}
 
class TransferSagaOrchestrator {
    async executeTransfer(
        fromAccount: string, 
        toAccount: string, 
        amount: number
    ): Promise<TransferResult> {
        const sagaId = generateUniqueId();
        
        // Define the saga steps with compensation actions
        const saga: TransferSaga = {
            sagaId,
            fromAccount,
            toAccount,
            amount,
            status: 'RUNNING',
            steps: [
                {
                    name: 'VALIDATE_SOURCE',
                    status: 'PENDING',
                    action: async () => {
                        const balance = await this.getBalance(fromAccount);
                        if (balance < amount) {
                            throw new Error('Insufficient funds');
                        }
                    },
                    compensation: async () => {
                        // Nothing to compensate - validation is read-only
                    }
                },
                {
                    name: 'DEBIT_SOURCE',
                    status: 'PENDING',
                    action: async () => {
                        await this.adjustBalance(fromAccount, -amount, sagaId);
                    },
                    compensation: async () => {
                        // Reverse the debit
                        await this.adjustBalance(fromAccount, amount, `${sagaId}-compensation`);
                    }
                },
                {
                    name: 'CREDIT_DESTINATION',
                    status: 'PENDING',
                    action: async () => {
                        await this.adjustBalance(toAccount, amount, sagaId);
                    },
                    compensation: async () => {
                        // Reverse the credit
                        await this.adjustBalance(toAccount, -amount, `${sagaId}-compensation`);
                    }
                },
                {
                    name: 'RECORD_TRANSFER',
                    status: 'PENDING',
                    action: async () => {
                        await this.recordTransfer({
                            sagaId,
                            fromAccount,
                            toAccount,
                            amount,
                            timestamp: Date.now()
                        });
                    },
                    compensation: async () => {
                        await this.recordTransfer({
                            sagaId: `${sagaId}-reversal`,
                            fromAccount: toAccount,  // Reversed
                            toAccount: fromAccount,
                            amount,
                            reason: 'SAGA_COMPENSATION',
                            timestamp: Date.now()
                        });
                    }
                }
            ]
        };
        
        // Execute saga
        await this.saveSaga(saga);
        
        try {
            // Execute steps in order
            for (let i = 0; i < saga.steps.length; i++) {
                const step = saga.steps[i];
                try {
                    await step.action();
                    step.status = 'COMPLETED';
                    await this.saveSaga(saga);
                } catch (error) {
                    step.status = 'FAILED';
                    saga.status = 'COMPENSATING';
                    await this.saveSaga(saga);
                    
                    // Compensate completed steps in reverse order
                    await this.compensate(saga, i);
                    throw error;
                }
            }
            
            saga.status = 'COMPLETED';
            await this.saveSaga(saga);
            return { success: true, sagaId };
            
        } catch (error) {
            return { success: false, sagaId, error: error.message };
        }
    }
    
    private async compensate(saga: TransferSaga, failedStep: number): Promise<void> {
        // Compensate in reverse order
        for (let i = failedStep - 1; i >= 0; i--) {
            const step = saga.steps[i];
            if (step.status === 'COMPLETED') {
                await step.compensation();
                step.status = 'COMPENSATED';
                await this.saveSaga(saga);
            }
        }
        saga.status = 'COMPENSATED';
        await this.saveSaga(saga);
    }
}

Sagas Are Not ACID Transactions

Sagas provide eventual consistency, not atomicity. During execution, the system is in an intermediate state (money debited but not yet credited). Design for this: don't show the 'available balance' during active transfers, or explicitly show 'pending' amounts.

Case Study: Collaborative Editing

Collaborative editing (like Google Docs) is a domain where eventual consistency isn't just acceptable—it's essential. Users expect real-time collaboration across the globe with minimal latency. Strong consistency would make this impossible.

The Challenge:

Multiple users edit the same document simultaneously
Each user sees their own changes immediately
Changes must merge without losing anyone's work
Network delays and partitions are common

Key Techniques:

Collaborative Editing Approaches
Technique	How It Works	Used By
Operational Transformation (OT)	Transform concurrent operations against each other	Google Docs, SharePoint
CRDTs	Conflict-free data structures that auto-merge	Apple Notes, Figma, Notion
Differential Sync	Exchange diffs, apply patches	Older systems
Last-Write-Wins	Simple but loses concurrent edits	Simple wikis

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
// Simplified CRDT for collaborative text editing
// Based on RGA (Replicated Growable Array)
 
interface CRDTCharacter {
    id: string;           // Globally unique ID
    value: string;        // The character
    deleted: boolean;     // Tombstone for deleted chars
    afterId: string | null; // Inserted after this character
}
 
class CRDTText {
    private characters: Map<string, CRDTCharacter> = new Map();
    private nodeId: string;  // This replica's unique ID
    private counter: number = 0;
    
    constructor(nodeId: string) {
        this.nodeId = nodeId;
    }
    
    // Generate unique, sortable ID
    private generateId(): string {
        return `${Date.now()}-${this.nodeId}-${++this.counter}`;
    }
    
    // Insert character after position
    insert(afterId: string | null, char: string): Operation {
        const newChar: CRDTCharacter = {
            id: this.generateId(),
            value: char,
            deleted: false,
            afterId
        };
        
        this.characters.set(newChar.id, newChar);
        
        // Return operation to broadcast to other replicas
        return { type: 'INSERT', character: newChar };
    }
    
    // Delete character (tombstone, don't actually remove)
    delete(charId: string): Operation {
        const char = this.characters.get(charId);
        if (char) {
            char.deleted = true;
        }
        return { type: 'DELETE', characterId: charId };
    }
    
    // Apply operation from remote replica
    applyRemote(op: Operation): void {
        if (op.type === 'INSERT') {
            // Idempotent: check if already exists
            if (!this.characters.has(op.character.id)) {
                this.characters.set(op.character.id, op.character);
            }
        } else if (op.type === 'DELETE') {
            const char = this.characters.get(op.characterId);
            if (char) {
                char.deleted = true;
            }
        }
    }
    
    // Merge entire state from another replica
    merge(otherState: Map<string, CRDTCharacter>): void {
        for (const [id, char] of otherState) {
            const existing = this.characters.get(id);
            if (!existing) {
                // New character, add it
                this.characters.set(id, char);
            } else if (char.deleted && !existing.deleted) {
                // Character was deleted on other replica
                existing.deleted = true;
            }
            // If both exist and neither is deleted, they're identical
        }
    }
    
    // Render the text by traversing the linked structure
    toString(): string {
        // Build adjacency list
        const afterMap = new Map<string | null, CRDTCharacter[]>();
        for (const char of this.characters.values()) {
            if (!afterMap.has(char.afterId)) {
                afterMap.set(char.afterId, []);
            }
            afterMap.get(char.afterId)!.push(char);
        }
        
        // Sort siblings by ID for deterministic ordering
        for (const siblings of afterMap.values()) {
            siblings.sort((a, b) => a.id.localeCompare(b.id));
        }
        
        // DFS to build string
        const result: string[] = [];
        const dfs = (afterId: string | null) => {
            const children = afterMap.get(afterId) || [];
            for (const child of children) {
                if (!child.deleted) {
                    result.push(child.value);
                }
                dfs(child.id);
            }
        };
        dfs(null);
        
        return result.join('');
    }
}
 
// Usage across replicas:
// User A types "Hi"
// User B types "Hey" 
// Both merge to "HHiey" or "HHeyi" - deterministic, no lost characters

CRDTs: The Heart of Collaborative Apps

CRDTs (Conflict-free Replicated Data Types) are data structures designed for eventual consistency. They guarantee that replicas can be merged in any order and will converge to the same state. While complex to implement, libraries like Yjs, Automerge, and Loro provide ready-to-use CRDTs for common use cases.

User Interface Patterns

Users don't care about consistency models—they care about whether the application feels fast and reliable. Good UI design can make eventual consistency invisible to users while maintaining a responsive experience.

Key UI Patterns:

UI Patterns for Eventual Consistency

•Optimistic Updates: Apply changes to the UI immediately, before server confirmation. Roll back if the server rejects.
•Pending States: Show indicators like 'Saving...', 'Syncing...', or a spinner when changes are in flight.
•Conflict Resolution UI: When conflicts are detected, present them to the user clearly with merge options.
•Stale Data Indicators: Show 'Last updated X ago' or visual cues for potentially stale data.
•Automatic Retry with Feedback: Retry failed operations automatically, showing progress to the user.
•Offline-First Design: Store changes locally, sync when online. Embrace rather than fight disconnection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
import { useState, useCallback } from 'react';
 
interface Task {
    id: string;
    title: string;
    completed: boolean;
    status: 'synced' | 'pending' | 'error';
}
 
function TaskList() {
    const [tasks, setTasks] = useState<Task[]>([]);
    
    const toggleComplete = useCallback(async (taskId: string) => {
        const task = tasks.find(t => t.id === taskId);
        if (!task) return;
        
        // 1. Optimistic update: immediately update UI
        setTasks(prev => prev.map(t => 
            t.id === taskId 
                ? { ...t, completed: !t.completed, status: 'pending' }
                : t
        ));
        
        try {
            // 2. Send to server
            await api.updateTask(taskId, { completed: !task.completed });
            
            // 3. Mark as synced
            setTasks(prev => prev.map(t =>
                t.id === taskId ? { ...t, status: 'synced' } : t
            ));
        } catch (error) {
            // 4. Rollback on failure
            setTasks(prev => prev.map(t =>
                t.id === taskId 
                    ? { ...t, completed: task.completed, status: 'error' }
                    : t
            ));
            
            // Show error notification
            showNotification({
                type: 'error',
                message: 'Failed to update task. Tap to retry.',
                action: () => toggleComplete(taskId)
            });
        }
    }, [tasks]);
    
    return (
        <ul className="task-list">
            {tasks.map(task => (
                <li 
                    key={task.id}
                    className={`task ${task.status}`}
                    onClick={() => toggleComplete(task.id)}
                >
                    <input 
                        type="checkbox" 
                        checked={task.completed}
                        readOnly
                    />
                    <span>{task.title}</span>
                    
                    {/* Status indicators */}
                    {task.status === 'pending' && <SyncingSpinner />}
                    {task.status === 'error' && <ErrorIcon />}
                </li>
            ))}
        </ul>
    );
}
 
// CSS for visual feedback
/*
.task.pending { opacity: 0.7; }
.task.error { border-left: 3px solid red; }
.task.synced { /* normal styling */ }
*/

Best Practices for EC-Friendly UIs:

Never block the UI waiting for server confirmation. Use optimistic updates.
Show sync status prominently but not intrusively. Users should know when they're offline or operations are pending.
Handle conflicts gracefully. Don't silently pick a winner—let users see and resolve conflicts when they matter.
Design for eventual visibility. If a user creates content, they should see it in lists even before other users see it (read-your-writes).
Provide retry mechanisms. Failed operations should be easy to retry without repeating the entire flow.

Mobile Apps Lead the Way

Mobile apps pioneered many EC-friendly UI patterns out of necessity—mobile networks are unreliable. Offline-first design, optimistic updates, and background sync are now best practices for all applications, not just mobile.

Compensating Actions and Recovery

In eventually consistent systems, you often make optimistic assumptions that later prove wrong. Compensating actions are the mechanism for correcting these situations—undoing or adjusting operations that shouldn't have succeeded.

When Compensation is Needed:

Compensation Scenarios

•Oversold Inventory: Accepted more orders than available stock. Compensate by cancelling excess orders with apology + discount.
•Failed Cross-System Transaction: Payment processed but shipping failed. Compensate by refunding payment.
•Stale Cache Decision: Made decision based on cached data that was outdated. Compensate by re-evaluating with fresh data.
•Conflict Resolution: Two users edited the same record. Compensate by merging or presenting conflict UI.
•Duplicate Processing: Message processed twice due to retry. Compensate by undoing the duplicate (if detected after).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
// Compensation for oversold inventory
class InventoryCompensation {
    async handleOversell(oversoldOrders: Order[]): Promise<void> {
        // Sort by priority (e.g., loyalty tier, order time)
        oversoldOrders.sort((a, b) => {
            // Keep orders from high-value customers
            if (a.customerTier !== b.customerTier) {
                return tierPriority(b.customerTier) - tierPriority(a.customerTier);
            }
            // Otherwise, first come first served
            return a.orderTime - b.orderTime;
        });
        
        // Cancel from the end (lowest priority)
        for (const order of oversoldOrders.reverse()) {
            await this.compensateOrder(order);
        }
    }
    
    private async compensateOrder(order: Order): Promise<void> {
        // 1. Cancel the order
        await this.orderService.cancel(order.id, 'INVENTORY_UNAVAILABLE');
        
        // 2. Refund payment
        await this.paymentService.refund(order.paymentId, order.totalAmount);
        
        // 3. Notify customer with apology
        await this.notificationService.send({
            userId: order.userId,
            template: 'ORDER_CANCELLED_STOCK',
            data: {
                orderNumber: order.orderNumber,
                productName: order.productName,
                discountCode: await this.generateApologyDiscount(order.userId)
            }
        });
        
        // 4. Log for analysis
        await this.analytics.logOversellCompensation({
            orderId: order.id,
            productId: order.productId,
            amount: order.totalAmount,
            timestamp: Date.now()
        });
    }
}
 
// Compensation for distributed saga failure
class BookingCompensation {
    async compensateFailedBooking(sagaId: string): Promise<void> {
        const saga = await this.getSaga(sagaId);
        
        // Get completed steps that need reversal
        const completedSteps = saga.steps.filter(s => s.status === 'COMPLETED');
        
        // Compensate in reverse order
        for (let i = completedSteps.length - 1; i >= 0; i--) {
            const step = completedSteps[i];
            
            switch (step.name) {
                case 'CHARGE_CREDIT_CARD':
                    await this.paymentService.refund(step.transactionId);
                    break;
                    
                case 'BOOK_FLIGHT':
                    await this.flightService.cancelReservation(step.reservationId);
                    break;
                    
                case 'BOOK_HOTEL':
                    await this.hotelService.cancelReservation(step.reservationId);
                    break;
                    
                case 'BOOK_CAR':
                    await this.carService.cancelReservation(step.reservationId);
                    break;
            }
            
            step.status = 'COMPENSATED';
            await this.saveSaga(saga);
        }
        
        saga.status = 'COMPENSATED';
        await this.saveSaga(saga);
    }
}

Compensation Must Be Idempotent

Compensation actions themselves may need to be retried. Design them to be idempotent—applying the same compensation twice should have the same effect as applying it once. Use unique IDs for compensation operations and check if already applied before executing.

Monitoring and Observability

You can't manage what you can't measure. Monitoring eventual consistency requires specific metrics and alerting strategies that differ from traditional systems.

Key Metrics for Eventually Consistent Systems
Metric	What It Measures	Alert Threshold (Example)
Replication Lag	Time for updates to reach replicas	5 seconds
Conflict Rate	Percentage of writes with conflicts	1%
Compensation Rate	How often compensations are needed	0.1%
Stale Read Rate	Reads returning outdated data	5%
Anti-Entropy Duration	Time to complete sync between replicas	1 minute
Pending Saga Count	Sagas in progress too long	100
Retry Rate	Operations requiring retries	10%

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
class ConsistencyMonitor {
    private metrics: MetricsClient;
    
    // Track replication lag
    async measureReplicationLag(): Promise<void> {
        const testKey = `__consistency_probe_${Date.now()}`;
        const testValue = { timestamp: Date.now(), probe: true };
        
        // Write to primary
        await this.db.write(testKey, testValue, { consistency: 'ONE' });
        
        // Read from each replica
        for (const replica of this.replicas) {
            const start = Date.now();
            let found = false;
            
            while (!found && Date.now() - start < 30000) {
                const value = await replica.read(testKey);
                if (value && value.timestamp === testValue.timestamp) {
                    found = true;
                    const lag = Date.now() - testValue.timestamp;
                    this.metrics.histogram('replication_lag_ms', lag, {
                        replica: replica.id
                    });
                } else {
                    await delay(10);
                }
            }
            
            if (!found) {
                this.metrics.increment('replication_timeout', { replica: replica.id });
            }
        }
        
        // Cleanup probe
        await this.db.delete(testKey);
    }
    
    // Track conflict rate
    onWriteConflict(key: string, resolution: ConflictResolution): void {
        this.metrics.increment('write_conflicts', {
            resolution: resolution.strategy,
            keyPrefix: key.split(':')[0]
        });
    }
    
    // Track compensation events
    onCompensation(type: string, original: string): void {
        this.metrics.increment('compensations', {
            type,
            originalOperation: original
        });
    }
    
    // Monitor saga health
    async checkStuckSagas(): Promise<void> {
        const stuckThreshold = Date.now() - 60 * 60 * 1000; // 1 hour
        
        const stuckSagas = await this.db.query('sagas', {
            status: 'RUNNING',
            startedAt: { $lt: stuckThreshold }
        });
        
        this.metrics.gauge('stuck_sagas', stuckSagas.length);
        
        if (stuckSagas.length > 10) {
            this.alerting.trigger({
                severity: 'HIGH',
                message: `${stuckSagas.length} sagas stuck for over 1 hour`,
                sagas: stuckSagas.map(s => s.sagaId)
            });
        }
    }
}

Dashboards for EC

Build dedicated dashboards for consistency health. Track replication lag percentiles (p50, p95, p99), conflict rates by data type, compensation frequencies, and saga completion rates. These metrics help you understand if your EC design is working well or needs adjustment.

Summary: Application-Level Handling

Building applications on eventually consistent infrastructure requires shifting responsibility from the database to the application. This isn't a burden—it's an opportunity to design systems that are more scalable, available, and user-friendly. Let's consolidate the key takeaways:

Key Takeaways

•Model for Resilience: Use append-only data, state machines, and idempotency keys to make your domain naturally tolerant of eventual consistency.
•Domain-Specific Patterns: Inventory uses reservations, financial operations use sagas, collaborative editing uses CRDTs—each domain has proven patterns.
•Optimistic UIs: Never block users waiting for server confirmation. Apply changes immediately and roll back if needed.
•Compensating Actions: Design every action with an undo. When optimistic assumptions fail, compensate gracefully.
•User Communication: Show sync status, handle conflicts explicitly, and be transparent about staleness.
•Comprehensive Monitoring: Track replication lag, conflict rates, and compensation frequency to ensure your EC design is healthy.

What's Next:

The final page of this module examines when eventual consistency works—the specific use cases, domains, and system characteristics where eventual consistency is the right choice, versus scenarios requiring stronger guarantees. We'll provide a decision framework for choosing consistency models.

Page Complete

You now understand how to build applications that work correctly with eventual consistency. These patterns—domain modeling, optimistic updates, compensating actions, and comprehensive monitoring—enable you to harness the scalability benefits of eventual consistency while maintaining application correctness.