System Design (HLD)High Availability

Availability vs Consistency Trade-offs

LevelAdvanced

Duration75 mins

TopicHigh Availability

4 / 5

Business Requirements Alignment

Translating Business Priorities into Technical Decisions

The consistency-availability trade-off isn't just a technical concern—it's fundamentally a business decision that engineers must help stakeholders understand and make. The best technical architecture is meaningless if it doesn't align with what the business actually needs.

This page teaches you to:

Extract consistency requirements from business language.
Quantify the business cost of both inconsistency and unavailability.
Communicate trade-offs to non-technical stakeholders clearly.
Document and maintain alignment between technical choices and business requirements.

These skills separate engineers who build systems from engineers who build the right systems.

What You Will Master

By the end of this page, you will be able to lead conversations with product managers, business analysts, and executives about consistency trade-offs, translating between technical and business vocabulary fluently.

From Business Requirements to Consistency Choices

Business stakeholders rarely say "we need linearizable consistency." They describe requirements in business terms that we must translate:

Business Language → Technical Requirement

Translating Business Requirements
Business Statement	Implied Consistency	Technical Translation
"The customer must never see a balance that's wrong"	Strong (CP)	Linearizable reads on account balances; reject reads if uncertain
"We can't sell more tickets than we have"	Strong (CP)	Serializable transactions for inventory; block sales if isolated
"Users should always be able to add to cart"	Eventual (AP)	Best-effort writes; merge conflicts later
"Search results need to be fast, not perfect"	Eventual (AP)	Stale indexes acceptable; prioritize response time
"A user should always see their own posts"	Session (RYW)	Read-your-writes guarantee; global consistency not required
"Reports don't need to be real-time"	Eventual (AP)	Async replication to reporting replica; minutes of lag acceptable
"Compliance audit must trace every action"	Strong (CP)	Durable, ordered writes; no dropped events

The Four Questions Framework

When gathering requirements, ask these questions explicitly:

1. "What happens if two users see different values for the same data at the same time?"

If answer involves money, legal, or safety concerns → CP
If answer involves slight confusion but is recoverable → AP is viable

2. "What happens if the system is temporarily unavailable?"

If transactions can wait (offline processing, retry) → CP is viable
If unavailability equals lost revenue or users → AP is necessary

3. "How quickly must changes propagate across all users?"

Milliseconds → Strong consistency required
Seconds → Session guarantees may suffice
Minutes → Eventual consistency is fine

4. "What's the recovery process if data conflicts occur?"

Business has clear conflict resolution (e.g., "most recent order wins") → AP viable
Conflicts require human investigation → CP preferred

requirements-gathering-template.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# Consistency Requirements Questionnaire
 
## Data Domain: [e.g., User Balances]
 
### Inconsistency Tolerance
1. If two users simultaneously view this data, can they see different values?
   - [ ] No - values must always match
   - [ ] Yes - for brief periods (< X seconds)
   - [ ] Yes - indefinitely, until explicitly synced
 
2. If a user makes a change, who must see it immediately?
   - [ ] Only that user
   - [ ] All users accessing the same record
   - [ ] All users globally
 
3. What's the maximum acceptable stale data age?
   - [ ] 0 (real-time)
   - [ ] Seconds
   - [ ] Minutes
   - [ ] Hours/Days
 
### Availability Tolerance
4. What happens if this feature is unavailable for 5 minutes?
   - [ ] Critical business impact (revenue, compliance)
   - [ ] Significant user frustration
   - [ ] Minor inconvenience
   - [ ] Acceptable (users can wait or retry)
 
5. During degraded conditions, what should happen?
   - [ ] Refuse operations (data integrity priority)
   - [ ] Allow operations with warning (availability priority)
   - [ ] Allow without warning (maximum availability)
 
### Conflict Resolution
6. If two users modify the same data simultaneously:
   - [ ] One must fail (no conflicts allowed)
   - [ ] Last write wins (simple resolution)
   - [ ] Merge changes (both preserved)
   - [ ] Flag for human review
 
### Compliance and Audit
7. Are there regulatory requirements for this data?
   - [ ] Yes - full audit trail required
   - [ ] Yes - accuracy requirements (financial, medical)
   - [ ] No specific requirements
 
---
## Summary
Based on responses, recommended consistency model: [Strong/Session/Eventual]
Key constraints: [List]
Risks of alternative models: [List]

Always Document the Decision

Create an Architecture Decision Record (ADR) for significant consistency choices. Include the business requirements that drove the decision, alternatives considered, and the trade-offs accepted. Future engineers will thank you.

Quantifying the Business Cost of Each Trade-off

Abstract discussions of "consistency vs availability" don't resonate with business stakeholders. Concrete cost analysis does. Your job is to quantify both sides of the trade-off in business terms.

Cost of Inconsistency

Inconsistency Cost Examples
Inconsistency Event	Business Impact	Example Quantification
Double-sold inventory	Operational cost + customer compensation + reputation damage	$50 per incident + 20% churn risk for affected customers
Stale product price displayed	Revenue loss if too low; trust loss if too high	1% of margin × affected transactions
Delayed notification	Reduced engagement; potential safety issue	0.1% engagement drop per 10 seconds delay
Inconsistent user profile	Minor confusion; user fixes it	Low cost; self-correcting
Financial audit failure	Regulatory fines; legal exposure	$10K-$10M depending on jurisdiction

Cost of Unavailability

Unavailability cost is often easier to quantify because it's more visible:

Unavailability Cost Examples
Unavailability Event	Business Impact	Example Quantification
E-commerce checkout down	Direct revenue loss	Revenue per minute × downtime minutes
API errors to partners	SLA violations; relationship damage	Penalty per minute + relationship cost
Search not responding	Users leave site; reduced conversions	Bounce rate × avg order value × traffic
Mobile app won't load	Negative reviews; uninstalls	Acquisition cost × churn rate increase
Internal tool down	Employee productivity loss	Employee cost per hour × affected employees

Building a Cost Model

For each consistency decision, build a simple cost model:

Scenario A (Prioritize Consistency):

Expected inconsistency events: 0
Expected unavailability: X hours/year
Cost of unavailability: $Y/hour
Annual cost: X × Y

Scenario B (Prioritize Availability):

Expected inconsistency events: Z/year
Cost per inconsistency: $W
Expected unavailability: 0
Annual cost: Z × W

Decision: Choose the lower expected annual cost, adjusted for risk tolerance.

cost-analysis-example.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// Cost analysis for inventory consistency decision
 
interface CostScenario {
  name: string;
  consistencyModel: 'CP' | 'AP';
  
  // Inconsistency costs
  expectedOversellsPerYear: number;
  costPerOversell: number;  // Customer compensation, fulfillment cost
  
  // Unavailability costs
  expectedUnavailabilityHoursPerYear: number;
  revenuePerHour: number;  // Lost sales during downtime
  
  // Indirect costs
  reputationImpact: number;  // Annual revenue at risk
  complianceRisk: number;    // Expected regulatory cost
}
 
function calculateAnnualCost(scenario: CostScenario): number {
  const inconsistencyCost = scenario.expectedOversellsPerYear * scenario.costPerOversell;
  const unavailabilityCost = scenario.expectedUnavailabilityHoursPerYear * scenario.revenuePerHour;
  
  return inconsistencyCost + unavailabilityCost + 
         scenario.reputationImpact + scenario.complianceRisk;
}
 
// Example: E-commerce inventory system
 
const cpScenario: CostScenario = {
  name: 'Strong Consistency (CP)',
  consistencyModel: 'CP',
  
  expectedOversellsPerYear: 0,  // Never oversell
  costPerOversell: 0,
  
  expectedUnavailabilityHoursPerYear: 8,  // Some downtime during partitions
  revenuePerHour: 50_000,
  
  reputationImpact: 0,  // No oversell = no angry customers
  complianceRisk: 0,
};
 
const apScenario: CostScenario = {
  name: 'Eventual Consistency (AP)',
  consistencyModel: 'AP',
  
  expectedOversellsPerYear: 500,  // Some overselling will occur
  costPerOversell: 150,  // Refund + shipping + customer service
  
  expectedUnavailabilityHoursPerYear: 0,  // Always available
  revenuePerHour: 0,
  
  reputationImpact: 25_000,  // Some customers unhappy with oversells
  complianceRisk: 0,
};
 
console.log('CP Cost:', calculateAnnualCost(cpScenario));
// $400,000 (8 hours × $50K/hour)
 
console.log('AP Cost:', calculateAnnualCost(apScenario));
// $100,000 (500 × $150 + $25K reputation)
 
// Decision: AP is cheaper for this specific scenario
// BUT: This ignores catastrophic risk (what if one oversell goes viral?)
// Need to also consider tail risk, not just expected value

Expected Value vs Tail Risk

Expected value calculations assume average outcomes. But a single catastrophic consistency failure (public embarrassment, regulatory intervention) can outweigh years of small savings. Include tail risk in your analysis, especially for financial, health, and safety-critical systems.

Communicating Trade-offs to Non-Technical Stakeholders

Technical jargon alienates stakeholders. Your goal is to explain the trade-off in terms they understand and care about.

Analogies That Work

Effective Analogies for CAP

•The ATM Analogy: "Imagine two ATMs at different bank branches. If they can't communicate (network partition), should they refuse to dispense cash (consistency) or give cash but risk overdrafts (availability)? For bank balances, we refuse. For displaying marketing material, we continue."
•The Inventory Warehouse: "If our New York and Los Angeles warehouses can't talk to each other, should they both accept orders (risking overselling) or should one pause (guaranteed correct but some orders fail)? The choice depends on the cost of each outcome."
•The Flight Status Board: "Airport boards can show 'Flight Delayed' based on partial information. If they had to be 100% accurate at all times, they'd sometimes show 'Information Unavailable.' Which is better for travelers?"
•The Restaurant Reservation: "If two people book the same table at the exact same moment via different systems, do we accept one and immediately reject the other (consistent), or accept both and sort it out later (available)? Fine dining might prefer the former; a casual restaurant the latter."

Framing the Conversation

Avoid: "We need to choose between CP and AP for the inventory service."

Instead: "There's a decision we need to make: In rare network issues, should the checkout refuse some orders to guarantee we never oversell, or should it accept all orders but occasionally need to manually fix an oversell? Here are the cost estimates for each approach..."

Visual Communication

Create simple diagrams that show:

Normal operation (everything works well)
Degraded operation (network issue occurs)
What each approach (CP vs AP) does during degradation
The business impact of each

stakeholder-presentation-outline.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Consistency Model Decision: [System Name]
 
## The Situation
We need to make an architectural decision about how [system] behaves 
when network issues occur. This affects both user experience and 
operational costs.
 
## Option A: Prioritize Accuracy
**What happens:** During network issues (~X times/year, lasting ~Y minutes each), 
some users will see temporary errors instead of completing their action.
 
**Business impact:**
- [Z] transactions affected per year
- $[A] estimated lost revenue
- Zero risk of [specific inconsistency problem]
 
**User experience:** "Please try again in a moment"
 
## Option B: Prioritize Availability  
**What happens:** During network issues, all users can continue their actions, 
but some may experience [specific inconsistency].
 
**Business impact:**
- Zero downtime during network issues
- [W] incidents per year requiring manual fix
- $[B] operational cost + customer compensation
 
**User experience:** Works normally but [inconsistency scenario] may occur
 
## Recommendation
Based on our cost analysis, we recommend [Option X] because:
- [Primary reason]
- [Secondary reason]
 
## What We Need From You
- Confirm the cost estimates are reasonable
- Validate which user experience is preferable
- Approve the recommended approach
 
## Questions?

Let Them Decide

Present options, provide analysis, make a recommendation—but let business stakeholders make the actual decision. They own the business priorities; you own the technical translation. Document their decision for future reference.

Industry-Specific Considerations

Different industries have vastly different tolerance for inconsistency and unavailability. Understanding your industry's norms and regulations is essential.

Financial Services

•Regulatory pressure: SOX, Dodd-Frank, PCI-DSS mandate accurate record-keeping. Audit trails must be complete.
•Inconsistency cost: Extremely high. Double-spend, incorrect balances, and missing transactions have legal consequences.
•Availability expectation: Important but secondary. Customers accept brief "service unavailable" over incorrect balances.
•Typical pattern: Strong consistency for transactions; eventual consistency for displays and notifications.

E-Commerce & Retail

•Regulatory pressure: Moderate. Consumer protection laws apply but less stringent than finance.
•Inconsistency cost: Medium. Overselling is costly but recoverable (refunds, apologies).
•Availability expectation: Very high. Every second of downtime is measurable revenue loss.
•Typical pattern: Strong consistency for payments and inventory; eventual consistency for catalogs, reviews, recommendations.

Social Media & Content

•Regulatory pressure: Low for content; higher for advertising spend.
•Inconsistency cost: Low. Slightly stale feeds, delayed likes, temporary duplicate posts are invisible to most users.
•Availability expectation: Extremely high. Users leave if the app doesn't load instantly.
•Typical pattern: Eventual consistency almost everywhere; strong consistency only for billing and authentication.

Healthcare & Life Sciences

•Regulatory pressure: Extremely high. HIPAA, FDA, clinical trial regulations mandate accuracy.
•Inconsistency cost: Potentially catastrophic. Wrong dosages, mixed-up records, or missing information can harm patients.
•Availability expectation: High for critical care, moderate for administrative functions.
•Typical pattern: Strong consistency for patient records, prescriptions, and clinical data. Availability sacrificed when necessary for safety.

Industry Consistency Patterns Summary
Industry	Default Approach	Consistency Priority	Availability Priority
Banking & Finance	CP	Transactions, balances, audit logs	Notifications, marketing content
E-Commerce	Mixed	Payments, inventory	Catalog, reviews, search
Social Media	AP	Billing, authentication	Everything else
Healthcare	CP	Patient data, prescriptions	Scheduling, non-clinical content
Gaming	Mixed	Purchases, rankings	Game state, chat, matchmaking
IoT / Telemetry	AP	Billing, alerts	Sensor data, metrics

Know Your Regulators

Before finalizing consistency decisions, consult with your compliance and legal teams. Regulations may mandate specific behaviors (e.g., audit completeness) that override pure business cost analysis.

Documenting and Maintaining Alignment

Consistency decisions are made once but have long-lasting implications. Documentation ensures decisions are understood, followed, and revisited when appropriate.

Architecture Decision Records (ADRs)

Every significant consistency choice should have an ADR that captures:

The decision context and problem statement.
Options considered with pros/cons.
The decision made and rationale.
Consequences and trade-offs accepted.
Date, stakeholders involved, and approval.

adr-template.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# ADR-042: Consistency Model for Order Management System
 
## Status
Accepted (2024-01-15)
 
## Context
The Order Management System (OMS) handles order creation, inventory 
reservation, and fulfillment tracking. We need to determine the consistency 
model for key operations, particularly during network partitions between 
our primary datacenter and disaster recovery site.
 
Key stakeholders: Product (Alice), Engineering (Bob), Compliance (Carol)
 
## Decision Drivers
- Business requires 99.9% uptime for order submission
- Compliance requires complete audit trail (SOX)
- Overselling has cost $150K in manual fixes over past year
- Current downtime costs ~$50K/hour in lost revenue
 
## Options Considered
 
### Option 1: Strong Consistency (CP)
- All writes require majority quorum across datacenters
- Orders would fail during cross-DC partition
- Pros: Never oversell; perfect audit trail
- Cons: Estimated 4 hours downtime/year during partitions
 
### Option 2: Eventual Consistency (AP)  
- Accept orders locally; sync asynchronously
- Merge conflicts with last-write-wins
- Pros: Zero order submission downtime
- Cons: Possible oversells; audit complexity
 
### Option 3: Hybrid Approach (Recommended)
- Strong consistency for inventory reservation
- Eventual consistency for order display and notifications
- Session guarantees for customer's view of their orders
 
## Decision
Adopt Option 3 (Hybrid Approach)
 
**Rationale:**
- Inventory is the critical constraint; gets strong consistency
- Order display is read-heavy and tolerates staleness
- Customer session guarantees prevent confusion ("where's my order?")
- Cost analysis: Hybrid has lowest expected annual cost
 
## Consequences
- Inventory service has ~5 hours/year of reduced availability during partitions
- Engineering must implement session-aware routing for customer reads
- Conflicting order updates during partition require manual review
- Quarterly review of oversell incidents to validate model
 
## Stakeholder Approval
- Product: Alice (approved)
- Engineering: Bob (approved)  
- Compliance: Carol (approved with condition: quarterly audit review)

Runbooks for Consistency Events

Document procedures for handling consistency-related incidents:

Partition detected: What alerts fire? Who responds? What manual interventions are allowed?
Conflict resolution: How are conflicts identified? What's the escalation path?
Post-partition reconciliation: What automated reconciliation runs? What manual verification is required?
Audit and reporting: How do we verify consistency was maintained during degraded operation?

Periodic Review

Consistency decisions should be reviewed periodically (annually or when major changes occur):

Have business priorities shifted?
Have incident costs matched our predictions?
Are there new technologies that change the trade-off?
Have regulatory requirements changed?

Living Documentation

ADRs should be versioned and updated, not replaced. When you change a decision, create a new ADR that references and supersedes the old one. This maintains the decision history and makes it clear how thinking evolved.

Handling Conflicting Requirements

Sometimes different stakeholders have contradictory requirements. Product wants maximum availability; Compliance demands perfect accuracy. Navigating these conflicts is a critical skill.

Resolution Strategies

•Separate the concerns: Often, different data within the same system can have different consistency models. Payments can be CP while notifications are AP. Both stakeholders get what they need for their domain.
•Quantify the trade-off: When separation isn't possible, present concrete cost analysis. "We can achieve 99.99% availability with occasional oversells, or 99.9% with no oversells. The cost difference is $X vs $Y."
•Escalate to decision-maker: If stakeholders can't agree, escalate to someone with authority over both domains. Come with data and a recommendation, not just a problem.
•Propose mitigation: If choosing AP to satisfy availability requirements, propose compensating controls. "We'll prioritize availability but add automated conflict detection and manual review for edge cases."
•Accept uncertainty: Technology is about trade-offs. Document the uncertainty, set up monitoring to learn from real-world behavior, and commit to revisiting the decision with actual data.

Common Conflict Patterns

Product vs Compliance:

Product wants features to never fail.
Compliance wants data to never be wrong.
Resolution: Identify which data is compliance-critical (usually a subset) and apply strong consistency there.

Engineering vs Business:

Engineering prefers simpler systems (often CP is simpler to reason about).
Business wants maximum availability.
Resolution: Quantify operational cost of each approach, including engineering time for incident response.

Short-term vs Long-term:

Short-term: "Just ship it, we'll fix consistency issues later."
Long-term: Technical debt from poor consistency choices is expensive.
Resolution: Calculate the cost of fixing consistency issues later vs building it right initially.

Beware False Dichotomies

Sometimes "conflicting" requirements aren't actually in conflict when you dig deeper. Product may accept brief unavailability if warned in advance. Compliance may accept eventual consistency with appropriate controls. Always probe whether the conflict is real before declaring it irresolvable.

Summary: Business Alignment Mastery

We've explored the often-neglected connection between technical consistency choices and business requirements. This is where senior engineers become technical leaders.

Key Takeaways

•Translate business language to technical requirements — Stakeholders express needs in business terms; your job is to identify the implied consistency model.
•Quantify both sides of the trade-off — Cost of inconsistency (errors, compliance, reputation) vs cost of unavailability (lost revenue, SLA penalties) must be concrete numbers.
•Use accessible analogies — ATMs, warehouses, and restaurants help stakeholders understand without technical jargon.
•Respect industry context — Finance, healthcare, and e-commerce have vastly different tolerance for each type of failure. Know your industry's norms.
•Document decisions formally — ADRs capture the reasoning, stakeholder approval, and trade-offs accepted for future reference.
•Navigate conflicts constructively — Separate concerns, quantify trade-offs, and escalate with data when stakeholders disagree.

What's Next

The final page of this module explores practical compromises—the real-world techniques that systems use to get the best of both worlds where possible, and gracefully handle the situations where trade-offs are inevitable.

Page Complete

You now have the skills to bridge the technical and business worlds around consistency decisions. This ability—to translate, quantify, communicate, and document trade-offs—is what differentiates engineers who build systems from engineers who build the right systems for their organizations.

4 / 5

Loading learning content...

System Design (HLD)High Availability

Availability vs Consistency Trade-offs

LevelAdvanced

Duration75 mins

TopicHigh Availability

4 / 5

Business Requirements Alignment

Translating Business Priorities into Technical Decisions

This page teaches you to:

Extract consistency requirements from business language.
Quantify the business cost of both inconsistency and unavailability.
Communicate trade-offs to non-technical stakeholders clearly.
Document and maintain alignment between technical choices and business requirements.

These skills separate engineers who build systems from engineers who build the right systems.

What You Will Master

From Business Requirements to Consistency Choices

Business stakeholders rarely say "we need linearizable consistency." They describe requirements in business terms that we must translate:

Business Language → Technical Requirement

Translating Business Requirements
Business Statement	Implied Consistency	Technical Translation
"The customer must never see a balance that's wrong"	Strong (CP)	Linearizable reads on account balances; reject reads if uncertain
"We can't sell more tickets than we have"	Strong (CP)	Serializable transactions for inventory; block sales if isolated
"Users should always be able to add to cart"	Eventual (AP)	Best-effort writes; merge conflicts later
"Search results need to be fast, not perfect"	Eventual (AP)	Stale indexes acceptable; prioritize response time
"A user should always see their own posts"	Session (RYW)	Read-your-writes guarantee; global consistency not required
"Reports don't need to be real-time"	Eventual (AP)	Async replication to reporting replica; minutes of lag acceptable
"Compliance audit must trace every action"	Strong (CP)	Durable, ordered writes; no dropped events

The Four Questions Framework

When gathering requirements, ask these questions explicitly:

1. "What happens if two users see different values for the same data at the same time?"

If answer involves money, legal, or safety concerns → CP
If answer involves slight confusion but is recoverable → AP is viable

2. "What happens if the system is temporarily unavailable?"

If transactions can wait (offline processing, retry) → CP is viable
If unavailability equals lost revenue or users → AP is necessary

3. "How quickly must changes propagate across all users?"

Milliseconds → Strong consistency required
Seconds → Session guarantees may suffice
Minutes → Eventual consistency is fine

4. "What's the recovery process if data conflicts occur?"

Business has clear conflict resolution (e.g., "most recent order wins") → AP viable
Conflicts require human investigation → CP preferred

requirements-gathering-template.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# Consistency Requirements Questionnaire
 
## Data Domain: [e.g., User Balances]
 
### Inconsistency Tolerance
1. If two users simultaneously view this data, can they see different values?
   - [ ] No - values must always match
   - [ ] Yes - for brief periods (< X seconds)
   - [ ] Yes - indefinitely, until explicitly synced
 
2. If a user makes a change, who must see it immediately?
   - [ ] Only that user
   - [ ] All users accessing the same record
   - [ ] All users globally
 
3. What's the maximum acceptable stale data age?
   - [ ] 0 (real-time)
   - [ ] Seconds
   - [ ] Minutes
   - [ ] Hours/Days
 
### Availability Tolerance
4. What happens if this feature is unavailable for 5 minutes?
   - [ ] Critical business impact (revenue, compliance)
   - [ ] Significant user frustration
   - [ ] Minor inconvenience
   - [ ] Acceptable (users can wait or retry)
 
5. During degraded conditions, what should happen?
   - [ ] Refuse operations (data integrity priority)
   - [ ] Allow operations with warning (availability priority)
   - [ ] Allow without warning (maximum availability)
 
### Conflict Resolution
6. If two users modify the same data simultaneously:
   - [ ] One must fail (no conflicts allowed)
   - [ ] Last write wins (simple resolution)
   - [ ] Merge changes (both preserved)
   - [ ] Flag for human review
 
### Compliance and Audit
7. Are there regulatory requirements for this data?
   - [ ] Yes - full audit trail required
   - [ ] Yes - accuracy requirements (financial, medical)
   - [ ] No specific requirements
 
---
## Summary
Based on responses, recommended consistency model: [Strong/Session/Eventual]
Key constraints: [List]
Risks of alternative models: [List]

Always Document the Decision

Quantifying the Business Cost of Each Trade-off

Cost of Inconsistency

Inconsistency Cost Examples
Inconsistency Event	Business Impact	Example Quantification
Double-sold inventory	Operational cost + customer compensation + reputation damage	$50 per incident + 20% churn risk for affected customers
Stale product price displayed	Revenue loss if too low; trust loss if too high	1% of margin × affected transactions
Delayed notification	Reduced engagement; potential safety issue	0.1% engagement drop per 10 seconds delay
Inconsistent user profile	Minor confusion; user fixes it	Low cost; self-correcting
Financial audit failure	Regulatory fines; legal exposure	$10K-$10M depending on jurisdiction

Cost of Unavailability

Unavailability cost is often easier to quantify because it's more visible:

Unavailability Cost Examples
Unavailability Event	Business Impact	Example Quantification
E-commerce checkout down	Direct revenue loss	Revenue per minute × downtime minutes
API errors to partners	SLA violations; relationship damage	Penalty per minute + relationship cost
Search not responding	Users leave site; reduced conversions	Bounce rate × avg order value × traffic
Mobile app won't load	Negative reviews; uninstalls	Acquisition cost × churn rate increase
Internal tool down	Employee productivity loss	Employee cost per hour × affected employees

Building a Cost Model

For each consistency decision, build a simple cost model:

Scenario A (Prioritize Consistency):

Expected inconsistency events: 0
Expected unavailability: X hours/year
Cost of unavailability: $Y/hour
Annual cost: X × Y

Scenario B (Prioritize Availability):

Expected inconsistency events: Z/year
Cost per inconsistency: $W
Expected unavailability: 0
Annual cost: Z × W

Decision: Choose the lower expected annual cost, adjusted for risk tolerance.

cost-analysis-example.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// Cost analysis for inventory consistency decision
 
interface CostScenario {
  name: string;
  consistencyModel: 'CP' | 'AP';
  
  // Inconsistency costs
  expectedOversellsPerYear: number;
  costPerOversell: number;  // Customer compensation, fulfillment cost
  
  // Unavailability costs
  expectedUnavailabilityHoursPerYear: number;
  revenuePerHour: number;  // Lost sales during downtime
  
  // Indirect costs
  reputationImpact: number;  // Annual revenue at risk
  complianceRisk: number;    // Expected regulatory cost
}
 
function calculateAnnualCost(scenario: CostScenario): number {
  const inconsistencyCost = scenario.expectedOversellsPerYear * scenario.costPerOversell;
  const unavailabilityCost = scenario.expectedUnavailabilityHoursPerYear * scenario.revenuePerHour;
  
  return inconsistencyCost + unavailabilityCost + 
         scenario.reputationImpact + scenario.complianceRisk;
}
 
// Example: E-commerce inventory system
 
const cpScenario: CostScenario = {
  name: 'Strong Consistency (CP)',
  consistencyModel: 'CP',
  
  expectedOversellsPerYear: 0,  // Never oversell
  costPerOversell: 0,
  
  expectedUnavailabilityHoursPerYear: 8,  // Some downtime during partitions
  revenuePerHour: 50_000,
  
  reputationImpact: 0,  // No oversell = no angry customers
  complianceRisk: 0,
};
 
const apScenario: CostScenario = {
  name: 'Eventual Consistency (AP)',
  consistencyModel: 'AP',
  
  expectedOversellsPerYear: 500,  // Some overselling will occur
  costPerOversell: 150,  // Refund + shipping + customer service
  
  expectedUnavailabilityHoursPerYear: 0,  // Always available
  revenuePerHour: 0,
  
  reputationImpact: 25_000,  // Some customers unhappy with oversells
  complianceRisk: 0,
};
 
console.log('CP Cost:', calculateAnnualCost(cpScenario));
// $400,000 (8 hours × $50K/hour)
 
console.log('AP Cost:', calculateAnnualCost(apScenario));
// $100,000 (500 × $150 + $25K reputation)
 
// Decision: AP is cheaper for this specific scenario
// BUT: This ignores catastrophic risk (what if one oversell goes viral?)
// Need to also consider tail risk, not just expected value

Expected Value vs Tail Risk

Communicating Trade-offs to Non-Technical Stakeholders

Technical jargon alienates stakeholders. Your goal is to explain the trade-off in terms they understand and care about.

Analogies That Work

Effective Analogies for CAP

•The ATM Analogy: "Imagine two ATMs at different bank branches. If they can't communicate (network partition), should they refuse to dispense cash (consistency) or give cash but risk overdrafts (availability)? For bank balances, we refuse. For displaying marketing material, we continue."
•The Inventory Warehouse: "If our New York and Los Angeles warehouses can't talk to each other, should they both accept orders (risking overselling) or should one pause (guaranteed correct but some orders fail)? The choice depends on the cost of each outcome."
•The Flight Status Board: "Airport boards can show 'Flight Delayed' based on partial information. If they had to be 100% accurate at all times, they'd sometimes show 'Information Unavailable.' Which is better for travelers?"
•The Restaurant Reservation: "If two people book the same table at the exact same moment via different systems, do we accept one and immediately reject the other (consistent), or accept both and sort it out later (available)? Fine dining might prefer the former; a casual restaurant the latter."

Framing the Conversation

Avoid: "We need to choose between CP and AP for the inventory service."

Visual Communication

Create simple diagrams that show:

Normal operation (everything works well)
Degraded operation (network issue occurs)
What each approach (CP vs AP) does during degradation
The business impact of each

stakeholder-presentation-outline.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Consistency Model Decision: [System Name]
 
## The Situation
We need to make an architectural decision about how [system] behaves 
when network issues occur. This affects both user experience and 
operational costs.
 
## Option A: Prioritize Accuracy
**What happens:** During network issues (~X times/year, lasting ~Y minutes each), 
some users will see temporary errors instead of completing their action.
 
**Business impact:**
- [Z] transactions affected per year
- $[A] estimated lost revenue
- Zero risk of [specific inconsistency problem]
 
**User experience:** "Please try again in a moment"
 
## Option B: Prioritize Availability  
**What happens:** During network issues, all users can continue their actions, 
but some may experience [specific inconsistency].
 
**Business impact:**
- Zero downtime during network issues
- [W] incidents per year requiring manual fix
- $[B] operational cost + customer compensation
 
**User experience:** Works normally but [inconsistency scenario] may occur
 
## Recommendation
Based on our cost analysis, we recommend [Option X] because:
- [Primary reason]
- [Secondary reason]
 
## What We Need From You
- Confirm the cost estimates are reasonable
- Validate which user experience is preferable
- Approve the recommended approach
 
## Questions?

Let Them Decide

Industry-Specific Considerations

Different industries have vastly different tolerance for inconsistency and unavailability. Understanding your industry's norms and regulations is essential.

Financial Services

•Regulatory pressure: SOX, Dodd-Frank, PCI-DSS mandate accurate record-keeping. Audit trails must be complete.
•Inconsistency cost: Extremely high. Double-spend, incorrect balances, and missing transactions have legal consequences.
•Availability expectation: Important but secondary. Customers accept brief "service unavailable" over incorrect balances.
•Typical pattern: Strong consistency for transactions; eventual consistency for displays and notifications.

E-Commerce & Retail

•Regulatory pressure: Moderate. Consumer protection laws apply but less stringent than finance.
•Inconsistency cost: Medium. Overselling is costly but recoverable (refunds, apologies).
•Availability expectation: Very high. Every second of downtime is measurable revenue loss.
•Typical pattern: Strong consistency for payments and inventory; eventual consistency for catalogs, reviews, recommendations.

Social Media & Content

•Regulatory pressure: Low for content; higher for advertising spend.
•Inconsistency cost: Low. Slightly stale feeds, delayed likes, temporary duplicate posts are invisible to most users.
•Availability expectation: Extremely high. Users leave if the app doesn't load instantly.
•Typical pattern: Eventual consistency almost everywhere; strong consistency only for billing and authentication.

Healthcare & Life Sciences

•Regulatory pressure: Extremely high. HIPAA, FDA, clinical trial regulations mandate accuracy.
•Inconsistency cost: Potentially catastrophic. Wrong dosages, mixed-up records, or missing information can harm patients.
•Availability expectation: High for critical care, moderate for administrative functions.
•Typical pattern: Strong consistency for patient records, prescriptions, and clinical data. Availability sacrificed when necessary for safety.

Industry Consistency Patterns Summary
Industry	Default Approach	Consistency Priority	Availability Priority
Banking & Finance	CP	Transactions, balances, audit logs	Notifications, marketing content
E-Commerce	Mixed	Payments, inventory	Catalog, reviews, search
Social Media	AP	Billing, authentication	Everything else
Healthcare	CP	Patient data, prescriptions	Scheduling, non-clinical content
Gaming	Mixed	Purchases, rankings	Game state, chat, matchmaking
IoT / Telemetry	AP	Billing, alerts	Sensor data, metrics

Know Your Regulators

Before finalizing consistency decisions, consult with your compliance and legal teams. Regulations may mandate specific behaviors (e.g., audit completeness) that override pure business cost analysis.

Documenting and Maintaining Alignment

Consistency decisions are made once but have long-lasting implications. Documentation ensures decisions are understood, followed, and revisited when appropriate.

Architecture Decision Records (ADRs)

Every significant consistency choice should have an ADR that captures:

The decision context and problem statement.
Options considered with pros/cons.
The decision made and rationale.
Consequences and trade-offs accepted.
Date, stakeholders involved, and approval.

adr-template.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# ADR-042: Consistency Model for Order Management System
 
## Status
Accepted (2024-01-15)
 
## Context
The Order Management System (OMS) handles order creation, inventory 
reservation, and fulfillment tracking. We need to determine the consistency 
model for key operations, particularly during network partitions between 
our primary datacenter and disaster recovery site.
 
Key stakeholders: Product (Alice), Engineering (Bob), Compliance (Carol)
 
## Decision Drivers
- Business requires 99.9% uptime for order submission
- Compliance requires complete audit trail (SOX)
- Overselling has cost $150K in manual fixes over past year
- Current downtime costs ~$50K/hour in lost revenue
 
## Options Considered
 
### Option 1: Strong Consistency (CP)
- All writes require majority quorum across datacenters
- Orders would fail during cross-DC partition
- Pros: Never oversell; perfect audit trail
- Cons: Estimated 4 hours downtime/year during partitions
 
### Option 2: Eventual Consistency (AP)  
- Accept orders locally; sync asynchronously
- Merge conflicts with last-write-wins
- Pros: Zero order submission downtime
- Cons: Possible oversells; audit complexity
 
### Option 3: Hybrid Approach (Recommended)
- Strong consistency for inventory reservation
- Eventual consistency for order display and notifications
- Session guarantees for customer's view of their orders
 
## Decision
Adopt Option 3 (Hybrid Approach)
 
**Rationale:**
- Inventory is the critical constraint; gets strong consistency
- Order display is read-heavy and tolerates staleness
- Customer session guarantees prevent confusion ("where's my order?")
- Cost analysis: Hybrid has lowest expected annual cost
 
## Consequences
- Inventory service has ~5 hours/year of reduced availability during partitions
- Engineering must implement session-aware routing for customer reads
- Conflicting order updates during partition require manual review
- Quarterly review of oversell incidents to validate model
 
## Stakeholder Approval
- Product: Alice (approved)
- Engineering: Bob (approved)  
- Compliance: Carol (approved with condition: quarterly audit review)

Runbooks for Consistency Events

Document procedures for handling consistency-related incidents:

Partition detected: What alerts fire? Who responds? What manual interventions are allowed?
Conflict resolution: How are conflicts identified? What's the escalation path?
Post-partition reconciliation: What automated reconciliation runs? What manual verification is required?
Audit and reporting: How do we verify consistency was maintained during degraded operation?

Periodic Review

Consistency decisions should be reviewed periodically (annually or when major changes occur):

Have business priorities shifted?
Have incident costs matched our predictions?
Are there new technologies that change the trade-off?
Have regulatory requirements changed?

Living Documentation

Handling Conflicting Requirements

Sometimes different stakeholders have contradictory requirements. Product wants maximum availability; Compliance demands perfect accuracy. Navigating these conflicts is a critical skill.

Resolution Strategies

•Separate the concerns: Often, different data within the same system can have different consistency models. Payments can be CP while notifications are AP. Both stakeholders get what they need for their domain.
•Quantify the trade-off: When separation isn't possible, present concrete cost analysis. "We can achieve 99.99% availability with occasional oversells, or 99.9% with no oversells. The cost difference is $X vs $Y."
•Escalate to decision-maker: If stakeholders can't agree, escalate to someone with authority over both domains. Come with data and a recommendation, not just a problem.
•Propose mitigation: If choosing AP to satisfy availability requirements, propose compensating controls. "We'll prioritize availability but add automated conflict detection and manual review for edge cases."
•Accept uncertainty: Technology is about trade-offs. Document the uncertainty, set up monitoring to learn from real-world behavior, and commit to revisiting the decision with actual data.

Common Conflict Patterns

Product vs Compliance:

Product wants features to never fail.
Compliance wants data to never be wrong.
Resolution: Identify which data is compliance-critical (usually a subset) and apply strong consistency there.

Engineering vs Business:

Engineering prefers simpler systems (often CP is simpler to reason about).
Business wants maximum availability.
Resolution: Quantify operational cost of each approach, including engineering time for incident response.

Short-term vs Long-term:

Short-term: "Just ship it, we'll fix consistency issues later."
Long-term: Technical debt from poor consistency choices is expensive.
Resolution: Calculate the cost of fixing consistency issues later vs building it right initially.

Beware False Dichotomies

Summary: Business Alignment Mastery

We've explored the often-neglected connection between technical consistency choices and business requirements. This is where senior engineers become technical leaders.

Key Takeaways

•Translate business language to technical requirements — Stakeholders express needs in business terms; your job is to identify the implied consistency model.
•Quantify both sides of the trade-off — Cost of inconsistency (errors, compliance, reputation) vs cost of unavailability (lost revenue, SLA penalties) must be concrete numbers.
•Use accessible analogies — ATMs, warehouses, and restaurants help stakeholders understand without technical jargon.
•Respect industry context — Finance, healthcare, and e-commerce have vastly different tolerance for each type of failure. Know your industry's norms.
•Document decisions formally — ADRs capture the reasoning, stakeholder approval, and trade-offs accepted for future reference.
•Navigate conflicts constructively — Separate concerns, quantify trade-offs, and escalate with data when stakeholders disagree.

What's Next

Page Complete

4 / 5