Choosing Between CP and AP — The Final Decision Guide

You've mastered the theory of CAP. You understand consistency, availability, and partition tolerance. You know that during network partitions, you must choose between C and A. But when you're standing at the whiteboard in a design session, the question becomes very concrete: Should this system be CP or AP?

This decision has profound implications. It affects your technology choices, your failure modes, your user experience during outages, and your operational complexity. Making this choice incorrectly leads to either:

• Wrong CP: Users can't complete critical actions during network issues that happen regularly
• Wrong AP: Silent data corruption, lost transactions, inconsistent state that's never detected

This page gives you the specific tools and frameworks to make this decision correctly.

The CP vs AP decision ultimately reduces to one fundamental question:

What is the worse outcome: showing incorrect data or showing nothing at all?

This question has different answers in different contexts. Let's formalize the criteria:

Criterion 1: Cost of Inconsistency

What happens if your system shows stale or conflicting data?

If the cost of inconsistency is High → Lean toward CP

Criterion 2: Cost of Unavailability

What happens if users can't complete their action?

If the cost of unavailability is High → Lean toward AP

Criterion 3: Conflict Resolution Complexity

If you choose AP and conflicts occur, how hard are they to resolve?

If conflicts are hard to resolve → Lean toward CP

Criterion 4: Detection and Recovery Capability

Can you detect and recover from inconsistency after the fact?

If inconsistency is undetectable or unrecoverable → CP is mandatory

Use this systematic framework to make and document your CP/AP decisions:

Step 1: Define the Data in Question

Be specific. Don't say "the database." Identify:

• What entities are we storing?
• What operations are performed?
• What are the access patterns?
• What are the consistency requirements per operation?

Step 2: Enumerate the Failure Scenarios

List what can go wrong during partitions:

• Which nodes might be partitioned?
• What operations could be in flight?
• What state might diverge?
• How long might partitions last?

Step 3: Evaluate Each Criterion (1-5 scale)

Step 4: Apply the Decision Matrix

Step 5: Document the Decision

Create an Architecture Decision Record (ADR):

# ADR-042: Session Store CAP Strategy

## Status: Accepted

## Context
Our session store handles user authentication state across 
multiple data centers. Users expect to remain logged in 
regardless of network conditions.

## Decision
We will use an AP strategy (Redis Cluster) for session storage.

## Scoring
- Cost of Inconsistency: 2/5
- Cost of Unavailability: 5/5
- Conflict Resolution: 1/5
- Detection/Recovery: 2/5

## Consequences
- Positive: High availability during partitions
- Positive: Low latency for session lookups
- Negative: Rare cases of seeing old session data
- Mitigation: Sticky sessions for read-your-writes guarantee

This documentation ensures your decision is understood, defensible, and reviewable.

Different domains have established patterns based on their unique requirements. Use these as starting points, not absolute rules.

Financial Services:

E-Commerce:

Healthcare: A Critical Case Study

Healthcare illustrates the stakes of CP/AP decisions:

Medication allergies → Strong CP If a patient's allergy record is inconsistent, they might receive a medication that causes anaphylaxis. Unavailability ("Cannot access allergy data") is far safer than inconsistency. Systems should block prescription decisions if allergy data is uncertain.

Appointment scheduling → Balanced Double-booking an appointment is bad but not dangerous. Being unable to schedule appointments during an outage is operationally problematic. Most systems lean AP with conflict resolution (call patient if double-booked).

Radiology results → CP for diagnosis, AP for viewing Oncologists making treatment decisions need the authoritative, consistent result. But for a patient to view their own results? Slightly stale is probably fine, and availability matters.

Often, neither a pure CP nor pure AP strategy is optimal. Hybrid approaches provide the best of both worlds—with added complexity.

Hybrid Strategy 1: Tiered Consistency

Different tiers of data get different treatment:

┌─────────────────────────────────────────────────────────────────┐
│  TIER 1 (Critical): CP                                          │
│  - User authentication tokens                                   │
│  - Payment authorizations                                       │
│  - Inventory commitments                                        │
│  → Use: Consensus-based system (etcd, PostgreSQL sync rep)      │
│                                                                 │
│  TIER 2 (Important): CP with degraded reads                     │
│  - Order history                                                │
│  - Account settings                                             │
│  - Notification preferences                                     │
│  → Use: Read replicas with staleness bounds                     │
│                                                                 │
│  TIER 3 (Eventual OK): AP                                       │
│  - Session data                                                 │
│  - Activity logs                                                │
│  - Analytics events                                             │
│  → Use: Async replication, eventual consistency                 │
└─────────────────────────────────────────────────────────────────┘

Hybrid Strategy 2: Write CP, Read AP

Writes go through a strongly consistent path; reads can use eventually consistent replicas:

• All writes go to a CP primary (ensures correctness)
• Reads can hit AP replicas (fast, available)
• Critical reads (after important writes) go to primary
• Non-critical reads accept potential staleness

Hybrid Strategy 3: The Reservation Pattern

This elegant pattern provides CP guarantees with AP-like availability:

Phase 1: Soft Reservation (AP)

• User adds item to cart
• System "soft reserves" inventory (eventual, optimistic)
• Multiple users might soft-reserve the same item
• Available, fast, no consistency guarantee

Phase 2: Hard Commitment (CP)

• User proceeds to checkout
• System attempts "hard reservation" with CP storage
• If inventory is actually available: succeed
• If conflicting (someone else committed first): fail gracefully
• Consistent, may fail, but failure is handled

Result: High availability for browsing/carting (most users), consistency only required at commitment (fewer users, can tolerate occasional failure).

Hybrid Strategy 4: Local-First with Server Reconciliation

Emerging pattern for mobile and collaborative applications:

1. Client writes to local storage (instant, always available)
2. Background sync attempts to replicate to server
3. Server uses CP logic to detect and resolve conflicts
4. Conflicts are resolved and synced back to client
5. User sees eventual consistency but always has a usable app

Let's examine how real systems have made and justified their CP/AP decisions.

Case Study 1: Amazon Shopping Cart (AP)

Amazon famously chose AP for shopping carts (described in the Dynamo paper):

Context: Users add items to carts; carts maintain state across sessions and devices.

Decision: AP (eventually consistent, vector clocks for conflict detection)

Reasoning:

• Cost of Unavailability: Very high. Cart unavailability = lost sales.
• Cost of Inconsistency: Low-medium. If items are duplicated, user removes them. If items are lost, user re-adds them.
• Conflict Resolution: Easy. Cart merging is straightforward (union of items).
• Detection/Recovery: Easy. User sees and corrects cart contents.

Outcome: Carts remain available during severe AWS outages. Occasional item duplication is a minor annoyance that users can fix. Sales are preserved.

Case Study 2: Google Spanner (CP)

Google designed Spanner for global strong consistency:

Context: Finance-grade transactional system for AdWords, Play Store, etc.

Decision: CP (synchronous Paxos replication, TrueTime for ordering)

Reasoning:

• Cost of Inconsistency: Extreme. Double-charging advertisers, incorrect royalty payments = lawsuits.
• Cost of Unavailability: High but manageable. Can tolerate brief write unavailability.
• Conflict Resolution: Impossible. Financial transactions cannot be merged.
• Detection/Recovery: Hard. Detecting incorrect financial state is complex.

Outcome: Global strong consistency with 99.999% availability. Achieves both through massive engineering investment (GPS/atomic clock time sync, multi-region Paxos). Latency cost is accepted.

Case Study 4: Stripe Payments (CP with Idempotency)

Context: Payment processing across global merchant base.

Decision: CP for actual charges, with idempotency keys to handle retries.

Reasoning:

• Double-charging a customer: Unacceptable
• Losing a charge: Unacceptable
• Brief unavailability: Acceptable (merchants will retry)

Solution:

• CP storage for payment records
• Idempotency keys ensure retries don't double-charge
• Clear error responses enable intelligent retry logic
• During partitions: return errors, not silent failures

Developer Experience:

stripe.Charge.create(
  amount=1000,
  currency='usd',
  idempotency_key='order-12345',  # Ensures at-most-once
)

If the request times out, the developer can safely retry with the same idempotency key. Stripe guarantees the charge happens exactly once.

Whether in a design review, interview, or architecture document, you need to communicate CAP decisions clearly and persuasively.

The CAP Decision Communication Template:

1. State the decision clearly "For [component], we've chosen a [CP/AP] strategy using [technology]."

2. Explain the primary driver "The primary driver is [cost of inconsistency / cost of unavailability] because [business reason]."

3. Acknowledge the trade-off "This means that during partitions, [what will happen]."

4. Describe the mitigation "We mitigate this by [specific strategy]."

5. Justify with concrete scenarios "For example, if [partition scenario], then [system behavior], which is preferable to [alternative]."

Example: Presenting an AP Decision

"For our session store, we've chosen an AP strategy using Redis Cluster.

The primary driver is availability because users who can't log in can't use any feature of our platform—it's a total blocker.

This means during partitions, users might briefly see outdated session data or preferences.

We mitigate this with sticky sessions to ensure users read from the node that received their writes, providing read-your-writes consistency.

For example, if our East and West datacenters are partitioned, a user on the East coast who updates their preferences will see those updates immediately. A user who later connects to West might see the old preferences until the partition heals—but they can still use the system."

In System Design Interviews:

Interviewers expect you to discuss CAP trade-offs. Use this structure:

1. Identify data types - "We have user data, transaction data, and analytics data."
2. Associate requirements - "Transactions need strong consistency; analytics can be eventual."
3. Make explicit choices - "For the order database, I'll use PostgreSQL with synchronous replication."
4. Explain partition behavior - "During partitions, writes will block or fail, but we'll never have inconsistent orders."
5. Acknowledge trade-offs - "This means we might see errors during network issues, but the UX for that is a retry message."

This demonstrates both theoretical understanding and practical application—exactly what interviewers want to see.

A CAP strategy that isn't tested is a CAP hope. Validate your choices with concrete experiments.

Testing CP Systems:

1. Partition minority of nodes

   • Expect: Minority refuses operations, returns errors
   • Verify: No inconsistent data written during partition

2. Partition during active write

   • Expect: Write times out or returns error
   • Verify: After partition heals, data is consistent (either write succeeded or didn't)

3. Rapid leader failover

   • Expect: Brief unavailability during election
   • Verify: No split-brain, single leader elected

Testing AP Systems:

1. Partition and write to both sides

   • Expect: Both writes succeed
   • Verify: After healing, conflict is detected and resolved correctly

2. Read from partitioned replica

   • Expect: Reads succeed but may return stale data
   • Verify: Eventually, reads converge to correct value

3. Concurrent conflicting writes

   • Expect: Both succeed during operation
   • Verify: Resolution strategy (LWW, merge, etc.) produces expected result

Chaos Engineering for CAP Validation:

Integrate CAP testing into your regular chaos engineering practice:

1. Scheduled partition tests Run automated partition injection weekly/monthly. Compare actual behavior to expected behavior.

2. Game days Simulate major partition events with full team participation. Practice incident response.

3. Continuous chaos Tools like Chaos Monkey, LitmusChaos, Gremlin can inject partitions continuously in staging/production.

4. Jepsen testing For custom distributed systems, invest in Jepsen-style testing that verifies linearizability or your expected consistency model.

Metrics to Capture:

• Error rate during injected partitions
• Latency percentiles during partitions
• Conflict rate in AP systems
• Recovery time after partition heals
• Data consistency verification results