Availability — Every Request Gets a Response

At 2:47 AM on a Monday morning, a customer in Tokyo tries to complete a purchase on your e-commerce platform. Their payment goes through, but the confirmation page hangs indefinitely. The customer doesn't know if they were charged. They might try again, potentially paying twice. Or they might abandon the purchase entirely and never return.

This scenario illustrates why availability matters. In the CAP theorem, availability means that every request received by a non-failing node must result in a response—not just eventually, but within a reasonable time. The system cannot simply hang, time out, or refuse to answer. Availability is the promise that your system is always there when users need it.

The CAP theorem defines availability precisely, and this precision matters.

CAP Availability (Formal Definition):

Every request received by a non-failing node in the system must result in a response.

Let's unpack this definition carefully:

"Every request": No exceptions. The system cannot reject requests or refuse to answer. Every valid request must receive a response.

"Received by a non-failing node": This qualification is crucial. A crashed node is offline—we don't expect responses from it. Availability applies to nodes that are operational and reachable.

"Must result in a response": The response doesn't have to be "success." It can be an error, a partial result, or stale data. But there must be a response. Hanging indefinitely violates availability.

What CAP availability notably does NOT guarantee:

• That the response is correct or current
• That the response arrives within a specific time
• That all nodes respond (only non-failing ones must)
• That the response is useful for the client's purpose

Why the distinction matters:

When designing systems, you must be clear about which definition you're targeting. A system can achieve CAP availability by always returning cached (potentially stale) data, but this might violate your business requirements for data freshness. Conversely, a system that blocks until fresh data is available might provide better practical availability for correctness-dependent use cases, even if it technically violates CAP availability during partitions.

Understanding this distinction prevents a common mistake: believing that because you've achieved one form of availability, you've achieved the other.

In industry, availability is typically expressed as a percentage—the famous "nines" that appear in every SLA discussion. Understanding these numbers and their implications is essential for system design.

The Nines Table:

Each additional nine represents a 10x improvement in availability—and typically a 10x increase in cost and complexity to achieve.

How to Calculate Availability:

Basic Formula:

Availability = (Total Time - Downtime) / Total Time × 100%

With MTBF and MTTR:

Availability = MTBF / (MTBF + MTTR)

Where:

• MTBF (Mean Time Between Failures): Average time the system operates before failing
• MTTR (Mean Time To Recovery): Average time to restore service after failure

Example: If MTBF = 720 hours (30 days) and MTTR = 1 hour: Availability = 720 / (720 + 1) = 99.86%

This formula reveals two paths to higher availability:

1. Increase MTBF: Make failures less frequent (better hardware, redundancy)
2. Decrease MTTR: Make recovery faster (automation, hot standby)

In practice, reducing MTTR often provides better ROI than increasing MTBF, because preventing all failures is impossible, but fast recovery is achievable.

Understanding what breaks availability is as important as knowing what enables it. These anti-patterns appear repeatedly in production incidents.

Anti-Pattern 1: Single Points of Failure (SPOF)

A SPOF is any component whose failure brings down the entire system. Common SPOFs:

• Single database server (even with backups—failover takes time)
• Single load balancer (ironic, since it's meant to distribute load)
• Single DNS provider (if DNS fails, no traffic reaches you)
• Single cloud region (region-wide outages happen)
• Single team member who understands the system ("bus factor")

Anti-Pattern 2: Synchronous Dependencies

When Service A calls Service B synchronously, A's availability is bounded by B's:

Availability(A) ≤ Availability(A_standalone) × Availability(B)

With 5 serial dependencies at 99.9% each:

System Availability = 0.999^5 = 99.5%

That's 43 hours of downtime per year—from services that are individually 99.9%!

Anti-Pattern 3: The Thundering Herd

After an outage, all clients retry simultaneously. This surge overwhelms the recovering system, causing another failure. The system oscillates between up and down states.

Solution: Implement exponential backoff with jitter. Each client waits a random amount before retrying, spreading the load over time.

Anti-Pattern 4: Correlated Failures

Two "independent" systems fail together because they share a hidden dependency:

• Both services use the same Redis cluster for caching
• Both deployed at the same time with the same buggy library
• Both hosted in the same data center that lost power
• Both use the same third-party API that went down

Solution: Audit dependencies rigorously. True independence is harder than it looks.

High availability isn't a feature you add—it's an architectural property you design for from the beginning. These strategies form the foundation of available systems.

Strategy 1: Redundancy at Every Layer

Every component should have at least one backup ready to take over:

• Load Balancers: Active-passive or active-active pairs
• Application Servers: Horizontally scaled behind load balancer
• Databases: Primary-replica, multi-primary, or consensus-based replication
• Networks: Multiple paths, multiple ISPs, multiple regions
• Storage: RAID, erasure coding, geo-replicated object storage

Strategy 2: Health Checks and Automatic Failover

Redundancy is worthless without detection and action:

• Liveness checks: Is the component running?
• Readiness checks: Is the component ready to serve traffic?
• Deep health checks: Are dependencies available?
• Automatic failover: Remove unhealthy nodes, route to healthy ones

Failover must be automatic and fast. Manual intervention that takes 15 minutes negates the value of 99.99% availability (which allows only 4.38 minutes of downtime per month).

Strategy 3: Graceful Degradation

Not all features are equally critical. Design your system to shed non-essential functionality under stress:

Example: E-commerce during Black Friday

This approach maximizes availability of the most important functionality.

Let's examine how major systems achieve high availability and what trade-offs they accept.

Case Study 1: AWS S3 (11 nines of durability)

S3 famously promises 99.999999999% durability. This means if you store 10 million objects, you might lose one every 10,000 years. How?

• Data automatically replicated across at least 3 AZs
• Each AZ has independent power, cooling, networking
• Integrity checking with automatic repair
• No single device failure can cause data loss
• Multiple layers of redundancy at every level

Note: S3's durability (11 nines) differs from its availability (99.99% for Standard, meaning ~52 minutes downtime/year). Durability means data isn't lost. Availability means you can access it.

Case Study 2: Google Spanner (External Consistency + High Availability)

Spanner achieves 99.999% availability while maintaining strong consistency. The secret:

• Paxos-based replication across multiple zones
• Synchronized clocks with GPS and atomic clocks (TrueTime)
• Automatic failover with sub-second leader election
• Massive engineering investment in every component

This demonstrates that high consistency and high availability can coexist—if you're willing to pay the engineering cost.

Case Study 3: The 2017 S3 Outage

On February 28, 2017, a single typo in an S3 maintenance command took down a significant portion of the internet for several hours. Services affected included Slack, Trello, Quora, and parts of AWS itself.

Lessons learned:

1. Dependencies cascade: Many services hard-depended on S3. When S3 failed, they failed.
2. Testing matters: The command that caused the outage hadn't been tested at scale in years.
3. Blast radius: A simple administrative error affected millions of users globally.
4. Recovery can be slow: Some S3 subsystems took hours to fully recover.

Changes Amazon made:

• Added safeguards preventing rapid capacity removal
• Improved tooling with better confirmation and rollback
• Reduced interdependencies between S3 subsystems
• Published detailed post-mortem to help others learn

We've explored consistency and availability separately. Now we confront the heart of the CAP theorem: under network partitions, you cannot have both.

The Partition Scenario:

Imagine a two-node distributed database. A network partition occurs—the nodes can't communicate. A client sends a write to Node A. What happens?

If you choose Consistency:

• Node A cannot confirm the write without Node B's acknowledgment
• But Node A can't reach Node B
• Node A must refuse the write (or block indefinitely)
• Result: System is unavailable during the partition

If you choose Availability:

• Node A accepts and acknowledges the write
• Meanwhile, Node B might be accepting different writes
• When the partition heals, the nodes have diverged
• Result: System is inconsistent until (and during) reconciliation

This isn't a design flaw—it's a fundamental impossibility proven by Brewer and formalized by Gilbert and Lynch.

Why can't we have both?

The proof is elegant:

1. Assume we want both linearizability (C) and availability (A)
2. During a partition, messages between nodes are lost
3. A client sends a write to Node A, which completes
4. A client sends a read to Node B
5. For availability: Node B must respond
6. For consistency: Node B must return the value just written to Node A
7. But Node B never received the write (partition!)
8. Contradiction: Node B cannot satisfy both requirements

The key insight:

The CAP theorem doesn't say you can never have consistency and availability. It says you can't have both during a partition. In the absence of partitions, you can have both. This is why the PACELC theorem (which we'll cover later) extends CAP to describe behavior when partitions aren't occurring.

Given that we must make trade-offs, how do we architect systems that maximize availability while managing consistency challenges?

Pattern 1: Eventual Consistency with Conflict Resolution

Accept that replicas may temporarily diverge, but implement mechanisms to detect and resolve conflicts:

• Last-Writer-Wins (LWW): Simple but loses data
• Multi-Value Registers: Return all concurrent values, let application choose
• CRDTs: Data structures that automatically merge without conflicts
• Custom merge functions: Application-specific resolution logic

Pattern 2: Read-Your-Writes Guarantee

Ensure clients always see their own writes, even if other clients see stale data:

• Route read requests to the same node that handled recent writes (sticky sessions)
• Include version tokens in responses; wait for replica to reach that version before reading
• Read from primary for critical operations, replicas for others

Pattern 3: Async Replication with Sync Fallback

Normally use async replication for speed, but allow callers to request sync replication for critical operations:

POST /orders
X-Consistency: strong  // Use sync replication for this request

This lets you tune consistency per-operation rather than per-system.

Pattern 4: The Outbox Pattern for Cross-Service Reliability

When you need to update a database AND publish an event, use the Outbox Pattern:

1. Write the event to an "outbox" table in the same database transaction as your data change
2. A separate process reads the outbox and publishes events to the message broker
3. If publishing fails, the process retries (the event is safely in the database)
4. After successful publish, mark the outbox entry as processed

This guarantees that if the database update succeeds, the event will eventually be published—achieving reliability without distributed transactions.