Circuit Breaker Pattern

At 2:47 AM on a quiet Monday morning, a payment processing service at a major e-commerce platform began experiencing elevated latency. The database behind it was struggling under an unusual load. Within 3 minutes, the payment service's response times climbed from 50ms to 15 seconds. Within 7 minutes, the order service—waiting synchronously for payment confirmations—had exhausted its thread pool. By the 10-minute mark, the product catalog service was unresponsive. At minute 15, the entire platform was down.

A single slow database had cascaded into a complete system failure affecting millions of users.

This scenario isn't hypothetical—it's a pattern that has taken down some of the world's largest platforms: Amazon, Netflix, Twitter, and countless others. The phenomenon is called cascade failure, and understanding how to prevent it is one of the most critical skills in distributed systems engineering.

To prevent cascade failures, we must first understand how they propagate. A cascade failure is not a single failure—it's a chain reaction where the failure of one component causes dependent components to fail, which in turn causes their dependents to fail, and so on until the entire system collapses.

The cascade failure lifecycle:

1. Initial Failure: A single component experiences an issue (database overload, network partition, bug, resource exhaustion)
2. Resource Accumulation: Callers waiting for the failed component begin accumulating (threads blocked, connections held, memory consumed)
3. Resource Exhaustion: Dependent services exhaust their resources waiting for responses that never come
4. Failure Propagation: Exhausted services become unable to handle even unrelated requests
5. System-Wide Collapse: The failure ripples through the dependency graph until nothing works

Why distributed systems are vulnerable:

Modern distributed architectures create intricate dependency webs. A single user request might traverse 10-50 different services. Each service call represents a potential failure point, and each failure point can initiate a cascade.

Consider a typical e-commerce request flow:

The dependency amplification effect:

Notice that failures amplify as they propagate upward. If the Inventory Service fails, the Cart Service cannot verify stock. If Cart fails, the Order Service cannot create orders. If Order fails, the entire checkout flow is broken—even if Payment, Pricing, and Product Catalog are perfectly healthy.

This amplification means that in a system with N services, your actual availability isn't the average of all service availabilities—it's closer to the product of them. If each of 50 services is 99.9% available independently, your system's theoretical availability is 0.999^50 ≈ 95.1%—far below what any individual service provides.

Before understanding the circuit breaker solution, it's instructive to examine why simpler approaches don't adequately address cascade failures.

Approach 1: Timeouts

Setting timeouts on all service calls seems like a straightforward solution—if a service doesn't respond within, say, 5 seconds, give up and move on. However, timeouts alone have critical limitations:

Approach 2: Health Checks

Periodic health checks can detect unhealthy services and remove them from the routing pool. But health checks have their own problems:

Approach 3: Retries with Backoff

Retrying failed requests with exponential backoff is a valuable pattern, but it doesn't prevent cascade failures—in fact, it can make them worse:

The circuit breaker pattern borrows its name from electrical engineering. In your home's electrical panel, circuit breakers protect your wiring from overloads. When current exceeds a safe threshold, the breaker "trips" and stops all current flow, preventing fires and equipment damage.

The electrical circuit breaker:

1. Normal operation: Current flows freely through the closed circuit
2. Overload detection: Current exceeds the rated capacity
3. Trip (Open): The breaker opens, stopping all current flow
4. Manual reset: An electrician investigates and resets the breaker
5. Current restored: Normal operation resumes

The software pattern works identically—but instead of electrical current, we're managing request flow, and instead of protecting wiring, we're protecting services and the resources they consume.

The software circuit breaker states:

Closed State (Normal Operation)

• Requests flow through to the downstream service
• Failures and successes are tracked
• The circuit remains closed until failure rate exceeds a threshold

Open State (Protecting the System)

• Requests fail immediately without calling the downstream service
• Fast failure means no resource consumption
• A timer counts down to the next recovery attempt

Half-Open State (Testing Recovery)

• A limited number of probe requests are allowed through
• If probes succeed, the circuit closes (service has recovered)
• If probes fail, the circuit opens again (service still unhealthy)

Let's revisit our earlier cascade failure scenario, but this time with circuit breakers in place.

Scenario: Payment Service Database Overload (With Circuit Breakers)

Minute 0-3: Payment service response times climb from 50ms to 15 seconds

• Order Service's circuit breaker to Payment Service begins tracking elevated latency
• Failure count increases as timeouts occur

Minute 3: Order Service circuit breaker trips (opens)

• Further requests to Payment Service fail immediately (10ms, not 15 seconds)
• Order Service threads are freed almost instantly
• Order Service can still handle orders requiring no payment (gift cards, account credits)

Minute 3-7: System stabilizes

• Order Service returns fast failures with appropriate error messages
• Product Catalog, Inventory, and other services remain fully functional
• Users see "Payment temporarily unavailable" instead of endless loading

Minute 7: Order Service circuit enters half-open state

• A single probe request is sent to Payment Service
• Probe fails → circuit reopens for another timeout period

Minute 15: Payment database recovers

• Half-open probe succeeds
• Circuit closes, normal operation resumes

Total impact: 15 minutes of degraded checkout experience Without circuit breakers: Complete platform outage lasting an hour or more

The Resource Protection Mechanism:

The fundamental way circuit breakers prevent cascades is through resource protection. When a circuit opens:

1. Threads are not blocked: Requests fail instantly, freeing threads to handle other work
2. Connections are not held: No new connections are opened to the failing service
3. Memory is not consumed: No request/response data accumulates waiting for timeouts
4. CPU is not wasted: No retry loops spinning against a dead service

This protection prevents the resource exhaustion that is the proximate cause of cascade failures. Even if a downstream service is completely dead, the calling service can continue operating normally for all other functionality.

At the heart of the circuit breaker pattern is the fail-fast principle: when failure is inevitable, fail immediately rather than after consuming resources and time.

Consider the mathematics of fail-fast:

The fail-fast economics:

Fail-fast isn't just about speed—it's about opportunity cost. Every thread waiting on a timeout is a thread that cannot serve other requests. Every connection held open to a failing service is a connection unavailable for healthy services.

In a system handling 10,000 requests/second:

• Slow failure (30s timeout): Each failing request consumes 30 seconds of thread time. Even a 1% failure rate to one service means 100 requests/second × 30 seconds = 3,000 thread-seconds consumed per second—likely more threads than you have.
• Fast failure (10ms): The same 100 failing requests × 0.01 seconds = 1 thread-second consumed per second—negligible impact.

Design implications of fail-fast:

Adopting fail-fast requires thoughtful system design:

1. Fallback strategies: What should happen when a circuit is open? Cached data? Default values? Honest error messages?

2. Partial response capability: Can your API return partial data when some downstream services are unavailable?

3. Client expectations: Are clients designed to handle fast failures gracefully, or will they retry aggressively?

4. Monitoring and alerting: Fast failures are invisible if not monitored. Operators need visibility into circuit state.

5. User experience: Error messages should be informative and actionable, not cryptic stack traces.

Understanding cascade failures in theory is valuable, but examining real-world incidents drives the lessons home. These case studies illustrate how major platforms have experienced cascade failures and what they learned.

Case Study 1: Amazon Web Services (2017)

On February 28, 2017, a significant portion of the internet went down when an Amazon S3 outage cascaded through services that depended on it.

• Root cause: An engineer running a maintenance script removed more S3 servers than intended
• Cascade mechanism: Services using S3 for storage, configuration, or state began failing
• Amplification: S3's status dashboard was itself hosted on S3, so AWS couldn't display status updates
• Duration: 4+ hours of significant disruption
• Lesson: Hidden dependencies (like dashboards on the same infrastructure) create unexpected failure paths

Case Study 2: Netflix (The Chaos Engineering Origin)

Netflix's early cloud migration exposed them to cascade failures that motivated their famous resilience engineering culture.

• Problem observed: A single service failure could propagate through 20+ dependent services
• Solution implemented: The Hystrix circuit breaker library (now in maintenance mode, superseded by Resilience4j)
• Key innovation: Embedding circuit breakers in every service-to-service call
• Result: Inspired the entire industry to adopt circuit breaker patterns

Case Study 3: Twitter (2016)

A configuration change in Twitter's image service cascaded into a multi-hour outage.

• Root cause: A change to image processing that caused increased latency
• Cascade mechanism: The tweet composition service sync-called image service; when it slowed, tweet creation slowed
• Amplification: Users retried tweet posts, creating more load; monitoring dashboards became slow to load
• Duration: 2+ hours of degraded service
• Lesson: Synchronous dependencies on non-critical services (images) can bring down critical flows (posting)

Circuit breakers are powerful but not universally applicable. Understanding when they add value—and when they might not—helps you deploy them effectively.

Strong candidates for circuit breakers:

Circuit breaker placement decisions:

Client-side vs. Server-side placement:

• Client-side (recommended for most cases): Each client has its own circuit breaker to the server. Failures are detected at the point of impact. Different clients can have different thresholds.

• Server-side (API gateway pattern): A gateway or load balancer implements circuit breakers for all clients. Simpler topology but less granular control.

Per-host vs. Per-service:

• Per-host: Separate circuits to each server instance. A single bad host doesn't open the circuit to all hosts. More accurate but more complex.

• Per-service: One circuit for all instances of a service. Simpler but a single bad host can open the circuit unnecessarily.

Most implementations use per-service circuits for simplicity, with health checks handling per-host removal.

We've established the foundation for understanding why circuit breakers exist and how they fundamentally address cascade failures in distributed systems.

What's next:

Now that we understand why circuit breakers are essential, the next page dives deep into how they work—examining the three circuit states (Closed, Open, Half-Open) and the transition logic between them. Understanding state transitions is crucial for configuring circuit breakers that balance protection with availability.