Failure Thresholds

A circuit breaker is only as good as its configuration. Set the failure threshold too high, and the circuit never trips—you'll experience full cascade failures before protection engages. Set it too low, and the circuit trips on transient noise—users experience unnecessary degradation.

Threshold configuration is not guesswork. It requires understanding your system's baseline behavior, the statistical properties of failure detection, and the trade-offs between sensitivity and stability. This page equips you with the analytical framework to configure circuit breaker thresholds correctly.

Incorrect threshold configuration is the most common reason circuit breakers fail to provide protection in production. Engineers often copy default values without understanding whether those defaults fit their specific use case. By the end of this page, you'll know how to reason about and tune each configuration parameter for your specific context.

The failure rate is the fundamental metric that circuit breakers use to assess dependency health. Before configuring thresholds, you must understand exactly how failure rate is calculated and what it represents.

The Basic Formula

Failure Rate = Failed Calls / Total Calls × 100%

Where:

• Failed Calls: Calls that resulted in a recorded failure (exceptions, timeouts, error responses)
• Total Calls: All calls within the measurement window (successes + failures)

Example Calculation

Over the last 100 calls:

• 80 succeeded
• 20 failed (timeouts or exceptions)

Failure Rate = 20 / 100 × 100% = 20%

The Measurement Window

Failure rate is not calculated over all time—it's calculated over a sliding window. This window can be:

1. Count-Based Window Track the last N calls. Oldest calls drop off as new ones arrive.

Example: Window size = 100 calls. Failure rate = failures in last 100 calls / 100.

Pros: Consistent sample size; predictable statistical properties. Cons: Time to fill window varies with traffic volume.

2. Time-Based Window Track calls within the last T seconds. All calls older than T are excluded.

Example: Window = 60 seconds. Failure rate = failures in last 60 seconds / total calls in last 60 seconds.

Pros: Consistent time horizon; failure rate ages out at predictable rate. Cons: Sample size varies with traffic; low traffic = high variance.

3. Hybrid: Time-Bucketed Windows Divide time into buckets (e.g., 10 × 6-second buckets for a 60-second window). Count successes/failures per bucket. Calculate aggregate rate.

Pros: Memory efficient; provides time decay. Cons: Bucket granularity affects precision.

Setting a failure rate threshold requires understanding basic statistics. What failure rate indicates a genuinely unhealthy service versus normal variance?

Baseline Failure Rate

Every service has a baseline failure rate under normal conditions. This is not zero—transient failures, client errors, and edge cases always produce some failures:

The Signal-to-Noise Challenge

If your baseline failure rate is 2% and you set your threshold at 5%, you need to reliably distinguish between:

• Normal variance around 2% (noise)
• Genuine degradation above 5% (signal)

With small sample sizes, normal variance can easily swing from 2% to 5% or higher by random chance.

Statistical Confidence

Consider the following scenario:

• True failure rate: 2% (20 failures per 1000 requests)
• Sample size: 50 requests
• Observed failures: 3 (6% in sample)

Is this 6% observed rate indicative of a problem, or just random variance in a small sample?

Confidence Interval Calculation

Using the binomial proportion confidence interval (Wilson score):

For a sample of 50 with 3 failures (6% observed):

• 95% confidence interval: approximately 1.9% to 15.8%

This means the true failure rate could plausibly be anywhere from 2% to 16%. A 50-request sample is insufficient to confidently detect a change from 2% to 6%.

The Minimum Sample Size

To reliably detect a doubling of failure rate (e.g., 2% → 4%) with 95% confidence and 80% power, statistical calculations suggest you need approximately:

• To detect 2% → 4%: ~1200 samples
• To detect 2% → 10%: ~150 samples
• To detect 2% → 20%: ~50 samples

This has critical implications: your window size determines what magnitude of change you can reliably detect.

Practical Guidance

For most services, the following guidance applies:

1. Measure your baseline failure rate during normal operation. Track it over days, not hours.

2. Set threshold significantly above baseline — at least 3-5x baseline rate, or a minimum of 30-50% failure rate for critical services. A 50% failure rate means half your requests are failing—that's unambiguously a problem.

3. Use window sizes of 50-100 minimum for count-based windows. Smaller windows have too much variance.

4. Accept that you're detecting major degradation, not subtle changes. Circuit breakers are for preventing cascades, not alerting on slight SLA degradation.

The minimum volume requirement is one of the most important—and most misunderstood—circuit breaker configuration parameters. Without it, circuits can trip inappropriately during low-traffic periods.

The Problem Without Minimum Volume

Consider a circuit with 50% failure threshold and no minimum volume requirement:

• During a quiet period, only 2 requests occur
• 1 request fails (perhaps a timeout due to network blip)
• Failure rate = 50%
• Circuit trips!

This is clearly wrong. One failure out of two requests is not statistically significant evidence of service degradation. Yet without minimum volume, the circuit breaker treats it as such.

The Solution: Minimum Calls Threshold

The minimum volume requirement specifies that failure rate evaluation should only occur when enough samples have been collected:

IF totalCallsInWindow >= minimumNumberOfCalls THEN
    evaluateFailureRate()
ELSE
    remainClosed()  // Not enough data to evaluate
END

Choosing the Minimum Volume

How high should the minimum volume be? Consider these factors:

1. Statistical Significance

As discussed earlier, you need enough samples for your failure rate calculation to be meaningful. Rule of thumb: minimum volume should be at least 20-50 requests.

2. Traffic Patterns

Minimum volume should be reachable during your lowest-traffic period:

3. Detection Latency

Higher minimum volume means longer time before the circuit can trip:

For low-traffic services, high minimum volumes can delay protection significantly.

The Relationship Between Window Size and Minimum Volume

These parameters interact:

• minimumNumberOfCalls ≤ slidingWindowSize (always)
• If minimum equals window size, you need a full window before evaluation
• If minimum is much smaller than window, you evaluate earlier but with less data

Recommended Configuration

For most use cases:

minimumNumberOfCalls = 0.2 × slidingWindowSize

With window size of 100:

• minimum = 20: Start evaluating once 20 calls recorded
• Provides some confidence while allowing quick detection

For critical services where false positives are costly:

minimumNumberOfCalls = 0.5 × slidingWindowSize

The sliding window size determines how much history the circuit breaker considers when evaluating health. It's perhaps the most impactful configuration parameter.

The Trade-off Matrix

The Detection Latency Issue

Window size directly impacts how quickly the circuit breaker detects degradation:

Scenario: Service starts failing 100% of requests

The Stale Failure Problem

Larger windows have a counterintuitive issue: old failures can keep the circuit tripped even after the dependency recovers.

Scenario:

1. Window size: 200 calls
2. Minutes 0-1: Service fails, 100 calls fail (100/100 = 100% failure rate)
3. Minute 1-2: Service recovers, 100 calls succeed
4. Current window: 100 failures + 100 successes = 50% failure rate
5. Circuit remains open due to stale failures!

The window must completely cycle before old failures age out.

Time-Based Window Considerations

For time-based windows, size is specified in seconds rather than call count:

Time-based windows have variable effective sample size based on traffic volume. During traffic spikes, you have more data. During lulls, you have less.

Practical Recommendations

Modern circuit breaker implementations like Resilience4j can trip based on slow calls, not just failures. This catches a critical failure mode: latency degradation that hasn't yet manifested as errors.

Why Slow Calls Matter

A service returning slowly is often worse than one returning errors:

A "successful" response that takes 10 seconds instead of 100ms still blocks the calling thread for 10 seconds. The cascade failure mechanics from our earlier discussion apply equally to slow successful calls.

Configuring Slow Call Detection

Determining the Slow Call Duration Threshold

The "slow" threshold should be significantly above your normal response time but below your timeout:

P50 Latency << Slow Threshold << Timeout

Example:

• P50 latency: 100ms
• P99 latency: 500ms
• Slow threshold: 2 seconds (4x P99)
• Timeout: 10 seconds

Guidance for setting slow threshold:

The threshold should catch genuine degradation while ignoring occasional slow calls that are within acceptable variance.

The Interaction with Timeouts

Slow call detection and timeouts work together:

1. Request starts — Timer begins
2. Slow threshold exceeded — Call counted as "slow" but continues
3. Call succeeds (slowly) — Slow call counter incremented, but it's a success
4. OR Timeout exceeded — Call fails, counted as both slow AND failed

Timeouts bound the maximum wait time. Slow call detection triggers protection earlier. Together, they provide comprehensive latency-based protection.

Armed with theory, let's develop a practical approach to threshold tuning for real services.

Step 1: Establish Baselines

Before configuring thresholds, collect baseline metrics:

SELECT 
    percentile_cont(0.50) WITHIN GROUP (ORDER BY duration_ms) as p50,
    percentile_cont(0.90) WITHIN GROUP (ORDER BY duration_ms) as p90,
    percentile_cont(0.99) WITHIN GROUP (ORDER BY duration_ms) as p99,
    COUNT(*) FILTER (WHERE status = 'error') * 100.0 / COUNT(*) as error_rate,
    COUNT(*) as total_calls
FROM service_call_logs
WHERE 
    service = 'inventory-service'
    AND timestamp > NOW() - INTERVAL '7 days'

Collect this data over a representative period (at least a week) covering:

• Peak traffic periods
• Off-peak periods
• Any regular batch jobs or maintenance windows

Step 2: Initial Configuration

Start with conservative settings:

Step 3: Deploy with Monitoring

Deploy the initial configuration with comprehensive monitoring:

1. Track circuit state (closed/open/half-open) over time
2. Log all state transitions with context
3. Alert on state changes (warning, not critical initially)
4. Correlate circuit events with dependency metrics

Step 4: Observe and Adjust

Analyze circuit behavior over weeks of operation:

If the circuit never trips:

• This might be correct (dependency is stable)
• Or thresholds are too conservative
• If known outages occurred without tripping, lower thresholds

If the circuit trips too often:

• Analyze each trip: was it justified?
• If trips occurred during normal operation, raise thresholds
• If trips preceded cascade failures, current thresholds are correct

If trips are appropriate but recovery is slow/fast:

• Adjust recovery timeout based on dependency recovery patterns
• If dependency recovers in 10s but timeout is 60s, reduce it
• If dependency needs 2 minutes but timeout is 30s, increase it

A common design question: should you have one circuit breaker per dependency, or multiple circuits for different operations on the same dependency?

The Global Circuit Approach

One circuit breaker for all calls to a dependency:

PaymentService → [Circuit Breaker] → Payment Gateway
  - authorize()
  - capture()
  - refund()
  - getTransaction()

Pros:

• Simple to configure and monitor
• Comprehensive protection for the dependency
• Lower operational overhead

Cons:

• A problem with one operation affects all operations
• Can't distinguish between operation-specific issues

The Per-Operation Approach

Separate circuit breakers for different operations:

PaymentService → [CB: authorize]  → Payment Gateway /authorize
              → [CB: capture]   → Payment Gateway /capture
              → [CB: refund]    → Payment Gateway /refund
              → [CB: query]     → Payment Gateway /transactions

Pros:

• Isolation: authorize failures don't affect refund
• Operation-specific thresholds based on operation characteristics
• More granular fallback behavior

Cons:

• More configuration and monitoring overhead
• May miss cross-operation correlation (e.g., database issues affecting all operations)

Hybrid Approach: Operation Groups

A middle ground is to group operations by characteristics:

PaymentService → [CB: critical-ops]   → authorize, capture
              → [CB: read-ops]       → getTransaction, listTransactions  
              → [CB: background-ops] → refund, void

This reduces configuration overhead while still providing isolation between fundamentally different operation types.

Guidance for Choosing