Combining Bulkhead with… — HLD: Bulkhead Pattern:… | OneNoughtOne

0/107

System Design (HLD)Bulkhead Pattern

Bulkhead Pattern: Failure Isolation in Distributed Systems

LevelAdvanced

Duration75 mins

TopicBulkhead Pattern

5 / 5

Combining Bulkhead with Circuit Breaker: Defense in Depth

Two Patterns, One Defense

Bulkheads and circuit breakers are often discussed separately, but in production systems, they work together as complementary layers of defense. Understanding how to combine them effectively is the difference between partial protection and comprehensive resilience. Consider this scenario: Your payment service is protected by a bulkhead with 50 threads. The payment provider becomes slow, responding in 30 seconds instead of 200ms. Without a circuit breaker, those 50 threads fill up, start rejecting requests, but keep trying to call the broken provider —burning resources on calls that will likely fail. With both patterns: the circuit breaker detects the failure pattern, opens, and stops making calls entirely . The bulkhead threads are freed immediately. Recovery is faster. Resources are preserved.

What You Will Learn

By the end of this page, you'll understand how bulkheads and circuit breakers complement each other, the correct order for applying these patterns, configuration strategies for combined use, and how to build comprehensive resilience with timeout, retry, bulkhead, and circuit breaker patterns working together.

Complementary Purposes: What Each Pattern Solves

While both patterns contribute to resilience, they address different failure modes: Bulkhead: Resource Isolation Prevents one failing dependency from consuming all resources Limits blast radius of failures Protects healthy paths from failing paths Works continuously, regardless of failure patterns Key metric: concurrent calls limited Circuit Breaker: Failure Response Detects repeated failures and stops making calls Gives failing systems time to recover Fails fast when system is known to be broken Works periodically, based on failure thresholds Key metric: failure rate triggers state change Critical Insight: Bulkheads protect the caller from resource exhaustion. Circuit breakers protect the callee from being overwhelmed and give it recovery time. Together, they create bidirectional protection.

Bulkhead vs Circuit Breaker Comparison
Aspect	Bulkhead	Circuit Breaker
Primary goal	Limit concurrent calls	Stop calls after failures
Failure mode addressed	Slow calls consuming resources	Repeated failures wasting effort
Action when triggered	Reject excess requests	Fail immediately, skip call
State	Stateless (counter)	Stateful (closed/open/half-open)
Recovery	Automatic when calls complete	Waits, then tests with probe requests
What it protects	Caller resources	Callee from overload
When it helps	Always (resource limit)	During failure patterns

pattern_coverage.txt

FAILURE MODE COVERAGE
 
Failure Mode                    Bulkhead    Circuit Breaker    Combined
──────────────────────────────────────────────────────────────────────
Slow dependency (high latency)     ✓             ○                ✓
  └─ Bulkhead limits threads used
  
Intermittent failures              ○             ✓                ✓
  └─ CB detects failure rate, stops calls
  
Complete outage                    ○             ✓                ✓
  └─ CB fails fast, preserves resources
  
Burst traffic overload             ✓             ○                ✓
  └─ Bulkhead rejects excess
  
Gradual degradation                ○             ✓                ✓
  └─ CB detects increasing errors
  
Resource exhaustion cascade        ✓             ○                ✓
  └─ Bulkhead isolates paths
  
Recovery flooding                  ○             ✓                ✓
  └─ CB half-open limits recovery probes
 
Legend: ✓ = Primary protection, ○ = Partial/no protection

Order of Application: Which Pattern First?

When combining patterns, the order matters significantly. The general recommendation is: Circuit Breaker \to Bulkhead \to Timeout \to Actual Call Or reading right-to-left (call stack order): Actual Call \leftarrow Timeout \leftarrow Bulkhead \leftarrow Circuit Breaker Why Circuit Breaker First (Outermost)? If circuit is open, fail immediately without touching the bulkhead Preserves bulkhead capacity for when circuit is closed Circuit breaker's fast failure is faster than bulkhead rejection Why Bulkhead Second? Limits concurrency after circuit breaker allows the call Provides secondary protection if circuit breaker hasn't tripped yet Ensures slow calls (that haven't failed enough to trip CB) don't exhaust resources Why Timeout Innermost? Timeout applies to the actual call duration Prevents individual calls from blocking bulkhead threads indefinitely Timeout failures feed into circuit breaker failure counting

pattern_ordering.txt

EXECUTION FLOW (Circuit Breaker → Bulkhead → Timeout → Call)
 
Request arrives
     │
     ▼
┌─────────────────────────────────────────────────────────────┐
│ CIRCUIT BREAKER                                             │
│  ├─ Is circuit OPEN?                                        │
│  │   └─ YES → Fail immediately (no bulkhead touched)        │
│  │   └─ NO  → Proceed to bulkhead                          │
│  └─ Records result for failure rate calculation             │
│                         │                                   │
│                         ▼                                   │
│  ┌───────────────────────────────────────────────────────┐ │
│  │ BULKHEAD                                               │ │
│  │  ├─ Is capacity available?                             │ │
│  │  │   └─ NO  → Reject (BulkheadFullException)          │ │
│  │  │   └─ YES → Acquire permit, proceed                  │ │
│  │  │                   │                                 │ │
│  │  │                   ▼                                 │ │
│  │  │  ┌─────────────────────────────────────────────┐   │ │
│  │  │  │ TIMEOUT                                     │   │ │
│  │  │  │  ├─ Start timer                             │   │ │
│  │  │  │  │         │                                │   │ │
│  │  │  │  │         ▼                                │   │ │
│  │  │  │  │  ┌────────────────────────────────┐     │   │ │
│  │  │  │  │  │ ACTUAL CALL                    │     │   │ │
│  │  │  │  │  │  └─ Execute HTTP/gRPC call     │     │   │ │
│  │  │  │  │  └────────────────────────────────┘     │   │ │
│  │  │  │  │  Check timer                             │   │ │
│  │  │  │  │   └─ Exceeded? → TimeoutException        │   │ │
│  │  │  │  └─ Return result or exception              │   │ │
│  │  │  └─────────────────────────────────────────────┘   │ │
│  │  │                   │                                 │ │
│  │  └─ Release permit   │                                 │ │
│  └───────────────────────────────────────────────────────┘ │
│                         │                                   │
│  Record outcome (success/failure) for CB calculation       │
└─────────────────────────────────────────────────────────────┘
     │
     ▼
Response returned

Incorrect Ordering Consequences

If bulkhead is outermost: Requests consume bulkhead capacity just to be rejected by an open circuit breaker—wasting isolation resources. If timeout is outermost: Timeout starts before bulkhead acquisition, so time spent waiting for a permit counts against the timeout—making timeouts behave unpredictably.

Implementation with Resilience4j

Resilience4j provides elegant composition of multiple resilience patterns through its Decorators API.

CombinedResilience.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
@Configuration
public class PaymentResilienceConfig {
    
    @Bean
    public CircuitBreaker paymentCircuitBreaker() {
        return CircuitBreaker.of("payment", CircuitBreakerConfig.custom()
            .failureRateThreshold(50)              // Open at 50% failure
            .slowCallRateThreshold(80)             // Also open if 80% slow
            .slowCallDurationThreshold(Duration.ofSeconds(2))
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(5)
            .slidingWindowSize(10)
            .build());
    }
    
    @Bean
    public Bulkhead paymentBulkhead() {
        return Bulkhead.of("payment", BulkheadConfig.custom()
            .maxConcurrentCalls(50)
            .maxWaitDuration(Duration.ZERO)  // Fail immediately
            .build());
    }
    
    @Bean
    public TimeLimiter paymentTimeLimiter() {
        return TimeLimiter.of(TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(3))
            .cancelRunningFuture(true)
            .build());
    }
}
 
@Service
public class PaymentService {
    
    private final CircuitBreaker circuitBreaker;
    private final Bulkhead bulkhead;
    private final TimeLimiter timeLimiter;
    private final PaymentGateway gateway;
    
    public PaymentResult processPayment(PaymentRequest request) {
        // Compose patterns in correct order: CB → Bulkhead → Timeout → Call
        Supplier<CompletableFuture<PaymentResult>> decoratedSupplier =
            Decorators.ofSupplier(() -> 
                    CompletableFuture.supplyAsync(() -> gateway.charge(request)))
                .withTimeLimiter(timeLimiter, scheduledExecutor)  // Innermost
                .withBulkhead(bulkhead)                           // Middle
                .withCircuitBreaker(circuitBreaker)               // Outermost
                .decorate();
        
        return Try.ofSupplier(decoratedSupplier)
            .map(CompletableFuture::join)
            .recover(CallNotPermittedException.class, 
                ex -> handleCircuitOpen(request))
            .recover(BulkheadFullException.class, 
                ex -> handleBulkheadFull(request))
            .recover(TimeoutException.class, 
                ex -> handleTimeout(request))
            .get();
    }
    
    private PaymentResult handleCircuitOpen(PaymentRequest request) {
        log.warn("Circuit breaker open, queueing payment: {}", request.getId());
        asyncPaymentQueue.enqueue(request);
        return PaymentResult.queued("Payment system recovering, queued for retry");
    }
    
    private PaymentResult handleBulkheadFull(PaymentRequest request) {
        log.warn("Bulkhead full, high load: {}", request.getId());
        return PaymentResult.retry("High demand, please retry in a moment");
    }
    
    private PaymentResult handleTimeout(PaymentRequest request) {
        log.error("Payment timeout: {}", request.getId());
        return PaymentResult.pending("Payment processing, check status later");
    }
}

application.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
resilience4j:
  circuitbreaker:
    instances:
      payment:
        registerHealthIndicator: true
        slidingWindowSize: 10
        failureRateThreshold: 50
        slowCallRateThreshold: 80
        slowCallDurationThreshold: 2s
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        recordExceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - com.example.BusinessValidationException
          
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 50
        maxWaitDuration: 0ms
        
  timelimiter:
    instances:
      payment:
        timeoutDuration: 3s
        cancelRunningFuture: true

Tuning Combined Configurations

When using bulkhead and circuit breaker together, their configurations must be coordinated. Misaligned settings can cause unexpected behavior. Key Coordination Points: 1. Sliding Window Size vs Bulkhead Size Circuit breaker's sliding window should be smaller than bulkhead concurrent capacity. Otherwise, you might trip the circuit before naturally exhausting the bulkhead. If bulkhead = 50 concurrent calls And sliding window = 100 Circuit might open before bulkhead is stressed 2. Slow Call Threshold vs Timeout Circuit breaker's slow call threshold should be less than or equal to the timeout. A call that times out is definitely slow. If timeout = 3 seconds Then slowCallDurationThreshold \leq 3 seconds 3. Wait Duration vs Recovery Time Circuit breaker's wait duration in open state should align with expected dependency recovery time. Too short and you probe too frequently; too long and you delay recovery unnecessarily.

configuration_alignment.txt

CONFIGURATION ALIGNMENT CHECKLIST
 
Payment Dependency Configuration
───────────────────────────────────────────────────────────────
 
Timeout Configuration:
  └── timeout = 3s
       │
       ├─► Aligns with: CB slowCallDurationThreshold ≤ 3s ✓
       ├─► Aligns with: Bulkhead sizing (threads × timeout = max blocking)
       └─► Result: Calls exceeding 3s are terminated and counted as failures
 
Bulkhead Configuration:
  └── maxConcurrentCalls = 50
       │
       ├─► Aligns with: Expected peak load × p95 latency × safety
       │    Peak=100 RPS, p95=0.4s, safety=1.25 → 50 ✓
       ├─► Aligns with: CB slidingWindowSize < 50
       │    slidingWindowSize=10 ✓
       └─► Result: Limits concurrent calls, feeds failures to CB
 
Circuit Breaker Configuration:
  └── slidingWindowSize = 10
      failureRateThreshold = 50%  (5 of 10 failures → open)
      slowCallRateThreshold = 80% (8 of 10 slow → open)
      waitDurationInOpenState = 30s
       │
       ├─► Aligns with: 5 failures reasonable given bulkhead size
       ├─► Aligns with: 30s matches dependency recovery time
       └─► Result: Opens quickly on problems, waits appropriate time
 
COORDINATION FORMULA:
  slidingWindowSize < maxConcurrentCalls
  slowCallDurationThreshold ≤ timeoutDuration
  waitDurationInOpenState ≈ expectedRecoveryTime
  permittedCallsInHalfOpen < maxConcurrentCalls / 5

Slow Call Detection

Circuit breakers that only track failures miss slow degradation. A dependency responding in 5 seconds 100% of the time isn't 'failing'—but it's destroying your latency SLA. Configure slowCallRateThreshold and slowCallDurationThreshold to detect and respond to latency degradation, not just errors.

The Complete Resilience Stack

For comprehensive resilience, additional patterns often complement bulkhead and circuit breaker: Retry : Handles transient failures by automatically retrying Timeout : Ensures calls don't block indefinitely Rate Limiter : Protects dependencies from being overwhelmed Cache/Fallback : Provides degraded service when dependencies fail Complete Pattern Order: Cache \to Rate Limiter \to Circuit Breaker \to Retry \to Bulkhead \to Timeout \to Call Reading right-to-left (execution order): Cache : Check if we already have the result Rate Limiter : Ensure we're not overwhelming the dependency Circuit Breaker : Check if dependency is healthy Retry : Wrapper that may re-execute on failure Bulkhead : Limit concurrent executions Timeout : Limit duration of each attempt Call : The actual remote call

CompleteResilienceStack.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
@Service
public class ResilientPaymentService {
    
    // Cache for fallback
    private final Cache<String, PaymentResult> resultCache;
    
    // Resilience components
    private final RateLimiter rateLimiter;
    private final CircuitBreaker circuitBreaker;
    private final Retry retry;
    private final Bulkhead bulkhead;
    private final TimeLimiter timeLimiter;
    
    public PaymentResult processWithFullResilience(PaymentRequest request) {
        // Check cache first (fastest path)
        PaymentResult cached = resultCache.getIfPresent(request.getIdempotencyKey());
        if (cached != null) {
            return cached;
        }
        
        // Build decorated call with all patterns
        Supplier<CompletableFuture<PaymentResult>> resilientCall =
            Decorators.ofSupplier(() -> 
                    CompletableFuture.supplyAsync(() -> gateway.charge(request)))
                // Innermost: Timeout on actual call
                .withTimeLimiter(timeLimiter, scheduler)
                // Limit concurrent calls
                .withBulkhead(bulkhead)
                // Retry transient failures (inside CB)
                .withRetry(retry)
                // Stop calling if failing
                .withCircuitBreaker(circuitBreaker)
                // Don't overwhelm dependency
                .withRateLimiter(rateLimiter)
                .decorate();
        
        try {
            PaymentResult result = resilientCall.get().join();
            // Cache successful results
            resultCache.put(request.getIdempotencyKey(), result);
            return result;
        } catch (Exception e) {
            return handleFailure(request, e);
        }
    }
    
    private PaymentResult handleFailure(PaymentRequest request, Exception e) {
        Throwable cause = e.getCause() != null ? e.getCause() : e;
        
        if (cause instanceof RequestNotPermitted) {
            // Rate limiter kicked in
            return PaymentResult.rateLimited("Too many requests");
        }
        if (cause instanceof CallNotPermittedException) {
            // Circuit breaker open - use fallback
            return getFallbackResult(request);
        }
        if (cause instanceof BulkheadFullException) {
            // System under heavy load
            return PaymentResult.retry("System busy, please retry");
        }
        if (cause instanceof TimeoutException) {
            // Call was too slow (after retries)
            return PaymentResult.pending("Processing - check status");
        }
        
        // Unknown failure
        log.error("Payment failed unexpectedly", e);
        return PaymentResult.failed("Payment unavailable");
    }
    
    private PaymentResult getFallbackResult(PaymentRequest request) {
        // Try cached result for this customer
        PaymentResult recent = recentPayments.getForCustomer(request.getCustomerId());
        if (recent != null && recent.isReusable()) {
            return recent.asRetry();
        }
        
        // Queue for async processing
        asyncQueue.enqueue(request);
        return PaymentResult.queued("Processing offline");
    }
}

Monitoring Combined Patterns

With multiple patterns, monitoring becomes more complex but also more informative. Each pattern contributes distinct signals. Key Metrics by Pattern: Circuit Breaker : State (closed/open/half-open), failure rate, call counts Bulkhead : Utilization, rejection rate, available capacity Retry : Retry attempts, eventually-succeeded rate Timeout : Timeout rate, actual latency distribution Combined Dashboard Essential Panels:

grafana_dashboard.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# Key metrics for combined resilience monitoring
 
# 1. Overall Health Summary
- title: "Dependency Health"
  metrics:
    - expr: 'resilience4j_circuitbreaker_state{name="payment"}'
      legend: "Circuit State (0=closed, 1=open, 2=half-open)"
    - expr: 'resilience4j_bulkhead_available_concurrent_calls{name="payment"}'
      legend: "Available Bulkhead Capacity"
 
# 2. Request Outcomes
- title: "Request Outcome Breakdown"
  type: stacked_percentage
  metrics:
    - expr: 'rate(requests_success_total{service="payment"}[5m])'
      legend: "Success"
    - expr: 'rate(circuitbreaker_calls_rejected_total{name="payment"}[5m])'
      legend: "Circuit Open Rejection"
    - expr: 'rate(bulkhead_calls_rejected_total{name="payment"}[5m])'
      legend: "Bulkhead Full Rejection"
    - expr: 'rate(timeout_count_total{name="payment"}[5m])'
      legend: "Timeout"
 
# 3. Circuit Breaker Detail
- title: "Circuit Breaker State Transitions"
  metrics:
    - expr: 'changes(resilience4j_circuitbreaker_state{name="payment"}[1h])'
      legend: "State Changes (last hour)"
    - expr: 'resilience4j_circuitbreaker_failure_rate{name="payment"}'
      legend: "Current Failure Rate %"
 
# 4. Bulkhead Utilization
- title: "Bulkhead Capacity"
  metrics:
    - expr: |
        1 - (resilience4j_bulkhead_available_concurrent_calls{name="payment"} /
             resilience4j_bulkhead_max_allowed_concurrent_calls{name="payment"})
      legend: "Utilization %"
 
# 5. Latency by Outcome
- title: "Latency Distribution"
  metrics:
    - expr: 'histogram_quantile(0.50, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p50"
    - expr: 'histogram_quantile(0.95, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p95"
    - expr: 'histogram_quantile(0.99, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p99"

Critical Alerts for Combined Patterns

•Circuit Open Alert — CB transitioned to open; dependency is failing
•Bulkhead >80% Utilization — Approaching capacity; consider scaling or investigating latency
•Bulkhead Rejections — Active rejections; load exceeds capacity
•High Retry Rate — Many transient failures; dependency may be unstable
•Timeout Rate Spike — Calls taking too long; investigate dependency latency
•Circuit Not Recovering — CB stuck open longer than expected; manual investigation needed

Summary: Defense in Depth with Combined Patterns

Key Takeaways

•Bulkhead and circuit breaker solve different problems — Bulkhead limits concurrency; CB stops calls after failures
•Order matters: CB → Bulkhead → Timeout → Call — Check circuit first to preserve bulkhead capacity
•Coordinate configurations — Sliding window < bulkhead size; slow threshold ≤ timeout
•Complete stack adds retry, rate limiter, cache — Each pattern handles specific failure modes
•Monitor each pattern's metrics — Combined dashboard shows full resilience picture
•Design fallbacks for each rejection type — Different responses for CB open vs bulkhead full vs timeout

Module Complete: You've now mastered the bulkhead pattern—from the foundational philosophy of failure isolation, through thread pool and semaphore implementations, sizing strategies, and integration with circuit breakers. This knowledge equips you to design systems that gracefully handle partial failures, protecting healthy functionality when individual dependencies degrade. The bulkhead pattern exemplifies a crucial principle in distributed systems: design for failure containment, not just failure prevention . Failures will happen; what matters is limiting their blast radius.

Module Complete: Bulkhead Pattern

Congratulations! You've completed the Bulkhead Pattern module. You understand failure isolation philosophy, thread pool and semaphore isolation, sizing strategies, and combined pattern integration. Apply these principles to build systems where failures stay contained and healthy functionality remains available.

System Design (HLD)Bulkhead Pattern

Bulkhead Pattern: Failure Isolation in Distributed Systems

LevelAdvanced

Duration75 mins

TopicBulkhead Pattern

5 / 5

Combining Bulkhead with Circuit Breaker: Defense in Depth

Two Patterns, One Defense

Bulkheads and circuit breakers are often discussed separately, but in production systems, they work together as complementary layers of defense. Understanding how to combine them effectively is the difference between partial protection and comprehensive resilience. Consider this scenario: Your payment service is protected by a bulkhead with 50 threads. The payment provider becomes slow, responding in 30 seconds instead of 200ms. Without a circuit breaker, those 50 threads fill up, start rejecting requests, but keep trying to call the broken provider —burning resources on calls that will likely fail. With both patterns: the circuit breaker detects the failure pattern, opens, and stops making calls entirely . The bulkhead threads are freed immediately. Recovery is faster. Resources are preserved.

What You Will Learn

By the end of this page, you'll understand how bulkheads and circuit breakers complement each other, the correct order for applying these patterns, configuration strategies for combined use, and how to build comprehensive resilience with timeout, retry, bulkhead, and circuit breaker patterns working together.

Complementary Purposes: What Each Pattern Solves

While both patterns contribute to resilience, they address different failure modes: Bulkhead: Resource Isolation Prevents one failing dependency from consuming all resources Limits blast radius of failures Protects healthy paths from failing paths Works continuously, regardless of failure patterns Key metric: concurrent calls limited Circuit Breaker: Failure Response Detects repeated failures and stops making calls Gives failing systems time to recover Fails fast when system is known to be broken Works periodically, based on failure thresholds Key metric: failure rate triggers state change Critical Insight: Bulkheads protect the caller from resource exhaustion. Circuit breakers protect the callee from being overwhelmed and give it recovery time. Together, they create bidirectional protection.

Bulkhead vs Circuit Breaker Comparison
Aspect	Bulkhead	Circuit Breaker
Primary goal	Limit concurrent calls	Stop calls after failures
Failure mode addressed	Slow calls consuming resources	Repeated failures wasting effort
Action when triggered	Reject excess requests	Fail immediately, skip call
State	Stateless (counter)	Stateful (closed/open/half-open)
Recovery	Automatic when calls complete	Waits, then tests with probe requests
What it protects	Caller resources	Callee from overload
When it helps	Always (resource limit)	During failure patterns

pattern_coverage.txt

FAILURE MODE COVERAGE
 
Failure Mode                    Bulkhead    Circuit Breaker    Combined
──────────────────────────────────────────────────────────────────────
Slow dependency (high latency)     ✓             ○                ✓
  └─ Bulkhead limits threads used
  
Intermittent failures              ○             ✓                ✓
  └─ CB detects failure rate, stops calls
  
Complete outage                    ○             ✓                ✓
  └─ CB fails fast, preserves resources
  
Burst traffic overload             ✓             ○                ✓
  └─ Bulkhead rejects excess
  
Gradual degradation                ○             ✓                ✓
  └─ CB detects increasing errors
  
Resource exhaustion cascade        ✓             ○                ✓
  └─ Bulkhead isolates paths
  
Recovery flooding                  ○             ✓                ✓
  └─ CB half-open limits recovery probes
 
Legend: ✓ = Primary protection, ○ = Partial/no protection

Order of Application: Which Pattern First?

When combining patterns, the order matters significantly. The general recommendation is: Circuit Breaker \to Bulkhead \to Timeout \to Actual Call Or reading right-to-left (call stack order): Actual Call \leftarrow Timeout \leftarrow Bulkhead \leftarrow Circuit Breaker Why Circuit Breaker First (Outermost)? If circuit is open, fail immediately without touching the bulkhead Preserves bulkhead capacity for when circuit is closed Circuit breaker's fast failure is faster than bulkhead rejection Why Bulkhead Second? Limits concurrency after circuit breaker allows the call Provides secondary protection if circuit breaker hasn't tripped yet Ensures slow calls (that haven't failed enough to trip CB) don't exhaust resources Why Timeout Innermost? Timeout applies to the actual call duration Prevents individual calls from blocking bulkhead threads indefinitely Timeout failures feed into circuit breaker failure counting

pattern_ordering.txt

EXECUTION FLOW (Circuit Breaker → Bulkhead → Timeout → Call)
 
Request arrives
     │
     ▼
┌─────────────────────────────────────────────────────────────┐
│ CIRCUIT BREAKER                                             │
│  ├─ Is circuit OPEN?                                        │
│  │   └─ YES → Fail immediately (no bulkhead touched)        │
│  │   └─ NO  → Proceed to bulkhead                          │
│  └─ Records result for failure rate calculation             │
│                         │                                   │
│                         ▼                                   │
│  ┌───────────────────────────────────────────────────────┐ │
│  │ BULKHEAD                                               │ │
│  │  ├─ Is capacity available?                             │ │
│  │  │   └─ NO  → Reject (BulkheadFullException)          │ │
│  │  │   └─ YES → Acquire permit, proceed                  │ │
│  │  │                   │                                 │ │
│  │  │                   ▼                                 │ │
│  │  │  ┌─────────────────────────────────────────────┐   │ │
│  │  │  │ TIMEOUT                                     │   │ │
│  │  │  │  ├─ Start timer                             │   │ │
│  │  │  │  │         │                                │   │ │
│  │  │  │  │         ▼                                │   │ │
│  │  │  │  │  ┌────────────────────────────────┐     │   │ │
│  │  │  │  │  │ ACTUAL CALL                    │     │   │ │
│  │  │  │  │  │  └─ Execute HTTP/gRPC call     │     │   │ │
│  │  │  │  │  └────────────────────────────────┘     │   │ │
│  │  │  │  │  Check timer                             │   │ │
│  │  │  │  │   └─ Exceeded? → TimeoutException        │   │ │
│  │  │  │  └─ Return result or exception              │   │ │
│  │  │  └─────────────────────────────────────────────┘   │ │
│  │  │                   │                                 │ │
│  │  └─ Release permit   │                                 │ │
│  └───────────────────────────────────────────────────────┘ │
│                         │                                   │
│  Record outcome (success/failure) for CB calculation       │
└─────────────────────────────────────────────────────────────┘
     │
     ▼
Response returned

Incorrect Ordering Consequences

If bulkhead is outermost: Requests consume bulkhead capacity just to be rejected by an open circuit breaker—wasting isolation resources. If timeout is outermost: Timeout starts before bulkhead acquisition, so time spent waiting for a permit counts against the timeout—making timeouts behave unpredictably.

Implementation with Resilience4j

Resilience4j provides elegant composition of multiple resilience patterns through its Decorators API.

CombinedResilience.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
@Configuration
public class PaymentResilienceConfig {
    
    @Bean
    public CircuitBreaker paymentCircuitBreaker() {
        return CircuitBreaker.of("payment", CircuitBreakerConfig.custom()
            .failureRateThreshold(50)              // Open at 50% failure
            .slowCallRateThreshold(80)             // Also open if 80% slow
            .slowCallDurationThreshold(Duration.ofSeconds(2))
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(5)
            .slidingWindowSize(10)
            .build());
    }
    
    @Bean
    public Bulkhead paymentBulkhead() {
        return Bulkhead.of("payment", BulkheadConfig.custom()
            .maxConcurrentCalls(50)
            .maxWaitDuration(Duration.ZERO)  // Fail immediately
            .build());
    }
    
    @Bean
    public TimeLimiter paymentTimeLimiter() {
        return TimeLimiter.of(TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(3))
            .cancelRunningFuture(true)
            .build());
    }
}
 
@Service
public class PaymentService {
    
    private final CircuitBreaker circuitBreaker;
    private final Bulkhead bulkhead;
    private final TimeLimiter timeLimiter;
    private final PaymentGateway gateway;
    
    public PaymentResult processPayment(PaymentRequest request) {
        // Compose patterns in correct order: CB → Bulkhead → Timeout → Call
        Supplier<CompletableFuture<PaymentResult>> decoratedSupplier =
            Decorators.ofSupplier(() -> 
                    CompletableFuture.supplyAsync(() -> gateway.charge(request)))
                .withTimeLimiter(timeLimiter, scheduledExecutor)  // Innermost
                .withBulkhead(bulkhead)                           // Middle
                .withCircuitBreaker(circuitBreaker)               // Outermost
                .decorate();
        
        return Try.ofSupplier(decoratedSupplier)
            .map(CompletableFuture::join)
            .recover(CallNotPermittedException.class, 
                ex -> handleCircuitOpen(request))
            .recover(BulkheadFullException.class, 
                ex -> handleBulkheadFull(request))
            .recover(TimeoutException.class, 
                ex -> handleTimeout(request))
            .get();
    }
    
    private PaymentResult handleCircuitOpen(PaymentRequest request) {
        log.warn("Circuit breaker open, queueing payment: {}", request.getId());
        asyncPaymentQueue.enqueue(request);
        return PaymentResult.queued("Payment system recovering, queued for retry");
    }
    
    private PaymentResult handleBulkheadFull(PaymentRequest request) {
        log.warn("Bulkhead full, high load: {}", request.getId());
        return PaymentResult.retry("High demand, please retry in a moment");
    }
    
    private PaymentResult handleTimeout(PaymentRequest request) {
        log.error("Payment timeout: {}", request.getId());
        return PaymentResult.pending("Payment processing, check status later");
    }
}

application.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
resilience4j:
  circuitbreaker:
    instances:
      payment:
        registerHealthIndicator: true
        slidingWindowSize: 10
        failureRateThreshold: 50
        slowCallRateThreshold: 80
        slowCallDurationThreshold: 2s
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        recordExceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - com.example.BusinessValidationException
          
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 50
        maxWaitDuration: 0ms
        
  timelimiter:
    instances:
      payment:
        timeoutDuration: 3s
        cancelRunningFuture: true

Tuning Combined Configurations

When using bulkhead and circuit breaker together, their configurations must be coordinated. Misaligned settings can cause unexpected behavior. Key Coordination Points: 1. Sliding Window Size vs Bulkhead Size Circuit breaker's sliding window should be smaller than bulkhead concurrent capacity. Otherwise, you might trip the circuit before naturally exhausting the bulkhead. If bulkhead = 50 concurrent calls And sliding window = 100 Circuit might open before bulkhead is stressed 2. Slow Call Threshold vs Timeout Circuit breaker's slow call threshold should be less than or equal to the timeout. A call that times out is definitely slow. If timeout = 3 seconds Then slowCallDurationThreshold \leq 3 seconds 3. Wait Duration vs Recovery Time Circuit breaker's wait duration in open state should align with expected dependency recovery time. Too short and you probe too frequently; too long and you delay recovery unnecessarily.

configuration_alignment.txt

CONFIGURATION ALIGNMENT CHECKLIST
 
Payment Dependency Configuration
───────────────────────────────────────────────────────────────
 
Timeout Configuration:
  └── timeout = 3s
       │
       ├─► Aligns with: CB slowCallDurationThreshold ≤ 3s ✓
       ├─► Aligns with: Bulkhead sizing (threads × timeout = max blocking)
       └─► Result: Calls exceeding 3s are terminated and counted as failures
 
Bulkhead Configuration:
  └── maxConcurrentCalls = 50
       │
       ├─► Aligns with: Expected peak load × p95 latency × safety
       │    Peak=100 RPS, p95=0.4s, safety=1.25 → 50 ✓
       ├─► Aligns with: CB slidingWindowSize < 50
       │    slidingWindowSize=10 ✓
       └─► Result: Limits concurrent calls, feeds failures to CB
 
Circuit Breaker Configuration:
  └── slidingWindowSize = 10
      failureRateThreshold = 50%  (5 of 10 failures → open)
      slowCallRateThreshold = 80% (8 of 10 slow → open)
      waitDurationInOpenState = 30s
       │
       ├─► Aligns with: 5 failures reasonable given bulkhead size
       ├─► Aligns with: 30s matches dependency recovery time
       └─► Result: Opens quickly on problems, waits appropriate time
 
COORDINATION FORMULA:
  slidingWindowSize < maxConcurrentCalls
  slowCallDurationThreshold ≤ timeoutDuration
  waitDurationInOpenState ≈ expectedRecoveryTime
  permittedCallsInHalfOpen < maxConcurrentCalls / 5

Slow Call Detection

Circuit breakers that only track failures miss slow degradation. A dependency responding in 5 seconds 100% of the time isn't 'failing'—but it's destroying your latency SLA. Configure slowCallRateThreshold and slowCallDurationThreshold to detect and respond to latency degradation, not just errors.

The Complete Resilience Stack

For comprehensive resilience, additional patterns often complement bulkhead and circuit breaker: Retry : Handles transient failures by automatically retrying Timeout : Ensures calls don't block indefinitely Rate Limiter : Protects dependencies from being overwhelmed Cache/Fallback : Provides degraded service when dependencies fail Complete Pattern Order: Cache \to Rate Limiter \to Circuit Breaker \to Retry \to Bulkhead \to Timeout \to Call Reading right-to-left (execution order): Cache : Check if we already have the result Rate Limiter : Ensure we're not overwhelming the dependency Circuit Breaker : Check if dependency is healthy Retry : Wrapper that may re-execute on failure Bulkhead : Limit concurrent executions Timeout : Limit duration of each attempt Call : The actual remote call

CompleteResilienceStack.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
@Service
public class ResilientPaymentService {
    
    // Cache for fallback
    private final Cache<String, PaymentResult> resultCache;
    
    // Resilience components
    private final RateLimiter rateLimiter;
    private final CircuitBreaker circuitBreaker;
    private final Retry retry;
    private final Bulkhead bulkhead;
    private final TimeLimiter timeLimiter;
    
    public PaymentResult processWithFullResilience(PaymentRequest request) {
        // Check cache first (fastest path)
        PaymentResult cached = resultCache.getIfPresent(request.getIdempotencyKey());
        if (cached != null) {
            return cached;
        }
        
        // Build decorated call with all patterns
        Supplier<CompletableFuture<PaymentResult>> resilientCall =
            Decorators.ofSupplier(() -> 
                    CompletableFuture.supplyAsync(() -> gateway.charge(request)))
                // Innermost: Timeout on actual call
                .withTimeLimiter(timeLimiter, scheduler)
                // Limit concurrent calls
                .withBulkhead(bulkhead)
                // Retry transient failures (inside CB)
                .withRetry(retry)
                // Stop calling if failing
                .withCircuitBreaker(circuitBreaker)
                // Don't overwhelm dependency
                .withRateLimiter(rateLimiter)
                .decorate();
        
        try {
            PaymentResult result = resilientCall.get().join();
            // Cache successful results
            resultCache.put(request.getIdempotencyKey(), result);
            return result;
        } catch (Exception e) {
            return handleFailure(request, e);
        }
    }
    
    private PaymentResult handleFailure(PaymentRequest request, Exception e) {
        Throwable cause = e.getCause() != null ? e.getCause() : e;
        
        if (cause instanceof RequestNotPermitted) {
            // Rate limiter kicked in
            return PaymentResult.rateLimited("Too many requests");
        }
        if (cause instanceof CallNotPermittedException) {
            // Circuit breaker open - use fallback
            return getFallbackResult(request);
        }
        if (cause instanceof BulkheadFullException) {
            // System under heavy load
            return PaymentResult.retry("System busy, please retry");
        }
        if (cause instanceof TimeoutException) {
            // Call was too slow (after retries)
            return PaymentResult.pending("Processing - check status");
        }
        
        // Unknown failure
        log.error("Payment failed unexpectedly", e);
        return PaymentResult.failed("Payment unavailable");
    }
    
    private PaymentResult getFallbackResult(PaymentRequest request) {
        // Try cached result for this customer
        PaymentResult recent = recentPayments.getForCustomer(request.getCustomerId());
        if (recent != null && recent.isReusable()) {
            return recent.asRetry();
        }
        
        // Queue for async processing
        asyncQueue.enqueue(request);
        return PaymentResult.queued("Processing offline");
    }
}

Monitoring Combined Patterns

With multiple patterns, monitoring becomes more complex but also more informative. Each pattern contributes distinct signals. Key Metrics by Pattern: Circuit Breaker : State (closed/open/half-open), failure rate, call counts Bulkhead : Utilization, rejection rate, available capacity Retry : Retry attempts, eventually-succeeded rate Timeout : Timeout rate, actual latency distribution Combined Dashboard Essential Panels:

grafana_dashboard.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# Key metrics for combined resilience monitoring
 
# 1. Overall Health Summary
- title: "Dependency Health"
  metrics:
    - expr: 'resilience4j_circuitbreaker_state{name="payment"}'
      legend: "Circuit State (0=closed, 1=open, 2=half-open)"
    - expr: 'resilience4j_bulkhead_available_concurrent_calls{name="payment"}'
      legend: "Available Bulkhead Capacity"
 
# 2. Request Outcomes
- title: "Request Outcome Breakdown"
  type: stacked_percentage
  metrics:
    - expr: 'rate(requests_success_total{service="payment"}[5m])'
      legend: "Success"
    - expr: 'rate(circuitbreaker_calls_rejected_total{name="payment"}[5m])'
      legend: "Circuit Open Rejection"
    - expr: 'rate(bulkhead_calls_rejected_total{name="payment"}[5m])'
      legend: "Bulkhead Full Rejection"
    - expr: 'rate(timeout_count_total{name="payment"}[5m])'
      legend: "Timeout"
 
# 3. Circuit Breaker Detail
- title: "Circuit Breaker State Transitions"
  metrics:
    - expr: 'changes(resilience4j_circuitbreaker_state{name="payment"}[1h])'
      legend: "State Changes (last hour)"
    - expr: 'resilience4j_circuitbreaker_failure_rate{name="payment"}'
      legend: "Current Failure Rate %"
 
# 4. Bulkhead Utilization
- title: "Bulkhead Capacity"
  metrics:
    - expr: |
        1 - (resilience4j_bulkhead_available_concurrent_calls{name="payment"} /
             resilience4j_bulkhead_max_allowed_concurrent_calls{name="payment"})
      legend: "Utilization %"
 
# 5. Latency by Outcome
- title: "Latency Distribution"
  metrics:
    - expr: 'histogram_quantile(0.50, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p50"
    - expr: 'histogram_quantile(0.95, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p95"
    - expr: 'histogram_quantile(0.99, sum(rate(request_duration_bucket[5m])) by (le))'
      legend: "p99"

Critical Alerts for Combined Patterns

•Circuit Open Alert — CB transitioned to open; dependency is failing
•Bulkhead >80% Utilization — Approaching capacity; consider scaling or investigating latency
•Bulkhead Rejections — Active rejections; load exceeds capacity
•High Retry Rate — Many transient failures; dependency may be unstable
•Timeout Rate Spike — Calls taking too long; investigate dependency latency
•Circuit Not Recovering — CB stuck open longer than expected; manual investigation needed

Summary: Defense in Depth with Combined Patterns

Key Takeaways

•Bulkhead and circuit breaker solve different problems — Bulkhead limits concurrency; CB stops calls after failures
•Order matters: CB → Bulkhead → Timeout → Call — Check circuit first to preserve bulkhead capacity
•Coordinate configurations — Sliding window < bulkhead size; slow threshold ≤ timeout
•Complete stack adds retry, rate limiter, cache — Each pattern handles specific failure modes
•Monitor each pattern's metrics — Combined dashboard shows full resilience picture
•Design fallbacks for each rejection type — Different responses for CB open vs bulkhead full vs timeout

Module Complete: You've now mastered the bulkhead pattern—from the foundational philosophy of failure isolation, through thread pool and semaphore implementations, sizing strategies, and integration with circuit breakers. This knowledge equips you to design systems that gracefully handle partial failures, protecting healthy functionality when individual dependencies degrade. The bulkhead pattern exemplifies a crucial principle in distributed systems: design for failure containment, not just failure prevention . Failures will happen; what matters is limiting their blast radius.

Module Complete: Bulkhead Pattern

Congratulations! You've completed the Bulkhead Pattern module. You understand failure isolation philosophy, thread pool and semaphore isolation, sizing strategies, and combined pattern integration. Apply these principles to build systems where failures stay contained and healthy functionality remains available.

Combining Bulkhead with Circuit Breaker: Defense in Depth

Two Patterns, One Defense

Complementary Purposes: What Each Pattern Solves

Order of Application: Which Pattern First?

Implementation with Resilience4j

Tuning Combined Configurations

The Complete Resilience Stack

Monitoring Combined Patterns

Summary: Defense in Depth with Combined Patterns

There’s more to understand.

Combining Bulkhead with Circuit Breaker: Defense in Depth

Two Patterns, One Defense

Complementary Purposes: What Each Pattern Solves

Order of Application: Which Pattern First?

Implementation with Resilience4j

Tuning Combined Configurations

The Complete Resilience Stack

Monitoring Combined Patterns

Summary: Defense in Depth with Combined Patterns

There’s more to understand.