Bulkhead Sizing: Capacity… — HLD: Bulkhead Pattern:… | OneNoughtOne

0/107

System Design (HLD)Bulkhead Pattern

Bulkhead Pattern: Failure Isolation in Distributed Systems

LevelAdvanced

Duration75 mins

TopicBulkhead Pattern

4 / 5

Bulkhead Sizing: Capacity Planning for Resilience

The Goldilocks Problem

Bulkhead sizing presents a classic Goldilocks problem: too small and valid requests are rejected during normal traffic; too large and the bulkhead fails to contain failures, defeating its purpose entirely. At one major e-commerce company, engineers initially sized their payment bulkhead at 10 threads—and found themselves rejecting checkout requests during modest traffic spikes. They overcorrected to 500 threads—which effectively meant no isolation, as a slow payment provider could now consume all their capacity. Finding the 'just right' size requires understanding traffic patterns, latency profiles, and failure modes.

What You Will Learn

By the end of this page, you'll understand the mathematical foundations of bulkhead sizing, techniques for gathering the data needed for sizing decisions, strategies for handling variable load, and approaches for adaptive sizing that adjusts to changing conditions.

The Mathematics of Bulkhead Sizing

Bulkhead sizing is fundamentally a queuing theory problem. Understanding the core formula helps make principled sizing decisions. Little's Law Foundation: Little's Law states that in a stable system: L = λ \times W Where: L = Average number of items in the system λ (lambda) = Average arrival rate W = Average time each item spends in the system Applied to Bulkheads: For a bulkhead protecting a dependency: Required Capacity = Request Rate \times Average Duration Example: Payment service receives 100 requests/second Average payment processing time is 200ms (0.2 seconds) Minimum capacity = 100 \times 0.2 = 20 concurrent slots But wait—this is just the minimum! This formula gives you the steady-state requirement. Real traffic has variance, latency has tails, and you need safety margins.

sizing_formula.txt

BULKHEAD SIZING FORMULA
 
Minimum Size (Steady State):
  Size_min = RPS × Latency_avg
 
Practical Size (with variance):
  Size_practical = RPS_peak × Latency_p95 × Safety_factor
 
Where:
  RPS_peak      = Peak requests per second (not average!)
  Latency_p95   = 95th percentile latency (not average!)
  Safety_factor = 1.2 to 2.0 depending on criticality
 
EXAMPLE CALCULATION:
─────────────────────────────────────────────────────────────
Service: Payment Gateway
 
Traffic Analysis:
  - Average RPS: 80 req/sec
  - Peak RPS: 200 req/sec (during flash sales)
  - Burst RPS: 400 req/sec (first minute of sale)
 
Latency Analysis:
  - p50 latency: 150ms (0.15s)
  - p95 latency: 400ms (0.40s)
  - p99 latency: 1200ms (1.2s)
 
Sizing Calculations:
                                                        
  Minimum (naive):
    Size = 80 × 0.15 = 12 threads
    ❌ Will reject during any spike!
  
  Adequate (peak × p95):
    Size = 200 × 0.40 = 80 threads
    ✓ Handles normal peaks
  
  Conservative (peak × p99 × safety):
    Size = 200 × 1.2 × 1.5 = 360 threads
    ⚠️ Very large, may defeat isolation purpose
  
  Recommended (peak × p95 × safety):
    Size = 200 × 0.40 × 1.5 = 120 threads
    ✅ Balances capacity and isolation

Use Percentiles, Not Averages

Averages hide tail latencies. A service with 100ms average but 5-second p99 needs capacity for those 5-second requests—not the 100ms average. Always size based on p95 or p99 latency for critical dependencies.

Gathering Data for Sizing Decisions

Accurate sizing requires accurate data. Here's how to gather the metrics needed for sizing decisions. Traffic Analysis: Measure actual RPS — Not theoretical capacity, but observed traffic Identify patterns — Daily cycles, weekly patterns, seasonal spikes Find peaks — Absolute maximum RPS observed Understand bursts — Short-duration spikes (flash sales, notifications) Latency Profiling: Capture full histogram — Not just average, but p50/p95/p99 Measure under load — Latency often increases at high load Include failure latency — Timeout duration for failed calls Track degradation patterns — How latency changes when dependency degrades

MetricsCollection.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
// Using Micrometer for comprehensive metrics
@Component
public class DependencyMetrics {
    
    private final MeterRegistry registry;
    private final AtomicInteger concurrentCalls = new AtomicInteger(0);
    
    public <T> T executeWithMetrics(String dependency, Supplier<T> call) {
        // Track concurrent calls (for sizing validation)
        int concurrent = concurrentCalls.incrementAndGet();
        registry.gauge("dependency.concurrent_calls", 
            Tags.of("dependency", dependency), concurrentCalls);
        
        // Track max observed concurrency
        registry.gauge("dependency.max_concurrent",
            Tags.of("dependency", dependency), 
            concurrent, c -> (double) c);
        
        Timer.Sample sample = Timer.start(registry);
        try {
            T result = call.get();
            sample.stop(registry.timer("dependency.duration",
                Tags.of("dependency", dependency, "outcome", "success")));
            return result;
        } catch (Exception e) {
            sample.stop(registry.timer("dependency.duration",
                Tags.of("dependency", dependency, "outcome", "failure")));
            throw e;
        } finally {
            concurrentCalls.decrementAndGet();
        }
    }
}
 
// Query for sizing data (Prometheus PromQL examples):
 
// Peak RPS over last 7 days
max_over_time(rate(dependency_duration_count{dependency="payment"}[1m])[7d:1m])
 
// p95 latency over last 7 days
histogram_quantile(0.95, 
  sum(rate(dependency_duration_bucket{dependency="payment"}[5m])) by (le)
)
 
// Max concurrent calls observed
max_over_time(dependency_concurrent_calls{dependency="payment"}[7d])

Essential Metrics for Bulkhead Sizing
Metric	Purpose	Where to Find	Typical Query
Peak RPS	Maximum throughput	APM, LB logs	max_over_time(rate(...)[7d])
p95 Latency	Tail latency for sizing	APM histograms	histogram_quantile(0.95,...)
p99 Latency	Worst case planning	APM histograms	histogram_quantile(0.99,...)
Max Concurrent	Actual concurrency observed	Custom gauge	max_over_time(concurrent[7d])
Error Rate	Failure frequency	APM, logs	sum(rate(errors[5m]))
Timeout Rate	Timeout frequency	APM	rate(timeout_count[5m])

Sizing Strategies for Different Scenarios

Different types of dependencies and traffic patterns require different sizing approaches. Strategy 1: Critical Fast Dependencies For dependencies that are fast (<50ms) and critical (must not fail): Size generously—rejection is worse than over-provisioning Use peak \times p99 \times 2.0 safety factor Consider semaphore instead of thread pool (lower overhead) Strategy 2: Slow External Dependencies For slow (>500ms) third-party services: Size conservatively—isolation is the priority Use peak \times p95 \times 1.2 safety factor Implement aggressive timeouts to cap latency Accept some rejection during spikes (graceful degradation) Strategy 3: Non-Critical Dependencies For services where failure is acceptable (recommendations, analytics): Size minimally—protect the system, not the feature Use average \times p95 \times 1.0 (no safety factor) Implement fallbacks that work without the dependency Strategy 4: Variable Traffic (Spiky) For services with high variance (flash sales, viral content): Size for expected peak, not observed historical peak Consider queue for burst absorption Implement dynamic scaling if infrastructure supports it

SizingConfigurations.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
@Configuration
public class BulkheadSizingConfig {
    
    // Strategy 1: Critical Fast Dependency (e.g., User Service)
    @Bean
    public Bulkhead userServiceBulkhead() {
        // Fast service: p95 = 30ms, peak = 500 RPS
        // Size = 500 × 0.03 × 2.0 = 30 (generous)
        return Bulkhead.of("user-service", BulkheadConfig.custom()
            .maxConcurrentCalls(50)  // Extra headroom for critical path
            .maxWaitDuration(Duration.ZERO)  // Fail fast
            .build());
    }
    
    // Strategy 2: Slow External Dependency (e.g., Payment Gateway)
    @Bean
    public ThreadPoolBulkhead paymentBulkhead() {
        // Slow external: p95 = 800ms, peak = 100 RPS
        // Size = 100 × 0.8 × 1.2 = 96, round to 100
        return ThreadPoolBulkhead.of("payment", ThreadPoolBulkheadConfig.custom()
            .maxThreadPoolSize(100)
            .coreThreadPoolSize(50)
            .queueCapacity(25)  // Small queue, prefer rejection
            .keepAliveDuration(Duration.ofSeconds(30))
            .build());
    }
    
    // Strategy 3: Non-Critical Dependency (e.g., Recommendations)
    @Bean
    public Bulkhead recommendationsBulkhead() {
        // Non-critical: p95 = 200ms, avg = 80 RPS
        // Size = 80 × 0.2 × 1.0 = 16, minimal
        return Bulkhead.of("recommendations", BulkheadConfig.custom()
            .maxConcurrentCalls(20)
            .maxWaitDuration(Duration.ZERO)
            .build());
    }
    
    // Strategy 4: Variable Traffic (e.g., Flash Sale Product Service)
    @Bean
    public ThreadPoolBulkhead flashSaleProductBulkhead() {
        // Spiky: normal peak = 200 RPS, flash sale = 2000 RPS, p95 = 100ms
        // Size for flash sale: 2000 × 0.1 × 1.5 = 300
        return ThreadPoolBulkhead.of("flash-sale-products", 
            ThreadPoolBulkheadConfig.custom()
                .maxThreadPoolSize(300)
                .coreThreadPoolSize(50)  // Normal operations
                .queueCapacity(200)  // Absorb burst
                .keepAliveDuration(Duration.ofSeconds(60))
                .build());
    }
}

Queue Sizing and Burst Absorption

For thread pool bulkheads, the queue provides burst absorption—handling short-term spikes that exceed thread capacity. Queue Sizing Principle: Queue should absorb seconds of burst, not minutes . Large queues cause latency to explode under sustained load. Formula: Queue Size = (Burst RPS - Sustainable RPS) \times Burst Duration \times Buffer Example: Sustainable RPS: 100 (thread pool can handle) Burst RPS: 300 (flash sale start) Burst Duration: 5 seconds Buffer: 1.2 Queue Size = (300 - 100) \times 5 \times 1.2 = 1,200 But wait—this seems too large! Let's reconsider.

queue_analysis.txt

QUEUE SIZING ANALYSIS
 
Scenario: Flash Sale Start
──────────────────────────────────────────────────────────
Thread Pool: 100 threads
Dependency Latency: 200ms average
Throughput: 100 threads × (1000ms / 200ms) = 500 RPS sustainable
Burst Traffic: 1000 RPS for 10 seconds
 
Without Queue:
  Rejection rate = (1000 - 500) / 1000 = 50% rejected
  ❌ Half of flash sale customers get errors
 
With Queue of 500:
  First 10 seconds: 
    - 500 RPS processed immediately
    - 500 RPS queued (5000 total after 10s... overflow!)
  ❌ Queue fills in 1 second, then rejections resume
 
With Queue of 5000:
  After 10 second burst:
    - Queue has 5000 waiting requests
    - Time to drain: 5000 / 500 RPS = 10 seconds
    - Last request waits 10+ seconds
  ❌ Unacceptable latency
 
BETTER APPROACH: Constrained Queue + Fast Rejection
──────────────────────────────────────────────────────────
Queue size: 100 (200ms latency contribution max)
Thread pool: 100
  
Behavior:
  - First burst absorbed by thread pool + queue
  - Excess immediately rejected with 429 (rate limited)
  - Client retries with backoff
  - Latency stays bounded (queue adds max 200ms)
  
Result:
  - Some requests rejected during burst ✓
  - Latency stays predictable ✓
  - System remains stable ✓
  
QUEUE SIZING RECOMMENDATION:
  Queue Size = Thread Pool Size × (Target Queue Latency / Avg Dependency Latency)
  
  Example: 100 threads × (200ms / 200ms) = 100 queue capacity

Large Queues Are Dangerous

A queue that's 'big enough to handle any burst' will instead accumulate minutes of latency during sustained overload. Prefer small queues with fast rejection and client-side retry. Latency is more damaging than rejections—a 30-second response is worse than an immediate retry prompt.

Adaptive and Dynamic Sizing

Static sizing works for stable workloads, but many systems benefit from dynamic adjustment based on observed conditions. Approach 1: Time-Based Scaling Adjust bulkhead size based on known traffic patterns (time of day, day of week, promotional calendar). Approach 2: Metrics-Based Scaling Adjust based on observed metrics—increase size when utilization is high, decrease when low. Approach 3: Event-Based Scaling Respond to specific events (marketing campaign start, product launch) with pre-configured size changes. Approach 4: Feedback Loop Scaling Continuously adjust based on rejection rate—if rejections exceed threshold, increase size; if utilization is consistently low, decrease.

AdaptiveSizing.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
@Component
public class AdaptiveBulkheadManager {
    
    private final BulkheadRegistry registry;
    private final MeterRegistry metrics;
    private final ScheduledExecutorService scheduler;
    
    // Configuration
    private static final double HIGH_UTILIZATION_THRESHOLD = 0.85;
    private static final double LOW_UTILIZATION_THRESHOLD = 0.30;
    private static final double REJECTION_RATE_THRESHOLD = 0.05;
    private static final int MIN_SIZE = 10;
    private static final int MAX_SIZE = 500;
    
    @PostConstruct
    public void startAdaptiveScaling() {
        scheduler.scheduleAtFixedRate(
            this::adjustBulkheadSizes,
            1, 1, TimeUnit.MINUTES
        );
    }
    
    private void adjustBulkheadSizes() {
        registry.getAllBulkheads().forEach(this::adjustBulkhead);
    }
    
    private void adjustBulkhead(Bulkhead bulkhead) {
        String name = bulkhead.getName();
        Bulkhead.Metrics m = bulkhead.getMetrics();
        
        int current = m.getMaxAllowedConcurrentCalls();
        int available = m.getAvailableConcurrentCalls();
        double utilization = 1.0 - ((double) available / current);
        
        // Get rejection rate from metrics
        double rejectionRate = getRejectionRate(name);
        
        int newSize = current;
        String reason = "stable";
        
        // Scale up if high utilization OR rejections
        if (utilization > HIGH_UTILIZATION_THRESHOLD || 
            rejectionRate > REJECTION_RATE_THRESHOLD) {
            newSize = Math.min(MAX_SIZE, (int)(current * 1.5));
            reason = String.format("scale-up: util=%.2f, reject=%.4f", 
                utilization, rejectionRate);
        }
        // Scale down if consistently underutilized
        else if (utilization < LOW_UTILIZATION_THRESHOLD && 
                 rejectionRate < 0.001) {
            newSize = Math.max(MIN_SIZE, (int)(current * 0.75));
            reason = String.format("scale-down: util=%.2f", utilization);
        }
        
        if (newSize != current) {
            log.info("Adjusting bulkhead {}: {} → {} ({})", 
                name, current, newSize, reason);
            bulkhead.changeConfig(BulkheadConfig.custom()
                .maxConcurrentCalls(newSize)
                .build());
        }
    }
}

Adaptive Sizing Cautions

Adaptive sizing adds complexity and can behave unexpectedly under novel conditions. Start with static sizing based on thorough analysis. Only add adaptive logic after you understand normal behavior and have robust monitoring. Always set hard min/max bounds to prevent runaway scaling.

Validating Sizing Through Testing

Calculated sizes are hypotheses—they must be validated through testing before production deployment. Load Testing: Simulate expected traffic patterns and verify: No rejections under normal peak load Graceful degradation (manageable rejection rate) under stress Latency remains bounded (queue doesn't accumulate) Bulkhead metrics match expectations Chaos Testing: Intentionally degrade dependencies and verify: Failures are contained to the bulkhead Other functionality remains available Recovery happens when dependency recovers Fallbacks work as designed Soak Testing: Run extended tests to detect: Thread leaks (not releasing back to pool) Memory growth (unbounded queues or context retention) Degradation over time (exhaustion patterns)

BulkheadLoadTest.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// Using Gatling for load testing bulkhead behavior
class BulkheadLoadTest extends Simulation {
    
    // Scenario 1: Normal Peak Load
    // Verify: Zero rejections, latency within SLA
    val normalPeak = scenario("Normal Peak")
        .exec(http("payment").post("/checkout")
            .body(paymentPayload)
            .check(status.is(200))
            .check(responseTimeInMillis.lt(3000))
        )
 
    // Scenario 2: Overload
    // Verify: Graceful rejection, no cascade
    val overload = scenario("Overload")
        .exec(http("payment").post("/checkout")
            .body(paymentPayload)
            .check(status.in(200, 429, 503))  // Accept rejections
        )
        .exec(http("search").get("/products")
            .check(status.is(200))  // Search must still work!
        )
 
    // Scenario 3: Slow Dependency
    // Verify: Rejections occur, other paths unaffected
    val slowDependency = scenario("Slow Dependency Simulation")
        .exec(http("payment").post("/checkout") // Hitting slow mock
            .check(status.in(200, 503))
            .check(responseTimeInMillis.lt(5000))  // Timeout works
        )
 
    setUp(
        normalPeak.inject(
            rampUsers(100).during(30.seconds),
            constantUsersPerSec(200).during(5.minutes)
        ).assertions(
            global.failedRequests.count.is(0),
            global.responseTime.percentile3.lt(2000)
        ),
        
        overload.inject(
            constantUsersPerSec(500).during(2.minutes)  // 2.5x capacity
        ).assertions(
            global.failedRequests.percent.lt(60),  // Max 60% rejection
            forAll.responseTime.percentile4.lt(5000)  // Even failures are fast
        )
    )
}

Summary: Bulkhead Sizing Mastery

Key Takeaways

•Size = Peak RPS × p95 Latency × Safety Factor — Use percentiles, not averages; use peak, not average traffic
•Different strategies for different dependencies — Critical gets generous sizing; non-critical gets minimal; slow gets conservative
•Queue size should be small — Absorb seconds of burst, not minutes; large queues cause latency explosion
•Gather real data before sizing — Historical metrics for traffic patterns and latency profiles
•Validate through testing — Load test normal and overload scenarios; chaos test failure containment
•Consider adaptive sizing for variable workloads — But start with static sizing and add adaptation carefully

What's Next: We've covered bulkhead isolation, thread pools, semaphores, and sizing. The final piece is understanding how bulkheads integrate with other resilience patterns. The next page explores combining bulkheads with circuit breakers —how these patterns complement each other and patterns for effective integration.

Page Complete

You now understand the mathematics and strategies behind bulkhead sizing. You've learned how to gather data, size for different dependency types, and validate through testing. Next, we'll explore combining bulkheads with circuit breakers for comprehensive resilience.

System Design (HLD)Bulkhead Pattern

Bulkhead Pattern: Failure Isolation in Distributed Systems

LevelAdvanced

Duration75 mins

TopicBulkhead Pattern

4 / 5

Bulkhead Sizing: Capacity Planning for Resilience

The Goldilocks Problem

Bulkhead sizing presents a classic Goldilocks problem: too small and valid requests are rejected during normal traffic; too large and the bulkhead fails to contain failures, defeating its purpose entirely. At one major e-commerce company, engineers initially sized their payment bulkhead at 10 threads—and found themselves rejecting checkout requests during modest traffic spikes. They overcorrected to 500 threads—which effectively meant no isolation, as a slow payment provider could now consume all their capacity. Finding the 'just right' size requires understanding traffic patterns, latency profiles, and failure modes.

What You Will Learn

By the end of this page, you'll understand the mathematical foundations of bulkhead sizing, techniques for gathering the data needed for sizing decisions, strategies for handling variable load, and approaches for adaptive sizing that adjusts to changing conditions.

The Mathematics of Bulkhead Sizing

Bulkhead sizing is fundamentally a queuing theory problem. Understanding the core formula helps make principled sizing decisions. Little's Law Foundation: Little's Law states that in a stable system: L = λ \times W Where: L = Average number of items in the system λ (lambda) = Average arrival rate W = Average time each item spends in the system Applied to Bulkheads: For a bulkhead protecting a dependency: Required Capacity = Request Rate \times Average Duration Example: Payment service receives 100 requests/second Average payment processing time is 200ms (0.2 seconds) Minimum capacity = 100 \times 0.2 = 20 concurrent slots But wait—this is just the minimum! This formula gives you the steady-state requirement. Real traffic has variance, latency has tails, and you need safety margins.

sizing_formula.txt

BULKHEAD SIZING FORMULA
 
Minimum Size (Steady State):
  Size_min = RPS × Latency_avg
 
Practical Size (with variance):
  Size_practical = RPS_peak × Latency_p95 × Safety_factor
 
Where:
  RPS_peak      = Peak requests per second (not average!)
  Latency_p95   = 95th percentile latency (not average!)
  Safety_factor = 1.2 to 2.0 depending on criticality
 
EXAMPLE CALCULATION:
─────────────────────────────────────────────────────────────
Service: Payment Gateway
 
Traffic Analysis:
  - Average RPS: 80 req/sec
  - Peak RPS: 200 req/sec (during flash sales)
  - Burst RPS: 400 req/sec (first minute of sale)
 
Latency Analysis:
  - p50 latency: 150ms (0.15s)
  - p95 latency: 400ms (0.40s)
  - p99 latency: 1200ms (1.2s)
 
Sizing Calculations:
                                                        
  Minimum (naive):
    Size = 80 × 0.15 = 12 threads
    ❌ Will reject during any spike!
  
  Adequate (peak × p95):
    Size = 200 × 0.40 = 80 threads
    ✓ Handles normal peaks
  
  Conservative (peak × p99 × safety):
    Size = 200 × 1.2 × 1.5 = 360 threads
    ⚠️ Very large, may defeat isolation purpose
  
  Recommended (peak × p95 × safety):
    Size = 200 × 0.40 × 1.5 = 120 threads
    ✅ Balances capacity and isolation

Use Percentiles, Not Averages

Averages hide tail latencies. A service with 100ms average but 5-second p99 needs capacity for those 5-second requests—not the 100ms average. Always size based on p95 or p99 latency for critical dependencies.

Gathering Data for Sizing Decisions

Accurate sizing requires accurate data. Here's how to gather the metrics needed for sizing decisions. Traffic Analysis: Measure actual RPS — Not theoretical capacity, but observed traffic Identify patterns — Daily cycles, weekly patterns, seasonal spikes Find peaks — Absolute maximum RPS observed Understand bursts — Short-duration spikes (flash sales, notifications) Latency Profiling: Capture full histogram — Not just average, but p50/p95/p99 Measure under load — Latency often increases at high load Include failure latency — Timeout duration for failed calls Track degradation patterns — How latency changes when dependency degrades

MetricsCollection.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
// Using Micrometer for comprehensive metrics
@Component
public class DependencyMetrics {
    
    private final MeterRegistry registry;
    private final AtomicInteger concurrentCalls = new AtomicInteger(0);
    
    public <T> T executeWithMetrics(String dependency, Supplier<T> call) {
        // Track concurrent calls (for sizing validation)
        int concurrent = concurrentCalls.incrementAndGet();
        registry.gauge("dependency.concurrent_calls", 
            Tags.of("dependency", dependency), concurrentCalls);
        
        // Track max observed concurrency
        registry.gauge("dependency.max_concurrent",
            Tags.of("dependency", dependency), 
            concurrent, c -> (double) c);
        
        Timer.Sample sample = Timer.start(registry);
        try {
            T result = call.get();
            sample.stop(registry.timer("dependency.duration",
                Tags.of("dependency", dependency, "outcome", "success")));
            return result;
        } catch (Exception e) {
            sample.stop(registry.timer("dependency.duration",
                Tags.of("dependency", dependency, "outcome", "failure")));
            throw e;
        } finally {
            concurrentCalls.decrementAndGet();
        }
    }
}
 
// Query for sizing data (Prometheus PromQL examples):
 
// Peak RPS over last 7 days
max_over_time(rate(dependency_duration_count{dependency="payment"}[1m])[7d:1m])
 
// p95 latency over last 7 days
histogram_quantile(0.95, 
  sum(rate(dependency_duration_bucket{dependency="payment"}[5m])) by (le)
)
 
// Max concurrent calls observed
max_over_time(dependency_concurrent_calls{dependency="payment"}[7d])

Essential Metrics for Bulkhead Sizing
Metric	Purpose	Where to Find	Typical Query
Peak RPS	Maximum throughput	APM, LB logs	max_over_time(rate(...)[7d])
p95 Latency	Tail latency for sizing	APM histograms	histogram_quantile(0.95,...)
p99 Latency	Worst case planning	APM histograms	histogram_quantile(0.99,...)
Max Concurrent	Actual concurrency observed	Custom gauge	max_over_time(concurrent[7d])
Error Rate	Failure frequency	APM, logs	sum(rate(errors[5m]))
Timeout Rate	Timeout frequency	APM	rate(timeout_count[5m])

Sizing Strategies for Different Scenarios

Different types of dependencies and traffic patterns require different sizing approaches. Strategy 1: Critical Fast Dependencies For dependencies that are fast (<50ms) and critical (must not fail): Size generously—rejection is worse than over-provisioning Use peak \times p99 \times 2.0 safety factor Consider semaphore instead of thread pool (lower overhead) Strategy 2: Slow External Dependencies For slow (>500ms) third-party services: Size conservatively—isolation is the priority Use peak \times p95 \times 1.2 safety factor Implement aggressive timeouts to cap latency Accept some rejection during spikes (graceful degradation) Strategy 3: Non-Critical Dependencies For services where failure is acceptable (recommendations, analytics): Size minimally—protect the system, not the feature Use average \times p95 \times 1.0 (no safety factor) Implement fallbacks that work without the dependency Strategy 4: Variable Traffic (Spiky) For services with high variance (flash sales, viral content): Size for expected peak, not observed historical peak Consider queue for burst absorption Implement dynamic scaling if infrastructure supports it

SizingConfigurations.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
@Configuration
public class BulkheadSizingConfig {
    
    // Strategy 1: Critical Fast Dependency (e.g., User Service)
    @Bean
    public Bulkhead userServiceBulkhead() {
        // Fast service: p95 = 30ms, peak = 500 RPS
        // Size = 500 × 0.03 × 2.0 = 30 (generous)
        return Bulkhead.of("user-service", BulkheadConfig.custom()
            .maxConcurrentCalls(50)  // Extra headroom for critical path
            .maxWaitDuration(Duration.ZERO)  // Fail fast
            .build());
    }
    
    // Strategy 2: Slow External Dependency (e.g., Payment Gateway)
    @Bean
    public ThreadPoolBulkhead paymentBulkhead() {
        // Slow external: p95 = 800ms, peak = 100 RPS
        // Size = 100 × 0.8 × 1.2 = 96, round to 100
        return ThreadPoolBulkhead.of("payment", ThreadPoolBulkheadConfig.custom()
            .maxThreadPoolSize(100)
            .coreThreadPoolSize(50)
            .queueCapacity(25)  // Small queue, prefer rejection
            .keepAliveDuration(Duration.ofSeconds(30))
            .build());
    }
    
    // Strategy 3: Non-Critical Dependency (e.g., Recommendations)
    @Bean
    public Bulkhead recommendationsBulkhead() {
        // Non-critical: p95 = 200ms, avg = 80 RPS
        // Size = 80 × 0.2 × 1.0 = 16, minimal
        return Bulkhead.of("recommendations", BulkheadConfig.custom()
            .maxConcurrentCalls(20)
            .maxWaitDuration(Duration.ZERO)
            .build());
    }
    
    // Strategy 4: Variable Traffic (e.g., Flash Sale Product Service)
    @Bean
    public ThreadPoolBulkhead flashSaleProductBulkhead() {
        // Spiky: normal peak = 200 RPS, flash sale = 2000 RPS, p95 = 100ms
        // Size for flash sale: 2000 × 0.1 × 1.5 = 300
        return ThreadPoolBulkhead.of("flash-sale-products", 
            ThreadPoolBulkheadConfig.custom()
                .maxThreadPoolSize(300)
                .coreThreadPoolSize(50)  // Normal operations
                .queueCapacity(200)  // Absorb burst
                .keepAliveDuration(Duration.ofSeconds(60))
                .build());
    }
}

Queue Sizing and Burst Absorption

For thread pool bulkheads, the queue provides burst absorption—handling short-term spikes that exceed thread capacity. Queue Sizing Principle: Queue should absorb seconds of burst, not minutes . Large queues cause latency to explode under sustained load. Formula: Queue Size = (Burst RPS - Sustainable RPS) \times Burst Duration \times Buffer Example: Sustainable RPS: 100 (thread pool can handle) Burst RPS: 300 (flash sale start) Burst Duration: 5 seconds Buffer: 1.2 Queue Size = (300 - 100) \times 5 \times 1.2 = 1,200 But wait—this seems too large! Let's reconsider.

queue_analysis.txt

QUEUE SIZING ANALYSIS
 
Scenario: Flash Sale Start
──────────────────────────────────────────────────────────
Thread Pool: 100 threads
Dependency Latency: 200ms average
Throughput: 100 threads × (1000ms / 200ms) = 500 RPS sustainable
Burst Traffic: 1000 RPS for 10 seconds
 
Without Queue:
  Rejection rate = (1000 - 500) / 1000 = 50% rejected
  ❌ Half of flash sale customers get errors
 
With Queue of 500:
  First 10 seconds: 
    - 500 RPS processed immediately
    - 500 RPS queued (5000 total after 10s... overflow!)
  ❌ Queue fills in 1 second, then rejections resume
 
With Queue of 5000:
  After 10 second burst:
    - Queue has 5000 waiting requests
    - Time to drain: 5000 / 500 RPS = 10 seconds
    - Last request waits 10+ seconds
  ❌ Unacceptable latency
 
BETTER APPROACH: Constrained Queue + Fast Rejection
──────────────────────────────────────────────────────────
Queue size: 100 (200ms latency contribution max)
Thread pool: 100
  
Behavior:
  - First burst absorbed by thread pool + queue
  - Excess immediately rejected with 429 (rate limited)
  - Client retries with backoff
  - Latency stays bounded (queue adds max 200ms)
  
Result:
  - Some requests rejected during burst ✓
  - Latency stays predictable ✓
  - System remains stable ✓
  
QUEUE SIZING RECOMMENDATION:
  Queue Size = Thread Pool Size × (Target Queue Latency / Avg Dependency Latency)
  
  Example: 100 threads × (200ms / 200ms) = 100 queue capacity

Large Queues Are Dangerous

A queue that's 'big enough to handle any burst' will instead accumulate minutes of latency during sustained overload. Prefer small queues with fast rejection and client-side retry. Latency is more damaging than rejections—a 30-second response is worse than an immediate retry prompt.

Adaptive and Dynamic Sizing

Static sizing works for stable workloads, but many systems benefit from dynamic adjustment based on observed conditions. Approach 1: Time-Based Scaling Adjust bulkhead size based on known traffic patterns (time of day, day of week, promotional calendar). Approach 2: Metrics-Based Scaling Adjust based on observed metrics—increase size when utilization is high, decrease when low. Approach 3: Event-Based Scaling Respond to specific events (marketing campaign start, product launch) with pre-configured size changes. Approach 4: Feedback Loop Scaling Continuously adjust based on rejection rate—if rejections exceed threshold, increase size; if utilization is consistently low, decrease.

AdaptiveSizing.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
@Component
public class AdaptiveBulkheadManager {
    
    private final BulkheadRegistry registry;
    private final MeterRegistry metrics;
    private final ScheduledExecutorService scheduler;
    
    // Configuration
    private static final double HIGH_UTILIZATION_THRESHOLD = 0.85;
    private static final double LOW_UTILIZATION_THRESHOLD = 0.30;
    private static final double REJECTION_RATE_THRESHOLD = 0.05;
    private static final int MIN_SIZE = 10;
    private static final int MAX_SIZE = 500;
    
    @PostConstruct
    public void startAdaptiveScaling() {
        scheduler.scheduleAtFixedRate(
            this::adjustBulkheadSizes,
            1, 1, TimeUnit.MINUTES
        );
    }
    
    private void adjustBulkheadSizes() {
        registry.getAllBulkheads().forEach(this::adjustBulkhead);
    }
    
    private void adjustBulkhead(Bulkhead bulkhead) {
        String name = bulkhead.getName();
        Bulkhead.Metrics m = bulkhead.getMetrics();
        
        int current = m.getMaxAllowedConcurrentCalls();
        int available = m.getAvailableConcurrentCalls();
        double utilization = 1.0 - ((double) available / current);
        
        // Get rejection rate from metrics
        double rejectionRate = getRejectionRate(name);
        
        int newSize = current;
        String reason = "stable";
        
        // Scale up if high utilization OR rejections
        if (utilization > HIGH_UTILIZATION_THRESHOLD || 
            rejectionRate > REJECTION_RATE_THRESHOLD) {
            newSize = Math.min(MAX_SIZE, (int)(current * 1.5));
            reason = String.format("scale-up: util=%.2f, reject=%.4f", 
                utilization, rejectionRate);
        }
        // Scale down if consistently underutilized
        else if (utilization < LOW_UTILIZATION_THRESHOLD && 
                 rejectionRate < 0.001) {
            newSize = Math.max(MIN_SIZE, (int)(current * 0.75));
            reason = String.format("scale-down: util=%.2f", utilization);
        }
        
        if (newSize != current) {
            log.info("Adjusting bulkhead {}: {} → {} ({})", 
                name, current, newSize, reason);
            bulkhead.changeConfig(BulkheadConfig.custom()
                .maxConcurrentCalls(newSize)
                .build());
        }
    }
}

Adaptive Sizing Cautions

Adaptive sizing adds complexity and can behave unexpectedly under novel conditions. Start with static sizing based on thorough analysis. Only add adaptive logic after you understand normal behavior and have robust monitoring. Always set hard min/max bounds to prevent runaway scaling.

Validating Sizing Through Testing

Calculated sizes are hypotheses—they must be validated through testing before production deployment. Load Testing: Simulate expected traffic patterns and verify: No rejections under normal peak load Graceful degradation (manageable rejection rate) under stress Latency remains bounded (queue doesn't accumulate) Bulkhead metrics match expectations Chaos Testing: Intentionally degrade dependencies and verify: Failures are contained to the bulkhead Other functionality remains available Recovery happens when dependency recovers Fallbacks work as designed Soak Testing: Run extended tests to detect: Thread leaks (not releasing back to pool) Memory growth (unbounded queues or context retention) Degradation over time (exhaustion patterns)

BulkheadLoadTest.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// Using Gatling for load testing bulkhead behavior
class BulkheadLoadTest extends Simulation {
    
    // Scenario 1: Normal Peak Load
    // Verify: Zero rejections, latency within SLA
    val normalPeak = scenario("Normal Peak")
        .exec(http("payment").post("/checkout")
            .body(paymentPayload)
            .check(status.is(200))
            .check(responseTimeInMillis.lt(3000))
        )
 
    // Scenario 2: Overload
    // Verify: Graceful rejection, no cascade
    val overload = scenario("Overload")
        .exec(http("payment").post("/checkout")
            .body(paymentPayload)
            .check(status.in(200, 429, 503))  // Accept rejections
        )
        .exec(http("search").get("/products")
            .check(status.is(200))  // Search must still work!
        )
 
    // Scenario 3: Slow Dependency
    // Verify: Rejections occur, other paths unaffected
    val slowDependency = scenario("Slow Dependency Simulation")
        .exec(http("payment").post("/checkout") // Hitting slow mock
            .check(status.in(200, 503))
            .check(responseTimeInMillis.lt(5000))  // Timeout works
        )
 
    setUp(
        normalPeak.inject(
            rampUsers(100).during(30.seconds),
            constantUsersPerSec(200).during(5.minutes)
        ).assertions(
            global.failedRequests.count.is(0),
            global.responseTime.percentile3.lt(2000)
        ),
        
        overload.inject(
            constantUsersPerSec(500).during(2.minutes)  // 2.5x capacity
        ).assertions(
            global.failedRequests.percent.lt(60),  // Max 60% rejection
            forAll.responseTime.percentile4.lt(5000)  // Even failures are fast
        )
    )
}

Summary: Bulkhead Sizing Mastery

Key Takeaways

•Size = Peak RPS × p95 Latency × Safety Factor — Use percentiles, not averages; use peak, not average traffic
•Different strategies for different dependencies — Critical gets generous sizing; non-critical gets minimal; slow gets conservative
•Queue size should be small — Absorb seconds of burst, not minutes; large queues cause latency explosion
•Gather real data before sizing — Historical metrics for traffic patterns and latency profiles
•Validate through testing — Load test normal and overload scenarios; chaos test failure containment
•Consider adaptive sizing for variable workloads — But start with static sizing and add adaptation carefully

What's Next: We've covered bulkhead isolation, thread pools, semaphores, and sizing. The final piece is understanding how bulkheads integrate with other resilience patterns. The next page explores combining bulkheads with circuit breakers —how these patterns complement each other and patterns for effective integration.

Page Complete

You now understand the mathematics and strategies behind bulkhead sizing. You've learned how to gather data, size for different dependency types, and validate through testing. Next, we'll explore combining bulkheads with circuit breakers for comprehensive resilience.

Bulkhead Sizing: Capacity Planning for Resilience

The Goldilocks Problem

The Mathematics of Bulkhead Sizing

Gathering Data for Sizing Decisions

Sizing Strategies for Different Scenarios

Queue Sizing and Burst Absorption

Adaptive and Dynamic Sizing

Validating Sizing Through Testing

Summary: Bulkhead Sizing Mastery

There’s more to understand.

Bulkhead Sizing: Capacity Planning for Resilience

The Goldilocks Problem

The Mathematics of Bulkhead Sizing

Gathering Data for Sizing Decisions

Sizing Strategies for Different Scenarios

Queue Sizing and Burst Absorption

Adaptive and Dynamic Sizing

Validating Sizing Through Testing

Summary: Bulkhead Sizing Mastery

There’s more to understand.