System DesignTimeout Patterns

Timeout Patterns in Synchronous Communication

LevelIntermediate

Duration60 mins

TopicTimeout Patterns

5 / 5

Timeout Tuning

The Art and Science of Timeout Selection

You understand why timeouts matter, the difference between connection and read timeouts, and how to propagate them across services. But there's still a crucial question we haven't fully answered: What value should you set? This question has no universal answer. A 500ms timeout that works perfectly for an in-memory cache would be catastrophic for a transatlantic payment processing call. A 30-second timeout that's appropriate for batch processing would destroy the user experience for an interactive API. Timeout tuning is the process of selecting timeout values that balance competing concerns: short enough to fail fast and prevent resource exhaustion, but long enough to succeed under normal conditions. This page provides a systematic approach to timeout tuning based on metrics, percentile analysis, and continuous refinement.

What You Will Learn

By the end of this page, you will understand how to analyze latency distributions to select timeout values, learn the relationship between timeouts and error budgets, see how to use percentile-based tuning, and develop a framework for ongoing timeout optimization.

Understanding Latency Distributions

Before tuning timeouts, you must understand how your system's latency behaves. Latency rarely follows a normal distribution—instead, it typically exhibits a long tail where most requests complete quickly but a small percentage take significantly longer. Key latency metrics: P50 (Median): Half of requests complete faster than this value P90: 90% of requests complete faster than this value P95: 95% of requests complete faster than this value P99: 99% of requests complete faster than this value P99.9: 99.9% of requests complete faster than this value Why averages are dangerous: The average (mean) latency is misleading for timeout selection. Consider this distribution: 900 requests at 50ms each 100 requests at 500ms each Mean: (900\times50 + 100\times500) / 1000 = 95ms P95: 50ms P99: 500ms If you set a timeout at 2\times the mean (190ms), you'd timeout 10% of your requests—those that naturally take 500ms. But if you set it at 2\times the P99 (1000ms), you'd allow successful completion while still protecting against truly stuck requests.

Latency Percentiles: What They Tell You
Percentile	Meaning	Use Case	Timeout Implication
P50	Typical request latency	User experience baseline	Not useful for timeout selection
P90	Most requests complete by here	Performance SLAs	Minimum viable timeout for high availability
P95	5% of requests are slower	Baseline for alerting	Conservative timeout floor
P99	1% of requests are slower	SLA boundaries	Good target for most timeouts
P99.9	0.1% are slower	Tail latency optimization	Maximum for latency-sensitive operations
Max	Slowest observed request	Capacity planning	Often too high; affected by outliers

Gathering latency data: To tune timeouts effectively, you need latency data from your actual production environment. Development or staging environments often have different latency characteristics due to: Different hardware and network topology Lower traffic volume (no contention) Missing production-only factors (authentication, logging, etc.) Different data sizes and patterns Sources of latency data: Application metrics: Instrument your code to record request duration Load balancer logs: ALB/ELB access logs include request duration Service mesh telemetry: Istio, Linkerd, and Envoy provide detailed latency histograms APM tools: Datadog, New Relic, Dynatrace track distributed latency Database slow query logs: For tuning database-specific timeouts

Collect Histograms, Not Just Aggregates

Ensure your metrics system captures latency as histograms, not just averages. Systems like Prometheus with histogram metrics allow you to compute any percentile after the fact. If you only capture averages, you've lost the tail data needed for timeout tuning.

The Percentile-Based Tuning Method

The most reliable method for selecting timeout values is based on observed percentile latency with appropriate headroom. The general formula is: Timeout = Percentile Latency \times Headroom Multiplier The percentile you choose depends on your tolerance for false positives (legitimate requests that timeout), and the multiplier provides buffer for normal latency variation.

Percentile Selection Guide
Percentile	Timeout at 1.5×	False Positive Rate	Best For
P90	Medium	~10%	Best-effort features, recommendations
P95	Medium-High	~5%	Non-critical APIs, background jobs
P99	High	~1%	Standard production APIs
P99.9	Very High	~0.1%	Critical transaction paths

Headroom multiplier selection: The multiplier accounts for latency variability. Higher multipliers reduce false positives but increase resource consumption during failures. 1.2\times - 1.5\times: Aggressive, for stable services with low latency variance 1.5\times - 2.0\times: Moderate, for typical production services 2.0\times - 3.0\times: Conservative, for services with high variability or external dependencies 3.0\times+: Very conservative, reduces timeout effectiveness Example calculation: Observed latency for payment service: P50: 200ms P95: 800ms P99: 1500ms P99.9: 3000ms For a standard API (P99 with 1.5x headroom): Timeout = 1500ms \times 1.5 = 2250ms \approx 2.5 seconds For a critical checkout flow (P99.9 with 2x headroom): Timeout = 3000ms \times 2.0 = 6000ms = 6 seconds For a non-critical recommendation (P95 with 1.3x headroom): Timeout = 800ms \times 1.3 = 1040ms \approx 1 second

timeout-calculator
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
interface LatencyData {
  p50: number;
  p90: number;
  p95: number;
  p99: number;
  p999: number;
}
 
interface TimeoutConfig {
  connectionTimeoutMs: number;
  readTimeoutMs: number;
  totalTimeoutMs: number;
}
 
type Criticality = 'best-effort' | 'standard' | 'critical';
 
/**
 * Calculate timeout values based on latency percentiles
 */
function calculateTimeouts(
  latency: LatencyData,
  criticality: Criticality = 'standard'
): TimeoutConfig {
  const config = {
    'best-effort': { percentile: 'p95', multiplier: 1.3 },
    'standard': { percentile: 'p99', multiplier: 1.5 },
    'critical': { percentile: 'p999', multiplier: 2.0 },
  }[criticality];
  
  const baseLatency = latency[config.percentile as keyof LatencyData];
  const readTimeout = Math.ceil(baseLatency * config.multiplier);
  
  // Connection timeout is typically 10-20% of read timeout, minimum 500ms
  const connectionTimeout = Math.max(500, Math.ceil(readTimeout * 0.15));
  
  // Total timeout is read timeout plus connection plus buffer
  const totalTimeout = connectionTimeout + readTimeout + 500;
  
  return {
    connectionTimeoutMs: connectionTimeout,
    readTimeoutMs: readTimeout,
    totalTimeoutMs: totalTimeout,
  };
}
 
// Example usage
const paymentServiceLatency: LatencyData = {
  p50: 200,
  p90: 600,
  p95: 800,
  p99: 1500,
  p999: 3000,
};
 
console.log('Best-effort:', calculateTimeouts(paymentServiceLatency, 'best-effort'));
// { connectionTimeoutMs: 500, readTimeoutMs: 1040, totalTimeoutMs: 2040 }
 
console.log('Standard:', calculateTimeouts(paymentServiceLatency, 'standard'));
// { connectionTimeoutMs: 500, readTimeoutMs: 2250, totalTimeoutMs: 3250 }
 
console.log('Critical:', calculateTimeouts(paymentServiceLatency, 'critical'));
// { connectionTimeoutMs: 900, readTimeoutMs: 6000, totalTimeoutMs: 7400 }

Account for Upstream Constraints

Your timeout can't exceed your caller's deadline. If the API Gateway has a 10-second timeout, internal services can't use 15-second timeouts effectively—they'll be cancelled by the gateway before completing. Work backwards from the overall request budget.

Timeout Budgeting Across Call Chains

When a request traverses multiple services, you must budget timeout across the entire chain. The sum of individual timeouts must be less than the overall request deadline. Budget allocation strategies:

Strategy 1: Proportional Allocation

•Method: Allocate timeout proportionally to expected duration of each step.
•Example: If step A typically takes 1s and step B takes 2s, give A 33% of budget and B 67%.
•Pros: Efficient use of budget; slower steps get more time.
•Cons: Requires understanding of typical durations; slow step A starves step B.

Strategy 2: Critical Path Priority

•Method: Give critical operations larger budgets; non-critical operations get minimal budget.
•Example: Payment gets 70% of budget; recommendations get 10%.
•Pros: Protects critical paths; non-critical can be skipped if slow.
•Cons: Non-critical features may timeout frequently.

Strategy 3: Fixed Maximum Per Operation

•Method: Each operation gets a maximum timeout regardless of remaining budget.
•Example: Each service call is capped at 5 seconds, regardless of downstream structure.
•Pros: Simple; predictable behavior; no complex budgeting.
•Cons: May not use full available budget; deep call chains accumulate timeouts.

Worked example: E-commerce checkout Total budget: 10 seconds (user-facing SLA) Checkout Flow: 1. Cart validation (local): ~100ms 2. Inventory reservation: ~500ms typical, 2s P99 3. Fraud check (external): ~1s typical, 3s P99 4. Payment processing (external): ~1.5s typical, 5s P99 5. Order creation (database): ~200ms typical, 1s P99 6. Notification trigger (async): fire-and-forget Budget allocation (Critical Path Priority): - Reserved for overhead: 500ms - Inventory reservation: 1.5s (P99 \times 0.75, non-critical) - Fraud check: 2s (important but can fallback) - Payment processing: 5s (critical, uses P99 exactly) - Order creation: 1s (P99, must succeed) - Total allocated: 10s ✓ Fallback behavior: - If fraud check timesout \to proceed with basic checks (accept slight risk) - If inventory timesout \to show error, don't charge card - If payment timesout \to show pending status, investigate

Parallel Calls Don't Stack

When making parallel calls, budget for the slowest one, not the sum. If you call Service A (3s timeout) and Service B (2s timeout) in parallel, budget 3 seconds (plus buffer), not 5 seconds.

SLA-Driven Timeout Tuning

Your SLAs (Service Level Agreements) and SLOs (Service Level Objectives) should directly inform timeout values. If you promise P99 latency under 500ms, requests timing out at 2 seconds have already violated the SLA. Timeout \leftrightarrow SLA relationship: SLO: P99 response time \leq 500ms Error budget: 1% of requests can exceed this Implication: - Requests targeting the P99 SLO must complete within 500ms - Timeouts must allow successful requests to finish - But timeout > SLO means successful requests still violate SLO Strategy: - Set timeout slightly above SLO (e.g., 600ms) - Requests that take 500-600ms: succeed but violate SLO (counted against budget) - Requests that take >600ms: timeout (also counted against budget) - This prioritizes quick failure over slow success

Timeout Tuning Based on SLO
SLO Target	Timeout Range	Rationale
P99 ≤ 100ms	100-150ms	Aggressive; any slow request is an SLO violation anyway
P99 ≤ 500ms	500-750ms	Allow slightly slow requests to complete
P99 ≤ 2s	2-3s	Give requests chance to complete; fail fast if hopeless
P99 ≤ 30s (batch)	30-45s	Long operations need proportionally more headroom

The error budget connection: Your error budget determines how much timeout-induced failure you can tolerate. If your SLO is 99.9% success rate (0.1% error budget), and timeouts cause 0.05% failures, you've consumed half your budget on timeouts alone. Balancing act: Shorter timeouts: Fail faster, reduce resource consumption, but increase false positives (legitimate slow requests timeout) Longer timeouts: Fewer false positives, but more resource consumption during failures and potentially violate SLAs The optimal timeout sits where false positive rate is acceptable and the latency is within SLA bounds.

sla-driven-timeout
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
interface SLOConfig {
  latencyTargetMs: number;  // e.g., 500 for P99 ≤ 500ms
  successRateTarget: number; // e.g., 0.999 for 99.9%
  allowedTimeoutRate: number; // Portion of error budget for timeouts
}
 
interface ObservedMetrics {
  latencyP99Ms: number;
  currentTimeoutMs: number;
  currentTimeoutRate: number; // % of requests timing out
  currentSuccessRate: number;
}
 
/**
 * Recommend timeout adjustments based on SLO compliance
 */
function recommendTimeoutAdjustment(
  slo: SLOConfig,
  observed: ObservedMetrics
): { newTimeoutMs: number; reasoning: string } {
  const maxTimeoutRate = (1 - slo.successRateTarget) * slo.allowedTimeoutRate;
  
  // Case 1: Timeouts too frequent, exceeding budget
  if (observed.currentTimeoutRate > maxTimeoutRate) {
    // Check if increasing timeout would help
    if (observed.latencyP99Ms < slo.latencyTargetMs) {
      // P99 is within SLA, so increasing timeout is safe
      const newTimeout = Math.min(
        observed.latencyP99Ms * 1.5,
        slo.latencyTargetMs * 1.2
      );
      return {
        newTimeoutMs: Math.ceil(newTimeout),
        reasoning: `Timeout rate (${(observed.currentTimeoutRate * 100).toFixed(2)}%) exceeds budget. P99 (${observed.latencyP99Ms}ms) is healthy. Increase timeout to ${Math.ceil(newTimeout)}ms.`
      };
    } else {
      // P99 exceeds SLA - the service is too slow
      return {
        newTimeoutMs: slo.latencyTargetMs,
        reasoning: `Timeout rate high, but P99 (${observed.latencyP99Ms}ms) exceeds SLO (${slo.latencyTargetMs}ms). Service needs optimization, not longer timeout.`
      };
    }
  }
  
  // Case 2: Timeouts rare, using little of error budget
  if (observed.currentTimeoutRate < maxTimeoutRate * 0.1) {
    // May be able to tighten timeout
    if (observed.currentTimeoutMs > observed.latencyP99Ms * 2) {
      const newTimeout = Math.max(
        observed.latencyP99Ms * 1.5,
        slo.latencyTargetMs
      );
      return {
        newTimeoutMs: Math.ceil(newTimeout),
        reasoning: `Timeout rate very low. Consider tightening from ${observed.currentTimeoutMs}ms to ${Math.ceil(newTimeout)}ms for faster failure detection.`
      };
    }
  }
  
  // Case 3: Current settings are good
  return {
    newTimeoutMs: observed.currentTimeoutMs,
    reasoning: 'Current timeout is well-tuned for observed latency and SLO.'
  };
}

Timeouts and Availability

Aggressive timeouts improve availability from the user's perspective: a fast error allows retry or fallback, while a slow hang provides no feedback. Consider timeout failures as a form of graceful degradation, not just errors.

The Iterative Tuning Process

Timeout tuning is not a one-time activity. As systems evolve—traffic patterns change, new features add latency, infrastructure scales—timeout values that were once optimal may become inappropriate. The continuous tuning cycle:

Converting Mermaid diagram...

Tuning Process Steps

•Baseline measurement: Collect at least 1 week of latency data to capture normal variation, including peak traffic periods.
•Initial timeout calculation: Using P99 × 1.5 (or appropriate percentile/multiplier for your criticality).
•Staged rollout: Deploy new timeouts to a canary first, then gradually roll out while monitoring.
•Monitor key metrics: Track timeout rate, error rate, latency percentiles, and resource consumption.
•Adjust based on observations: If timeout rate is too high, increase timeout or investigate latency. If very low, consider tightening.
•Document decisions: Record why timeouts are set to current values for future engineers.
•Schedule review: Set reminders to review timeouts quarterly or after significant changes.

When to re-tune: After deployment of new features that might affect latency After infrastructure changes (new regions, different instance types) When timeout error rates change significantly When SLOs change (tighter latency requirements) Seasonal traffic patterns (higher load \to higher latency \to potentially need different timeouts) After optimizations (if P99 dropped significantly, timeout might be too generous) Automated tuning (advanced): Some organizations implement automated timeout adjustment based on observed latency. This is powerful but risky—ensure guardrails exist to prevent runaway adjustments:

auto-tuning-guardrails
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
interface AutoTuningConfig {
  // Bounds to prevent runaway adjustments
  minTimeoutMs: number;
  maxTimeoutMs: number;
  
  // Rate limiting on changes
  maxAdjustmentPercentPerWindow: number;  // e.g., 20%
  adjustmentWindowHours: number;           // e.g., 24 hours
  
  // Confidence requirements
  minSampleSize: number;                   // Need enough data
  stabilityWindowHours: number;            // Latency must be stable
}
 
interface TimeoutAdjustment {
  currentTimeoutMs: number;
  proposedTimeoutMs: number;
  adjustmentPercent: number;
  confidence: 'high' | 'medium' | 'low';
  blocked: boolean;
  blockReason?: string;
}
 
function proposeTimeoutAdjustment(
  currentTimeout: number,
  targetTimeout: number,
  config: AutoTuningConfig,
  metrics: { sampleSize: number; latencyStdDev: number }
): TimeoutAdjustment {
  const adjustmentPercent = ((targetTimeout - currentTimeout) / currentTimeout) * 100;
  
  // Guard 1: Min/max bounds
  let proposedTimeout = Math.max(
    config.minTimeoutMs,
    Math.min(config.maxTimeoutMs, targetTimeout)
  );
  
  // Guard 2: Rate limit adjustments
  if (Math.abs(adjustmentPercent) > config.maxAdjustmentPercentPerWindow) {
    const cappedAdjustment = Math.sign(adjustmentPercent) * 
      config.maxAdjustmentPercentPerWindow / 100 * currentTimeout;
    proposedTimeout = currentTimeout + cappedAdjustment;
    
    return {
      currentTimeoutMs: currentTimeout,
      proposedTimeoutMs: Math.round(proposedTimeout),
      adjustmentPercent: (proposedTimeout - currentTimeout) / currentTimeout * 100,
      confidence: 'medium',
      blocked: false,
      blockReason: `Capped adjustment from ${adjustmentPercent.toFixed(1)}% to ${config.maxAdjustmentPercentPerWindow}%`
    };
  }
  
  // Guard 3: Require sufficient sample size
  if (metrics.sampleSize < config.minSampleSize) {
    return {
      currentTimeoutMs: currentTimeout,
      proposedTimeoutMs: currentTimeout,
      adjustmentPercent: 0,
      confidence: 'low',
      blocked: true,
      blockReason: `Insufficient samples: ${metrics.sampleSize} < ${config.minSampleSize}`
    };
  }
  
  // Guard 4: Block if latency is too variable
  const coefficientOfVariation = metrics.latencyStdDev / targetTimeout;
  if (coefficientOfVariation > 0.5) {
    return {
      currentTimeoutMs: currentTimeout,
      proposedTimeoutMs: currentTimeout,
      adjustmentPercent: 0,
      confidence: 'low',
      blocked: true,
      blockReason: `Latency too variable (CV: ${coefficientOfVariation.toFixed(2)})`
    };
  }
  
  return {
    currentTimeoutMs: currentTimeout,
    proposedTimeoutMs: Math.round(proposedTimeout),
    adjustmentPercent: (proposedTimeout - currentTimeout) / currentTimeout * 100,
    confidence: 'high',
    blocked: false
  };
}

Human Oversight Required

Even with automated tuning, maintain human oversight. Automated systems can make locally optimal decisions that are globally harmful. Review automated timeout changes regularly and investigate unexpected adjustments.

Common Timeout Tuning Mistakes

Even experienced engineers make mistakes when tuning timeouts. Here are the most common pitfalls and how to avoid them.

Common Mistakes

•Using average latency — Ignores tail behavior
•Copying from other services — Different services have different needs
•Setting once and forgetting — Latency evolves
•Very long timeouts 'to be safe' — Defeats timeout purpose
•Same timeout for all endpoints — /health check ≠ /report
•Not testing timeout behavior — Discovery in production

Better Approaches

•Use P99/P99.9 — Account for tail latency
•Measure your own latency — Base on actual data
•Review quarterly — Adjust for changes
•Size to SLA + headroom — Fail fast philosophy
•Per-endpoint configuration — Match operation characteristics
•Chaos testing — Verify behavior under failure

Mistake deep dive: The 30-second timeout trap Many systems default to 30-second timeouts because it feels "generous" and unlikely to cause false positives. This is problematic: Resource exhaustion: If a dependency is slow, you'll hold threads for 30 seconds each. At 100 requests/second, you need 3,000 threads just for waiting. Cascade propagation: Your 30-second timeout means your callers must wait 30+ seconds. Their callers wait longer. The entire system slows. User experience: No user wants to wait 30 seconds. If they're still waiting, they've already left. False sense of security: 30 seconds isn't "safe"—it's just a different kind of failure. You've traded fast failure for slow failure. The fix: If operations genuinely take 30+ seconds, they shouldn't be synchronous. Use async/background processing with status polling.

Timeout Anti-Patterns
Anti-Pattern	Symptom	Fix
30+ second default	Thread pool exhaustion during dependency issues	Reduce to P99 × 1.5 or move to async
Single global timeout	Slow endpoints block fast ones	Per-endpoint timeout configuration
Timeout = SLA exactly	No room for response transmission	Timeout = SLA + overhead buffer
Ignoring retries in budget	Total wait = attempts × timeout	Budget across retries, not per attempt
No timeout on cache calls	Cache becomes system SPOF	Even 'fast' systems need timeouts

Start Aggressive, Loosen Carefully

It's easier to increase a timeout that's too short (you'll see timeouts in monitoring) than to decrease one that's too long (you won't notice the waste until a crisis). Start with aggressive timeouts and loosen based on data.

Observability for Timeout Tuning

Effective timeout tuning requires robust observability. You need to see latency distributions, timeout rates, and their correlation with system health. Essential metrics:

Latency Metrics

•request_duration_seconds — Histogram of request latency, labeled by endpoint/operation
•downstream_call_duration_seconds — Histogram of calls to each downstream service
•connection_time_seconds — Time spent establishing connections
•queue_wait_time_seconds — Time requests wait for resources (thread pool, connection pool)

Timeout Metrics

•timeout_total — Counter of timeout events, labeled by operation and timeout type
•timeout_rate — Percentage of requests that timeout (derived metric)
•timeout_at_ms — Record what timeout value was configured when timeout occurred
•time_to_timeout_ms — How long request ran before timing out (for near-misses)

timeout-metrics
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Example Prometheus metrics for timeout tuning
 
# Latency histogram - enables percentile calculation
- name: http_request_duration_seconds
  type: histogram
  labels: [method, path, status]
  buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
 
# Downstream call latency
- name: downstream_request_duration_seconds
  type: histogram
  labels: [target_service, method]
  buckets: [.01, .05, .1, .25, .5, 1, 2.5, 5, 10, 30]
 
# Timeout counter
- name: request_timeout_total
  type: counter
  labels: [operation, timeout_type] # timeout_type: connection, read, total
 
# Configured timeout value (for correlation)
- name: timeout_configured_seconds
  type: gauge
  labels: [target_service, timeout_type]
 
# Near-miss tracking (requests that almost timed out)
- name: request_time_to_deadline_seconds
  type: histogram
  labels: [operation]
  buckets: [.01, .05, .1, .25, .5, 1, 2, 5] # Time remaining when complete

Dashboard Essentials

Create a dashboard that shows, for each downstream service: (1) current configured timeout, (2) P50/P90/P99 latency, (3) timeout rate, and (4) latency vs timeout as a time series. This visualization immediately shows when timeouts are misconfigured.

5 / 5

Latency Percentiles: What They Tell You

Percentile

Meaning

Use Case

Timeout Implication

P50

Typical request latency

User experience baseline

Not useful for timeout selection

P90

Most requests complete by here

Performance SLAs

Minimum viable timeout for high availability

P95

5% of requests are slower

Baseline for alerting

Conservative timeout floor

P99

1% of requests are slower

SLA boundaries

Good target for most timeouts

P99.9

0.1% are slower

Tail latency optimization

Maximum for latency-sensitive operations

Max

Slowest observed request

Capacity planning

Often too high; affected by outliers

Percentile Selection Guide

Percentile

Timeout at 1.5×

False Positive Rate

Best For

P90

Medium

~10%

Best-effort features, recommendations

P95

Medium-High

~5%

Non-critical APIs, background jobs

P99

High

~1%

Standard production APIs

P99.9

Very High

~0.1%

Critical transaction paths

Observed latency for payment service: P50: 200ms P95: 800ms P99: 1500ms P99.9: 3000ms For a standard API (P99 with 1.5x headroom): Timeout = 1500ms × 1.5 = 2250ms ≈ 2.5 seconds For a critical checkout flow (P99.9 with 2x headroom): Timeout = 3000ms × 2.0 = 6000ms = 6 seconds For a non-critical recommendation (P95 with 1.3x headroom): Timeout = 800ms × 1.3 = 1040ms ≈ 1 second

interface LatencyData { p50: number; p90: number; p95: number; p99: number; p999: number; } interface TimeoutConfig { connectionTimeoutMs: number; readTimeoutMs: number; totalTimeoutMs: number; } type Criticality = 'best-effort' | 'standard' | 'critical'; /** * Calculate timeout values based on latency percentiles */ function calculateTimeouts( latency: LatencyData, criticality: Criticality = 'standard' ): TimeoutConfig { const config = { 'best-effort': { percentile: 'p95', multiplier: 1.3 }, 'standard': { percentile: 'p99', multiplier: 1.5 }, 'critical': { percentile: 'p999', multiplier: 2.0 }, }[criticality]; const baseLatency = latency[config.percentile as keyof LatencyData]; const readTimeout = Math.ceil(baseLatency * config.multiplier); // Connection timeout is typically 10-20% of read timeout, minimum 500ms const connectionTimeout = Math.max(500, Math.ceil(readTimeout * 0.15)); // Total timeout is read timeout plus connection plus buffer const totalTimeout = connectionTimeout + readTimeout + 500; return { connectionTimeoutMs: connectionTimeout, readTimeoutMs: readTimeout, totalTimeoutMs: totalTimeout, }; } // Example usage const paymentServiceLatency: LatencyData = { p50: 200, p90: 600, p95: 800, p99: 1500, p999: 3000, }; console.log('Best-effort:', calculateTimeouts(paymentServiceLatency, 'best-effort')); // { connectionTimeoutMs: 500, readTimeoutMs: 1040, totalTimeoutMs: 2040 } console.log('Standard:', calculateTimeouts(paymentServiceLatency, 'standard')); // { connectionTimeoutMs: 500, readTimeoutMs: 2250, totalTimeoutMs: 3250 } console.log('Critical:', calculateTimeouts(paymentServiceLatency, 'critical')); // { connectionTimeoutMs: 900, readTimeoutMs: 6000, totalTimeoutMs: 7400 }

Checkout Flow: 1. Cart validation (local): ~100ms 2. Inventory reservation: ~500ms typical, 2s P99 3. Fraud check (external): ~1s typical, 3s P99 4. Payment processing (external): ~1.5s typical, 5s P99 5. Order creation (database): ~200ms typical, 1s P99 6. Notification trigger (async): fire-and-forget Budget allocation (Critical Path Priority): - Reserved for overhead: 500ms - Inventory reservation: 1.5s (P99 × 0.75, non-critical) - Fraud check: 2s (important but can fallback) - Payment processing: 5s (critical, uses P99 exactly) - Order creation: 1s (P99, must succeed) - Total allocated: 10s ✓ Fallback behavior: - If fraud check timesout → proceed with basic checks (accept slight risk) - If inventory timesout → show error, don't charge card - If payment timesout → show pending status, investigate

SLO: P99 response time ≤ 500ms Error budget: 1% of requests can exceed this Implication: - Requests targeting the P99 SLO must complete within 500ms - Timeouts must allow successful requests to finish - But timeout > SLO means successful requests still violate SLO Strategy: - Set timeout slightly above SLO (e.g., 600ms) - Requests that take 500-600ms: succeed but violate SLO (counted against budget) - Requests that take >600ms: timeout (also counted against budget) - This prioritizes quick failure over slow success

Timeout Tuning Based on SLO

SLO Target

Timeout Range

Rationale

P99 ≤ 100ms

100-150ms

Aggressive; any slow request is an SLO violation anyway

P99 ≤ 500ms

500-750ms

Allow slightly slow requests to complete

P99 ≤ 2s

2-3s

Give requests chance to complete; fail fast if hopeless

P99 ≤ 30s (batch)

30-45s

Long operations need proportionally more headroom

interface SLOConfig { latencyTargetMs: number; // e.g., 500 for P99 ≤ 500ms successRateTarget: number; // e.g., 0.999 for 99.9% allowedTimeoutRate: number; // Portion of error budget for timeouts } interface ObservedMetrics { latencyP99Ms: number; currentTimeoutMs: number; currentTimeoutRate: number; // % of requests timing out currentSuccessRate: number; } /** * Recommend timeout adjustments based on SLO compliance */ function recommendTimeoutAdjustment( slo: SLOConfig, observed: ObservedMetrics ): { newTimeoutMs: number; reasoning: string } { const maxTimeoutRate = (1 - slo.successRateTarget) * slo.allowedTimeoutRate; // Case 1: Timeouts too frequent, exceeding budget if (observed.currentTimeoutRate > maxTimeoutRate) { // Check if increasing timeout would help if (observed.latencyP99Ms < slo.latencyTargetMs) { // P99 is within SLA, so increasing timeout is safe const newTimeout = Math.min( observed.latencyP99Ms * 1.5, slo.latencyTargetMs * 1.2 ); return { newTimeoutMs: Math.ceil(newTimeout), reasoning: `Timeout rate (${(observed.currentTimeoutRate * 100).toFixed(2)}%) exceeds budget. P99 (${observed.latencyP99Ms}ms) is healthy. Increase timeout to ${Math.ceil(newTimeout)}ms.` }; } else { // P99 exceeds SLA - the service is too slow return { newTimeoutMs: slo.latencyTargetMs, reasoning: `Timeout rate high, but P99 (${observed.latencyP99Ms}ms) exceeds SLO (${slo.latencyTargetMs}ms). Service needs optimization, not longer timeout.` }; } } // Case 2: Timeouts rare, using little of error budget if (observed.currentTimeoutRate < maxTimeoutRate * 0.1) { // May be able to tighten timeout if (observed.currentTimeoutMs > observed.latencyP99Ms * 2) { const newTimeout = Math.max( observed.latencyP99Ms * 1.5, slo.latencyTargetMs ); return { newTimeoutMs: Math.ceil(newTimeout), reasoning: `Timeout rate very low. Consider tightening from ${observed.currentTimeoutMs}ms to ${Math.ceil(newTimeout)}ms for faster failure detection.` }; } } // Case 3: Current settings are good return { newTimeoutMs: observed.currentTimeoutMs, reasoning: 'Current timeout is well-tuned for observed latency and SLO.' }; }

interface AutoTuningConfig { // Bounds to prevent runaway adjustments minTimeoutMs: number; maxTimeoutMs: number; // Rate limiting on changes maxAdjustmentPercentPerWindow: number; // e.g., 20% adjustmentWindowHours: number; // e.g., 24 hours // Confidence requirements minSampleSize: number; // Need enough data stabilityWindowHours: number; // Latency must be stable } interface TimeoutAdjustment { currentTimeoutMs: number; proposedTimeoutMs: number; adjustmentPercent: number; confidence: 'high' | 'medium' | 'low'; blocked: boolean; blockReason?: string; } function proposeTimeoutAdjustment( currentTimeout: number, targetTimeout: number, config: AutoTuningConfig, metrics: { sampleSize: number; latencyStdDev: number } ): TimeoutAdjustment { const adjustmentPercent = ((targetTimeout - currentTimeout) / currentTimeout) * 100; // Guard 1: Min/max bounds let proposedTimeout = Math.max( config.minTimeoutMs, Math.min(config.maxTimeoutMs, targetTimeout) ); // Guard 2: Rate limit adjustments if (Math.abs(adjustmentPercent) > config.maxAdjustmentPercentPerWindow) { const cappedAdjustment = Math.sign(adjustmentPercent) * config.maxAdjustmentPercentPerWindow / 100 * currentTimeout; proposedTimeout = currentTimeout + cappedAdjustment; return { currentTimeoutMs: currentTimeout, proposedTimeoutMs: Math.round(proposedTimeout), adjustmentPercent: (proposedTimeout - currentTimeout) / currentTimeout * 100, confidence: 'medium', blocked: false, blockReason: `Capped adjustment from ${adjustmentPercent.toFixed(1)}% to ${config.maxAdjustmentPercentPerWindow}%` }; } // Guard 3: Require sufficient sample size if (metrics.sampleSize < config.minSampleSize) { return { currentTimeoutMs: currentTimeout, proposedTimeoutMs: currentTimeout, adjustmentPercent: 0, confidence: 'low', blocked: true, blockReason: `Insufficient samples: ${metrics.sampleSize} < ${config.minSampleSize}` }; } // Guard 4: Block if latency is too variable const coefficientOfVariation = metrics.latencyStdDev / targetTimeout; if (coefficientOfVariation > 0.5) { return { currentTimeoutMs: currentTimeout, proposedTimeoutMs: currentTimeout, adjustmentPercent: 0, confidence: 'low', blocked: true, blockReason: `Latency too variable (CV: ${coefficientOfVariation.toFixed(2)})` }; } return { currentTimeoutMs: currentTimeout, proposedTimeoutMs: Math.round(proposedTimeout), adjustmentPercent: (proposedTimeout - currentTimeout) / currentTimeout * 100, confidence: 'high', blocked: false }; }

Timeout Anti-Patterns

Anti-Pattern

Symptom

Fix

30+ second default

Thread pool exhaustion during dependency issues

Reduce to P99 × 1.5 or move to async

Single global timeout

Slow endpoints block fast ones

Per-endpoint timeout configuration

Timeout = SLA exactly

No room for response transmission

Timeout = SLA + overhead buffer

Ignoring retries in budget

Total wait = attempts × timeout

Budget across retries, not per attempt

No timeout on cache calls

Cache becomes system SPOF

Even 'fast' systems need timeouts

# Example Prometheus metrics for timeout tuning # Latency histogram - enables percentile calculation - name: http_request_duration_seconds type: histogram labels: [method, path, status] buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10] # Downstream call latency - name: downstream_request_duration_seconds type: histogram labels: [target_service, method] buckets: [.01, .05, .1, .25, .5, 1, 2.5, 5, 10, 30] # Timeout counter - name: request_timeout_total type: counter labels: [operation, timeout_type] # timeout_type: connection, read, total # Configured timeout value (for correlation) - name: timeout_configured_seconds type: gauge labels: [target_service, timeout_type] # Near-miss tracking (requests that almost timed out) - name: request_time_to_deadline_seconds type: histogram labels: [operation] buckets: [.01, .05, .1, .25, .5, 1, 2, 5] # Time remaining when complete

Timeout Tuning

The Art and Science of Timeout Selection

Understanding Latency Distributions

The Percentile-Based Tuning Method

Timeout Budgeting Across Call Chains

SLA-Driven Timeout Tuning

The Iterative Tuning Process

Common Timeout Tuning Mistakes

Observability for Timeout Tuning

There’s more to understand.

Timeout Tuning

The Art and Science of Timeout Selection

Understanding Latency Distributions

The Percentile-Based Tuning Method

Timeout Budgeting Across Call Chains

SLA-Driven Timeout Tuning

The Iterative Tuning Process

Common Timeout Tuning Mistakes

Observability for Timeout Tuning

There’s more to understand.