System Design (HLD)Retry with Backoff

Retry with Backoff: Building Resilient Distributed Systems

LevelAdvanced

Duration75 mins

TopicRetry with Backoff

1 / 5

When to Retry: The Decision Framework

The Fundamental Question of Distributed Reliability

In a perfect world, every remote call would succeed on the first attempt. Reality, however, is far more complex. Networks experience transient congestion, services momentarily overload, and infrastructure components undergo brief interruptions. The question facing every distributed systems engineer isn't whether failures will occur, but how to respond when they do. Retry logic is one of the most powerful—and most dangerous—tools in your resilience toolkit. Implemented correctly, retries allow systems to automatically recover from transient failures without human intervention, providing seamless resilience that users never notice. Implemented incorrectly, retries can transform minor hiccups into cascading catastrophes, amplifying load during outages and turning recoverable situations into system-wide meltdowns. This page establishes the fundamental decision framework for retry logic: understanding when to retry, why that distinction matters, and how to classify failures accurately. Before you ever consider backoff algorithms or jitter strategies, you must first master this critical question: Should you retry at all?

What You Will Learn

By the end of this page, you will understand the distinction between transient and permanent failures, master a systematic framework for retry decisions, recognize the dangers of inappropriate retries, and know how to classify different failure types for optimal retry strategies. This knowledge forms the foundation for all subsequent retry pattern discussions.

The Nature of Failures in Distributed Systems

Before discussing retry strategies, we must understand the taxonomy of failures in distributed systems. Not all failures are created equal, and treating them identically leads to either missed recovery opportunities or catastrophic retry storms. The Fundamental Distinction: Transient vs Permanent Every failure in a distributed system falls into one of two broad categories: Transient failures are temporary conditions that resolve spontaneously over time. The underlying cause—network congestion, temporary resource exhaustion, brief service restart—is self-correcting. If you simply wait and try again, the request will likely succeed. These are the ideal candidates for retry. Permanent failures represent conditions that will not change regardless of how long you wait or how many times you retry. A malformed request, an invalid authentication token, a deleted resource—these errors indicate fundamental problems that require different handling: code changes, user intervention, or graceful degradation.

Transient vs Permanent Failure Characteristics
Characteristic	Transient Failures	Permanent Failures
Self-correcting	Yes - resolve without intervention	No - require external action
Time-dependent	Likely to succeed if retried later	Will fail indefinitely
Root cause	Temporary system conditions	Fundamental request/state problems
Retry value	High - automatic recovery	None - wastes resources
Correct response	Retry with backoff	Fail fast, escalate, or degrade
Examples	Network timeout, 503 Service Unavailable	400 Bad Request, 404 Not Found

Why This Distinction Matters Operationally Misclassifying failures has severe consequences in both directions: Treating permanent failures as transient (retry what shouldn't be retried): Wastes compute resources on futile attempts Increases latency for users waiting for inevitable failure Generates unnecessary load on already-stressed dependencies Pollutes logs and metrics with noise May exhaust retry budgets needed for genuine transient failures Treating transient failures as permanent (fail fast when you should retry): Converts recoverable situations into user-visible errors Reduces system availability unnecessarily Forces human intervention for self-healing issues Undermines user confidence in system reliability May trigger unnecessary on-call pages

The Classification Challenge

In practice, classifying failures isn't always straightforward. A timeout could indicate transient network congestion (retry) or a permanently deadlocked service (don't retry). An HTTP 500 might be a momentary glitch or a fundamental bug. The art of retry policy design lies in making classification decisions that are correct in the common case while minimizing harm when classification is wrong.

Transient Failures: The Retry Sweet Spot

Transient failures are the primary target for retry logic—situations where the same request, made moments later, has a reasonable probability of success. Understanding common transient failure patterns helps you design effective retry policies. Network-Level Transient Failures Network infrastructure introduces numerous opportunities for transient failures: Connection timeouts : The remote server exists and is healthy, but momentary network congestion prevents establishing a connection within the timeout window. Read/write timeouts : A connection was established but data transfer stalled due to temporary bandwidth constraints or packet loss. Connection resets : TCP RST packets received due to transient network equipment issues, NAT table expirations, or brief service interruptions. DNS resolution failures : Temporary DNS server unreachability or propagation delays. TLS handshake failures : Transient issues during secure connection establishment, often related to clock skew or temporary certificate validation issues.

Common Transient Failure Scenarios

•Service Overload (429, 503) — The service is temporarily overwhelmed but actively shedding load. Retrying after a delay allows time for recovery.
•Resource Contention — Database connection pools exhausted, thread pools full, or memory pressure. These conditions often resolve as competing requests complete.
•Deployment Transitions — Brief unavailability during rolling deployments when old instances terminate before new ones become healthy.
•Load Balancer Re-routing — Momentary failures as load balancers detect unhealthy backends and route to healthy alternatives.
•Circuit Breaker Reset — Upstream circuit breakers temporarily rejecting requests but will soon allow testing traffic.
•Rate Limit Bucket Replenishment — Rate limits that reset on a time window boundary, allowing retries after the window rolls.
•Leader Election — Distributed systems temporarily unavailable during consensus leader changes.
•Garbage Collection Pauses — JVM-based services experiencing stop-the-world GC events causing temporary unresponsiveness.

Identifying Transient Failures Through HTTP Status Codes HTTP response codes provide valuable signals for retry decisions, though they must be interpreted carefully: Clearly Retryable (Transient): 408 Request Timeout — Server timed out waiting for request 429 Too Many Requests — Rate limiting; often includes Retry-After header 502 Bad Gateway — Upstream server returned invalid response 503 Service Unavailable — Server temporarily unable to handle request 504 Gateway Timeout — Upstream server didn't respond in time Contextually Retryable: 500 Internal Server Error — May be transient (OOM, deadlock) or permanent (bug) Connection refused — May be transient (restart) or permanent (misconfiguration) Generally Not Retryable: 400 Bad Request — Request is malformed; retry will produce same result 401 Unauthorized — Authentication failed; retry won't fix credentials 403 Forbidden — Authorization denied; retry won't grant permissions 404 Not Found — Resource doesn't exist; retry won't create it 409 Conflict — State conflict that likely requires resolution 422 Unprocessable Entity — Semantic errors in request payload

The Retry-After Header

When services return 429 or 503 responses, they often include a Retry-After header indicating how long to wait before retrying. Always respect this header when present—it provides server-side intelligence about when retry is appropriate. Ignoring it and retrying immediately often makes the situation worse.

Permanent Failures: When Retries Harm

Not all failures deserve retries. Permanent failures—conditions that will persist regardless of time or retry attempts—must be handled differently. Retrying permanent failures is not merely wasteful; it's actively harmful. Categories of Permanent Failures Client-Side Errors (4xx responses) Most 4xx HTTP status codes indicate problems with the request itself. The same request, made identically, will fail identically: Validation failures (400) : Request syntax is malformed or violates business rules Authentication failures (401) : Credentials are missing, expired, or invalid Authorization failures (403) : User lacks permission for the requested operation Resource not found (404) : The target resource doesn't exist Method not allowed (405) : The HTTP method isn't supported for this resource Payload too large (413) : Request body exceeds server limits Unsupported media type (415) : Content-Type not accepted

Why Retrying Permanent Failures Is Harmful

•Amplified Load on Failing Services — A service returning errors is already under stress. Retry storms multiply the number of requests it must process and reject, potentially converting partial outages into complete failures.
•Delayed User Feedback — Users waiting for retries to exhaust experience increased latency with no benefit. A request that will ultimately fail should fail fast.
•Wasted Compute Resources — Every futile retry consumes CPU, memory, network bandwidth, and connection pool slots that could serve valid requests.
•Polluted Observability — Logs fill with repeated failures, metrics spike with noise, and genuine signals become harder to distinguish from retry artifacts.
•Exhausted Retry Budgets — Some systems limit total retries. Burning retries on permanent failures leaves none for genuine transient issues.
•Cascading Resource Starvation — Threads blocked waiting for retry delays can starve other operations, spreading failure beyond the original scope.

Semantic Errors and Business Logic Failures Beyond HTTP status codes, many permanent failures are semantic or business-logic level: Insufficient funds : Retrying a payment won't change the account balance Inventory exhaustion : Retrying a purchase won't restock the item Duplicate operations : Retrying a unique constraint violation won't make it unique Expired offers : Retrying after a deadline won't extend the deadline Version conflicts : Retrying optimistic locking failures without re-reading state won't resolve the conflict The Fail-Fast Imperative For permanent failures, the correct strategy is fail fast : immediately propagate the error to the caller, allowing them to handle it appropriately. This might mean: Returning a clear error message to the user Triggering compensating transactions Logging for debugging and alerting Falling back to alternative flows Queuing for human review Fail-fast preserves resources, provides quicker feedback, and allows the system to focus on requests that can actually succeed.

The Retry Storm Catastrophe

During a major cloud provider outage, a single misconfigured retry policy multiplied error traffic by 100x. A service returning 503 errors was bombarded with retries from thousands of clients, each configured to retry 10 times with minimal delay. The resulting traffic prevented the service from recovering even after the underlying issue was resolved. The outage extended from minutes to hours solely due to inappropriate retry behavior.

The Retry Decision Framework

With the understanding of transient vs permanent failures, we can establish a systematic decision framework for retry behavior. This framework should be applied consistently across your distributed system. Step 1: Classify the Failure Before deciding to retry, classify the failure using all available information: HTTP status code : The most immediate signal, but not always definitive Error response body : Often contains error codes or messages indicating permanence Error type/exception : Programming language exceptions may indicate transience (timeout exceptions) or permanence (validation exceptions) Request context : Was this a read or write? Reads are generally safer to retry Service-specific knowledge : Some services define custom retryable error codes

retry-decision.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
// Retry decision framework implementation
interface RetryDecision {
    shouldRetry: boolean;
    reason: string;
    suggestedDelayMs?: number;
}
 
type FailureCategory = 'transient' | 'permanent' | 'unknown';
 
interface FailureClassification {
    category: FailureCategory;
    confidence: 'high' | 'medium' | 'low';
    evidence: string;
}
 
function classifyFailure(
    statusCode: number | undefined,
    errorCode: string | undefined,
    errorMessage: string | undefined,
    exception: Error | undefined
): FailureClassification {
    // HTTP status code classification
    if (statusCode !== undefined) {
        // Definitely transient
        if ([408, 429, 502, 503, 504].includes(statusCode)) {
            return {
                category: 'transient',
                confidence: 'high',
                evidence: `HTTP ${statusCode} is explicitly transient`
            };
        }
        
        // Definitely permanent (client errors)
        if (statusCode >= 400 && statusCode < 500 && statusCode !== 408 && statusCode !== 429) {
            return {
                category: 'permanent',
                confidence: 'high',
                evidence: `HTTP ${statusCode} indicates client error`
            };
        }
        
        // Ambiguous server error
        if (statusCode === 500) {
            return {
                category: 'unknown',
                confidence: 'low',
                evidence: 'HTTP 500 may be transient or permanent'
            };
        }
    }
    
    // Exception type classification
    if (exception) {
        const exceptionName = exception.name || exception.constructor.name;
        
        const transientExceptions = [
            'TimeoutError', 'ConnectionError', 'NetworkError',
            'ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED'
        ];
        
        if (transientExceptions.some(te => exceptionName.includes(te))) {
            return {
                category: 'transient',
                confidence: 'high',
                evidence: `Exception ${exceptionName} is typically transient`
            };
        }
        
        const permanentExceptions = [
            'ValidationError', 'AuthenticationError',
            'AuthorizationError', 'NotFoundError'
        ];
        
        if (permanentExceptions.some(pe => exceptionName.includes(pe))) {
            return {
                category: 'permanent',
                confidence: 'high',
                evidence: `Exception ${exceptionName} is typically permanent`
            };
        }
    }
    
    // Error code classification (service-specific)
    if (errorCode) {
        const transientCodes = ['RESOURCE_EXHAUSTED', 'UNAVAILABLE', 'DEADLINE_EXCEEDED'];
        const permanentCodes = ['INVALID_ARGUMENT', 'NOT_FOUND', 'PERMISSION_DENIED'];
        
        if (transientCodes.includes(errorCode)) {
            return { category: 'transient', confidence: 'high', evidence: `Error code ${errorCode}` };
        }
        if (permanentCodes.includes(errorCode)) {
            return { category: 'permanent', confidence: 'high', evidence: `Error code ${errorCode}` };
        }
    }
    
    // Default to unknown with low confidence
    return {
        category: 'unknown',
        confidence: 'low',
        evidence: 'Unable to classify failure from available signals'
    };
}

Step 2: Apply the Retry Decision Matrix Once classified, apply the decision matrix based on failure category and operation characteristics:

Retry Decision Matrix
Failure Category	Operation Type	Decision	Rationale
Transient (high confidence)	Read	Retry	Safe, likely to succeed
Transient (high confidence)	Write (idempotent)	Retry	Safe with idempotency key
Transient (high confidence)	Write (non-idempotent)	Conditional retry	Only if timeout/connection failure
Permanent (high confidence)	Any	Fail fast	Retry will not help
Unknown (low confidence)	Read	Limited retry	1-2 retries to probe
Unknown (low confidence)	Write	No retry	Risk of duplication too high

Step 3: Check Retry Preconditions Even when failure classification suggests retry, additional preconditions must be met: Retry budget remaining : Haven't exceeded maximum retry attempts Time budget remaining : Total elapsed time within acceptable latency bounds Circuit breaker open : Upstream circuit isn't open (would reject retry immediately) Idempotency assured : For writes, idempotency key is present or operation is naturally idempotent Client still waiting : Connection hasn't been closed or cancelled Retry-After respected : If server provided delay, sufficient time has passed

The Unknown Category Strategy

When failure classification is uncertain, adopt a conservative posture. For unknown failures on read operations, allow 1-2 retries with backoff—this provides opportunity to recover from transient issues while limiting damage if permanent. For unknown failures on write operations, fail fast unless idempotency is guaranteed. The cost of duplicate writes typically exceeds the cost of a false negative on retry.

Operation Characteristics and Retry Safety

Beyond failure classification, the nature of the operation itself profoundly influences retry safety. The same transient failure may warrant retry for one operation but not another. Read vs Write Operations Read operations (GET, HEAD, OPTIONS) are inherently safer to retry because they don't modify state. Even if a read is executed multiple times due to retry, the result is merely redundant work, not data corruption. This is why HTTP considers these methods safe and idempotent . Write operations (POST, PUT, DELETE, PATCH) carry the risk of duplicate execution. If a request succeeds but the response is lost (timeout, connection reset), the client doesn't know whether to retry. Retrying could create duplicate records, double-charge payments, or send multiple notifications.

Safe to Retry

•All read operations — GET, HEAD, OPTIONS, TRACE
•Naturally idempotent writes — PUT that sets absolute state, DELETE of specific resource
•Writes with idempotency keys — POST/PATCH with client-generated unique identifier
•Timeout failures on reads — The operation may or may not have completed, but re-reading is safe
•Connection failures before request sent — Request definitely not processed

Dangerous to Retry

•Non-idempotent writes without keys — POST creating resources, triggering workflows
•Timeout failures on writes — Request may have succeeded; retry = duplicate
•Partial success scenarios — Some side effects completed before failure
•Stateful sequences — Operations that depend on prior state changes
•Financial transactions — Payments, transfers without deduplication

The Timeout Dilemma Timeout failures present the most challenging retry decision because the request's fate is unknown . Consider this scenario: Client sends POST request to create an order Server receives and processes request, creates the order Server begins sending response Network interruption: response never reaches client Client experiences timeout after 30 seconds From the client's perspective, it's impossible to distinguish this case (order created) from one where the server never received the request (order not created). Retrying could create a duplicate order. Resolution strategies: Idempotency keys : Include a unique request ID; server rejects/returns duplicates Conditional operations : Use ETags or version numbers for concurrency control Two-phase operations : Reserve, then commit with confirmation Query before retry : Check if the operation succeeded before retrying Accept eventual consistency : Design downstream to handle duplicates gracefully

The Idempotency Prerequisite

As a general rule: never implement aggressive retry policies for write operations unless you've also implemented idempotency. The two must be designed together. We'll explore idempotency requirements in depth in a later page of this module.

Context-Aware Retry Policies

Effective retry policies aren't one-size-fits-all. Different operations, services, and contexts warrant different retry behaviors. This section explores how to design context-aware policies. Service-Level Differentiation Not all dependencies are equally important or equally likely to experience transient failures: Critical path services (payment, authentication): May warrant more aggressive retry for transient failures, but stricter classification to avoid retry storms Best-effort services (analytics, recommendations): May tolerate faster failure with fewer retries External APIs : Often have rate limits that require respecting Retry-After headers Internal microservices : May allow more aggressive retry given controlled environment

context-aware-retry.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
// Context-aware retry policy configuration
interface RetryPolicyConfig {
    maxAttempts: number;
    baseDelayMs: number;
    maxDelayMs: number;
    retryableStatuses: number[];
    retryableExceptions: string[];
    respectRetryAfter: boolean;
    nonRetryableStatuses: number[];
}
 
// Different policies for different contexts
const retryPolicies: Record<string, RetryPolicyConfig> = {
    // Critical payment service: more retries, careful classification
    paymentService: {
        maxAttempts: 5,
        baseDelayMs: 100,
        maxDelayMs: 5000,
        retryableStatuses: [408, 429, 502, 503, 504],
        retryableExceptions: ['TimeoutError', 'ConnectionError'],
        respectRetryAfter: true,
        nonRetryableStatuses: [400, 401, 402, 403, 404, 409, 422],
    },
    
    // Best-effort analytics: fast fail, minimal retry
    analyticsService: {
        maxAttempts: 2,
        baseDelayMs: 50,
        maxDelayMs: 500,
        retryableStatuses: [502, 503, 504],
        retryableExceptions: ['TimeoutError'],
        respectRetryAfter: false,
        nonRetryableStatuses: [400, 401, 403, 404, 429], // Don't retry rate limiting
    },
    
    // External third-party API: respect their limits
    externalApi: {
        maxAttempts: 3,
        baseDelayMs: 1000,
        maxDelayMs: 60000,
        retryableStatuses: [429, 500, 502, 503, 504],
        retryableExceptions: ['TimeoutError', 'ConnectionError', 'NetworkError'],
        respectRetryAfter: true, // Critical for external APIs
        nonRetryableStatuses: [400, 401, 403, 404, 405],
    },
    
    // Internal microservice: more tolerant
    internalService: {
        maxAttempts: 4,
        baseDelayMs: 100,
        maxDelayMs: 2000,
        retryableStatuses: [408, 429, 500, 502, 503, 504],
        retryableExceptions: ['TimeoutError', 'ConnectionError', 'ECONNRESET'],
        respectRetryAfter: true,
        nonRetryableStatuses: [400, 401, 403, 404],
    },
};
 
// Policy selection based on context
function selectRetryPolicy(
    serviceName: string,
    operationType: 'read' | 'write',
    isCriticalPath: boolean
): RetryPolicyConfig {
    const basePolicy = retryPolicies[serviceName] || retryPolicies.internalService;
    
    // Reduce retry attempts for writes unless idempotent
    if (operationType === 'write') {
        return {
            ...basePolicy,
            maxAttempts: Math.min(basePolicy.maxAttempts, 3),
        };
    }
    
    // Increase attempts for critical path operations
    if (isCriticalPath) {
        return {
            ...basePolicy,
            maxAttempts: Math.min(basePolicy.maxAttempts + 2, 7),
        };
    }
    
    return basePolicy;
}

Request-Level Context Beyond service-level differentiation, individual requests may warrant different treatment: User-facing vs background : User-facing requests may prefer fast failure to keep UI responsive; background jobs may retry more aggressively Remaining deadline : A request near its deadline shouldn't start new retries that can't complete in time Prior failures in session : If a user's session has experienced many failures, additional retries may not improve experience Request priority : High-priority requests may get more generous retry budgets

Deadline-Aware Retries

Always consider the remaining time budget before initiating a retry. If a request has a 5-second deadline and 4.5 seconds have elapsed, starting a retry (with its own potential timeout) is wasteful. The request should fail fast and let the caller decide whether to abandon or start fresh. We'll explore deadline propagation patterns in the Timeout and Deadline module.

Failure Detection and Classification Infrastructure

Accurate failure classification requires proper infrastructure. Without consistent error handling and propagation, retry policies can't make informed decisions. Standardized Error Responses Services should return structured error responses that enable accurate classification: { "error": { "code": "RESOURCE_EXHAUSTED", "message": "Rate limit exceeded for API calls", "details": { "retryable": true, "retryAfterSeconds": 30, "quotaLimit": 1000, "quotaRemaining": 0, "quotaResetsAt": "2024-01-15T10:00:00Z" } } } Key elements for retry classification: Machine-readable error code: Not just HTTP status, but semantic codes like RESOURCE_EXHAUSTED, INVALID_ARGUMENT, DEADLINE_EXCEEDED Explicit retryability signal: When possible, services should indicate whether the error is retryable Retry timing hints: How long to wait before retrying (Retry-After header or response field) Debug information: Context for logging/debugging without affecting retry logic

Building Reliable Classification Infrastructure

•Consistent error taxonomy: Define a standard set of error codes across all services with clear retryability semantics
•HTTP status code discipline: Use status codes consistently to indicate error categories (4xx = don't retry, 503 = retry encouraged)
•Structured logging: Log failures with classifiable metadata, enabling analysis of retry behavior
•Error code registries: Maintain centralized documentation of error codes and their retry implications
•Client library standards: Provide shared retry infrastructure in client libraries to ensure consistent behavior
•Observability integration: Track retry decisions, success rates, and time-to-success for tuning policies

Exception Mapping and Wrapping In code, different infrastructure layers throw different exception types. Establishing a mapping to retryability domains simplifies policy implementation:

exception-classification.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
// Exception classification system
enum ErrorCategory {
    TRANSIENT = 'transient',
    PERMANENT = 'permanent',
    UNKNOWN = 'unknown',
}
 
interface ClassifiedError extends Error {
    category: ErrorCategory;
    originalError?: Error;
    httpStatus?: number;
    errorCode?: string;
    retryAfterMs?: number;
}
 
// Map infrastructure exceptions to categories
function classifyException(error: Error): ClassifiedError {
    const classified: ClassifiedError = Object.assign(
        new Error(error.message),
        {
            name: error.name,
            stack: error.stack,
            category: ErrorCategory.UNKNOWN,
            originalError: error,
        }
    );
    
    // Node.js system errors
    const errorCode = (error as NodeJS.ErrnoException).code;
    if (errorCode === 'ECONNRESET' || 
        errorCode === 'ETIMEDOUT' || 
        errorCode === 'ECONNREFUSED' ||
        errorCode === 'EPIPE' ||
        errorCode === 'EHOSTUNREACH') {
        classified.category = ErrorCategory.TRANSIENT;
        return classified;
    }
    
    // HTTP client errors (from fetch, axios, etc.)
    const httpStatus = (error as any).response?.status;
    if (httpStatus) {
        classified.httpStatus = httpStatus;
        
        if ([408, 429, 502, 503, 504].includes(httpStatus)) {
            classified.category = ErrorCategory.TRANSIENT;
            
            // Extract Retry-After if present
            const retryAfter = (error as any).response?.headers?.['retry-after'];
            if (retryAfter) {
                classified.retryAfterMs = parseRetryAfter(retryAfter);
            }
        } else if (httpStatus >= 400 && httpStatus < 500) {
            classified.category = ErrorCategory.PERMANENT;
        } else if (httpStatus === 500) {
            classified.category = ErrorCategory.UNKNOWN; // 500 is ambiguous
        }
        
        return classified;
    }
    
    // gRPC status codes
    const grpcCode = (error as any).code;
    if (grpcCode !== undefined) {
        const transientGrpcCodes = [
            'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'UNAVAILABLE', 'ABORTED'
        ];
        const permanentGrpcCodes = [
            'INVALID_ARGUMENT', 'NOT_FOUND', 'ALREADY_EXISTS',
            'PERMISSION_DENIED', 'UNAUTHENTICATED'
        ];
        
        if (transientGrpcCodes.includes(grpcCode)) {
            classified.category = ErrorCategory.TRANSIENT;
        } else if (permanentGrpcCodes.includes(grpcCode)) {
            classified.category = ErrorCategory.PERMANENT;
        }
    }
    
    return classified;
}
 
function parseRetryAfter(value: string): number {
    // Retry-After can be seconds or HTTP date
    const seconds = parseInt(value, 10);
    if (!isNaN(seconds)) {
        return seconds * 1000;
    }
    
    const date = new Date(value);
    if (!isNaN(date.getTime())) {
        return Math.max(0, date.getTime() - Date.now());
    }
    
    return 0;
}

1 / 5

Transient vs Permanent Failure Characteristics

Characteristic

Transient Failures

Permanent Failures

Self-correcting

Yes - resolve without intervention

No - require external action

Time-dependent

Likely to succeed if retried later

Will fail indefinitely

Root cause

Temporary system conditions

Fundamental request/state problems

Retry value

High - automatic recovery

None - wastes resources

Correct response

Retry with backoff

Fail fast, escalate, or degrade

Examples

Network timeout, 503 Service Unavailable

400 Bad Request, 404 Not Found

Identifying Transient Failures Through HTTP Status Codes

HTTP response codes provide valuable signals for retry decisions, though they must be interpreted carefully:

Clearly Retryable (Transient):

408 Request Timeout — Server timed out waiting for request
429 Too Many Requests — Rate limiting; often includes Retry-After header
502 Bad Gateway — Upstream server returned invalid response
503 Service Unavailable — Server temporarily unable to handle request
504 Gateway Timeout — Upstream server didn't respond in time

Contextually Retryable:

500 Internal Server Error — May be transient (OOM, deadlock) or permanent (bug)
Connection refused — May be transient (restart) or permanent (misconfiguration)

Generally Not Retryable:

400 Bad Request — Request is malformed; retry will produce same result
401 Unauthorized — Authentication failed; retry won't fix credentials
403 Forbidden — Authorization denied; retry won't grant permissions
404 Not Found — Resource doesn't exist; retry won't create it
409 Conflict — State conflict that likely requires resolution
422 Unprocessable Entity — Semantic errors in request payload

// Retry decision framework implementation interface RetryDecision { shouldRetry: boolean; reason: string; suggestedDelayMs?: number; } type FailureCategory = 'transient' | 'permanent' | 'unknown'; interface FailureClassification { category: FailureCategory; confidence: 'high' | 'medium' | 'low'; evidence: string; } function classifyFailure( statusCode: number | undefined, errorCode: string | undefined, errorMessage: string | undefined, exception: Error | undefined ): FailureClassification { // HTTP status code classification if (statusCode !== undefined) { // Definitely transient if ([408, 429, 502, 503, 504].includes(statusCode)) { return { category: 'transient', confidence: 'high', evidence: `HTTP ${statusCode} is explicitly transient` }; } // Definitely permanent (client errors) if (statusCode >= 400 && statusCode < 500 && statusCode !== 408 && statusCode !== 429) { return { category: 'permanent', confidence: 'high', evidence: `HTTP ${statusCode} indicates client error` }; } // Ambiguous server error if (statusCode === 500) { return { category: 'unknown', confidence: 'low', evidence: 'HTTP 500 may be transient or permanent' }; } } // Exception type classification if (exception) { const exceptionName = exception.name || exception.constructor.name; const transientExceptions = [ 'TimeoutError', 'ConnectionError', 'NetworkError', 'ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED' ]; if (transientExceptions.some(te => exceptionName.includes(te))) { return { category: 'transient', confidence: 'high', evidence: `Exception ${exceptionName} is typically transient` }; } const permanentExceptions = [ 'ValidationError', 'AuthenticationError', 'AuthorizationError', 'NotFoundError' ]; if (permanentExceptions.some(pe => exceptionName.includes(pe))) { return { category: 'permanent', confidence: 'high', evidence: `Exception ${exceptionName} is typically permanent` }; } } // Error code classification (service-specific) if (errorCode) { const transientCodes = ['RESOURCE_EXHAUSTED', 'UNAVAILABLE', 'DEADLINE_EXCEEDED']; const permanentCodes = ['INVALID_ARGUMENT', 'NOT_FOUND', 'PERMISSION_DENIED']; if (transientCodes.includes(errorCode)) { return { category: 'transient', confidence: 'high', evidence: `Error code ${errorCode}` }; } if (permanentCodes.includes(errorCode)) { return { category: 'permanent', confidence: 'high', evidence: `Error code ${errorCode}` }; } } // Default to unknown with low confidence return { category: 'unknown', confidence: 'low', evidence: 'Unable to classify failure from available signals' }; }

Retry Decision Matrix

Failure Category

Operation Type

Decision

Rationale

Transient (high confidence)

Read

Retry

Safe, likely to succeed

Transient (high confidence)

Write (idempotent)

Retry

Safe with idempotency key

Transient (high confidence)

Write (non-idempotent)

Conditional retry

Only if timeout/connection failure

Permanent (high confidence)

Any

Fail fast

Retry will not help

Unknown (low confidence)

Read

Limited retry

1-2 retries to probe

Unknown (low confidence)

Write

No retry

Risk of duplication too high

// Context-aware retry policy configuration interface RetryPolicyConfig { maxAttempts: number; baseDelayMs: number; maxDelayMs: number; retryableStatuses: number[]; retryableExceptions: string[]; respectRetryAfter: boolean; nonRetryableStatuses: number[]; } // Different policies for different contexts const retryPolicies: Record<string, RetryPolicyConfig> = { // Critical payment service: more retries, careful classification paymentService: { maxAttempts: 5, baseDelayMs: 100, maxDelayMs: 5000, retryableStatuses: [408, 429, 502, 503, 504], retryableExceptions: ['TimeoutError', 'ConnectionError'], respectRetryAfter: true, nonRetryableStatuses: [400, 401, 402, 403, 404, 409, 422], }, // Best-effort analytics: fast fail, minimal retry analyticsService: { maxAttempts: 2, baseDelayMs: 50, maxDelayMs: 500, retryableStatuses: [502, 503, 504], retryableExceptions: ['TimeoutError'], respectRetryAfter: false, nonRetryableStatuses: [400, 401, 403, 404, 429], // Don't retry rate limiting }, // External third-party API: respect their limits externalApi: { maxAttempts: 3, baseDelayMs: 1000, maxDelayMs: 60000, retryableStatuses: [429, 500, 502, 503, 504], retryableExceptions: ['TimeoutError', 'ConnectionError', 'NetworkError'], respectRetryAfter: true, // Critical for external APIs nonRetryableStatuses: [400, 401, 403, 404, 405], }, // Internal microservice: more tolerant internalService: { maxAttempts: 4, baseDelayMs: 100, maxDelayMs: 2000, retryableStatuses: [408, 429, 500, 502, 503, 504], retryableExceptions: ['TimeoutError', 'ConnectionError', 'ECONNRESET'], respectRetryAfter: true, nonRetryableStatuses: [400, 401, 403, 404], }, }; // Policy selection based on context function selectRetryPolicy( serviceName: string, operationType: 'read' | 'write', isCriticalPath: boolean ): RetryPolicyConfig { const basePolicy = retryPolicies[serviceName] || retryPolicies.internalService; // Reduce retry attempts for writes unless idempotent if (operationType === 'write') { return { ...basePolicy, maxAttempts: Math.min(basePolicy.maxAttempts, 3), }; } // Increase attempts for critical path operations if (isCriticalPath) { return { ...basePolicy, maxAttempts: Math.min(basePolicy.maxAttempts + 2, 7), }; } return basePolicy; }

{ "error": { "code": "RESOURCE_EXHAUSTED", "message": "Rate limit exceeded for API calls", "details": { "retryable": true, "retryAfterSeconds": 30, "quotaLimit": 1000, "quotaRemaining": 0, "quotaResetsAt": "2024-01-15T10:00:00Z" } } }

// Exception classification system enum ErrorCategory { TRANSIENT = 'transient', PERMANENT = 'permanent', UNKNOWN = 'unknown', } interface ClassifiedError extends Error { category: ErrorCategory; originalError?: Error; httpStatus?: number; errorCode?: string; retryAfterMs?: number; } // Map infrastructure exceptions to categories function classifyException(error: Error): ClassifiedError { const classified: ClassifiedError = Object.assign( new Error(error.message), { name: error.name, stack: error.stack, category: ErrorCategory.UNKNOWN, originalError: error, } ); // Node.js system errors const errorCode = (error as NodeJS.ErrnoException).code; if (errorCode === 'ECONNRESET' || errorCode === 'ETIMEDOUT' || errorCode === 'ECONNREFUSED' || errorCode === 'EPIPE' || errorCode === 'EHOSTUNREACH') { classified.category = ErrorCategory.TRANSIENT; return classified; } // HTTP client errors (from fetch, axios, etc.) const httpStatus = (error as any).response?.status; if (httpStatus) { classified.httpStatus = httpStatus; if ([408, 429, 502, 503, 504].includes(httpStatus)) { classified.category = ErrorCategory.TRANSIENT; // Extract Retry-After if present const retryAfter = (error as any).response?.headers?.['retry-after']; if (retryAfter) { classified.retryAfterMs = parseRetryAfter(retryAfter); } } else if (httpStatus >= 400 && httpStatus < 500) { classified.category = ErrorCategory.PERMANENT; } else if (httpStatus === 500) { classified.category = ErrorCategory.UNKNOWN; // 500 is ambiguous } return classified; } // gRPC status codes const grpcCode = (error as any).code; if (grpcCode !== undefined) { const transientGrpcCodes = [ 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'UNAVAILABLE', 'ABORTED' ]; const permanentGrpcCodes = [ 'INVALID_ARGUMENT', 'NOT_FOUND', 'ALREADY_EXISTS', 'PERMISSION_DENIED', 'UNAUTHENTICATED' ]; if (transientGrpcCodes.includes(grpcCode)) { classified.category = ErrorCategory.TRANSIENT; } else if (permanentGrpcCodes.includes(grpcCode)) { classified.category = ErrorCategory.PERMANENT; } } return classified; } function parseRetryAfter(value: string): number { // Retry-After can be seconds or HTTP date const seconds = parseInt(value, 10); if (!isNaN(seconds)) { return seconds * 1000; } const date = new Date(value); if (!isNaN(date.getTime())) { return Math.max(0, date.getTime() - Date.now()); } return 0; }

When to Retry: The Decision Framework

The Fundamental Question of Distributed Reliability

The Nature of Failures in Distributed Systems

Transient Failures: The Retry Sweet Spot

Permanent Failures: When Retries Harm

The Retry Decision Framework

Operation Characteristics and Retry Safety

Context-Aware Retry Policies

Failure Detection and Classification Infrastructure

There’s more to understand.

When to Retry: The Decision Framework

The Fundamental Question of Distributed Reliability

The Nature of Failures in Distributed Systems

Transient Failures: The Retry Sweet Spot

Permanent Failures: When Retries Harm

The Retry Decision Framework

Operation Characteristics and Retry Safety

Context-Aware Retry Policies

Failure Detection and Classification Infrastructure

There’s more to understand.