Payment System - Learning Module

Loading content...

0/273

Idempotency Guarantees

The Most Critical Property in Payment Systems

Imagine this scenario: A customer clicks "Pay Now," your server sends the payment request to the gateway, the gateway successfully charges the card, but the response times out before reaching your server. Your system doesn't know if the payment succeeded or failed. What do you do?

If you retry: You might charge the customer twice. If you don't retry: You might lose the payment entirely.

This is the idempotency problem, and it's the single most important design challenge in payment systems. Unlike most distributed systems where "at-least-once" semantics are acceptable (with eventual reconciliation), payment systems require exactly-once semantics. Charging a customer twice isn't a bug you can fix later—it's a customer service disaster, a regulatory violation, and a potential lawsuit.

The Double-Charge Epidemic

Double charges are the #1 source of payment-related customer complaints. Studies show that even a single double-charge experience makes 40% of customers less likely to purchase again. At scale, poorly implemented idempotency can cost millions in refunds, chargebacks, and lost customer lifetime value.

What You Will Learn

By the end of this page, you will understand idempotency fundamentals, database patterns for idempotency storage, the complete request lifecycle with idempotency keys, handling timeout and retry scenarios safely, and distributed idempotency across multiple services.

Understanding Idempotency

Idempotency is the property that an operation produces the same result regardless of how many times it's performed. In mathematical terms: f(f(x)) = f(x).

For payment systems, this means:

Making the same payment request twice should result in exactly one charge
The second (or third, or hundredth) request should return the same response as the first
No side effects should occur on duplicate requests

Naturally Idempotent Operations

•GET /payments/{id} — Reading data is always idempotent
•PUT /payments/{id} — Replace with same data = same result
•DELETE /payments/{id} — Deleting twice = deleted
•Setting a value: balance = 100

NOT Naturally Idempotent

•POST /payments — Create new = new ID each time
•Increment: balance += 100
•Append: log.push(entry)
•Any operation that creates side effects

Why Do We Need Explicit Idempotency?

In a perfect world, every request succeeds or fails clearly. In reality:

Network partitions — The client sends a request but never receives a response
Timeouts — The request takes too long; client gives up but server continues
Retries — Client SDKs, load balancers, and proxies may retry automatically
User behavior — Users click "Pay" multiple times out of impatience
Crash recovery — Server processes request, crashes before responding

Without idempotency, each of these scenarios can result in duplicate operations.

Converting Mermaid diagram...

The Worst Case Scenario

The most dangerous situation is when the timeout occurs AFTER the gateway successfully charges the card but BEFORE you record the success. Your database shows no payment, but the customer's card has been charged. Without proper idempotency handling, a retry will charge them again.

Idempotency Keys: The Foundation

An idempotency key is a unique identifier provided by the client that allows the server to recognize duplicate requests. If the same key is seen twice, the server returns the cached response from the first request instead of processing again.

Key Properties:

•Client-generated — The client must generate the key, not the server. The server can't distinguish duplicates if it generates IDs.
•Globally unique — Keys must be unique across all clients, all time. UUIDs (v4 or v7) are the standard choice.
•Request-scoped — One key per intended operation. Separate payment attempts = separate keys.
•Immutable binding — Once a key is associated with request parameters, those parameters cannot change.

idempotency-key-design.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// Idempotency Key Generation Strategies
 
// Option 1: Pure UUID (simple, always unique)
const idempotencyKey = crypto.randomUUID();
// Example: "550e8400-e29b-41d4-a716-446655440000"
 
// Option 2: Semantic key with UUID suffix (debuggable)
const idempotencyKey = `order_${orderId}_payment_${crypto.randomUUID()}`;
// Example: "order_12345_payment_550e8400-e29b-41d4"
 
// Option 3: Deterministic key from request context (automatic dedup)
const idempotencyKey = crypto
  .createHash('sha256')
  .update(`${userId}:${orderId}:${amount}:${currency}`)
  .digest('hex');
// Example: "a3f2b8c9d4e5..." (same input = same key)
 
// ⚠️ Danger: Pure timestamp is NOT safe
const badKey = Date.now().toString();
// Race condition: Multiple requests in same millisecond
 
// ⚠️ Danger: Order ID alone is NOT safe for retries
const badKey2 = orderId;
// If first payment fails, second attempt with same order needs new key
 
// ✅ Best Practice: Combination of order context + attempt UUID
interface PaymentIdempotencyKey {
  orderId: string;
  attemptId: string;  // New UUID for each payment attempt
}
 
function generatePaymentIdempotencyKey(orderId: string): string {
  return `pay_${orderId}_${crypto.randomUUID()}`;
}
 
// For retries of the SAME attempt (network retry), reuse the key
// For NEW attempts (after decline), generate new key

Retry vs. New Attempt

Distinguish between retries (same request due to timeout/error, same idempotency key) and new attempts (user tries different card after decline, new idempotency key). Retries should be idempotent; new attempts are distinct payments.

Database Design for Idempotency

Idempotency requires persistent storage to track which keys have been processed and their results. The database design must handle:

Fast lookups — Check if key exists on every request
Atomic creation — Prevent race conditions where duplicate requests both insert
Response storage — Cache the full response for replay
Expiration — Avoid unlimited growth of idempotency records

idempotency-schema.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
-- Idempotency Keys Table
CREATE TABLE idempotency_keys (
    -- The idempotency key itself (primary lookup)
    idempotency_key VARCHAR(255) PRIMARY KEY,
    
    -- Request fingerprint (detect parameter mismatches)
    request_hash VARCHAR(64) NOT NULL,
    
    -- Processing state
    status VARCHAR(20) NOT NULL DEFAULT 'processing',
    -- 'processing' | 'completed' | 'failed'
    
    -- Cached response for replay
    response_body JSONB,
    response_status_code INTEGER,
    
    -- The created resource ID (if any)
    resource_type VARCHAR(50),
    resource_id VARCHAR(255),
    
    -- Timing
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    completed_at TIMESTAMP WITH TIME ZONE,
    expires_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() + INTERVAL '24 hours',
    
    -- For locking during processing
    locked_until TIMESTAMP WITH TIME ZONE,
    lock_token VARCHAR(64)
);
 
-- Index for expiration cleanup
CREATE INDEX idx_idempotency_expires 
    ON idempotency_keys(expires_at) 
    WHERE status = 'completed';
 
-- Index for finding stuck requests
CREATE INDEX idx_idempotency_processing 
    ON idempotency_keys(locked_until) 
    WHERE status = 'processing';
 
-- Prevent table bloat with automatic cleanup
-- (Run via cron or pg_cron)
CREATE OR REPLACE FUNCTION cleanup_expired_idempotency_keys()
RETURNS void AS $$
BEGIN
    DELETE FROM idempotency_keys 
    WHERE expires_at < NOW() - INTERVAL '1 hour';
END;
$$ LANGUAGE plpgsql;

Key Schema Decisions Explained:

•request_hash — Hash of request body. If client sends same key with different parameters, reject with 400 error. Prevents misuse.
•status = 'processing' — Critical for handling concurrent duplicates. First request sets 'processing'; duplicates wait or return 409.
•response_body as JSONB — Store complete response for replay. Client gets exact same response on retry.
•locked_until + lock_token — Pessimistic locking for long-running operations. Prevents stuck 'processing' states.
•expires_at = 24 hours — Balance between supporting delayed retries and preventing unbounded table growth.

idempotency-store.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
interface IdempotencyRecord {
  idempotencyKey: string;
  requestHash: string;
  status: 'processing' | 'completed' | 'failed';
  responseBody?: object;
  responseStatusCode?: number;
  resourceType?: string;
  resourceId?: string;
  createdAt: Date;
  completedAt?: Date;
  expiresAt: Date;
  lockedUntil?: Date;
  lockToken?: string;
}
 
enum IdempotencyResult {
  NEW_REQUEST,           // First time seeing this key
  DUPLICATE_PROCESSING,  // Another request is processing this key
  DUPLICATE_COMPLETED,   // Request completed, return cached response
  KEY_MISMATCH,         // Same key but different request params
  LOCK_EXPIRED,         // Previous request timed out, can retry
}
 
class IdempotencyStore {
  constructor(private db: Database) {}
  
  /**
   * Attempt to acquire the idempotency key for processing
   */
  async acquire(
    key: string,
    requestHash: string,
    ttlSeconds: number = 30
  ): Promise<{ result: IdempotencyResult; record?: IdempotencyRecord }> {
    
    const lockToken = crypto.randomUUID();
    const now = new Date();
    const lockedUntil = new Date(now.getTime() + ttlSeconds * 1000);
    
    // Attempt atomic insert-or-update
    const result = await this.db.query(`
      INSERT INTO idempotency_keys (
        idempotency_key, 
        request_hash, 
        status, 
        locked_until, 
        lock_token
      )
      VALUES ($1, $2, 'processing', $3, $4)
      ON CONFLICT (idempotency_key) DO UPDATE SET
        -- Only update if lock expired
        locked_until = CASE 
          WHEN idempotency_keys.status = 'processing' 
               AND idempotency_keys.locked_until < NOW()
          THEN $3
          ELSE idempotency_keys.locked_until
        END,
        lock_token = CASE
          WHEN idempotency_keys.status = 'processing'
               AND idempotency_keys.locked_until < NOW()
          THEN $4
          ELSE idempotency_keys.lock_token
        END
      RETURNING *, 
        (xmax = 0) AS was_inserted,
        (lock_token = $4) AS lock_acquired
    `, [key, requestHash, lockedUntil, lockToken]);
    
    const record = result.rows[0];
    
    // Case 1: New request (inserted successfully)
    if (record.was_inserted) {
      return { result: IdempotencyResult.NEW_REQUEST, record };
    }
    
    // Case 2: Request hash mismatch
    if (record.request_hash !== requestHash) {
      return { result: IdempotencyResult.KEY_MISMATCH, record };
    }
    
    // Case 3: Already completed
    if (record.status === 'completed' || record.status === 'failed') {
      return { result: IdempotencyResult.DUPLICATE_COMPLETED, record };
    }
    
    // Case 4: Still processing, but we got the lock (previous timed out)
    if (record.lock_acquired) {
      return { result: IdempotencyResult.LOCK_EXPIRED, record };
    }
    
    // Case 5: Still processing, lock held by another request
    return { result: IdempotencyResult.DUPLICATE_PROCESSING, record };
  }
  
  /**
   * Mark request as completed with cached response
   */
  async complete(
    key: string,
    lockToken: string,
    response: {
      statusCode: number;
      body: object;
      resourceType?: string;
      resourceId?: string;
    }
  ): Promise<void> {
    await this.db.query(`
      UPDATE idempotency_keys 
      SET 
        status = 'completed',
        response_status_code = $3,
        response_body = $4,
        resource_type = $5,
        resource_id = $6,
        completed_at = NOW(),
        locked_until = NULL,
        lock_token = NULL
      WHERE idempotency_key = $1 
        AND lock_token = $2
    `, [key, lockToken, response.statusCode, response.body, 
        response.resourceType, response.resourceId]);
  }
  
  /**
   * Mark request as failed (allow retry with same key)
   */
  async fail(key: string, lockToken: string, error: object): Promise<void> {
    await this.db.query(`
      UPDATE idempotency_keys 
      SET 
        status = 'failed',
        response_body = $3,
        completed_at = NOW(),
        locked_until = NULL,
        lock_token = NULL
      WHERE idempotency_key = $1 
        AND lock_token = $2
    `, [key, lockToken, error]);
  }
}

Lock Timeout Is Critical

The lock timeout (30 seconds above) must be longer than your maximum expected processing time but short enough to recover from crashes quickly. If a server crashes mid-processing, the lock expires and another request can retry safely.

The Complete Idempotent Request Lifecycle

Let's trace through the complete lifecycle of an idempotent payment request, handling all edge cases.

Converting Mermaid diagram...

idempotent-payment-handler.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
class IdempotentPaymentHandler {
  constructor(
    private idempotencyStore: IdempotencyStore,
    private paymentService: PaymentService,
    private metrics: MetricsClient
  ) {}
  
  async handlePayment(
    request: PaymentRequest,
    idempotencyKey: string
  ): Promise<PaymentResponse> {
    
    // Step 1: Hash the request for mismatch detection
    const requestHash = this.hashRequest(request);
    
    // Step 2: Attempt to acquire the idempotency key
    const acquisition = await this.idempotencyStore.acquire(
      idempotencyKey,
      requestHash,
      30 // 30 second lock TTL
    );
    
    // Step 3: Handle based on acquisition result
    switch (acquisition.result) {
      
      case IdempotencyResult.DUPLICATE_COMPLETED:
        // Return cached response
        this.metrics.increment('idempotency.cache_hit');
        return {
          ...acquisition.record!.responseBody as PaymentResponse,
          idempotentReplayed: true,
        };
        
      case IdempotencyResult.DUPLICATE_PROCESSING:
        // Another request is processing - wait and retry
        this.metrics.increment('idempotency.concurrent_request');
        await this.sleep(1000); // Wait 1 second
        return this.handlePayment(request, idempotencyKey);
        
      case IdempotencyResult.KEY_MISMATCH:
        // Same key, different request - client bug
        this.metrics.increment('idempotency.key_mismatch');
        throw new IdempotencyKeyMismatchError(
          idempotencyKey,
          'Request parameters do not match previous request with this key'
        );
        
      case IdempotencyResult.NEW_REQUEST:
      case IdempotencyResult.LOCK_EXPIRED:
        // Process the payment
        return this.processPaymentWithIdempotency(
          request,
          idempotencyKey,
          acquisition.record!.lockToken!
        );
    }
  }
  
  private async processPaymentWithIdempotency(
    request: PaymentRequest,
    idempotencyKey: string,
    lockToken: string
  ): Promise<PaymentResponse> {
    
    try {
      // Process the actual payment
      const result = await this.paymentService.processPayment({
        ...request,
        // Pass idempotency key to gateway too!
        gatewayIdempotencyKey: idempotencyKey,
      });
      
      // Store successful response
      await this.idempotencyStore.complete(idempotencyKey, lockToken, {
        statusCode: 200,
        body: result,
        resourceType: 'payment',
        resourceId: result.id,
      });
      
      return result;
      
    } catch (error) {
      if (this.isRetriableError(error)) {
        // Don't store failed state for retriable errors
        // Lock will expire and allow retry
        throw error;
      }
      
      // Store permanent failure
      await this.idempotencyStore.fail(idempotencyKey, lockToken, {
        error: error.message,
        code: error.code,
      });
      
      throw error;
    }
  }
  
  private hashRequest(request: PaymentRequest): string {
    // Hash only the immutable parts of the request
    const toHash = {
      amount: request.amount,
      currency: request.currency,
      paymentMethodId: request.paymentMethodId,
      merchantId: request.merchantId,
    };
    
    return crypto
      .createHash('sha256')
      .update(JSON.stringify(toHash))
      .digest('hex');
  }
  
  private isRetriableError(error: unknown): boolean {
    // Retriable: timeouts, 5xx, network errors
    // Not retriable: 4xx, business logic errors
    return error instanceof TimeoutError 
        || error instanceof NetworkError
        || (error instanceof GatewayError && error.statusCode >= 500);
  }
  
  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Hash Only Immutable Fields

When hashing the request for mismatch detection, only include fields that MUST match for the request to be considered the same. Don't include timestamps, request IDs, or other generated fields. Otherwise, legitimate retries will fail the hash check.

Gateway-Level Idempotency

Idempotency at your application level prevents duplicate requests from creating duplicate payments. But what if the duplicate occurs between your server and the gateway? You need two layers of idempotency:

Application-level — Prevents duplicate calls to YOUR system
Gateway-level — Prevents duplicate calls to the GATEWAY

Converting Mermaid diagram...

Gateway Idempotency Support
Gateway	Idempotency Header	TTL	Scope
Stripe	Idempotency-Key header	24 hours	Per API key
Adyen	X-Idempotency-Key header	N/A (use reference)	Per merchant account
Braintree	Built into transaction ID	Indefinite	Per transaction
PayPal	PayPal-Request-Id header	14 days	Per API credential
Square	Idempotency-Key header	45 days	Per Square account

gateway-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// Stripe example: Pass idempotency key to SDK
async function chargeWithStripe(
  request: PaymentRequest,
  idempotencyKey: string
): Promise<StripePaymentIntent> {
  
  return stripe.paymentIntents.create(
    {
      amount: request.amount,
      currency: request.currency,
      payment_method: request.paymentMethodId,
      confirm: true,
    },
    {
      idempotencyKey: idempotencyKey,  // Critical!
    }
  );
}
 
// Adyen example: Use reference field
async function chargeWithAdyen(
  request: PaymentRequest,
  idempotencyKey: string
): Promise<AdyenPaymentResponse> {
  
  return fetch('https://checkout.adyen.com/v70/payments', {
    method: 'POST',
    headers: {
      'X-API-Key': ADYEN_API_KEY,
      'X-Idempotency-Key': idempotencyKey,  // Critical!
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      amount: { value: request.amount, currency: request.currency },
      reference: `${request.orderId}:${idempotencyKey}`,  // Also use in reference
      paymentMethod: { storedPaymentMethodId: request.paymentMethodId },
      merchantAccount: ADYEN_MERCHANT_ACCOUNT,
    }),
  }).then(r => r.json());
}
 
// Key propagation strategy
function generateGatewayIdempotencyKey(
  applicationKey: string,
  gatewayName: string,
  attemptNumber: number
): string {
  // Include gateway name so different gateways don't conflict
  // Include attempt number for failover scenarios
  return `${applicationKey}:${gatewayName}:attempt${attemptNumber}`;
}

Failover Idempotency Keys

When failing over to a different gateway, you MUST use a different idempotency key suffix. Otherwise, if Gateway A actually succeeded but timed out, Gateway B's idempotency key won't be the same—you'll have duplicate charges. Use key:stripeAttempt1 for Stripe and key:adyenAttempt1 for Adyen failover.

Distributed Idempotency Challenges

In a microservices architecture, a single payment request may involve multiple services: Payment Service, Fraud Service, Notification Service, Inventory Service. Each may need idempotency guarantees. How do you coordinate?

Distributed Idempotency Challenges

•Saga Coordination — A payment saga may fail at step 3. On retry, should steps 1 and 2 execute again?
•Cross-Service Consistency — Service A and B must agree on whether a transaction is 'completed'.
•Timeout Handling — If the orchestrator times out, how do participants know to rollback?
•Ordering — Messages may arrive out of order. Service B might see 'retry' before 'original'.

distributed-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
// Pattern: Hierarchical Idempotency Keys
 
interface PaymentSaga {
  sagaId: string;  // Top-level idempotency key
  steps: SagaStep[];
}
 
interface SagaStep {
  stepId: string;
  service: string;
  operation: string;
  idempotencyKey: string;  // Derived from sagaId
  status: 'pending' | 'completed' | 'failed' | 'compensated';
}
 
class PaymentSagaOrchestrator {
  async executeSaga(
    request: PaymentRequest,
    masterIdempotencyKey: string
  ): Promise<PaymentResult> {
    
    // Create saga with derived idempotency keys
    const saga: PaymentSaga = {
      sagaId: masterIdempotencyKey,
      steps: [
        {
          stepId: '1_fraud_check',
          service: 'fraud',
          operation: 'checkTransaction',
          idempotencyKey: `${masterIdempotencyKey}:fraud:check`,
          status: 'pending',
        },
        {
          stepId: '2_reserve_inventory',
          service: 'inventory',
          operation: 'reserve',
          idempotencyKey: `${masterIdempotencyKey}:inventory:reserve`,
          status: 'pending',
        },
        {
          stepId: '3_charge_payment',
          service: 'payment',
          operation: 'charge',
          idempotencyKey: `${masterIdempotencyKey}:payment:charge`,
          status: 'pending',
        },
        {
          stepId: '4_confirm_inventory',
          service: 'inventory',
          operation: 'confirm',
          idempotencyKey: `${masterIdempotencyKey}:inventory:confirm`,
          status: 'pending',
        },
      ],
    };
    
    // Each service uses its derived key for idempotency
    for (const step of saga.steps) {
      try {
        await this.executeStep(step, request);
        step.status = 'completed';
      } catch (error) {
        step.status = 'failed';
        await this.compensate(saga, step);
        throw error;
      }
    }
    
    return this.buildResult(saga);
  }
  
  private async compensate(saga: PaymentSaga, failedStep: SagaStep) {
    // Execute compensation in reverse order
    const completedSteps = saga.steps
      .filter(s => s.status === 'completed')
      .reverse();
    
    for (const step of completedSteps) {
      // Compensation also needs idempotency!
      const compensationKey = `${step.idempotencyKey}:compensate`;
      await this.executeCompensation(step, compensationKey);
      step.status = 'compensated';
    }
  }
}
 
// Each service checks its derived idempotency key
class FraudService {
  async checkTransaction(
    request: TransactionCheckRequest,
    idempotencyKey: string  // e.g., "saga123:fraud:check"
  ): Promise<FraudCheckResult> {
    
    // Use same idempotency pattern as payment service
    const acquisition = await this.idempotencyStore.acquire(idempotencyKey);
    
    if (acquisition.result === IdempotencyResult.DUPLICATE_COMPLETED) {
      return acquisition.record.responseBody as FraudCheckResult;
    }
    
    const result = await this.performFraudCheck(request);
    
    await this.idempotencyStore.complete(idempotencyKey, result);
    return result;
  }
}

Derive Don't Duplicate

Derive child idempotency keys from the parent key deterministically. This ensures that on retry, each service sees the same idempotency key it saw before. Pattern: {parentKey}:{serviceName}:{operation}.

Testing Idempotency Implementations

Idempotency bugs often only manifest under specific race conditions in production. Thorough testing is essential. Here are the critical test scenarios:

idempotency-tests.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
describe('Idempotency', () => {
  
  describe('Basic Idempotency', () => {
    it('should return same response on retry', async () => {
      const key = 'test-key-1';
      const request = { amount: 1000, currency: 'USD' };
      
      const response1 = await handler.processPayment(request, key);
      const response2 = await handler.processPayment(request, key);
      
      expect(response2.id).toEqual(response1.id);
      expect(response2.idempotentReplayed).toBe(true);
    });
    
    it('should reject different request with same key', async () => {
      const key = 'test-key-2';
      
      await handler.processPayment({ amount: 1000 }, key);
      
      await expect(
        handler.processPayment({ amount: 2000 }, key)  // Different amount!
      ).rejects.toThrow(IdempotencyKeyMismatchError);
    });
  });
  
  describe('Concurrent Requests', () => {
    it('should process only once with parallel requests', async () => {
      const key = 'test-key-3';
      const request = { amount: 1000 };
      
      // Fire 10 concurrent requests with same key
      const promises = Array(10).fill(null).map(() =>
        handler.processPayment(request, key)
      );
      
      const responses = await Promise.all(promises);
      
      // All should have same payment ID
      const ids = new Set(responses.map(r => r.id));
      expect(ids.size).toBe(1);
      
      // Gateway should only be called once
      expect(mockGateway.charge).toHaveBeenCalledTimes(1);
    });
  });
  
  describe('Failure Recovery', () => {
    it('should allow retry after gateway timeout', async () => {
      const key = 'test-key-4';
      
      // First request times out at gateway
      mockGateway.charge.mockRejectedValueOnce(new TimeoutError());
      
      await expect(
        handler.processPayment({ amount: 1000 }, key)
      ).rejects.toThrow(TimeoutError);
      
      // Wait for lock to expire
      await sleep(35000);
      
      // Retry should succeed
      mockGateway.charge.mockResolvedValueOnce({ id: 'pay_123' });
      
      const response = await handler.processPayment({ amount: 1000 }, key);
      expect(response.id).toBe('pay_123');
    });
    
    it('should prevent retry after business error', async () => {
      const key = 'test-key-5';
      
      // First request is declined
      mockGateway.charge.mockRejectedValueOnce(
        new CardDeclinedError('insufficient_funds')
      );
      
      await expect(
        handler.processPayment({ amount: 1000 }, key)
      ).rejects.toThrow(CardDeclinedError);
      
      // Retry should return cached decline
      const retryResult = await handler.processPayment({ amount: 1000 }, key);
      expect(retryResult.status).toBe('failed');
      expect(retryResult.idempotentReplayed).toBe(true);
      
      // Gateway should NOT be called again
      expect(mockGateway.charge).toHaveBeenCalledTimes(1);
    });
  });
  
  describe('Edge Cases', () => {
    it('should handle key expiration', async () => {
      const key = 'test-key-6';
      
      await handler.processPayment({ amount: 1000 }, key);
      
      // Simulate key expiration
      await db.query(
        "UPDATE idempotency_keys SET expires_at = NOW() - INTERVAL '1 hour'"
      );
      
      // Should process as new request after expiration
      const response2 = await handler.processPayment({ amount: 1000 }, key);
      expect(response2.idempotentReplayed).toBe(false);
    });
  });
});

Chaos Testing Is Essential

Beyond unit tests, inject failures in staging: random timeouts, process kills during processing, database connection drops. Tools like Chaos Monkey or Gremlin can simulate these scenarios. If your idempotency survives chaos testing, it'll survive production.

Summary: Idempotency Guarantees

Idempotency is the cornerstone of reliable payment systems. Let's consolidate the key principles:

Key Takeaways

•Idempotency is non-negotiable — Payment systems must guarantee exactly-once processing. At-least-once with reconciliation is not acceptable.
•Client-generated keys — Idempotency keys must be generated by the client, unique per intended operation, and reused for retries of the same operation.
•Database-backed storage — Persist idempotency state with proper locking, response caching, and expiration handling.
•Two-layer idempotency — Implement idempotency at both application layer AND gateway layer to prevent duplicates at every hop.
•Handle all edge cases — Concurrent requests, timeouts, lock expiration, and parameter mismatches all need explicit handling.
•Derived keys for distributed systems — In saga patterns, derive child idempotency keys from the parent key deterministically.
•Test extensively — Concurrent request tests, failure recovery tests, and chaos testing are essential for confidence.

What's Next:

With idempotency ensuring we never double-charge customers, the next challenge is preventing malicious actors from exploiting our system. The next page covers fraud detection—how to identify and prevent fraudulent transactions while minimizing false positives that reject legitimate customers.

Page Complete

You now understand how to implement bullet-proof idempotency in payment systems. This knowledge directly prevents the most common and damaging failure mode in payment processing: double charges.

Idempotency Guarantees

The Most Critical Property in Payment Systems

If you retry: You might charge the customer twice. If you don't retry: You might lose the payment entirely.

The Double-Charge Epidemic

What You Will Learn

Understanding Idempotency

Idempotency is the property that an operation produces the same result regardless of how many times it's performed. In mathematical terms: f(f(x)) = f(x).

For payment systems, this means:

Making the same payment request twice should result in exactly one charge
The second (or third, or hundredth) request should return the same response as the first
No side effects should occur on duplicate requests

Naturally Idempotent Operations

•GET /payments/{id} — Reading data is always idempotent
•PUT /payments/{id} — Replace with same data = same result
•DELETE /payments/{id} — Deleting twice = deleted
•Setting a value: balance = 100

NOT Naturally Idempotent

•POST /payments — Create new = new ID each time
•Increment: balance += 100
•Append: log.push(entry)
•Any operation that creates side effects

Why Do We Need Explicit Idempotency?

In a perfect world, every request succeeds or fails clearly. In reality:

Network partitions — The client sends a request but never receives a response
Timeouts — The request takes too long; client gives up but server continues
Retries — Client SDKs, load balancers, and proxies may retry automatically
User behavior — Users click "Pay" multiple times out of impatience
Crash recovery — Server processes request, crashes before responding

Without idempotency, each of these scenarios can result in duplicate operations.

Converting Mermaid diagram...

The Worst Case Scenario

Idempotency Keys: The Foundation

Key Properties:

•Client-generated — The client must generate the key, not the server. The server can't distinguish duplicates if it generates IDs.
•Globally unique — Keys must be unique across all clients, all time. UUIDs (v4 or v7) are the standard choice.
•Request-scoped — One key per intended operation. Separate payment attempts = separate keys.
•Immutable binding — Once a key is associated with request parameters, those parameters cannot change.

idempotency-key-design.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// Idempotency Key Generation Strategies
 
// Option 1: Pure UUID (simple, always unique)
const idempotencyKey = crypto.randomUUID();
// Example: "550e8400-e29b-41d4-a716-446655440000"
 
// Option 2: Semantic key with UUID suffix (debuggable)
const idempotencyKey = `order_${orderId}_payment_${crypto.randomUUID()}`;
// Example: "order_12345_payment_550e8400-e29b-41d4"
 
// Option 3: Deterministic key from request context (automatic dedup)
const idempotencyKey = crypto
  .createHash('sha256')
  .update(`${userId}:${orderId}:${amount}:${currency}`)
  .digest('hex');
// Example: "a3f2b8c9d4e5..." (same input = same key)
 
// ⚠️ Danger: Pure timestamp is NOT safe
const badKey = Date.now().toString();
// Race condition: Multiple requests in same millisecond
 
// ⚠️ Danger: Order ID alone is NOT safe for retries
const badKey2 = orderId;
// If first payment fails, second attempt with same order needs new key
 
// ✅ Best Practice: Combination of order context + attempt UUID
interface PaymentIdempotencyKey {
  orderId: string;
  attemptId: string;  // New UUID for each payment attempt
}
 
function generatePaymentIdempotencyKey(orderId: string): string {
  return `pay_${orderId}_${crypto.randomUUID()}`;
}
 
// For retries of the SAME attempt (network retry), reuse the key
// For NEW attempts (after decline), generate new key

Retry vs. New Attempt

Database Design for Idempotency

Idempotency requires persistent storage to track which keys have been processed and their results. The database design must handle:

Fast lookups — Check if key exists on every request
Atomic creation — Prevent race conditions where duplicate requests both insert
Response storage — Cache the full response for replay
Expiration — Avoid unlimited growth of idempotency records

idempotency-schema.sql
SQL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
-- Idempotency Keys Table
CREATE TABLE idempotency_keys (
    -- The idempotency key itself (primary lookup)
    idempotency_key VARCHAR(255) PRIMARY KEY,
    
    -- Request fingerprint (detect parameter mismatches)
    request_hash VARCHAR(64) NOT NULL,
    
    -- Processing state
    status VARCHAR(20) NOT NULL DEFAULT 'processing',
    -- 'processing' | 'completed' | 'failed'
    
    -- Cached response for replay
    response_body JSONB,
    response_status_code INTEGER,
    
    -- The created resource ID (if any)
    resource_type VARCHAR(50),
    resource_id VARCHAR(255),
    
    -- Timing
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    completed_at TIMESTAMP WITH TIME ZONE,
    expires_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() + INTERVAL '24 hours',
    
    -- For locking during processing
    locked_until TIMESTAMP WITH TIME ZONE,
    lock_token VARCHAR(64)
);
 
-- Index for expiration cleanup
CREATE INDEX idx_idempotency_expires 
    ON idempotency_keys(expires_at) 
    WHERE status = 'completed';
 
-- Index for finding stuck requests
CREATE INDEX idx_idempotency_processing 
    ON idempotency_keys(locked_until) 
    WHERE status = 'processing';
 
-- Prevent table bloat with automatic cleanup
-- (Run via cron or pg_cron)
CREATE OR REPLACE FUNCTION cleanup_expired_idempotency_keys()
RETURNS void AS $$
BEGIN
    DELETE FROM idempotency_keys 
    WHERE expires_at < NOW() - INTERVAL '1 hour';
END;
$$ LANGUAGE plpgsql;

Key Schema Decisions Explained:

•request_hash — Hash of request body. If client sends same key with different parameters, reject with 400 error. Prevents misuse.
•status = 'processing' — Critical for handling concurrent duplicates. First request sets 'processing'; duplicates wait or return 409.
•response_body as JSONB — Store complete response for replay. Client gets exact same response on retry.
•locked_until + lock_token — Pessimistic locking for long-running operations. Prevents stuck 'processing' states.
•expires_at = 24 hours — Balance between supporting delayed retries and preventing unbounded table growth.

idempotency-store.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
interface IdempotencyRecord {
  idempotencyKey: string;
  requestHash: string;
  status: 'processing' | 'completed' | 'failed';
  responseBody?: object;
  responseStatusCode?: number;
  resourceType?: string;
  resourceId?: string;
  createdAt: Date;
  completedAt?: Date;
  expiresAt: Date;
  lockedUntil?: Date;
  lockToken?: string;
}
 
enum IdempotencyResult {
  NEW_REQUEST,           // First time seeing this key
  DUPLICATE_PROCESSING,  // Another request is processing this key
  DUPLICATE_COMPLETED,   // Request completed, return cached response
  KEY_MISMATCH,         // Same key but different request params
  LOCK_EXPIRED,         // Previous request timed out, can retry
}
 
class IdempotencyStore {
  constructor(private db: Database) {}
  
  /**
   * Attempt to acquire the idempotency key for processing
   */
  async acquire(
    key: string,
    requestHash: string,
    ttlSeconds: number = 30
  ): Promise<{ result: IdempotencyResult; record?: IdempotencyRecord }> {
    
    const lockToken = crypto.randomUUID();
    const now = new Date();
    const lockedUntil = new Date(now.getTime() + ttlSeconds * 1000);
    
    // Attempt atomic insert-or-update
    const result = await this.db.query(`
      INSERT INTO idempotency_keys (
        idempotency_key, 
        request_hash, 
        status, 
        locked_until, 
        lock_token
      )
      VALUES ($1, $2, 'processing', $3, $4)
      ON CONFLICT (idempotency_key) DO UPDATE SET
        -- Only update if lock expired
        locked_until = CASE 
          WHEN idempotency_keys.status = 'processing' 
               AND idempotency_keys.locked_until < NOW()
          THEN $3
          ELSE idempotency_keys.locked_until
        END,
        lock_token = CASE
          WHEN idempotency_keys.status = 'processing'
               AND idempotency_keys.locked_until < NOW()
          THEN $4
          ELSE idempotency_keys.lock_token
        END
      RETURNING *, 
        (xmax = 0) AS was_inserted,
        (lock_token = $4) AS lock_acquired
    `, [key, requestHash, lockedUntil, lockToken]);
    
    const record = result.rows[0];
    
    // Case 1: New request (inserted successfully)
    if (record.was_inserted) {
      return { result: IdempotencyResult.NEW_REQUEST, record };
    }
    
    // Case 2: Request hash mismatch
    if (record.request_hash !== requestHash) {
      return { result: IdempotencyResult.KEY_MISMATCH, record };
    }
    
    // Case 3: Already completed
    if (record.status === 'completed' || record.status === 'failed') {
      return { result: IdempotencyResult.DUPLICATE_COMPLETED, record };
    }
    
    // Case 4: Still processing, but we got the lock (previous timed out)
    if (record.lock_acquired) {
      return { result: IdempotencyResult.LOCK_EXPIRED, record };
    }
    
    // Case 5: Still processing, lock held by another request
    return { result: IdempotencyResult.DUPLICATE_PROCESSING, record };
  }
  
  /**
   * Mark request as completed with cached response
   */
  async complete(
    key: string,
    lockToken: string,
    response: {
      statusCode: number;
      body: object;
      resourceType?: string;
      resourceId?: string;
    }
  ): Promise<void> {
    await this.db.query(`
      UPDATE idempotency_keys 
      SET 
        status = 'completed',
        response_status_code = $3,
        response_body = $4,
        resource_type = $5,
        resource_id = $6,
        completed_at = NOW(),
        locked_until = NULL,
        lock_token = NULL
      WHERE idempotency_key = $1 
        AND lock_token = $2
    `, [key, lockToken, response.statusCode, response.body, 
        response.resourceType, response.resourceId]);
  }
  
  /**
   * Mark request as failed (allow retry with same key)
   */
  async fail(key: string, lockToken: string, error: object): Promise<void> {
    await this.db.query(`
      UPDATE idempotency_keys 
      SET 
        status = 'failed',
        response_body = $3,
        completed_at = NOW(),
        locked_until = NULL,
        lock_token = NULL
      WHERE idempotency_key = $1 
        AND lock_token = $2
    `, [key, lockToken, error]);
  }
}

Lock Timeout Is Critical

The Complete Idempotent Request Lifecycle

Let's trace through the complete lifecycle of an idempotent payment request, handling all edge cases.

Converting Mermaid diagram...

idempotent-payment-handler.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
class IdempotentPaymentHandler {
  constructor(
    private idempotencyStore: IdempotencyStore,
    private paymentService: PaymentService,
    private metrics: MetricsClient
  ) {}
  
  async handlePayment(
    request: PaymentRequest,
    idempotencyKey: string
  ): Promise<PaymentResponse> {
    
    // Step 1: Hash the request for mismatch detection
    const requestHash = this.hashRequest(request);
    
    // Step 2: Attempt to acquire the idempotency key
    const acquisition = await this.idempotencyStore.acquire(
      idempotencyKey,
      requestHash,
      30 // 30 second lock TTL
    );
    
    // Step 3: Handle based on acquisition result
    switch (acquisition.result) {
      
      case IdempotencyResult.DUPLICATE_COMPLETED:
        // Return cached response
        this.metrics.increment('idempotency.cache_hit');
        return {
          ...acquisition.record!.responseBody as PaymentResponse,
          idempotentReplayed: true,
        };
        
      case IdempotencyResult.DUPLICATE_PROCESSING:
        // Another request is processing - wait and retry
        this.metrics.increment('idempotency.concurrent_request');
        await this.sleep(1000); // Wait 1 second
        return this.handlePayment(request, idempotencyKey);
        
      case IdempotencyResult.KEY_MISMATCH:
        // Same key, different request - client bug
        this.metrics.increment('idempotency.key_mismatch');
        throw new IdempotencyKeyMismatchError(
          idempotencyKey,
          'Request parameters do not match previous request with this key'
        );
        
      case IdempotencyResult.NEW_REQUEST:
      case IdempotencyResult.LOCK_EXPIRED:
        // Process the payment
        return this.processPaymentWithIdempotency(
          request,
          idempotencyKey,
          acquisition.record!.lockToken!
        );
    }
  }
  
  private async processPaymentWithIdempotency(
    request: PaymentRequest,
    idempotencyKey: string,
    lockToken: string
  ): Promise<PaymentResponse> {
    
    try {
      // Process the actual payment
      const result = await this.paymentService.processPayment({
        ...request,
        // Pass idempotency key to gateway too!
        gatewayIdempotencyKey: idempotencyKey,
      });
      
      // Store successful response
      await this.idempotencyStore.complete(idempotencyKey, lockToken, {
        statusCode: 200,
        body: result,
        resourceType: 'payment',
        resourceId: result.id,
      });
      
      return result;
      
    } catch (error) {
      if (this.isRetriableError(error)) {
        // Don't store failed state for retriable errors
        // Lock will expire and allow retry
        throw error;
      }
      
      // Store permanent failure
      await this.idempotencyStore.fail(idempotencyKey, lockToken, {
        error: error.message,
        code: error.code,
      });
      
      throw error;
    }
  }
  
  private hashRequest(request: PaymentRequest): string {
    // Hash only the immutable parts of the request
    const toHash = {
      amount: request.amount,
      currency: request.currency,
      paymentMethodId: request.paymentMethodId,
      merchantId: request.merchantId,
    };
    
    return crypto
      .createHash('sha256')
      .update(JSON.stringify(toHash))
      .digest('hex');
  }
  
  private isRetriableError(error: unknown): boolean {
    // Retriable: timeouts, 5xx, network errors
    // Not retriable: 4xx, business logic errors
    return error instanceof TimeoutError 
        || error instanceof NetworkError
        || (error instanceof GatewayError && error.statusCode >= 500);
  }
  
  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Hash Only Immutable Fields

Gateway-Level Idempotency

Application-level — Prevents duplicate calls to YOUR system
Gateway-level — Prevents duplicate calls to the GATEWAY

Converting Mermaid diagram...

Gateway Idempotency Support
Gateway	Idempotency Header	TTL	Scope
Stripe	Idempotency-Key header	24 hours	Per API key
Adyen	X-Idempotency-Key header	N/A (use reference)	Per merchant account
Braintree	Built into transaction ID	Indefinite	Per transaction
PayPal	PayPal-Request-Id header	14 days	Per API credential
Square	Idempotency-Key header	45 days	Per Square account

gateway-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// Stripe example: Pass idempotency key to SDK
async function chargeWithStripe(
  request: PaymentRequest,
  idempotencyKey: string
): Promise<StripePaymentIntent> {
  
  return stripe.paymentIntents.create(
    {
      amount: request.amount,
      currency: request.currency,
      payment_method: request.paymentMethodId,
      confirm: true,
    },
    {
      idempotencyKey: idempotencyKey,  // Critical!
    }
  );
}
 
// Adyen example: Use reference field
async function chargeWithAdyen(
  request: PaymentRequest,
  idempotencyKey: string
): Promise<AdyenPaymentResponse> {
  
  return fetch('https://checkout.adyen.com/v70/payments', {
    method: 'POST',
    headers: {
      'X-API-Key': ADYEN_API_KEY,
      'X-Idempotency-Key': idempotencyKey,  // Critical!
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      amount: { value: request.amount, currency: request.currency },
      reference: `${request.orderId}:${idempotencyKey}`,  // Also use in reference
      paymentMethod: { storedPaymentMethodId: request.paymentMethodId },
      merchantAccount: ADYEN_MERCHANT_ACCOUNT,
    }),
  }).then(r => r.json());
}
 
// Key propagation strategy
function generateGatewayIdempotencyKey(
  applicationKey: string,
  gatewayName: string,
  attemptNumber: number
): string {
  // Include gateway name so different gateways don't conflict
  // Include attempt number for failover scenarios
  return `${applicationKey}:${gatewayName}:attempt${attemptNumber}`;
}

Failover Idempotency Keys

Distributed Idempotency Challenges

•Saga Coordination — A payment saga may fail at step 3. On retry, should steps 1 and 2 execute again?
•Cross-Service Consistency — Service A and B must agree on whether a transaction is 'completed'.
•Timeout Handling — If the orchestrator times out, how do participants know to rollback?
•Ordering — Messages may arrive out of order. Service B might see 'retry' before 'original'.

distributed-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
// Pattern: Hierarchical Idempotency Keys
 
interface PaymentSaga {
  sagaId: string;  // Top-level idempotency key
  steps: SagaStep[];
}
 
interface SagaStep {
  stepId: string;
  service: string;
  operation: string;
  idempotencyKey: string;  // Derived from sagaId
  status: 'pending' | 'completed' | 'failed' | 'compensated';
}
 
class PaymentSagaOrchestrator {
  async executeSaga(
    request: PaymentRequest,
    masterIdempotencyKey: string
  ): Promise<PaymentResult> {
    
    // Create saga with derived idempotency keys
    const saga: PaymentSaga = {
      sagaId: masterIdempotencyKey,
      steps: [
        {
          stepId: '1_fraud_check',
          service: 'fraud',
          operation: 'checkTransaction',
          idempotencyKey: `${masterIdempotencyKey}:fraud:check`,
          status: 'pending',
        },
        {
          stepId: '2_reserve_inventory',
          service: 'inventory',
          operation: 'reserve',
          idempotencyKey: `${masterIdempotencyKey}:inventory:reserve`,
          status: 'pending',
        },
        {
          stepId: '3_charge_payment',
          service: 'payment',
          operation: 'charge',
          idempotencyKey: `${masterIdempotencyKey}:payment:charge`,
          status: 'pending',
        },
        {
          stepId: '4_confirm_inventory',
          service: 'inventory',
          operation: 'confirm',
          idempotencyKey: `${masterIdempotencyKey}:inventory:confirm`,
          status: 'pending',
        },
      ],
    };
    
    // Each service uses its derived key for idempotency
    for (const step of saga.steps) {
      try {
        await this.executeStep(step, request);
        step.status = 'completed';
      } catch (error) {
        step.status = 'failed';
        await this.compensate(saga, step);
        throw error;
      }
    }
    
    return this.buildResult(saga);
  }
  
  private async compensate(saga: PaymentSaga, failedStep: SagaStep) {
    // Execute compensation in reverse order
    const completedSteps = saga.steps
      .filter(s => s.status === 'completed')
      .reverse();
    
    for (const step of completedSteps) {
      // Compensation also needs idempotency!
      const compensationKey = `${step.idempotencyKey}:compensate`;
      await this.executeCompensation(step, compensationKey);
      step.status = 'compensated';
    }
  }
}
 
// Each service checks its derived idempotency key
class FraudService {
  async checkTransaction(
    request: TransactionCheckRequest,
    idempotencyKey: string  // e.g., "saga123:fraud:check"
  ): Promise<FraudCheckResult> {
    
    // Use same idempotency pattern as payment service
    const acquisition = await this.idempotencyStore.acquire(idempotencyKey);
    
    if (acquisition.result === IdempotencyResult.DUPLICATE_COMPLETED) {
      return acquisition.record.responseBody as FraudCheckResult;
    }
    
    const result = await this.performFraudCheck(request);
    
    await this.idempotencyStore.complete(idempotencyKey, result);
    return result;
  }
}

Derive Don't Duplicate

Testing Idempotency Implementations

Idempotency bugs often only manifest under specific race conditions in production. Thorough testing is essential. Here are the critical test scenarios:

idempotency-tests.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
describe('Idempotency', () => {
  
  describe('Basic Idempotency', () => {
    it('should return same response on retry', async () => {
      const key = 'test-key-1';
      const request = { amount: 1000, currency: 'USD' };
      
      const response1 = await handler.processPayment(request, key);
      const response2 = await handler.processPayment(request, key);
      
      expect(response2.id).toEqual(response1.id);
      expect(response2.idempotentReplayed).toBe(true);
    });
    
    it('should reject different request with same key', async () => {
      const key = 'test-key-2';
      
      await handler.processPayment({ amount: 1000 }, key);
      
      await expect(
        handler.processPayment({ amount: 2000 }, key)  // Different amount!
      ).rejects.toThrow(IdempotencyKeyMismatchError);
    });
  });
  
  describe('Concurrent Requests', () => {
    it('should process only once with parallel requests', async () => {
      const key = 'test-key-3';
      const request = { amount: 1000 };
      
      // Fire 10 concurrent requests with same key
      const promises = Array(10).fill(null).map(() =>
        handler.processPayment(request, key)
      );
      
      const responses = await Promise.all(promises);
      
      // All should have same payment ID
      const ids = new Set(responses.map(r => r.id));
      expect(ids.size).toBe(1);
      
      // Gateway should only be called once
      expect(mockGateway.charge).toHaveBeenCalledTimes(1);
    });
  });
  
  describe('Failure Recovery', () => {
    it('should allow retry after gateway timeout', async () => {
      const key = 'test-key-4';
      
      // First request times out at gateway
      mockGateway.charge.mockRejectedValueOnce(new TimeoutError());
      
      await expect(
        handler.processPayment({ amount: 1000 }, key)
      ).rejects.toThrow(TimeoutError);
      
      // Wait for lock to expire
      await sleep(35000);
      
      // Retry should succeed
      mockGateway.charge.mockResolvedValueOnce({ id: 'pay_123' });
      
      const response = await handler.processPayment({ amount: 1000 }, key);
      expect(response.id).toBe('pay_123');
    });
    
    it('should prevent retry after business error', async () => {
      const key = 'test-key-5';
      
      // First request is declined
      mockGateway.charge.mockRejectedValueOnce(
        new CardDeclinedError('insufficient_funds')
      );
      
      await expect(
        handler.processPayment({ amount: 1000 }, key)
      ).rejects.toThrow(CardDeclinedError);
      
      // Retry should return cached decline
      const retryResult = await handler.processPayment({ amount: 1000 }, key);
      expect(retryResult.status).toBe('failed');
      expect(retryResult.idempotentReplayed).toBe(true);
      
      // Gateway should NOT be called again
      expect(mockGateway.charge).toHaveBeenCalledTimes(1);
    });
  });
  
  describe('Edge Cases', () => {
    it('should handle key expiration', async () => {
      const key = 'test-key-6';
      
      await handler.processPayment({ amount: 1000 }, key);
      
      // Simulate key expiration
      await db.query(
        "UPDATE idempotency_keys SET expires_at = NOW() - INTERVAL '1 hour'"
      );
      
      // Should process as new request after expiration
      const response2 = await handler.processPayment({ amount: 1000 }, key);
      expect(response2.idempotentReplayed).toBe(false);
    });
  });
});

Chaos Testing Is Essential

Summary: Idempotency Guarantees

Idempotency is the cornerstone of reliable payment systems. Let's consolidate the key principles:

Key Takeaways

•Idempotency is non-negotiable — Payment systems must guarantee exactly-once processing. At-least-once with reconciliation is not acceptable.
•Client-generated keys — Idempotency keys must be generated by the client, unique per intended operation, and reused for retries of the same operation.
•Database-backed storage — Persist idempotency state with proper locking, response caching, and expiration handling.
•Two-layer idempotency — Implement idempotency at both application layer AND gateway layer to prevent duplicates at every hop.
•Handle all edge cases — Concurrent requests, timeouts, lock expiration, and parameter mismatches all need explicit handling.
•Derived keys for distributed systems — In saga patterns, derive child idempotency keys from the parent key deterministically.
•Test extensively — Concurrent request tests, failure recovery tests, and chaos testing are essential for confidence.

What's Next:

Page Complete

You now understand how to implement bullet-proof idempotency in payment systems. This knowledge directly prevents the most common and damaging failure mode in payment processing: double charges.