Choreography vs Orchestration — The Saga Pattern:… | OneNoughtOne

0/105

System Design (HLD)Saga Pattern

The Saga Pattern: Managing Distributed Transactions

LevelAdvanced

Duration90 mins

TopicSaga Pattern

2 / 5

Choreography vs Orchestration

Two Philosophies of Saga Coordination

Once you've decided to implement the Saga pattern, you face a critical architectural decision: how will the saga steps be coordinated? There are exactly two approaches, each representing a fundamentally different philosophy of distributed system design: Choreography — Decentralized coordination through events. Each service publishes events; other services react to those events. There is no central authority—coordination emerges from the collective behavior of autonomous services. Orchestration — Centralized coordination through commands. A central orchestrator explicitly tells each service what to do and when. The orchestrator maintains the saga's state and makes all decisions about flow control. These approaches aren't merely implementation details—they reflect deep architectural philosophies about coupling, autonomy, observability, and failure handling. Understanding their trade-offs is essential for making the right choice for your system.

What You Will Learn

By the end of this page, you will understand both choreography and orchestration in depth, including their execution models, implementation patterns, failure handling approaches, and the specific conditions under which each excels. You'll be equipped to make an informed architectural decision for any saga requirement.

Choreography: The Decentralized Dance

In choreography, there is no central controller. Instead, each participant service knows only about its immediate predecessors and successors in the saga. Services communicate by publishing and subscribing to domain events. The Dance Metaphor: Imagine a group of dancers performing a routine. In choreography, there's no dance instructor calling out steps. Each dancer watches the others and knows their cue—when dancer A finishes a spin, dancer B knows to start their leap. The coordination emerges from shared understanding of the sequence. Execution Model:

Converting Mermaid diagram...

Key Characteristics: Event-Driven : Services react to events, not commands Peer-to-Peer : No hierarchical relationship between services Emergent Coordination : The saga flow 'emerges' from individual service behaviors Distributed Knowledge : Each service only knows about events it cares about Asynchronous : All communication is asynchronous by nature

choreography-implementation.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
// Choreography-style Saga Implementation
 
// ============================================
// ORDER SERVICE
// ============================================
class OrderService {
  constructor(
    private eventBus: EventBus,
    private orderRepository: OrderRepository
  ) {
    // Subscribe to relevant events
    this.eventBus.subscribe('PaymentFailed', this.handlePaymentFailed.bind(this));
    this.eventBus.subscribe('InventoryReservationFailed', this.handleInventoryFailed.bind(this));
    this.eventBus.subscribe('NotificationSent', this.handleSagaComplete.bind(this));
  }
 
  async createOrder(orderData: CreateOrderRequest): Promise<Order> {
    // T1: Create order (LOCAL transaction)
    const order = await this.orderRepository.create({
      ...orderData,
      status: 'PENDING',
      sagaState: 'IN_PROGRESS'
    });
 
    // Publish event - Inventory Service is listening
    await this.eventBus.publish({
      type: 'OrderCreated',
      payload: {
        orderId: order.id,
        customerId: order.customerId,
        items: order.items,
        timestamp: new Date()
      }
    });
 
    return order;
  }
 
  // C1: Compensating transaction for order creation
  private async handlePaymentFailed(event: PaymentFailedEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CANCELLED',
      sagaState: 'COMPENSATED',
      cancellationReason: 'Payment failed'
    });
    
    // Notify customer of cancellation
    await this.eventBus.publish({
      type: 'OrderCancelled',
      payload: { orderId: event.orderId, reason: 'Payment failed' }
    });
  }
 
  private async handleInventoryFailed(event: InventoryFailedEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CANCELLED',
      sagaState: 'COMPENSATED',
      cancellationReason: 'Item out of stock'
    });
  }
 
  private async handleSagaComplete(event: NotificationSentEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CONFIRMED',
      sagaState: 'COMPLETED'
    });
  }
}
 
// ============================================
// INVENTORY SERVICE
// ============================================
class InventoryService {
  constructor(
    private eventBus: EventBus,
    private inventoryRepository: InventoryRepository
  ) {
    this.eventBus.subscribe('OrderCreated', this.handleOrderCreated.bind(this));
    this.eventBus.subscribe('PaymentFailed', this.handlePaymentFailed.bind(this));
    this.eventBus.subscribe('OrderCancelled', this.handleOrderCancelled.bind(this));
  }
 
  // T2: Reserve inventory
  private async handleOrderCreated(event: OrderCreatedEvent) {
    try {
      for (const item of event.items) {
        await this.inventoryRepository.reserve(item.productId, item.quantity, event.orderId);
      }
 
      await this.eventBus.publish({
        type: 'InventoryReserved',
        payload: {
          orderId: event.orderId,
          items: event.items,
          timestamp: new Date()
        }
      });
    } catch (error) {
      // Reservation failed - publish failure event
      await this.eventBus.publish({
        type: 'InventoryReservationFailed',
        payload: {
          orderId: event.orderId,
          reason: error.message
        }
      });
    }
  }
 
  // C2: Compensating transaction - release reservation
  private async handlePaymentFailed(event: PaymentFailedEvent) {
    await this.inventoryRepository.releaseReservation(event.orderId);
    
    // Note: We don't publish an event here because the OrderCancelled
    // event will be published by OrderService
  }
 
  private async handleOrderCancelled(event: OrderCancelledEvent) {
    await this.inventoryRepository.releaseReservation(event.orderId);
  }
}
 
// ============================================
// PAYMENT SERVICE
// ============================================
class PaymentService {
  constructor(
    private eventBus: EventBus,
    private paymentGateway: PaymentGateway,
    private paymentRepository: PaymentRepository
  ) {
    this.eventBus.subscribe('InventoryReserved', this.handleInventoryReserved.bind(this));
    this.eventBus.subscribe('ShipmentFailed', this.handleShipmentFailed.bind(this));
  }
 
  // T3: Process payment
  private async handleInventoryReserved(event: InventoryReservedEvent) {
    try {
      const payment = await this.paymentGateway.charge({
        orderId: event.orderId,
        amount: this.calculateTotal(event.items)
      });
 
      await this.paymentRepository.create({
        orderId: event.orderId,
        transactionId: payment.transactionId,
        amount: payment.amount,
        status: 'CAPTURED'
      });
 
      await this.eventBus.publish({
        type: 'PaymentProcessed',
        payload: {
          orderId: event.orderId,
          transactionId: payment.transactionId,
          amount: payment.amount
        }
      });
    } catch (error) {
      await this.eventBus.publish({
        type: 'PaymentFailed',
        payload: {
          orderId: event.orderId,
          reason: error.message
        }
      });
    }
  }
 
  // C3: Compensating transaction - refund
  private async handleShipmentFailed(event: ShipmentFailedEvent) {
    const payment = await this.paymentRepository.findByOrderId(event.orderId);
    
    if (payment) {
      await this.paymentGateway.refund(payment.transactionId);
      await this.paymentRepository.update(payment.id, { status: 'REFUNDED' });
    }
  }
}

Choreography: Implementation Considerations

Implementing choreography correctly requires careful attention to several critical concerns:

Critical Implementation Concerns

•Event Ordering Guarantees — Events may arrive out of order due to network delays or parallel processing. Services must handle PaymentProcessed arriving before InventoryReserved (if using a partitioned message queue incorrectly).
•Event Idempotency — The same event may be delivered multiple times. Every event handler must be idempotent—processing the same event twice must not cause incorrect state.
•Correlation — Events must carry correlation IDs (typically the saga ID or order ID) so services can associate events with the correct saga instance.
•Event Schema Evolution — As the system evolves, event schemas change. Services must handle both old and new event formats during transition periods.
•Dead Letter Handling — When event processing fails repeatedly, events must be moved to dead letter queues for manual investigation.

choreography-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
// Idempotent Event Processing Pattern
 
interface ProcessedEvent {
  eventId: string;
  eventType: string;
  processedAt: Date;
  result: 'SUCCESS' | 'FAILED' | 'SKIPPED';
}
 
class IdempotentEventProcessor {
  constructor(
    private processedEventRepo: Repository<ProcessedEvent>,
    private db: Database
  ) {}
 
  async processEvent<T>(
    event: DomainEvent<T>,
    handler: (event: DomainEvent<T>) => Promise<void>
  ): Promise<void> {
    
    // Check if already processed (idempotency check)
    const existing = await this.processedEventRepo.findById(event.id);
    if (existing) {
      console.log(`Event ${event.id} already processed, skipping`);
      return;
    }
 
    // Use transaction to ensure atomic "check-then-process"
    await this.db.transaction(async (tx) => {
      // Double-check within transaction (handles race conditions)
      const existsInTx = await tx.processedEvents.findById(event.id);
      if (existsInTx) {
        return; // Skip if another process handled it
      }
 
      // Record that we're processing this event
      await tx.processedEvents.create({
        eventId: event.id,
        eventType: event.type,
        processedAt: new Date(),
        result: 'PROCESSING'
      });
 
      try {
        // Execute the actual handler
        await handler(event);
        
        // Mark as successfully processed
        await tx.processedEvents.update(event.id, { result: 'SUCCESS' });
      } catch (error) {
        // Mark as failed (will be retried or sent to DLQ)
        await tx.processedEvents.update(event.id, { result: 'FAILED' });
        throw error;
      }
    });
  }
}
 
// Usage in Inventory Service:
class InventoryService {
  constructor(
    private processor: IdempotentEventProcessor,
    private inventoryRepo: InventoryRepository
  ) {}
 
  async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    await this.processor.processEvent(event, async (e) => {
      // This code runs at most once per event.id
      for (const item of e.items) {
        await this.inventoryRepo.reserve(
          item.productId,
          item.quantity,
          e.orderId
        );
      }
    });
  }
}

The Dual-Write Danger

A common choreography pitfall is the dual-write problem : updating the database AND publishing an event in separate steps. If the service crashes between these operations, you either lose the event (database updated, event not published) or publish a ghost event (event published, database not updated). Solutions include the Transactional Outbox Pattern (covered in Module 4) or Event Sourcing (covered in Module 2).

Orchestration: The Centralized Conductor

In orchestration, a dedicated Saga Orchestrator (also called Saga Execution Coordinator or SEC) explicitly controls the saga flow. The orchestrator knows the complete saga definition and tells each participant service exactly what to do. The Orchestra Metaphor: Imagine a symphony orchestra. Musicians don't watch each other—they watch the conductor. The conductor has the complete score and signals each section when to play. The coordination is explicit and centralized. Execution Model:

Converting Mermaid diagram...

Key Characteristics: Command-Driven : Orchestrator sends explicit commands, not implicit events Centralized Control : Single component owns the saga definition Explicit Flow : The saga sequence is clearly defined in one place State Machine : Orchestrator maintains saga state explicitly Request-Reply : Communication follows command/reply pattern (though still async)

orchestration-implementation.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
// Orchestration-style Saga Implementation
 
// ============================================
// SAGA DEFINITION (Declarative)
// ============================================
interface SagaStep<TInput, TOutput> {
  name: string;
  execute: (input: TInput, context: SagaContext) => Promise<TOutput>;
  compensate: (input: TInput, context: SagaContext) => Promise<void>;
}
 
interface OrderSagaData {
  orderId?: string;
  customerId: string;
  items: OrderItem[];
  reservationId?: string;
  paymentId?: string;
  shipmentId?: string;
}
 
// The saga definition - a sequence of steps
const orderSagaDefinition: SagaStep<OrderSagaData, OrderSagaData>[] = [
  {
    name: 'CreateOrder',
    execute: async (data, ctx) => {
      const result = await ctx.services.order.createOrder({
        customerId: data.customerId,
        items: data.items
      });
      return { ...data, orderId: result.orderId };
    },
    compensate: async (data, ctx) => {
      if (data.orderId) {
        await ctx.services.order.cancelOrder(data.orderId);
      }
    }
  },
  {
    name: 'ReserveInventory',
    execute: async (data, ctx) => {
      const result = await ctx.services.inventory.reserve({
        orderId: data.orderId!,
        items: data.items
      });
      return { ...data, reservationId: result.reservationId };
    },
    compensate: async (data, ctx) => {
      if (data.reservationId) {
        await ctx.services.inventory.releaseReservation(data.reservationId);
      }
    }
  },
  {
    name: 'ProcessPayment',
    execute: async (data, ctx) => {
      const result = await ctx.services.payment.processPayment({
        orderId: data.orderId!,
        customerId: data.customerId,
        amount: calculateTotal(data.items)
      });
      return { ...data, paymentId: result.paymentId };
    },
    compensate: async (data, ctx) => {
      if (data.paymentId) {
        await ctx.services.payment.refund(data.paymentId);
      }
    }
  },
  {
    name: 'ScheduleShipment',
    execute: async (data, ctx) => {
      const result = await ctx.services.shipping.scheduleShipment({
        orderId: data.orderId!,
        items: data.items
      });
      return { ...data, shipmentId: result.shipmentId };
    },
    compensate: async (data, ctx) => {
      if (data.shipmentId) {
        await ctx.services.shipping.cancelShipment(data.shipmentId);
      }
    }
  }
];
 
// ============================================
// SAGA ORCHESTRATOR ENGINE
// ============================================
interface SagaInstance {
  id: string;
  definition: string;
  currentStep: number;
  state: 'RUNNING' | 'COMPLETED' | 'COMPENSATING' | 'FAILED';
  data: Record<string, unknown>;
  completedSteps: string[];
  error?: string;
}
 
class SagaOrchestrator {
  constructor(
    private sagaRepository: SagaRepository,
    private serviceContext: SagaContext
  ) {}
 
  async executeSaga<T>(
    sagaId: string,
    definition: SagaStep<T, T>[],
    initialData: T
  ): Promise<SagaResult<T>> {
    
    // Create saga instance
    let instance = await this.sagaRepository.create({
      id: sagaId,
      definition: 'OrderSaga',
      currentStep: 0,
      state: 'RUNNING',
      data: initialData as Record<string, unknown>,
      completedSteps: []
    });
 
    let data = initialData;
 
    try {
      // Execute each step in sequence
      for (let i = 0; i < definition.length; i++) {
        const step = definition[i];
        
        console.log(`Executing step ${i + 1}/${definition.length}: ${step.name}`);
        
        // Execute the step
        data = await step.execute(data, this.serviceContext);
        
        // Persist progress after each step
        instance = await this.sagaRepository.update(instance.id, {
          currentStep: i + 1,
          data: data as Record<string, unknown>,
          completedSteps: [...instance.completedSteps, step.name]
        });
      }
 
      // Mark as completed
      await this.sagaRepository.update(instance.id, { state: 'COMPLETED' });
      
      return { success: true, data };
 
    } catch (error) {
      console.error(`Saga failed at step ${instance.currentStep}: ${error}`);
      
      // Begin compensation
      await this.compensate(instance, definition, data);
      
      return { success: false, error: error.message };
    }
  }
 
  private async compensate<T>(
    instance: SagaInstance,
    definition: SagaStep<T, T>[],
    data: T
  ): Promise<void> {
    
    await this.sagaRepository.update(instance.id, { state: 'COMPENSATING' });
 
    // Compensate in REVERSE order, starting from the last completed step
    for (let i = instance.currentStep - 1; i >= 0; i--) {
      const step = definition[i];
      
      console.log(`Compensating step ${i + 1}: ${step.name}`);
      
      try {
        await step.compensate(data, this.serviceContext);
      } catch (compensationError) {
        // Compensation failure is serious - log and continue
        console.error(`Compensation failed for ${step.name}: ${compensationError}`);
        // In production: alert, manual intervention queue, etc.
      }
    }
 
    await this.sagaRepository.update(instance.id, { state: 'FAILED' });
  }
}
 
// ============================================
// USAGE
// ============================================
const orchestrator = new SagaOrchestrator(sagaRepo, serviceContext);
 
const result = await orchestrator.executeSaga(
  'saga-' + uuid(),
  orderSagaDefinition,
  {
    customerId: 'customer-123',
    items: [
      { productId: 'prod-1', quantity: 2, price: 29.99 },
      { productId: 'prod-2', quantity: 1, price: 49.99 }
    ]
  }
);
 
if (result.success) {
  console.log('Order saga completed:', result.data);
} else {
  console.log('Order saga failed and compensated:', result.error);
}

Orchestration: Advanced Patterns

Production-grade orchestration requires sophisticated patterns beyond the basic execution loop:

Advanced Orchestration Patterns

•Saga State Persistence — The orchestrator must persist saga state after each step. If it crashes, it must resume from the last persisted state, not restart from scratch.
•Asynchronous Service Communication — Despite the request-reply model, communication with participant services should be asynchronous. The orchestrator sends a command, waits for a reply event, with timeout handling.
•Parallel Step Execution — Some steps can execute in parallel (e.g., notify customer AND update analytics). Orchestrators should support step dependencies and parallel execution.
•Conditional Branching — Real sagas often have conditional logic: 'if premium customer, skip fraud check'. Orchestrators must support dynamic flow control.
•Step Retry with Backoff — Transient failures should be retried with exponential backoff before triggering compensation.

advanced-orchestrator.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
// Advanced Orchestrator with State Machine
 
type SagaState = 
  | 'STARTED'
  | 'ORDER_CREATED'
  | 'INVENTORY_RESERVED'
  | 'PAYMENT_PROCESSED'
  | 'SHIPMENT_SCHEDULED'
  | 'COMPLETED'
  | 'COMPENSATING_SHIPMENT'
  | 'COMPENSATING_PAYMENT'
  | 'COMPENSATING_INVENTORY'
  | 'COMPENSATING_ORDER'
  | 'FAILED';
 
interface StateTransition {
  from: SagaState;
  event: string;
  to: SagaState;
  action: (context: SagaContext) => Promise<void>;
}
 
// Define saga as explicit state machine
const orderSagaStateMachine: StateTransition[] = [
  // Happy path transitions
  {
    from: 'STARTED',
    event: 'CREATE_ORDER',
    to: 'ORDER_CREATED',
    action: async (ctx) => {
      const order = await ctx.services.order.create(ctx.data);
      ctx.data.orderId = order.id;
      await ctx.emit('RESERVE_INVENTORY');
    }
  },
  {
    from: 'ORDER_CREATED',
    event: 'INVENTORY_RESERVED_SUCCESS',
    to: 'INVENTORY_RESERVED',
    action: async (ctx) => {
      ctx.data.reservationId = ctx.event.reservationId;
      await ctx.emit('PROCESS_PAYMENT');
    }
  },
  {
    from: 'INVENTORY_RESERVED',
    event: 'PAYMENT_SUCCESS',
    to: 'PAYMENT_PROCESSED',
    action: async (ctx) => {
      ctx.data.paymentId = ctx.event.paymentId;
      await ctx.emit('SCHEDULE_SHIPMENT');
    }
  },
  {
    from: 'PAYMENT_PROCESSED',
    event: 'SHIPMENT_SCHEDULED_SUCCESS',
    to: 'SHIPMENT_SCHEDULED',
    action: async (ctx) => {
      ctx.data.shipmentId = ctx.event.shipmentId;
      await ctx.emit('COMPLETE_SAGA');
    }
  },
  {
    from: 'SHIPMENT_SCHEDULED',
    event: 'COMPLETE_SAGA',
    to: 'COMPLETED',
    action: async (ctx) => {
      await ctx.services.notification.sendConfirmation(ctx.data.orderId);
      console.log('Saga completed successfully');
    }
  },
 
  // Compensation transitions (failure handling)
  {
    from: 'PAYMENT_PROCESSED',
    event: 'SHIPMENT_FAILED',
    to: 'COMPENSATING_PAYMENT',
    action: async (ctx) => {
      await ctx.services.payment.refund(ctx.data.paymentId);
      await ctx.emit('PAYMENT_REFUNDED');
    }
  },
  {
    from: 'COMPENSATING_PAYMENT',
    event: 'PAYMENT_REFUNDED',
    to: 'COMPENSATING_INVENTORY',
    action: async (ctx) => {
      await ctx.services.inventory.release(ctx.data.reservationId);
      await ctx.emit('INVENTORY_RELEASED');
    }
  },
  {
    from: 'COMPENSATING_INVENTORY',
    event: 'INVENTORY_RELEASED',
    to: 'COMPENSATING_ORDER',
    action: async (ctx) => {
      await ctx.services.order.cancel(ctx.data.orderId);
      await ctx.emit('ORDER_CANCELLED');
    }
  },
  {
    from: 'COMPENSATING_ORDER',
    event: 'ORDER_CANCELLED',
    to: 'FAILED',
    action: async (ctx) => {
      await ctx.services.notification.sendFailure(ctx.data.orderId, ctx.error);
      console.log('Saga failed and fully compensated');
    }
  },
 
  // Direct failure transitions from any step
  {
    from: 'INVENTORY_RESERVED',
    event: 'PAYMENT_FAILED',
    to: 'COMPENSATING_INVENTORY',
    action: async (ctx) => {
      await ctx.services.inventory.release(ctx.data.reservationId);
      await ctx.emit('INVENTORY_RELEASED');
    }
  },
  {
    from: 'ORDER_CREATED',
    event: 'INVENTORY_RESERVATION_FAILED',
    to: 'COMPENSATING_ORDER',
    action: async (ctx) => {
      await ctx.services.order.cancel(ctx.data.orderId);
      await ctx.emit('ORDER_CANCELLED');
    }
  }
];
 
// State machine executor
class StateMachineOrchestrator {
  private transitions: Map<string, StateTransition> = new Map();
 
  constructor(definition: StateTransition[]) {
    for (const t of definition) {
      const key = `${t.from}:${t.event}`;
      this.transitions.set(key, t);
    }
  }
 
  async processEvent(saga: SagaInstance, event: SagaEvent): Promise<SagaState> {
    const transitionKey = `${saga.state}:${event.type}`;
    const transition = this.transitions.get(transitionKey);
 
    if (!transition) {
      throw new Error(
        `No transition from state '${saga.state}' for event '${event.type}'`
      );
    }
 
    const context: SagaContext = {
      data: saga.data,
      event: event.payload,
      services: this.serviceClients,
      emit: async (eventType: string) => {
        await this.sagaEventQueue.publish({
          sagaId: saga.id,
          type: eventType,
          payload: saga.data
        });
      }
    };
 
    // Execute the transition action
    await transition.action(context);
 
    // Update and persist the new state
    saga.state = transition.to;
    saga.data = context.data;
    await this.sagaRepository.update(saga.id, saga);
 
    return transition.to;
  }
}

Choreography vs Orchestration: The Complete Comparison

Now that we understand both approaches deeply, let's compare them across all relevant dimensions:

Comprehensive Choreography vs Orchestration Comparison
Dimension	Choreography	Orchestration
Coupling	Loose - services only know about events	Tighter - orchestrator knows all services
Single Point of Failure	None - fully distributed	Orchestrator is SPOF (mitigated by HA)
Flow Visibility	Scattered across services	Centralized in orchestrator
Debugging Complexity	Hard - must trace events across services	Easier - saga state visible in one place
Testing	Complex - requires integration testing	Simpler - can unit test saga logic
Adding New Steps	Modify only affected services	Modify orchestrator and add service
Cyclic Dependencies	Risk of event loops	Not possible - orchestrator controls flow
Saga State	Implicit in service states	Explicit in orchestrator
Recovery	Complex - reconstruct from events	Simple - resume from persisted state
Parallel Steps	Natural with event fanout	Requires explicit coordination
Team Autonomy	High - services evolve independently	Lower - orchestrator changes affect all
Coordination Overhead	Low - just event bus	Higher - orchestrator infrastructure
Best For	Simple sagas, 3-5 steps	Complex sagas, many steps/branches

Choose Choreography When

• Saga has 3-5 simple, linear steps • Team autonomy is paramount • Services already use an event-driven architecture • Saga logic rarely changes • You have strong distributed tracing capabilities • Each service team owns their saga participation

Choose Orchestration When

• Saga has many steps (5+) with complex branching • Business logic changes frequently • You need clear visibility into saga state • Multiple teams must coordinate closely • Sagas span organizational boundaries • You're using a saga framework (Temporal, Camunda)

Industry Saga Frameworks

Building saga infrastructure from scratch is complex. Production systems typically leverage battle-tested frameworks that handle the difficult aspects: state persistence, retry logic, timeout handling, and observability.

Temporal (formerly Cadence at Uber) is the leading orchestration framework for complex workflows including sagas. Key Features: Durable Execution : Workflow state survives crashes, restarts, deployments Saga Pattern Built-in : Native support for compensations Language SDKs : Go, Java, TypeScript, Python, PHP Visibility : Full workflow history and real-time state Scalability : Powers Uber's 5,000+ microservices Used By : Uber, Netflix, Snap, Coinbase, Box, Stripe

TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// Temporal Saga Workflow Example
import { proxyActivities, ApplicationFailure } from '@temporalio/workflow';
import type * as activities from './activities';
 
const { 
  createOrder, 
  reserveInventory, 
  processPayment, 
  scheduleShipment,
  // Compensations
  cancelOrder,
  releaseInventory,
  refundPayment
} = proxyActivities<typeof activities>({
  startToCloseTimeout: '30 seconds',
  retry: { maximumAttempts: 3 }
});
 
export async function orderSaga(orderData: OrderData): Promise<OrderResult> {
  const compensations: Array<() => Promise<void>> = [];
 
  try {
    // Step 1: Create Order
    const orderId = await createOrder(orderData);
    compensations.push(() => cancelOrder(orderId));
 
    // Step 2: Reserve Inventory  
    const reservationId = await reserveInventory(orderId, orderData.items);
    compensations.push(() => releaseInventory(reservationId));
 
    // Step 3: Process Payment
    const paymentId = await processPayment(orderId, orderData.amount);
    compensations.push(() => refundPayment(paymentId));
 
    // Step 4: Schedule Shipment
    const shipmentId = await scheduleShipment(orderId, orderData.address);
    // No compensation needed - if we got here, we're done
 
    return { success: true, orderId, shipmentId };
 
  } catch (error) {
    // Execute compensations in reverse order
    for (const compensate of compensations.reverse()) {
      try {
        await compensate();
      } catch (compensationError) {
        // Log but continue with other compensations
        console.error('Compensation failed:', compensationError);
      }
    }
 
    throw ApplicationFailure.nonRetryable(
      'Order saga failed and was compensated',
      'SAGA_FAILED',
      { originalError: error }
    );
  }
}

Hybrid Approaches: The Best of Both Worlds

In practice, many systems use hybrid approaches that combine choreography and orchestration strategically: Pattern 1: Orchestration Within, Choreography Between Each bounded context uses internal orchestration for complex workflows, but contexts communicate via events (choreography).

Converting Mermaid diagram...

Pattern 2: Choreography for Happy Path, Orchestration for Compensation The forward flow uses event choreography for its simplicity and loose coupling. When failures occur, a compensation orchestrator is triggered to coordinate the rollback. Pattern 3: Saga Choreography with Saga Coordinator Events drive the saga forward, but a lightweight coordinator service listens to all events for: Monitoring saga progress Detecting stuck sagas (timeouts) Triggering compensation when needed Providing saga visibility This preserves choreography's loose coupling while adding centralized observability.

Real-World Pragmatism

Production systems rarely fit neatly into 'choreography OR orchestration'. Start with the simpler approach (usually choreography for simple sagas), add orchestration infrastructure when complexity demands it. The goal is managing complexity, not architectural purity.

System Design (HLD)Saga Pattern

The Saga Pattern: Managing Distributed Transactions

LevelAdvanced

Duration90 mins

TopicSaga Pattern

2 / 5

Choreography vs Orchestration

Two Philosophies of Saga Coordination

Once you've decided to implement the Saga pattern, you face a critical architectural decision: how will the saga steps be coordinated? There are exactly two approaches, each representing a fundamentally different philosophy of distributed system design: Choreography — Decentralized coordination through events. Each service publishes events; other services react to those events. There is no central authority—coordination emerges from the collective behavior of autonomous services. Orchestration — Centralized coordination through commands. A central orchestrator explicitly tells each service what to do and when. The orchestrator maintains the saga's state and makes all decisions about flow control. These approaches aren't merely implementation details—they reflect deep architectural philosophies about coupling, autonomy, observability, and failure handling. Understanding their trade-offs is essential for making the right choice for your system.

What You Will Learn

By the end of this page, you will understand both choreography and orchestration in depth, including their execution models, implementation patterns, failure handling approaches, and the specific conditions under which each excels. You'll be equipped to make an informed architectural decision for any saga requirement.

Choreography: The Decentralized Dance

In choreography, there is no central controller. Instead, each participant service knows only about its immediate predecessors and successors in the saga. Services communicate by publishing and subscribing to domain events. The Dance Metaphor: Imagine a group of dancers performing a routine. In choreography, there's no dance instructor calling out steps. Each dancer watches the others and knows their cue—when dancer A finishes a spin, dancer B knows to start their leap. The coordination emerges from shared understanding of the sequence. Execution Model:

Converting Mermaid diagram...

Key Characteristics: Event-Driven : Services react to events, not commands Peer-to-Peer : No hierarchical relationship between services Emergent Coordination : The saga flow 'emerges' from individual service behaviors Distributed Knowledge : Each service only knows about events it cares about Asynchronous : All communication is asynchronous by nature

choreography-implementation.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
// Choreography-style Saga Implementation
 
// ============================================
// ORDER SERVICE
// ============================================
class OrderService {
  constructor(
    private eventBus: EventBus,
    private orderRepository: OrderRepository
  ) {
    // Subscribe to relevant events
    this.eventBus.subscribe('PaymentFailed', this.handlePaymentFailed.bind(this));
    this.eventBus.subscribe('InventoryReservationFailed', this.handleInventoryFailed.bind(this));
    this.eventBus.subscribe('NotificationSent', this.handleSagaComplete.bind(this));
  }
 
  async createOrder(orderData: CreateOrderRequest): Promise<Order> {
    // T1: Create order (LOCAL transaction)
    const order = await this.orderRepository.create({
      ...orderData,
      status: 'PENDING',
      sagaState: 'IN_PROGRESS'
    });
 
    // Publish event - Inventory Service is listening
    await this.eventBus.publish({
      type: 'OrderCreated',
      payload: {
        orderId: order.id,
        customerId: order.customerId,
        items: order.items,
        timestamp: new Date()
      }
    });
 
    return order;
  }
 
  // C1: Compensating transaction for order creation
  private async handlePaymentFailed(event: PaymentFailedEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CANCELLED',
      sagaState: 'COMPENSATED',
      cancellationReason: 'Payment failed'
    });
    
    // Notify customer of cancellation
    await this.eventBus.publish({
      type: 'OrderCancelled',
      payload: { orderId: event.orderId, reason: 'Payment failed' }
    });
  }
 
  private async handleInventoryFailed(event: InventoryFailedEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CANCELLED',
      sagaState: 'COMPENSATED',
      cancellationReason: 'Item out of stock'
    });
  }
 
  private async handleSagaComplete(event: NotificationSentEvent) {
    await this.orderRepository.update(event.orderId, {
      status: 'CONFIRMED',
      sagaState: 'COMPLETED'
    });
  }
}
 
// ============================================
// INVENTORY SERVICE
// ============================================
class InventoryService {
  constructor(
    private eventBus: EventBus,
    private inventoryRepository: InventoryRepository
  ) {
    this.eventBus.subscribe('OrderCreated', this.handleOrderCreated.bind(this));
    this.eventBus.subscribe('PaymentFailed', this.handlePaymentFailed.bind(this));
    this.eventBus.subscribe('OrderCancelled', this.handleOrderCancelled.bind(this));
  }
 
  // T2: Reserve inventory
  private async handleOrderCreated(event: OrderCreatedEvent) {
    try {
      for (const item of event.items) {
        await this.inventoryRepository.reserve(item.productId, item.quantity, event.orderId);
      }
 
      await this.eventBus.publish({
        type: 'InventoryReserved',
        payload: {
          orderId: event.orderId,
          items: event.items,
          timestamp: new Date()
        }
      });
    } catch (error) {
      // Reservation failed - publish failure event
      await this.eventBus.publish({
        type: 'InventoryReservationFailed',
        payload: {
          orderId: event.orderId,
          reason: error.message
        }
      });
    }
  }
 
  // C2: Compensating transaction - release reservation
  private async handlePaymentFailed(event: PaymentFailedEvent) {
    await this.inventoryRepository.releaseReservation(event.orderId);
    
    // Note: We don't publish an event here because the OrderCancelled
    // event will be published by OrderService
  }
 
  private async handleOrderCancelled(event: OrderCancelledEvent) {
    await this.inventoryRepository.releaseReservation(event.orderId);
  }
}
 
// ============================================
// PAYMENT SERVICE
// ============================================
class PaymentService {
  constructor(
    private eventBus: EventBus,
    private paymentGateway: PaymentGateway,
    private paymentRepository: PaymentRepository
  ) {
    this.eventBus.subscribe('InventoryReserved', this.handleInventoryReserved.bind(this));
    this.eventBus.subscribe('ShipmentFailed', this.handleShipmentFailed.bind(this));
  }
 
  // T3: Process payment
  private async handleInventoryReserved(event: InventoryReservedEvent) {
    try {
      const payment = await this.paymentGateway.charge({
        orderId: event.orderId,
        amount: this.calculateTotal(event.items)
      });
 
      await this.paymentRepository.create({
        orderId: event.orderId,
        transactionId: payment.transactionId,
        amount: payment.amount,
        status: 'CAPTURED'
      });
 
      await this.eventBus.publish({
        type: 'PaymentProcessed',
        payload: {
          orderId: event.orderId,
          transactionId: payment.transactionId,
          amount: payment.amount
        }
      });
    } catch (error) {
      await this.eventBus.publish({
        type: 'PaymentFailed',
        payload: {
          orderId: event.orderId,
          reason: error.message
        }
      });
    }
  }
 
  // C3: Compensating transaction - refund
  private async handleShipmentFailed(event: ShipmentFailedEvent) {
    const payment = await this.paymentRepository.findByOrderId(event.orderId);
    
    if (payment) {
      await this.paymentGateway.refund(payment.transactionId);
      await this.paymentRepository.update(payment.id, { status: 'REFUNDED' });
    }
  }
}

Choreography: Implementation Considerations

Implementing choreography correctly requires careful attention to several critical concerns:

Critical Implementation Concerns

•Event Ordering Guarantees — Events may arrive out of order due to network delays or parallel processing. Services must handle PaymentProcessed arriving before InventoryReserved (if using a partitioned message queue incorrectly).
•Event Idempotency — The same event may be delivered multiple times. Every event handler must be idempotent—processing the same event twice must not cause incorrect state.
•Correlation — Events must carry correlation IDs (typically the saga ID or order ID) so services can associate events with the correct saga instance.
•Event Schema Evolution — As the system evolves, event schemas change. Services must handle both old and new event formats during transition periods.
•Dead Letter Handling — When event processing fails repeatedly, events must be moved to dead letter queues for manual investigation.

choreography-idempotency.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
// Idempotent Event Processing Pattern
 
interface ProcessedEvent {
  eventId: string;
  eventType: string;
  processedAt: Date;
  result: 'SUCCESS' | 'FAILED' | 'SKIPPED';
}
 
class IdempotentEventProcessor {
  constructor(
    private processedEventRepo: Repository<ProcessedEvent>,
    private db: Database
  ) {}
 
  async processEvent<T>(
    event: DomainEvent<T>,
    handler: (event: DomainEvent<T>) => Promise<void>
  ): Promise<void> {
    
    // Check if already processed (idempotency check)
    const existing = await this.processedEventRepo.findById(event.id);
    if (existing) {
      console.log(`Event ${event.id} already processed, skipping`);
      return;
    }
 
    // Use transaction to ensure atomic "check-then-process"
    await this.db.transaction(async (tx) => {
      // Double-check within transaction (handles race conditions)
      const existsInTx = await tx.processedEvents.findById(event.id);
      if (existsInTx) {
        return; // Skip if another process handled it
      }
 
      // Record that we're processing this event
      await tx.processedEvents.create({
        eventId: event.id,
        eventType: event.type,
        processedAt: new Date(),
        result: 'PROCESSING'
      });
 
      try {
        // Execute the actual handler
        await handler(event);
        
        // Mark as successfully processed
        await tx.processedEvents.update(event.id, { result: 'SUCCESS' });
      } catch (error) {
        // Mark as failed (will be retried or sent to DLQ)
        await tx.processedEvents.update(event.id, { result: 'FAILED' });
        throw error;
      }
    });
  }
}
 
// Usage in Inventory Service:
class InventoryService {
  constructor(
    private processor: IdempotentEventProcessor,
    private inventoryRepo: InventoryRepository
  ) {}
 
  async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
    await this.processor.processEvent(event, async (e) => {
      // This code runs at most once per event.id
      for (const item of e.items) {
        await this.inventoryRepo.reserve(
          item.productId,
          item.quantity,
          e.orderId
        );
      }
    });
  }
}

The Dual-Write Danger

A common choreography pitfall is the dual-write problem : updating the database AND publishing an event in separate steps. If the service crashes between these operations, you either lose the event (database updated, event not published) or publish a ghost event (event published, database not updated). Solutions include the Transactional Outbox Pattern (covered in Module 4) or Event Sourcing (covered in Module 2).

Orchestration: The Centralized Conductor

In orchestration, a dedicated Saga Orchestrator (also called Saga Execution Coordinator or SEC) explicitly controls the saga flow. The orchestrator knows the complete saga definition and tells each participant service exactly what to do. The Orchestra Metaphor: Imagine a symphony orchestra. Musicians don't watch each other—they watch the conductor. The conductor has the complete score and signals each section when to play. The coordination is explicit and centralized. Execution Model:

Converting Mermaid diagram...

Key Characteristics: Command-Driven : Orchestrator sends explicit commands, not implicit events Centralized Control : Single component owns the saga definition Explicit Flow : The saga sequence is clearly defined in one place State Machine : Orchestrator maintains saga state explicitly Request-Reply : Communication follows command/reply pattern (though still async)

orchestration-implementation.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
// Orchestration-style Saga Implementation
 
// ============================================
// SAGA DEFINITION (Declarative)
// ============================================
interface SagaStep<TInput, TOutput> {
  name: string;
  execute: (input: TInput, context: SagaContext) => Promise<TOutput>;
  compensate: (input: TInput, context: SagaContext) => Promise<void>;
}
 
interface OrderSagaData {
  orderId?: string;
  customerId: string;
  items: OrderItem[];
  reservationId?: string;
  paymentId?: string;
  shipmentId?: string;
}
 
// The saga definition - a sequence of steps
const orderSagaDefinition: SagaStep<OrderSagaData, OrderSagaData>[] = [
  {
    name: 'CreateOrder',
    execute: async (data, ctx) => {
      const result = await ctx.services.order.createOrder({
        customerId: data.customerId,
        items: data.items
      });
      return { ...data, orderId: result.orderId };
    },
    compensate: async (data, ctx) => {
      if (data.orderId) {
        await ctx.services.order.cancelOrder(data.orderId);
      }
    }
  },
  {
    name: 'ReserveInventory',
    execute: async (data, ctx) => {
      const result = await ctx.services.inventory.reserve({
        orderId: data.orderId!,
        items: data.items
      });
      return { ...data, reservationId: result.reservationId };
    },
    compensate: async (data, ctx) => {
      if (data.reservationId) {
        await ctx.services.inventory.releaseReservation(data.reservationId);
      }
    }
  },
  {
    name: 'ProcessPayment',
    execute: async (data, ctx) => {
      const result = await ctx.services.payment.processPayment({
        orderId: data.orderId!,
        customerId: data.customerId,
        amount: calculateTotal(data.items)
      });
      return { ...data, paymentId: result.paymentId };
    },
    compensate: async (data, ctx) => {
      if (data.paymentId) {
        await ctx.services.payment.refund(data.paymentId);
      }
    }
  },
  {
    name: 'ScheduleShipment',
    execute: async (data, ctx) => {
      const result = await ctx.services.shipping.scheduleShipment({
        orderId: data.orderId!,
        items: data.items
      });
      return { ...data, shipmentId: result.shipmentId };
    },
    compensate: async (data, ctx) => {
      if (data.shipmentId) {
        await ctx.services.shipping.cancelShipment(data.shipmentId);
      }
    }
  }
];
 
// ============================================
// SAGA ORCHESTRATOR ENGINE
// ============================================
interface SagaInstance {
  id: string;
  definition: string;
  currentStep: number;
  state: 'RUNNING' | 'COMPLETED' | 'COMPENSATING' | 'FAILED';
  data: Record<string, unknown>;
  completedSteps: string[];
  error?: string;
}
 
class SagaOrchestrator {
  constructor(
    private sagaRepository: SagaRepository,
    private serviceContext: SagaContext
  ) {}
 
  async executeSaga<T>(
    sagaId: string,
    definition: SagaStep<T, T>[],
    initialData: T
  ): Promise<SagaResult<T>> {
    
    // Create saga instance
    let instance = await this.sagaRepository.create({
      id: sagaId,
      definition: 'OrderSaga',
      currentStep: 0,
      state: 'RUNNING',
      data: initialData as Record<string, unknown>,
      completedSteps: []
    });
 
    let data = initialData;
 
    try {
      // Execute each step in sequence
      for (let i = 0; i < definition.length; i++) {
        const step = definition[i];
        
        console.log(`Executing step ${i + 1}/${definition.length}: ${step.name}`);
        
        // Execute the step
        data = await step.execute(data, this.serviceContext);
        
        // Persist progress after each step
        instance = await this.sagaRepository.update(instance.id, {
          currentStep: i + 1,
          data: data as Record<string, unknown>,
          completedSteps: [...instance.completedSteps, step.name]
        });
      }
 
      // Mark as completed
      await this.sagaRepository.update(instance.id, { state: 'COMPLETED' });
      
      return { success: true, data };
 
    } catch (error) {
      console.error(`Saga failed at step ${instance.currentStep}: ${error}`);
      
      // Begin compensation
      await this.compensate(instance, definition, data);
      
      return { success: false, error: error.message };
    }
  }
 
  private async compensate<T>(
    instance: SagaInstance,
    definition: SagaStep<T, T>[],
    data: T
  ): Promise<void> {
    
    await this.sagaRepository.update(instance.id, { state: 'COMPENSATING' });
 
    // Compensate in REVERSE order, starting from the last completed step
    for (let i = instance.currentStep - 1; i >= 0; i--) {
      const step = definition[i];
      
      console.log(`Compensating step ${i + 1}: ${step.name}`);
      
      try {
        await step.compensate(data, this.serviceContext);
      } catch (compensationError) {
        // Compensation failure is serious - log and continue
        console.error(`Compensation failed for ${step.name}: ${compensationError}`);
        // In production: alert, manual intervention queue, etc.
      }
    }
 
    await this.sagaRepository.update(instance.id, { state: 'FAILED' });
  }
}
 
// ============================================
// USAGE
// ============================================
const orchestrator = new SagaOrchestrator(sagaRepo, serviceContext);
 
const result = await orchestrator.executeSaga(
  'saga-' + uuid(),
  orderSagaDefinition,
  {
    customerId: 'customer-123',
    items: [
      { productId: 'prod-1', quantity: 2, price: 29.99 },
      { productId: 'prod-2', quantity: 1, price: 49.99 }
    ]
  }
);
 
if (result.success) {
  console.log('Order saga completed:', result.data);
} else {
  console.log('Order saga failed and compensated:', result.error);
}

Orchestration: Advanced Patterns

Production-grade orchestration requires sophisticated patterns beyond the basic execution loop:

Advanced Orchestration Patterns

•Saga State Persistence — The orchestrator must persist saga state after each step. If it crashes, it must resume from the last persisted state, not restart from scratch.
•Asynchronous Service Communication — Despite the request-reply model, communication with participant services should be asynchronous. The orchestrator sends a command, waits for a reply event, with timeout handling.
•Parallel Step Execution — Some steps can execute in parallel (e.g., notify customer AND update analytics). Orchestrators should support step dependencies and parallel execution.
•Conditional Branching — Real sagas often have conditional logic: 'if premium customer, skip fraud check'. Orchestrators must support dynamic flow control.
•Step Retry with Backoff — Transient failures should be retried with exponential backoff before triggering compensation.

advanced-orchestrator.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
// Advanced Orchestrator with State Machine
 
type SagaState = 
  | 'STARTED'
  | 'ORDER_CREATED'
  | 'INVENTORY_RESERVED'
  | 'PAYMENT_PROCESSED'
  | 'SHIPMENT_SCHEDULED'
  | 'COMPLETED'
  | 'COMPENSATING_SHIPMENT'
  | 'COMPENSATING_PAYMENT'
  | 'COMPENSATING_INVENTORY'
  | 'COMPENSATING_ORDER'
  | 'FAILED';
 
interface StateTransition {
  from: SagaState;
  event: string;
  to: SagaState;
  action: (context: SagaContext) => Promise<void>;
}
 
// Define saga as explicit state machine
const orderSagaStateMachine: StateTransition[] = [
  // Happy path transitions
  {
    from: 'STARTED',
    event: 'CREATE_ORDER',
    to: 'ORDER_CREATED',
    action: async (ctx) => {
      const order = await ctx.services.order.create(ctx.data);
      ctx.data.orderId = order.id;
      await ctx.emit('RESERVE_INVENTORY');
    }
  },
  {
    from: 'ORDER_CREATED',
    event: 'INVENTORY_RESERVED_SUCCESS',
    to: 'INVENTORY_RESERVED',
    action: async (ctx) => {
      ctx.data.reservationId = ctx.event.reservationId;
      await ctx.emit('PROCESS_PAYMENT');
    }
  },
  {
    from: 'INVENTORY_RESERVED',
    event: 'PAYMENT_SUCCESS',
    to: 'PAYMENT_PROCESSED',
    action: async (ctx) => {
      ctx.data.paymentId = ctx.event.paymentId;
      await ctx.emit('SCHEDULE_SHIPMENT');
    }
  },
  {
    from: 'PAYMENT_PROCESSED',
    event: 'SHIPMENT_SCHEDULED_SUCCESS',
    to: 'SHIPMENT_SCHEDULED',
    action: async (ctx) => {
      ctx.data.shipmentId = ctx.event.shipmentId;
      await ctx.emit('COMPLETE_SAGA');
    }
  },
  {
    from: 'SHIPMENT_SCHEDULED',
    event: 'COMPLETE_SAGA',
    to: 'COMPLETED',
    action: async (ctx) => {
      await ctx.services.notification.sendConfirmation(ctx.data.orderId);
      console.log('Saga completed successfully');
    }
  },
 
  // Compensation transitions (failure handling)
  {
    from: 'PAYMENT_PROCESSED',
    event: 'SHIPMENT_FAILED',
    to: 'COMPENSATING_PAYMENT',
    action: async (ctx) => {
      await ctx.services.payment.refund(ctx.data.paymentId);
      await ctx.emit('PAYMENT_REFUNDED');
    }
  },
  {
    from: 'COMPENSATING_PAYMENT',
    event: 'PAYMENT_REFUNDED',
    to: 'COMPENSATING_INVENTORY',
    action: async (ctx) => {
      await ctx.services.inventory.release(ctx.data.reservationId);
      await ctx.emit('INVENTORY_RELEASED');
    }
  },
  {
    from: 'COMPENSATING_INVENTORY',
    event: 'INVENTORY_RELEASED',
    to: 'COMPENSATING_ORDER',
    action: async (ctx) => {
      await ctx.services.order.cancel(ctx.data.orderId);
      await ctx.emit('ORDER_CANCELLED');
    }
  },
  {
    from: 'COMPENSATING_ORDER',
    event: 'ORDER_CANCELLED',
    to: 'FAILED',
    action: async (ctx) => {
      await ctx.services.notification.sendFailure(ctx.data.orderId, ctx.error);
      console.log('Saga failed and fully compensated');
    }
  },
 
  // Direct failure transitions from any step
  {
    from: 'INVENTORY_RESERVED',
    event: 'PAYMENT_FAILED',
    to: 'COMPENSATING_INVENTORY',
    action: async (ctx) => {
      await ctx.services.inventory.release(ctx.data.reservationId);
      await ctx.emit('INVENTORY_RELEASED');
    }
  },
  {
    from: 'ORDER_CREATED',
    event: 'INVENTORY_RESERVATION_FAILED',
    to: 'COMPENSATING_ORDER',
    action: async (ctx) => {
      await ctx.services.order.cancel(ctx.data.orderId);
      await ctx.emit('ORDER_CANCELLED');
    }
  }
];
 
// State machine executor
class StateMachineOrchestrator {
  private transitions: Map<string, StateTransition> = new Map();
 
  constructor(definition: StateTransition[]) {
    for (const t of definition) {
      const key = `${t.from}:${t.event}`;
      this.transitions.set(key, t);
    }
  }
 
  async processEvent(saga: SagaInstance, event: SagaEvent): Promise<SagaState> {
    const transitionKey = `${saga.state}:${event.type}`;
    const transition = this.transitions.get(transitionKey);
 
    if (!transition) {
      throw new Error(
        `No transition from state '${saga.state}' for event '${event.type}'`
      );
    }
 
    const context: SagaContext = {
      data: saga.data,
      event: event.payload,
      services: this.serviceClients,
      emit: async (eventType: string) => {
        await this.sagaEventQueue.publish({
          sagaId: saga.id,
          type: eventType,
          payload: saga.data
        });
      }
    };
 
    // Execute the transition action
    await transition.action(context);
 
    // Update and persist the new state
    saga.state = transition.to;
    saga.data = context.data;
    await this.sagaRepository.update(saga.id, saga);
 
    return transition.to;
  }
}

Choreography vs Orchestration: The Complete Comparison

Now that we understand both approaches deeply, let's compare them across all relevant dimensions:

Comprehensive Choreography vs Orchestration Comparison
Dimension	Choreography	Orchestration
Coupling	Loose - services only know about events	Tighter - orchestrator knows all services
Single Point of Failure	None - fully distributed	Orchestrator is SPOF (mitigated by HA)
Flow Visibility	Scattered across services	Centralized in orchestrator
Debugging Complexity	Hard - must trace events across services	Easier - saga state visible in one place
Testing	Complex - requires integration testing	Simpler - can unit test saga logic
Adding New Steps	Modify only affected services	Modify orchestrator and add service
Cyclic Dependencies	Risk of event loops	Not possible - orchestrator controls flow
Saga State	Implicit in service states	Explicit in orchestrator
Recovery	Complex - reconstruct from events	Simple - resume from persisted state
Parallel Steps	Natural with event fanout	Requires explicit coordination
Team Autonomy	High - services evolve independently	Lower - orchestrator changes affect all
Coordination Overhead	Low - just event bus	Higher - orchestrator infrastructure
Best For	Simple sagas, 3-5 steps	Complex sagas, many steps/branches

Choose Choreography When

• Saga has 3-5 simple, linear steps • Team autonomy is paramount • Services already use an event-driven architecture • Saga logic rarely changes • You have strong distributed tracing capabilities • Each service team owns their saga participation

Choose Orchestration When

• Saga has many steps (5+) with complex branching • Business logic changes frequently • You need clear visibility into saga state • Multiple teams must coordinate closely • Sagas span organizational boundaries • You're using a saga framework (Temporal, Camunda)

Industry Saga Frameworks

Building saga infrastructure from scratch is complex. Production systems typically leverage battle-tested frameworks that handle the difficult aspects: state persistence, retry logic, timeout handling, and observability.

Temporal (formerly Cadence at Uber) is the leading orchestration framework for complex workflows including sagas. Key Features: Durable Execution : Workflow state survives crashes, restarts, deployments Saga Pattern Built-in : Native support for compensations Language SDKs : Go, Java, TypeScript, Python, PHP Visibility : Full workflow history and real-time state Scalability : Powers Uber's 5,000+ microservices Used By : Uber, Netflix, Snap, Coinbase, Box, Stripe

TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// Temporal Saga Workflow Example
import { proxyActivities, ApplicationFailure } from '@temporalio/workflow';
import type * as activities from './activities';
 
const { 
  createOrder, 
  reserveInventory, 
  processPayment, 
  scheduleShipment,
  // Compensations
  cancelOrder,
  releaseInventory,
  refundPayment
} = proxyActivities<typeof activities>({
  startToCloseTimeout: '30 seconds',
  retry: { maximumAttempts: 3 }
});
 
export async function orderSaga(orderData: OrderData): Promise<OrderResult> {
  const compensations: Array<() => Promise<void>> = [];
 
  try {
    // Step 1: Create Order
    const orderId = await createOrder(orderData);
    compensations.push(() => cancelOrder(orderId));
 
    // Step 2: Reserve Inventory  
    const reservationId = await reserveInventory(orderId, orderData.items);
    compensations.push(() => releaseInventory(reservationId));
 
    // Step 3: Process Payment
    const paymentId = await processPayment(orderId, orderData.amount);
    compensations.push(() => refundPayment(paymentId));
 
    // Step 4: Schedule Shipment
    const shipmentId = await scheduleShipment(orderId, orderData.address);
    // No compensation needed - if we got here, we're done
 
    return { success: true, orderId, shipmentId };
 
  } catch (error) {
    // Execute compensations in reverse order
    for (const compensate of compensations.reverse()) {
      try {
        await compensate();
      } catch (compensationError) {
        // Log but continue with other compensations
        console.error('Compensation failed:', compensationError);
      }
    }
 
    throw ApplicationFailure.nonRetryable(
      'Order saga failed and was compensated',
      'SAGA_FAILED',
      { originalError: error }
    );
  }
}

Hybrid Approaches: The Best of Both Worlds

In practice, many systems use hybrid approaches that combine choreography and orchestration strategically: Pattern 1: Orchestration Within, Choreography Between Each bounded context uses internal orchestration for complex workflows, but contexts communicate via events (choreography).

Converting Mermaid diagram...

Pattern 2: Choreography for Happy Path, Orchestration for Compensation The forward flow uses event choreography for its simplicity and loose coupling. When failures occur, a compensation orchestrator is triggered to coordinate the rollback. Pattern 3: Saga Choreography with Saga Coordinator Events drive the saga forward, but a lightweight coordinator service listens to all events for: Monitoring saga progress Detecting stuck sagas (timeouts) Triggering compensation when needed Providing saga visibility This preserves choreography's loose coupling while adding centralized observability.

Real-World Pragmatism

Production systems rarely fit neatly into 'choreography OR orchestration'. Start with the simpler approach (usually choreography for simple sagas), add orchestration infrastructure when complexity demands it. The goal is managing complexity, not architectural purity.

Choreography vs Orchestration

Two Philosophies of Saga Coordination

Choreography: The Decentralized Dance

Choreography: Implementation Considerations

Orchestration: The Centralized Conductor

Orchestration: Advanced Patterns

Choreography vs Orchestration: The Complete Comparison

Industry Saga Frameworks

Hybrid Approaches: The Best of Both Worlds

There’s more to understand.

Choreography vs Orchestration

Two Philosophies of Saga Coordination

Choreography: The Decentralized Dance

Choreography: Implementation Considerations

Orchestration: The Centralized Conductor

Orchestration: Advanced Patterns

Choreography vs Orchestration: The Complete Comparison

Industry Saga Frameworks

Hybrid Approaches: The Best of Both Worlds

There’s more to understand.