Distributed Transactions
Participant Role
Read aloud is unavailable in this browser.
The Footsoldiers of Distributed Consensus
While the coordinator orchestrates the Two-Phase Commit protocol, it is the participants (also called resource managers or cohorts) that hold the actual data and perform the real work. Each participant manages a portion of the distributed database—executing transactions locally, acquiring locks, maintaining durability, and responding to coordinator directives.
The participant's role is deceptively complex. It must balance local autonomy with global coordination, maintain consistency despite failures, and manage the precarious PREPARED state where it has promised to follow the coordinator's decision but doesn't yet know what that decision will be. Understanding the participant's responsibilities, state machine, and recovery procedures is essential for implementing correct distributed transaction processing.
By the end of this page, you will have a comprehensive understanding of the participant's responsibilities throughout the distributed transaction lifecycle. You'll understand local transaction execution, the vote decision process, the critical PREPARED state, lock management during 2PC, uncertainty resolution, and participant recovery procedures.
Participant Identity and Registration
A participant is any database node that holds data accessed by a distributed transaction. Before the commit process begins, participants must register with the transaction coordinator so that the coordinator knows to include them in the prepare phase.
Registration Mechanisms:
Implicit Registration: When the transaction first accesses data at a node, that node automatically registers with the coordinator. This is transparent to the application—the database infrastructure handles registration behind the scenes.
Explicit Registration:
The application or middleware explicitly enlists participants in the transaction using APIs like XA's xa_start and xa_end. This gives the application more control but requires awareness of distributed transaction semantics.
Registration Information:
When registering, a participant typically provides:
- Participant Identifier: A unique identifier for this node
- Network Endpoint: How the coordinator can reach the participant
- Resource Identifier: Which database or resource the participant represents
- Capabilities: What features the participant supports (e.g., read-only optimization)
| System/Standard | Registration Method | Registration Point | Notes |
|---|---|---|---|
| X/Open XA | Explicit (xa_start/xa_end) | Before accessing resource | Industry standard for TMs |
| PostgreSQL 2PC | Implicit via PREPARE | At PREPARE TRANSACTION | Single-node prepares explicitly |
| MySQL/XA | Explicit (XA START/XA END) | Before queries | Mirrors X/Open model |
| CockroachDB | Implicit | First access | Internal transaction coordinator |
| Spanner | Implicit | First access per paxos group | Paxos-replicated participants |
Tracking the Coordinator:
While the coordinator tracks participants, each participant must also track information about the coordinator:
- Coordinator Identity: Who is coordinating this transaction
- Coordinator Endpoint: How to contact the coordinator
- Transaction ID: The global transaction identifier
This information is critical for the participant's recovery process. If the participant crashes while in the PREPARED state, upon recovery it must contact the coordinator to learn the transaction's outcome.
Local Transaction Execution
Before the commit protocol begins, the participant executes the local portion of the distributed transaction. This involves all the normal transaction processing operations: parsing queries, acquiring locks, reading data, writing modifications, and maintaining transaction isolation.
Local Execution Responsibilities:
1. Lock Acquisition and Management
The participant acquires locks on all data items it accesses, following the database's concurrency control protocol (2PL, MVCC, etc.). These locks:
- Prevent conflicts with other local transactions
- Must be held until the distributed transaction commits or aborts
- Are NOT released when voting VOTE_COMMIT—they persist through the PREPARED state
- Are only released when the global decision (COMMIT or ABORT) is received
2. Logging for Local Recovery
As the transaction modifies data, the participant writes redo and undo information to its local log:
- Redo Information: How to reapply changes if the transaction commits
- Undo Information: How to reverse changes if the transaction aborts
This logging follows write-ahead logging (WAL) rules—log records are written before data modifications.
The participant maintains a local transaction manager that handles local ACID properties. The distributed transaction coordinator layers on top of this—it doesn't replace local transaction semantics but coordinates them across nodes. The local transaction manager knows how to execute, commit, and abort transactions; the coordinator tells it when to do so.
3. Maintaining Transaction State
The participant tracks the state of the distributed transaction:
- ACTIVE: Transaction is executing, modifications are being made
- PREPARED: Voted VOTE_COMMIT, waiting for global decision
- COMMITTED: Global COMMIT received, changes made permanent
- ABORTED: Either voted ABORT or received GLOBAL_ABORT
4. Read-Write Set Tracking
For conflict detection and validation, the participant may track:
- Read Set: All data items read by the transaction
- Write Set: All data items modified by the transaction
This information supports optimistic concurrency control and may be used during the vote decision.
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283
interface ParticipantTransactionContext { // Global transaction identifier from coordinator globalTxId: string; // Local transaction identifier localTxId: string; // Coordinator information for recovery coordinator: { id: string; endpoint: string; }; // Current state in participant state machine state: 'ACTIVE' | 'PREPARED' | 'COMMITTED' | 'ABORTED'; // Locks held by this transaction heldLocks: Set<LockHandle>; // Read set for validation readSet: Map<DataItemId, Version>; // Write set with old/new values writeSet: Map<DataItemId, { oldValue: any; newValue: any }>; // Local log position for undo/redo logSequenceNumber: number;} class ParticipantTransactionManager { private activeTransactions: Map<string, ParticipantTransactionContext>; /** * Execute a local operation within a distributed transaction */ async executeOperation( globalTxId: string, operation: Operation ): Promise<OperationResult> { const ctx = this.activeTransactions.get(globalTxId); if (!ctx || ctx.state !== 'ACTIVE') { throw new Error(`Transaction ${globalTxId} not active`); } // Acquire necessary locks const locks = await this.lockManager.acquireLocks( operation.requiredLocks, globalTxId ); ctx.heldLocks = new Set([...ctx.heldLocks, ...locks]); if (operation.type === 'READ') { // Track read for validation const item = await this.storage.read(operation.itemId); ctx.readSet.set(operation.itemId, item.version); return { data: item.value }; } else if (operation.type === 'WRITE') { // Log old value for undo const oldItem = await this.storage.read(operation.itemId); // Log redo/undo information ctx.logSequenceNumber = await this.log.writeRedoUndo({ transactionId: globalTxId, itemId: operation.itemId, oldValue: oldItem.value, newValue: operation.value }); // Apply modification (in buffer, not yet durable) await this.storage.write(operation.itemId, operation.value); // Track in write set ctx.writeSet.set(operation.itemId, { oldValue: oldItem.value, newValue: operation.value }); return { success: true }; } }}Processing the PREPARE Request
When the participant receives a PREPARE message from the coordinator, it must make a critical decision: Can this transaction commit locally? This decision has binding consequences—once a participant votes VOTE_COMMIT, it has promised to follow the coordinator's final decision.
The Vote Decision Process:
The participant evaluates whether the transaction can be committed by checking several conditions:
- •All constraints satisfied: Foreign keys, unique constraints, check constraints are all valid after applying the transaction's modifications.
- •No deadlocks detected: The transaction is not involved in a deadlock cycle that requires its abort.
- •Resources available: Sufficient disk space, memory, and other resources exist to complete the transaction.
- •Transaction is valid: The transaction hasn't been marked for rollback due to errors.
- •Locks are held: All necessary locks are still held (no lock timeout).
- •Validation passes (for OCC): If using optimistic concurrency control, the validation phase succeeds.
Voting VOTE_COMMIT:
If all conditions are satisfied, the participant:
-
Force-writes the PREPARED record to stable storage
- This record contains the transaction ID, coordinator info, and all undo/redo information
- The force-write ensures the promise survives crashes
-
Transitions to PREPARED state
- The participant is now in the 'uncertain' or 'in-doubt' state
- It cannot unilaterally abort or commit
-
Sends VOTE_COMMIT to the coordinator
- This is the participant's promise to follow the coordinator's decision
-
Continues holding all locks
- Locks are NOT released when voting COMMIT
- Other transactions are blocked if they conflict
Voting VOTE_ABORT:
If any condition fails, the participant:
- Writes ABORT record to stable storage
- Rolls back local modifications using undo information
- Releases all locks held by this transaction
- Sends VOTE_ABORT to coordinator
- Cleans up local transaction state
Note: A participant that votes ABORT doesn't need to wait for the coordinator's decision—the transaction WILL abort regardless of other votes.
Voting VOTE_COMMIT is a serious commitment. The participant is saying: 'I CAN commit this transaction, I WILL commit if told to, I WILL abort if told to, and I WILL wait as long as necessary to learn the decision.' This promise holds even across crashes—the participant must recover and honor it.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103
class ParticipantTransactionManager { /** * Handle PREPARE request from coordinator */ async handlePrepare(globalTxId: string): Promise<'VOTE_COMMIT' | 'VOTE_ABORT'> { const ctx = this.activeTransactions.get(globalTxId); if (!ctx) { // Unknown transaction - vote abort return 'VOTE_ABORT'; } if (ctx.state !== 'ACTIVE') { // Already processed - should not happen throw new Error(`Unexpected PREPARE for ${globalTxId} in state ${ctx.state}`); } // CHECK 1: Verify all constraints are satisfied const constraintResult = await this.checkConstraints(ctx); if (!constraintResult.satisfied) { return this.voteAbort(ctx, `Constraint violation: ${constraintResult.reason}`); } // CHECK 2: Check for deadlock involvement if (this.deadlockDetector.isInvolvedInDeadlock(globalTxId)) { return this.voteAbort(ctx, 'Selected as deadlock victim'); } // CHECK 3: Verify sufficient resources const resourceCheck = await this.checkResources(ctx); if (!resourceCheck.available) { return this.voteAbort(ctx, `Insufficient resources: ${resourceCheck.reason}`); } // CHECK 4: Validation for optimistic concurrency control if (this.usesOCC) { const validationResult = await this.validateReadSet(ctx); if (!validationResult.valid) { return this.voteAbort(ctx, 'Validation failed - read set was modified'); } } // All checks passed - vote COMMIT return this.voteCommit(ctx); } /** * Vote to commit the transaction */ private async voteCommit(ctx: ParticipantTransactionContext): Promise<'VOTE_COMMIT'> { // CRITICAL: Force-write PREPARED record BEFORE sending vote // This record must contain enough info to redo OR undo the transaction await this.log.forceWrite({ type: 'PREPARED', transactionId: ctx.globalTxId, coordinator: ctx.coordinator, writeSet: ctx.writeSet, // For redo // Undo information was logged during execution }); // Transition to PREPARED state ctx.state = 'PREPARED'; // Keep all locks - do NOT release them! // Other transactions will block on these locks console.log(`Transaction ${ctx.globalTxId} entering PREPARED state`); return 'VOTE_COMMIT'; } /** * Vote to abort the transaction */ private async voteAbort( ctx: ParticipantTransactionContext, reason: string ): Promise<'VOTE_ABORT'> { console.log(`Transaction ${ctx.globalTxId} voting ABORT: ${reason}`); // Log abort await this.log.forceWrite({ type: 'ABORT', transactionId: ctx.globalTxId, reason: reason }); // Rollback local changes await this.rollbackTransaction(ctx); // Release all locks await this.lockManager.releaseAll(ctx.globalTxId); ctx.heldLocks.clear(); // Update state ctx.state = 'ABORTED'; // Clean up this.activeTransactions.delete(ctx.globalTxId); return 'VOTE_ABORT'; }}The PREPARED State: Living with Uncertainty
The PREPARED state (also called the uncertain, in-doubt, or limbo state) is the most critical and dangerous phase for a participant. In this state:
- The participant has voted VOTE_COMMIT
- It has promised to follow the coordinator's decision
- It does NOT know what the decision will be
- It cannot unilaterally abort or commit
- It is holding locks that block other transactions
- It must WAIT for the coordinator or until it can learn the outcome
This state is inherently uncomfortable—the participant is exposed to uncertainty. If the coordinator fails or becomes unreachable, the participant may be stuck indefinitely.
A participant in the PREPARED state is BLOCKED. It cannot proceed without learning the coordinator's decision. If the coordinator has crashed, the participant may wait indefinitely—holding locks that block other transactions. This is the fundamental weakness of the Two-Phase Commit protocol.
What the Participant is Waiting For:
In the PREPARED state, the participant is waiting to receive one of two messages:
GLOBAL_COMMIT: The coordinator has decided to commit. The participant:
- Writes COMMIT record to stable storage
- Makes changes permanent
- Releases all locks
- Sends ACK to coordinator
- Cleans up transaction state
GLOBAL_ABORT: The coordinator has decided to abort. The participant:
- Writes ABORT record to stable storage
- Rolls back all modifications
- Releases all locks
- Sends ACK to coordinator
- Cleans up transaction state
Why the Participant Cannot Decide Unilaterally:
Suppose the participant decides to abort unilaterally after some timeout:
- Problem: The coordinator might have decided COMMIT and already told other participants
- Result: Some participants committed, this participant aborted → inconsistency!
Alternatively, if the participant decides to commit unilaterally:
- Problem: Another participant might have voted ABORT
- Result: This participant committed, others aborted → inconsistency!
The participant MUST wait for authoritative information about the outcome.
Managing the PREPARED State:
Production systems implement several measures to make the PREPARED state manageable:
-
Timeout and Query: After a timeout, the participant polls the coordinator for the decision. The coordinator's log is the authoritative source.
-
Cooperative Termination Protocol: If the coordinator is unreachable, participants can contact each other. If any participant knows the outcome (has received GLOBAL_COMMIT or GLOBAL_ABORT), it can share this information.
-
Prepared Transaction Monitoring: DBAs monitor prepared transactions and can manually resolve them after consulting other participants.
-
Maximum Prepare Duration: Some systems set a maximum time a transaction can remain in PREPARED state before administrators are alerted.
Processing the Global Decision
When the participant finally receives the coordinator's global decision, it must act on it promptly and correctly. The handling differs based on whether the decision is COMMIT or ABORT.
Handling GLOBAL_COMMIT:
When the participant receives GLOBAL_COMMIT:
- •Force-write COMMIT record: Write <COMMIT T> to stable storage. This makes the commit decision durable.
- •Make changes permanent: Depending on the storage engine, this might involve marking dirty pages as durable, flushing buffers, or updating data files.
- •Release all locks: Free locks held for this transaction, allowing blocked transactions to proceed.
- •Send acknowledgment: Reply ACK to the coordinator so it knows the participant has committed.
- •Clean up resources: Remove transaction context, free memory, close cursors.
Handling GLOBAL_ABORT:
When the participant receives GLOBAL_ABORT:
- •Force-write ABORT record: Write <ABORT T> to stable storage (for recovery).
- •Rollback modifications: Use the undo information logged during execution to reverse all changes made by this transaction.
- •Release all locks: Free locks, allowing blocked transactions to proceed.
- •Send acknowledgment: Reply ACK to the coordinator.
- •Clean up resources: Remove transaction context and free memory.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116
class ParticipantTransactionManager { /** * Handle the global decision from coordinator */ async handleGlobalDecision( globalTxId: string, decision: 'GLOBAL_COMMIT' | 'GLOBAL_ABORT' ): Promise<void> { const ctx = this.activeTransactions.get(globalTxId); if (!ctx) { // Transaction not found - might have been cleaned up already // This is okay - idempotent handling console.log(`Decision for unknown tx ${globalTxId} - already completed`); return; } if (ctx.state === 'COMMITTED' || ctx.state === 'ABORTED') { // Already processed - idempotent console.log(`Decision for ${globalTxId} already processed`); return; } if (ctx.state !== 'PREPARED') { // Unexpected state - might be a late PREPARE race condition throw new Error(`Unexpected decision for ${globalTxId} in state ${ctx.state}`); } if (decision === 'GLOBAL_COMMIT') { await this.executeCommit(ctx); } else { await this.executeAbort(ctx); } } /** * Execute local commit after receiving GLOBAL_COMMIT */ private async executeCommit(ctx: ParticipantTransactionContext): Promise<void> { console.log(`Committing transaction ${ctx.globalTxId}`); // Step 1: Force-write COMMIT record await this.log.forceWrite({ type: 'COMMIT', transactionId: ctx.globalTxId, timestamp: Date.now() }); // Step 2: Make changes permanent // In practice, this might flush buffer pool pages or // trigger checkpoint behavior depending on the storage engine await this.storage.makeChangesPermanent(ctx.globalTxId); // Step 3: Release all locks await this.lockManager.releaseAll(ctx.globalTxId); ctx.heldLocks.clear(); // Step 4: Update state ctx.state = 'COMMITTED'; // Step 5: Clean up this.activeTransactions.delete(ctx.globalTxId); console.log(`Transaction ${ctx.globalTxId} committed successfully`); } /** * Execute local abort after receiving GLOBAL_ABORT */ private async executeAbort(ctx: ParticipantTransactionContext): Promise<void> { console.log(`Aborting transaction ${ctx.globalTxId}`); // Step 1: Force-write ABORT record await this.log.forceWrite({ type: 'ABORT', transactionId: ctx.globalTxId, timestamp: Date.now() }); // Step 2: Rollback all modifications using undo log await this.rollbackTransaction(ctx); // Step 3: Release all locks await this.lockManager.releaseAll(ctx.globalTxId); ctx.heldLocks.clear(); // Step 4: Update state ctx.state = 'ABORTED'; // Step 5: Clean up this.activeTransactions.delete(ctx.globalTxId); console.log(`Transaction ${ctx.globalTxId} aborted successfully`); } /** * Rollback transaction modifications using undo log */ private async rollbackTransaction(ctx: ParticipantTransactionContext): Promise<void> { // Read undo log records in reverse order (most recent first) const undoRecords = await this.log.getUndoRecords(ctx.globalTxId); for (const record of undoRecords.reverse()) { // Restore old value await this.storage.write(record.itemId, record.oldValue); // Log the undo operation (for recovery if we crash during rollback) await this.log.write({ type: 'UNDO', transactionId: ctx.globalTxId, itemId: record.itemId, restoredValue: record.oldValue }); } }}Lock Management During 2PC
Lock management is critical during the Two-Phase Commit protocol. The participant must maintain strict lock discipline to ensure isolation and prevent the 'dirty read' and 'lost update' problems that could compromise transaction integrity.
Lock Duration in 2PC:
In normal (non-distributed) strict 2PL, locks are held until the transaction commits or aborts. In 2PC, this extends further:
| Phase | Lock State | Duration | Impact on Other Transactions |
|---|---|---|---|
| Execution (ACTIVE) | Acquiring locks | Until PREPARE or rollback | Concurrent access blocked |
| Vote COMMIT (→ PREPARED) | Locks held | Entire PREPARED duration | Blocked indefinitely if stuck |
| Vote ABORT | Locks released | Immediate | Others can proceed |
| PREPARED state | Locks fully held | Until decision arrives | Potential indefinite blocking |
| GLOBAL_COMMIT received | Locks released | After commit completes | Others can now access |
| GLOBAL_ABORT received | Locks released | After rollback completes | Others can now access |
During the PREPARED state, locks are held but no useful work is being done—the participant is just waiting. If the coordinator is slow or has failed, these locks block other transactions. This is why 2PC is criticized for poor availability: a single coordinator failure can cascade to block many transactions across the system.
Why Locks Must Be Held in PREPARED State:
Consider what happens if locks were released when entering PREPARED state:
- Transaction T1 updates row R and enters PREPARED state
- T1 releases lock on R
- Transaction T2 reads row R (sees T1's uncommitted changes)
- Coordinator decides ABORT for T1
- T1 rolls back, but T2 has already read the aborted value
- Dirty read has occurred!
Even worse with writes:
- T1 updates row R to value A and enters PREPARED state
- T1 releases lock on R
- T2 updates row R to value B and commits
- Coordinator decides COMMIT for T1
- T1 commits value A, overwriting T2's value B
- Lost update has occurred!
Holding locks through the PREPARED state prevents these anomalies.
Lock Timeout Considerations:
Many databases support lock timeouts to prevent indefinite waiting. However, in 2PC:
- Lock waiters may timeout after waiting for a PREPARED transaction's locks
- The PREPARED transaction itself should NOT timeout its locks—it must wait for the coordinator
- This asymmetry means lock waiters may abort due to timeout, but the PREPARED transaction keeps its locks
Participant Recovery
When a participant crashes and recovers, it must handle in-flight distributed transactions correctly. The recovery procedure depends on what state each transaction was in when the crash occurred.
Recovery by Transaction State:
Case 1: Transaction in ACTIVE state (only execution records, no PREPARED)
- The transaction was executing but had not yet voted
- Action: Abort the transaction locally (rollback using undo log, release any locks)
- Rationale: The coordinator will timeout waiting for our vote and decide ABORT anyway
Case 2: Transaction in PREPARED state (PREPARED record present, no COMMIT/ABORT)
- The transaction voted COMMIT and was waiting for the global decision
- Action: Re-enter PREPARED state, re-acquire locks, query coordinator for decision
- This is the critical case—we MUST learn the outcome from the coordinator
Case 3: COMMIT record present
- We received GLOBAL_COMMIT but may not have fully applied it
- Action: Complete the commit (redo if necessary), release locks
Case 4: ABORT record present
- We decided to abort (locally or received GLOBAL_ABORT)
- Action: Complete the abort (undo if necessary), release locks
For transactions in PREPARED state, the participant must reconstruct the locks that were held before the crash. The PREPARED log record should contain sufficient information (the write set) to determine what locks need to be re-acquired. This ensures that other transactions cannot access in-doubt data during recovery.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134
class ParticipantRecovery { /** * Recover participant state after crash */ async recover(): Promise<void> { console.log('Starting participant recovery...'); // Scan log and identify in-flight transactions const transactions = await this.classifyTransactions(); // Handle each transaction based on its state for (const [txId, state] of transactions) { await this.recoverTransaction(txId, state); } console.log('Participant recovery complete'); } /** * Recover a single transaction */ private async recoverTransaction( txId: string, state: RecoveryTransactionState ): Promise<void> { switch (state.lastRecord) { case 'COMMIT': // Complete the commit await this.redoCommit(txId, state); break; case 'ABORT': // Complete the abort await this.undoAbort(txId, state); break; case 'PREPARED': // Re-enter prepared state and query coordinator await this.recoverPrepared(txId, state); break; default: // Only execution records - abort await this.abortIncomplete(txId, state); } } /** * Recover a transaction that was in PREPARED state */ private async recoverPrepared( txId: string, state: RecoveryTransactionState ): Promise<void> { console.log(`Recovering PREPARED transaction ${txId}`); // Re-acquire locks based on write set const writeSet = state.preparedRecord!.writeSet; for (const itemId of writeSet.keys()) { await this.lockManager.acquireLock(itemId, 'EXCLUSIVE', txId); } // Create transaction context in PREPARED state const ctx: ParticipantTransactionContext = { globalTxId: txId, localTxId: state.preparedRecord!.localTxId, coordinator: state.preparedRecord!.coordinator, state: 'PREPARED', heldLocks: new Set(), // Will be populated by lock acquisition readSet: new Map(), writeSet: writeSet, logSequenceNumber: state.preparedRecord!.lsn }; this.activeTransactions.set(txId, ctx); // Query coordinator for decision await this.queryCoordinatorForDecision(txId, ctx); } /** * Query coordinator to learn transaction outcome */ private async queryCoordinatorForDecision( txId: string, ctx: ParticipantTransactionContext ): Promise<void> { const retryInterval = 5000; // 5 seconds while (ctx.state === 'PREPARED') { try { const decision = await this.sendDecisionQuery( ctx.coordinator.endpoint, txId ); if (decision === 'COMMIT') { await this.executeCommit(ctx); } else if (decision === 'ABORT') { await this.executeAbort(ctx); } // else: coordinator doesn't know yet, keep waiting } catch (error) { console.log(`Cannot reach coordinator for ${txId}: ${error}`); // Will retry after interval } if (ctx.state === 'PREPARED') { await this.sleep(retryInterval); } } } /** * Abort a transaction that never reached PREPARED state */ private async abortIncomplete( txId: string, state: RecoveryTransactionState ): Promise<void> { console.log(`Aborting incomplete transaction ${txId}`); // Undo any modifications await this.undoAbort(txId, state); // Log abort await this.log.forceWrite({ type: 'ABORT', transactionId: txId, reason: 'Recovery: never prepared' }); }}KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.