For decades, distributed transaction support was the holy grail of database systems—and the graveyard of many attempts. The fundamental challenge: how do you ensure that a transaction affecting data in Tokyo, Frankfurt, and Virginia either commits everywhere or nowhere, without unacceptable latency or availability loss?

Traditional two-phase commit (2PC) protocols solve the atomicity problem but create new ones: they require all participants to be available, they hold locks during coordination, and coordinator failures can leave transactions in limbo. At global scale, these problems become severe.

Spanner's distributed transaction protocol represents a synthesis of multiple innovations:

• Paxos groups for fault-tolerant replication within each shard
• Two-phase commit for atomicity across shards
• TrueTime for globally meaningful timestamps
• Wound-wait deadlock prevention for lock management

The result is a system that provides serializable, externally consistent transactions spanning the globe—something that was considered impractical before Spanner demonstrated it at scale.

Before diving into the distributed case, let's establish how transactions work within a single Paxos group.

Single-Group Transactions:

When a transaction only touches data managed by one Paxos group (common for well-designed schemas using interleaved tables), the protocol is straightforward:

1. Read Phase: Client reads data, acquiring read locks
2. Write Buffering: Writes are buffered in the client until commit
3. Prepare: Client sends writes to the leader
4. Leader Processing: Leader acquires write locks, assigns timestamp
5. Paxos Commit: Leader replicates transaction via Paxos
6. Commit-Wait: Leader waits until assigned timestamp passes
7. Apply and Release: Transaction applied, locks released
8. Response: Client receives commit confirmation with timestamp

This single-group path is highly optimized. The Paxos replication ensures durability (transaction survives server failures), and the commit-wait ensures external consistency.

Transaction Types:

Spanner supports several transaction types, each optimized for its use case:

1. Read-Write Transactions: Full ACID transactions that can read and modify data. Required for any writes.

2. Read-Only Transactions: Transactions that only read data. Execute at a consistent snapshot without blocking writes.

3. DML Statements: Individual SQL statements (INSERT, UPDATE, DELETE) executed as implicit transactions.

4. Partitioned DML: Bulk operations that execute in parallel across shards, with weaker atomicity guarantees but higher throughput.

Concurrency Control: Two-Phase Locking:

Spanner uses strict two-phase locking (S2PL) for concurrency control:

• Growing Phase: Transaction acquires all needed locks (shared for reads, exclusive for writes)
• Shrinking Phase: Transaction releases all locks at once after commit

Locks are held until transaction completion to ensure serializability. This means:

• Readers don't block other readers (shared locks are compatible)
• Writers block readers and other writers (exclusive locks)
• Read-only transactions use snapshot isolation and don't acquire locks

Lock Granularity:

Spanner supports multiple lock granularities:

• Row-level locks: Standard for OLTP workloads
• Key range locks: Block inserts in a key range, preventing phantoms
• Table-level locks: Used for DDL operations

When a transaction spans multiple Paxos groups—for example, transferring money between two users in different regions—Spanner uses two-phase commit (2PC) to ensure atomicity.

The Participants:

• Coordinator: One Paxos group (typically the one receiving the first write) is designated as the coordinator
• Participants: All other Paxos groups involved in the transaction

Phase 1: Prepare

1. Coordinator sends PREPARE to all participants

2. Each participant:

   • Acquires locks on its data
   • Writes transaction data to its Paxos log (ensuring durability)
   • Responds with PREPARED (or ABORT if something fails)

3. Coordinator waits for all participants to respond

Phase 2: Commit

1. If all participants responded PREPARED:

   • Coordinator assigns a commit timestamp
   • Coordinator writes COMMIT to its Paxos log (the commit point)
   • Coordinator sends COMMIT with timestamp to all participants

2. If any participant responded ABORT:

   • Coordinator sends ABORT to all participants
   • All participants roll back and release locks

The Critical Innovation: Paxos-Backed 2PC

Traditional 2PC has a critical vulnerability: if the coordinator fails between receiving all PREPARED votes and writing COMMIT, the transaction is stuck. Participants can't safely commit (they don't know if all participants prepared) or abort (another participant might have already committed).

Spanner solves this by making the coordinator itself a Paxos group:

Timestamp Coordination in 2PC:

A subtle but critical aspect of distributed transactions is timestamp assignment. The commit timestamp must be:

1. Greater than any timestamp seen by the transaction (to respect causality)
2. Greater than TT.now().latest at commit time (for external consistency)
3. Greater than the prepare timestamps from all participants (for global ordering)

During prepare, each participant reports the maximum timestamp it has seen. The coordinator takes the maximum of all these values and TT.now().latest to assign the commit timestamp.

Commit-Wait in Distributed Transactions:

The coordinator performs commit-wait after assigning the timestamp but before sending the COMMIT message. This means:

1. The commit-wait overlaps with Paxos logging
2. By the time COMMIT reaches participants, the timestamp has definitely passed
3. Participants can apply immediately without additional waiting

This optimization hides much of the commit-wait latency within the distributed protocol overhead.

Locking creates the possibility of deadlocks: Transaction A holds lock X and waits for lock Y, while Transaction B holds lock Y and waits for lock X. Neither can proceed.

Spanner prevents deadlocks using the wound-wait protocol—a priority-based scheme using transaction timestamps:

The Wound-Wait Rules:

1. Each transaction receives a timestamp when it starts
2. Older transactions (lower timestamps) have higher priority
3. When a transaction T1 wants a lock held by T2:
   • If T1 is older (higher priority): T1 wounds T2, forcing T2 to abort
   • If T1 is younger (lower priority): T1 waits for T2 to release

Why This Prevents Deadlocks:

Consider the potential deadlock scenario:

• T1 (started at t=100) holds lock X, wants lock Y
• T2 (started at t=200) holds lock Y, wants lock X

With wound-wait:

• T1 (older) wanting lock Y from T2 (younger): T1 wounds T2 → T2 aborts
• T2 releases lock Y, T1 acquires it and completes

Alternatively:

• T2 (younger) wanting lock X from T1 (older): T2 waits
• T1 continues, releases X when done, T2 then acquires it

In no scenario do both transactions wait for each other.

Implementation Details:

Retry Behavior:

When a transaction is aborted due to being wounded:

1. Timestamp Preservation: The transaction retries with its original timestamp, maintaining its priority position
2. Exponential Backoff: Retries use exponential backoff to reduce contention
3. Starvation Prevention: Because aborted transactions keep their timestamp, they eventually become the "oldest" and complete

The Cost of Wound-Wait:

Wound-wait has a cost: transactions may be aborted and retried. In high-contention scenarios, this can cause:

• Wasted work (aborted transactions' reads and writes discarded)
• Increased latency (due to retries)
• Reduced throughput (resources spent on work that's discarded)

However, for most workloads, wound-wait's overhead is minimal compared to deadlock detection and resolution approaches used by other systems.

Let's trace through a complete read-write transaction lifecycle, from client request to committed response.

Pre-Transaction: Session and Context

Before executing transactions, clients establish a session with Spanner. Sessions maintain:

• Connection state and authentication
• Transaction contexts
• Read timestamps for snapshot reads
• Client-side result caching

The Transaction Lifecycle:

Multi-Group Transaction Flow:

When the transaction spans multiple Paxos groups (e.g., users in one group, audit_log in another), the flow adds 2PC coordination:

1. Participant Prepare: All Paxos groups receive PREPARE with their mutations
2. Participant Response: Each group logs PREPARED and responds to coordinator
3. Timestamp Selection: Coordinator selects max timestamp from all participants
4. Commit-Wait: Coordinator waits for timestamp to pass
5. Commit Broadcast: Coordinator sends COMMIT to all participants
6. Parallel Apply: All participants apply mutations and release locks

Latency Impact of Distribution:

Multi-group transactions add:

• Round-trip to participants during prepare
• Possible additional round-trip for commit acknowledgment
• Coordinator wait for slowest participant

For geographically distributed groups, this can add 100-200ms compared to single-group transactions.

Read-only transactions are a powerful optimization in Spanner. They provide serializable isolation without acquiring locks, enabling high read throughput without blocking writers.

How Read-Only Transactions Work:

1. Timestamp Selection: A read timestamp is chosen (either strong, bounded-stale, or exact-stale)
2. Snapshot Execution: All reads execute against the snapshot at that timestamp
3. No Locks Needed: Since we're reading a fixed point in time, there's no conflict with concurrent writes
4. Any Replica: Reads can be served by any replica that has caught up to the read timestamp

The Power of Lock-Free Reads:

Consider a report that reads from multiple tables:

SELECT SUM(balance) FROM accounts WHERE region = 'US';
SELECT COUNT(*) FROM transactions WHERE date = TODAY;
SELECT * FROM audit_log WHERE timestamp > YESTERDAY;

With a read-only transaction:

• All three queries see a consistent snapshot
• No locks are held during execution
• Concurrent write transactions continue unimpeded
• Reads can be parallelized across replicas

Strong vs. Stale Reads:

The timestamp selection for read-only transactions creates different consistency-latency tradeoffs:

Strong Reads:

• Timestamp = current time (TT.now().latest)
• Sees all transactions committed before the read began
• May need to wait briefly for replicas to catch up
• May need to contact leader for very recent data

Bounded Staleness:

• Timestamp = current time - staleness bound
• Can always be served immediately from local replica
• May miss transactions committed in the last N seconds
• Ideal for dashboards, analytics, search indexes

Exact Staleness:

• Timestamp = specific value (e.g., start of day)
• Reproducible: same query at same timestamp = same results
• Ideal for batch processing, reconciliation

Multi-Region Optimization:

For globally deployed applications, stale reads provide dramatic performance improvements:

• User in Tokyo, data leader in Virginia
• Strong read: ~150ms (round-trip to Virginia)
• 10-second stale read: ~5ms (local Tokyo replica)

For read-heavy workloads where slight staleness is acceptable, this 30x improvement transforms user experience.

Understanding Spanner's performance characteristics helps you design schemas and applications that work with the system rather than against it.

Latency Components:

Total transaction latency comprises several components:

1. Network Round-Trips: Client to Spanner frontend, frontend to Paxos groups
2. Lock Acquisition: Wait time if locks are held by other transactions
3. Paxos Replication: Time to replicate across replicas (depends on geographic spread)
4. Commit-Wait: TrueTime uncertainty period
5. 2PC Overhead: For multi-group transactions, coordinator-participant round-trips

Throughput Considerations:

Spanner scales horizontally—adding more nodes increases capacity. However, several factors affect throughput:

1. Hot Spots: If many transactions access the same rows, lock contention limits throughput. Solutions:

• Use UUID keys instead of sequential IDs
• Partition hot data across multiple rows
• Use read-only transactions where possible

2. Geographic Distribution: Multi-region configurations have lower per-transaction throughput due to coordination overhead. However, total system throughput may be higher due to geographic distribution of load.

3. Transaction Size: Larger transactions (more rows read/written) hold locks longer, reducing concurrency. Keep transactions small and focused.

4. 2PC Overhead: Multi-group transactions have lower throughput than single-group. Design schemas to keep related data together.

Benchmarks and Real-World Performance:

Google has published performance data showing:

• Thousands of nodes handling millions of QPS
• 99th percentile latencies under 100ms for regional configurations
• Linear scalability as nodes are added
• Recovery times under 10 seconds for node failures

We've explored the sophisticated transaction machinery that enables Spanner's globally distributed ACID guarantees. Let's consolidate the key insights:

What's Next:

With transactions covered, we'll explore how Spanner automatically manages data distribution. The next page covers Automatic Sharding—how Spanner dynamically partitions, rebalances, and moves data without application involvement or downtime.