Google Spanner: The Globally Distributed Database

Ask any engineer who has managed a sharded database about their experience, and you'll hear war stories: resharding projects that took months, application changes needed for every shard topology change, data migration scripts that failed at 3 AM, and the constant mental overhead of remembering which data lives where.

Manual sharding—dividing a database into pieces that different servers manage—works, but it imposes tremendous operational burden:

• Application Awareness: Applications must know shard topology and route queries accordingly
• Rebalancing Pain: Adding shards requires migrating data and updating routing logic
• Hotspot Handling: Hot shards require manual intervention to split
• Cross-Shard Queries: Joins across shards become the application's problem

Google Spanner eliminates this operational overhead through automatic sharding. The database handles data partitioning, movement, and rebalancing transparently. Applications interact with a single logical database while Spanner manages the distributed reality underneath.

This isn't just convenience—it's a fundamental shift in database operations. Engineers focus on application logic while Spanner ensures data is optimally distributed across a possibly massive fleet of servers.

To understand automatic sharding, we need to understand Spanner's data organization hierarchy.

Key Ranges and Splits:

All data in Spanner is stored in primary key order. A split is a contiguous range of the key space—for example, all rows with primary keys from "A" to "M". Splits are the fundamental unit of sharding.

Tablets:

A tablet is the physical manifestation of a split on a single server. Each tablet:

• Contains data for one key range
• Is managed by one spanserver
• Has its own Paxos group for replication
• Can be moved between servers

Directories:

A directory is a set of contiguous rows that share:

• Common replication configuration
• Common access patterns
• Locality properties (they're always stored together)

Directories are the unit of data movement. When Spanner moves data for load balancing or failure recovery, it moves entire directories.

The Hierarchy:

Why This Hierarchy Matters:

1. Directories preserve locality: Related data (a user and their orders) stays together, even as splits change.

2. Splits enable parallelism: Different splits can be served by different servers, allowing horizontal scaling.

3. Tablets enable distribution: By placing tablets on different spanservers, load is distributed across the fleet.

4. Paxos groups per split: Each split has its own consensus group, allowing independent commits.

Split Boundaries:

Spanner chooses split boundaries based on:

• Size: Splits should be similar sizes (typically 1-4 GB each)
• Load: Heavily accessed key ranges may be split finer
• Directory boundaries: Splits try to respect directory boundaries to avoid cross-split directories

Spanner continuously monitors splits and automatically adjusts boundaries to maintain optimal performance.

Split Triggers:

Spanner splits a key range when:

1. Size Threshold: Split exceeds target size (typically 1-4 GB)
2. CPU Hotspot: Split is consuming disproportionate CPU time
3. Disk Hotspot: Split has disproportionate I/O traffic
4. Latency Anomaly: Operations on split have elevated latency

The Split Process:

Splitting is designed to be transparent and non-disruptive:

1. Boundary Selection: Spanner selects a split point in the middle of the key range (respecting directory boundaries when possible)

2. Metadata Update: Split metadata is updated—the old split becomes two new splits with new key ranges

3. Tablet Creation: New tablets are created on spanservers to manage the new splits

4. Paxos Reconfiguration: The old Paxos group becomes two new groups, each with its own leader

5. Traffic Redirection: Clients are seamlessly redirected to the appropriate new split

Zero Downtime:

The split operation doesn't block reads or writes:

• Operations in progress complete against the old split
• New operations are routed to the appropriate new split
• There's no moment where data is unavailable

Merge Triggers:

Spanner merges adjacent splits when:

1. Size Drops: Combined size of adjacent splits falls well below threshold
2. Load Decreases: Both splits have low utilization
3. Directory Efficiency: Merge would reunite a split directory

Merges are the inverse of splits—two Paxos groups become one, and tablets are consolidated.

Split/Merge Decisions:

Spanner's control plane continuously analyzes metrics and makes split/merge decisions:

• Decisions are made centrally with global visibility
• Changes are rate-limited to avoid oscillation
• Hysteresis prevents rapid split-merge cycles
• Time-of-day patterns are considered (don't split right before expected load increase)

Beyond splitting key ranges, Spanner actively manages which spanservers host which tablets to balance load across the fleet.

The Placement Problem:

Given:

• N spanservers with varying capacities
• M tablets with varying load requirements
• Replication constraints (replicas must be in different zones)
• Leader placement preferences (leaders should be in certain regions)

Find: An assignment of tablets to spanservers that:

• Balances CPU, memory, and I/O load
• Respects fault tolerance requirements
• Minimizes latency for common access patterns
• Allows for future growth

The Placement Manager:

Spanner's placement manager continuously optimizes tablet placement:

1. Metric Collection: Each spanserver reports CPU, memory, I/O, and per-tablet metrics

2. Imbalance Detection: Placement manager identifies servers that are over/under loaded

3. Move Calculation: For overloaded servers, placement manager calculates which tablets to move and where

4. Move Execution: Tablets are moved to rebalance load

5. Monitoring: Results are monitored and adjustments made if needed

Movement Types:

1. Follower Movement (Cheapest):

• Moving a follower replica to a different server
• New follower catches up from existing replicas
• Minimal disruption—Paxos continues with existing quorum

2. Leader Movement:

• Moving the leader replica to a different server
• Requires leader election after move
• Brief pause in writes during election (~100ms)

3. Split Redistribution:

• After a split, new tablets are placed on less-loaded servers
• Spreads hot data across more servers

Movement Constraints:

Not all movements are safe or desirable:

• Quorum Safety: Never move so many replicas that quorum is lost
• Rate Limiting: Don't move too many tablets simultaneously
• Cooldown: Recently moved tablets aren't immediately moved again
• Preference Respect: Honor region/zone placement preferences

Hotspots—key ranges that receive disproportionate traffic—are the nemesis of distributed databases. Spanner's automatic sharding includes sophisticated hotspot detection and mitigation.

Types of Hotspots:

1. Write Hotspots:

• Many transactions writing to the same key range
• Often caused by sequential key generation (auto-increment IDs)
• Particularly problematic because writes require leader coordination

2. Read Hotspots:

• Many queries reading the same key range
• Popular content, trending items, shared configuration
• Less severe because reads can be distributed across replicas

3. Lock Contention Hotspots:

• Many transactions competing for locks on the same rows
• Metrics show low CPU but high transaction abort rates

Hotspot Detection:

Spanner monitors multiple signals:

• Per-split CPU utilization
• Per-split QPS and latency percentiles
• Lock wait times and abort rates
• Key access patterns (via sampling)

When metrics exceed thresholds, the split is flagged as a hotspot.

Prevention Through Schema Design:

The best hotspot mitigation is prevention. Schema design choices that prevent hotspots:

1. UUID or Random Keys:

-- Instead of:
CREATE TABLE orders (order_id INT64 AUTO_INCREMENT, ...);
-- Hotspot: All new orders go to highest key range

-- Use:
CREATE TABLE orders (order_id STRING(36), ...);  -- UUID
-- Orders distributed across entire key space

2. Key Prefixing for Distribution:

-- Instead of:
-- PRIMARY KEY (timestamp)  -- all recent data in one range

-- Use:
-- PRIMARY KEY (shard_id, timestamp)
-- where shard_id = hash(some_attribute) % num_shards

3. Separate Hot and Cold Data:

-- Hot table: recent/active data, smaller, more splits
CREATE TABLE active_sessions (...);  

-- Cold table: historical data, larger, fewer splits  
CREATE TABLE archived_sessions (...);

While Spanner automatically manages split boundaries and tablet placement, it also provides fine-grained control over replication at the directory level.

Placement Groups:

Spanner allows defining placement groups—sets of databases or tables that share replication configuration:

-- All user data replicated to EU for GDPR
ALTER TABLE users SET OPTIONS (
  placement_group = 'eu_primary'
);

-- Analytics data replicated globally for low-latency reads
ALTER TABLE analytics SET OPTIONS (
  placement_group = 'global_read_optimized'
);

Per-Row Replication:

For advanced use cases, Spanner can even vary replication at the row level:

-- European users' data stays in EU
-- American users' data stays in Americas
-- Based on user.region column

This enables:

• Data residency compliance (GDPR, data sovereignty laws)
• Performance optimization (data near users who access it)
• Cost optimization (less replication for less critical data)

Movedir and Directory Movement:

Spanner's internal movedir operation handles directory movement:

1. Coordination: Central coordinator identifies source and destination zones
2. Replication Setup: New replica created at destination
3. Catch-Up: New replica catches up from source
4. Cutover: Traffic redirected to new replica
5. Cleanup: Old replica removed (if moving, not adding replica)

Automated Locality Optimization:

Spanner can automatically optimize directory placement based on access patterns:

• Track where reads/writes originate
• Move directories closer to their access sources
• Adjust leader placement for write-heavy directories
• Add read replicas for read-heavy directories

This optimization is continuous—as access patterns change, data placement adapts.

Google's operational experience with Spanner's automatic sharding provides insights into real-world behavior at massive scale.

Scale Characteristics:

• Google's internal Spanner deployment spans multiple continents
• Thousands of spanservers managing millions of splits
• Continuous rebalancing handles changing workloads
• System maintains SLAs during organic growth and sudden traffic spikes

Emergent Behaviors:

At scale, interesting patterns emerge:

1. Diurnal Patterns:

• Workloads follow time-of-day patterns
• Spanner pre-emptively rebalances before predicted load increases
• Leader placement may shift between regions as "follow the sun"

2. Event-Driven Scaling:

• Marketing campaigns cause traffic spikes
• Automatic splitting handles increased load
• Merging happens gradually after event ends

3. Long-Tail Key Ranges:

• Some key ranges are naturally quieter
• These may consolidate into larger splits
• System keeps reserve capacity for unexpected growth

What Spanner Doesn't Do Automatically:

While Spanner automates much of sharding, some decisions remain with operators:

1. Schema Design: Spanner can't fix a poorly designed schema. Key selection and interleaving decisions have lasting impact.

2. Replication Configuration: Multi-region vs. single-region, leader placement preferences—these are business decisions.

3. Capacity Planning: Spanner scales within provisioned resources. You must provision sufficient nodes.

4. Application-Level Sharding: If you need to shard at the application level (multi-tenancy, separate databases), that's your design.

5. Hotspot Prevention via Design: The best hotspot handling is prevention through good schema design.

We've explored how Spanner transforms sharding from an operational burden into an automated system behavior. Let's consolidate the key insights:

What's Next:

We've covered Spanner's architecture—global distribution, TrueTime, transactions, and sharding. The final page brings it all together: When to Use Spanner. We'll explore the use cases where Spanner shines, when alternatives might be better, and how to evaluate Spanner for your specific requirements.