We've explored the foundational concepts of distributed databases: why we distribute (motivation), how we divide data (fragmentation), how we maintain copies (replication), and how we hide complexity (transparency). Now we examine how these concepts assemble into coherent system designs—the architecture of distributed databases.

Architecture defines how components are organized, how they communicate, what they share, and how they coordinate. Different architectural choices lead to radically different system characteristics in terms of scalability, availability, consistency, and operational complexity.

There is no single "best" architecture. Each makes trade-offs appropriate for different workloads, scale requirements, and operational contexts. Understanding these architectures allows you to select the right system for your needs and understand the behaviors you'll observe.

Before examining specific architectures, let's understand the dimensions along which architectures differ:

1. Resource Sharing Model

What resources do nodes share?

• Shared-Nothing: Nodes share nothing; each has private CPU, memory, disk
• Shared-Disk: Nodes share storage but have private CPU and memory
• Shared-Everything: All resources shared (traditional SMP, not truly distributed)

2. Data Distribution Model

How is data placed across nodes?

• Partitioned: Each piece of data on one node (or replicated set of nodes)
• Replicated: All data on all nodes
• Hybrid: Some data partitioned, some replicated (e.g., reference data replicated)

3. Coordination Model

How do nodes coordinate operations?

• Centralized: One coordinator node makes decisions
• Decentralized: All nodes participate in coordination (consensus)
• Hierarchical: Tree of coordinators for scalability

4. Consistency Model

What guarantees does the system provide?

• Strong: All reads see most recent write (linearizable)
• Session: Consistency within a session
• Eventual: Replicas converge over time

5. Query Execution Model

How are queries distributed?

• Push-down: Sub-queries pushed to data nodes, results aggregated
• Centralized: All data pulled to central query processor
• Hybrid: Compute where efficient, move when necessary

Shared-nothing architecture assigns each node its own private CPU, memory, and storage. Nodes communicate only via network messages—there's no shared memory or shared disk. Each node is responsible for a subset of the data (its partition) and processes queries against that data locally.

Characteristics

• Independence: Nodes operate independently; one node's failure doesn't affect others' data
• Scale-out: Adding nodes adds capacity linearly (assuming good partitioning)
• Data locality: Queries execute where data resides, minimizing network transfer
• No contention: No shared resources to become bottlenecks

How It Works

1. Data is partitioned across nodes based on partition key
2. Queries are routed to nodes containing relevant data
3. Parallel execution on multiple nodes when query spans partitions
4. Coordinator aggregates results from participating nodes
5. Replication provides redundancy (each partition copied to multiple nodes)

Representative Systems

Shared-disk (or shared-storage) architecture gives each node private CPU and memory, but all nodes access a common storage layer. The storage is typically a SAN (Storage Area Network), network-attached storage, or cloud block storage.

Characteristics

• Single data copy: All nodes see the same storage; no partition placement decisions
• Flexible compute scaling: Add/remove compute nodes without moving data
• Simpler data management: No rebalancing, no data movement when scaling
• Coordination required: Cache coherence and locking across nodes essential

How It Works

1. All nodes connect to shared storage (SAN, cloud storage)
2. Each node caches data in private memory (buffer pool)
3. Coordination layer manages cache coherence (e.g., block locking)
4. Queries can execute on any node—data accessible everywhere
5. Write coordination prevents conflicts
6. Storage layer provides durability and (optionally) replication

Cache Coherence Challenge

When multiple nodes cache the same data, modifications must be coordinated:

• Write invalidation: When Node 1 modifies data, Node 2's cache is invalidated
• Distributed lock manager: Ensures only one node can modify a page
• Global buffer cache: Shared memory across nodes (rare, complex)

Cache coherence overhead limits the scalability of shared-disk compared to shared-nothing.

Representative Systems

Federated (or multi-database) architecture integrates multiple autonomous databases into a coherent system. Each component database retains independence but participates in a larger federation that enables unified queries and transactions.

Characteristics

• Heterogeneity: Component databases may be different vendors, versions, schemas
• Autonomy: Each database is independently administered and controlled
• Integration layer: Middleware provides unified query and transaction interface
• Schema mapping: Global schema maps to local schemas of component databases

Why Federation?

1. Legacy integration: Connect existing databases without replacing them
2. Organizational boundaries: Different departments own different databases
3. Best-of-breed: Use specialized databases for different workloads
4. Incremental migration: Gradually move to new systems without big-bang cutover
5. Regulatory requirements: Keep certain data in specific systems for compliance

Federation Components

Global Schema

A unified schema that abstracts local database schemas:

• Global entities map to local tables
• Schema conflicts resolved (naming, types, semantics)
• May be read-only or fully updatable

Query Decomposition

Global queries are split into local queries:

• Identify which local databases have relevant data
• Rewrite global query in local SQL dialects
• Optimize for local database capabilities

Result Integration

Local results are combined into global results:

• Schema translation back to global schema
• Joins across databases (expensive!)
• Aggregations, sorting, deduplication

Distributed Transactions

Coordinate transactions spanning databases:

• Two-phase commit across heterogeneous systems
• Limited by least capable participant
• Often avoided due to complexity

Multi-primary (or multi-master) architecture allows writes at multiple nodes, unlike primary-secondary where only one node accepts writes. Each primary independently accepts writes, with changes replicated between primaries.

Why Multi-Primary?

1. Write availability: If one primary fails, others continue accepting writes
2. Geographic writes: Users write to nearby primary; no cross-region latency
3. Load distribution: Write load spread across primaries

The Conflict Challenge

Multi-primary's fundamental challenge is conflicts. When two primaries concurrently modify the same data:

• Primary A sets balance = 100
• Primary B sets balance = 150
• After replication, which value wins?

Both writes are valid; there's no single "correct" answer. Conflict resolution is required.

Conflict Resolution Strategies

1. Last-Writer-Wins (LWW)

Highest timestamp wins. Simple but:

• Clock skew can pick "wrong" winner
• Concurrent writes are silently discarded
• Works well when conflicts are rare

2. Application-Defined Merge

Application provides merge logic:

• Shopping cart: union items
• Counter: add deltas
• Document: three-way merge

Requires careful application design.

3. CRDTs (Conflict-free Replicated Data Types)

Data structures that mathematically guarantee convergence:

• Grow-only counters
• Observed-remove sets
• Merge semantics built into data type

4. Conflict Detection and Resolution

Detect conflicts; let humans or application resolve:

• Flag conflicting versions
• Present to user for selection
• Store conflict metadata for audit

Multi-Primary Topologies

All-to-All Replication

Every primary replicates to every other. Simple but O(n²) connections.

Ring Replication

Each primary replicates to next in ring. Efficient but failure propagation risk.

Hub-and-Spoke

Central hub receives and redistributes. Single point of failure but simple.

Representative Systems

Cloud-native database architecture is designed from the ground up for cloud infrastructure, leveraging cloud primitives (object storage, elastic compute, managed networking) rather than adapting traditional designs.

Key Characteristics

1. Compute-Storage Separation

Compute (query processing) and storage (data persistence) are separate, independently scalable tiers:

• Scale compute for query load without touching storage
• Scale storage for data growth without touching compute
• Pay for each tier based on actual usage

2. Serverless Operation

Resources automatically scale with demand:

• Automatic provisioning during load spikes
• Scale to zero during idle periods
• No capacity planning required

3. Multi-tenant Isolation

Cloud databases serve multiple tenants on shared infrastructure:

• Workload isolation prevents noisy neighbors
• Resource quotas ensure fair sharing
• Data isolation for security

Cloud-Native Benefits

• Elastic scaling: Handle 100× traffic spike without pre-provisioning
• Cost efficiency: Pay only for what you use (compute, storage, I/O)
• Operational simplicity: Cloud provider manages infrastructure
• Global availability: Deploy in any region with clicks
• Storage economics: Cloud object storage is cheap ($0.02/GB/month)

Representative Systems

Selecting the right architecture depends on workload characteristics, scale requirements, operational constraints, and organizational context.

Decision Framework

Questions to Guide Selection

1. What's my scale target? 1TB or 1PB? 100 TPS or 1M TPS?
2. What consistency do I need? Can my application tolerate stale reads?
3. What's my availability requirement? Can I tolerate downtime during failover?
4. What's my geographic distribution? Single region, multi-region, global?
5. What's my operational capacity? Dedicated DBA team or developers doing everything?
6. What's my budget profile? Fixed capacity or pay-per-use?
7. What's my integration landscape? Greenfield or connecting existing systems?

Architecture is how distributed database concepts combine into functional systems. Let's consolidate the key concepts:

Module Complete: Distributed Database Concepts

You've now completed the foundational module on distributed database concepts. You understand:

• Why we distribute databases (motivation)
• How we divide data (fragmentation)
• How we maintain copies (replication)
• How we hide complexity (transparency)
• How these combine into systems (architecture)

With this foundation, you're prepared to explore specific distributed database techniques in subsequent modules: fragmentation strategies, distributed transactions, CAP theorem implications, and sharding patterns.