Distributed System Concepts

The defining goal of a distributed system is to appear as a single coherent system despite being composed of multiple autonomous nodes. Achieving this illusion requires transparency—the concealment of the system's distributed nature from users and applications.

When you access a website, you don't think about which server in which data center is handling your request. When you save a file to cloud storage, you don't consider which physical disk across which geographic region stores your data. This seamless experience is transparency at work.

However, transparency is not binary—it exists in multiple dimensions. A system might transparently handle server locations but expose replication delays. Understanding these different transparency types allows system designers to make informed decisions about which complexities to hide and which to expose.

Transparency in distributed systems refers to hiding from users and application programmers the separation of components so that the system is perceived as a whole rather than a collection of independent pieces.

The International Organization for Standardization's Reference Model for Open Distributed Processing (ISO RM-ODP) defines a framework of transparency types that articulate different aspects of distribution that can be hidden:

Why Transparency Matters:

1. Usability — Users shouldn't need to understand system internals to use it effectively
2. Portability — Applications should work regardless of underlying infrastructure changes
3. Simplicity — Hiding complexity reduces cognitive load and development effort
4. Abstraction — Clean interfaces enable modular system design

The Transparency Challenge:

Complete transparency is often impossible or undesirable. Network latency, partial failures, and consistency constraints are physical realities that cannot be entirely hidden. Attempting to create the illusion of a single, instantly-responsive, never-failing system can lead to poor user experiences (mysterious delays) or incorrect programs (ignoring failures).

The key insight is that transparency should be applied thoughtfully. Each type of transparency involves tradeoffs between simplicity and control, abstraction and awareness.

Access transparency hides differences in data representation and how resources are accessed. It enables local and remote resources to be accessed using identical operations, without the user or application being aware of whether a resource is local or remote.

The Problem Access Transparency Solves:

Different computers may have different:

• Data representations (big-endian vs. little-endian byte ordering)
• Character encodings (ASCII, UTF-8, UTF-16)
• Floating-point formats (IEEE 754 variations)
• Programming language conventions (how structures are laid out in memory)

Without access transparency, every application would need to handle these differences explicitly when communicating with remote systems.

How Access Transparency Works:

Access transparency is typically achieved through:

1. Standardized Data Representation

   • External Data Representation (XDR) for Sun RPC
   • Protocol Buffers for gRPC
   • JSON/XML for web services
   • ASN.1 for telecommunications

2. Marshalling/Serialization

   • Local data structures converted to standard wire format for transmission
   • Remote data structures reconstructed from wire format
   • Language-specific stubs hide this conversion

3. Unified Interface Definitions

   • IDL (Interface Definition Language) describes APIs
   • Same interface used for local and remote calls
   • Compiler generates language-specific bindings

Location transparency hides where a resource is physically located. Users and applications can access resources without knowing their physical or network location. The same name provides access to the resource regardless of where it resides.

The Problem Location Transparency Solves:

Physical locations of resources change:

• Servers migrate between data centers
• Services scale across regions
• Cloud instances are ephemeral
• Load balancing distributes requests

Hardcoding physical locations (IP addresses, machine names) into applications creates brittle systems that break when infrastructure changes.

How Location Transparency Works:

1. Naming Systems

   • DNS (Domain Name System): maps human-readable names to IP addresses
   • Service Discovery: dynamic location of service instances
   • URIs: Uniform Resource Identifiers abstract location

2. Indirection Layers

   • Load balancers: single address proxies to multiple backends
   • Reverse proxies: abstract backend topology
   • Service meshes: route based on service names, not locations

3. Virtual Addresses

   • Virtual IPs (VIPs): floating addresses that migrate between machines
   • Anycast: same IP address routed to nearest/best server

Location Transparency in Practice:

Example: Cloud Object Storage

When you store a file in AWS S3 with URL s3://my-bucket/reports/2024/quarterly.pdf, you have no knowledge of:

• Which physical data center stores the file
• Which disk array contains the bits
• Whether the data is replicated across regions
• If the file recently migrated for load balancing

The logical path (my-bucket/reports/2024/quarterly.pdf) remains constant regardless of physical location.

Limits of Location Transparency:

Location transparency cannot hide the fundamental reality of physics. A resource in Tokyo cannot be accessed from New York with the same latency as a local resource. For latency-sensitive applications, some location awareness may be necessary (e.g., selecting the nearest CDN edge node). The transparency provides logical abstraction; performance characteristics may still vary by location.

Migration transparency hides the fact that resources may move from one location to another. This goes beyond location transparency by ensuring that ongoing access remains unaffected when resources relocate.

The Distinction from Location Transparency:

• Location transparency: resources can be accessed without knowing their location
• Migration transparency: resources can move without affecting ongoing access

Location transparency addresses static location, while migration transparency addresses dynamic relocation.

Why Migration Transparency Matters:

Modern infrastructure requires frequent resource movement:

• Hardware maintenance requires migrating VMs to other hosts
• Load balancing may relocate services to less loaded machines
• Disaster recovery requires geographic failover
• Container orchestration (Kubernetes) constantly reschedules pods
• Database shards migrate as data distributions change

How Migration Transparency Works:

1. Stable Identifiers

   • Resources have persistent identifiers independent of location
   • Identifiers resolve to current location dynamically
   • Example: Kubernetes pod IDs persist through rescheduling

2. Connection Handoff

   • Active connections transfer during migration
   • State synchronized between old and new locations
   • Example: Live VM migration with TCP connection preservation

3. Session Persistence

   • User sessions detached from specific instances
   • State stored externally (Redis, distributed cache)
   • Any instance can resume a session

Relocation transparency extends migration transparency by hiding that resources may be moved while being accessed. Migration transparency ensures moves don't break existing references; relocation transparency ensures they don't interrupt active operations.

The Distinction:

• Migration transparency: references remain valid after a move
• Relocation transparency: operations in progress are not disrupted during a move

Technical Requirements:

Relocation transparency requires sophisticated coordination:

1. State Synchronization

   • All state must transfer to the new location
   • Including in-flight requests and their interim results
   • No data loss or duplication during transfer

2. Request Routing

   • Requests in transit must reach the new location
   • No requests can be lost during the transition
   • Ordering must be preserved when significant

3. Minimal Interruption

   • Operations should experience minimal delay
   • Ideally, clients perceive no interruption
   • Certainly no visible errors or inconsistency

Practical Implementation: Database Shard Migration

Consider migrating a database shard (a partition of data) while the system continues serving requests:

Phase 1: Preparation

• New shard instance initialized
• Bulk data copied to new location
• New instance catches up to current state

Phase 2: Dual-Write Mode

• Writes go to both old and new locations
• Reads continue from old location
• Both replicas stay synchronized

Phase 3: Cutover

• Brief pause (milliseconds) to drain in-flight writes
• Routing updated to point to new location
• Old location becomes backup/garbage

Phase 4: Cleanup

• Old shard decommissioned after confirmation
• Routing references removed

From clients' perspectives, the shard simply became slightly slower during cutover, then resumed normal operation. No errors, no rerequests, no awareness that data physically moved.

Replication transparency hides that multiple copies of a resource exist. Users and applications can access the resource as if there were a single copy, without concern for which replica they're accessing or how replicas are kept synchronized.

Why Replication Exists:

Distributed systems replicate data for critical reasons:

1. Fault Tolerance — If one replica fails, others continue serving
2. Performance — Replicas can be placed near users, reducing latency
3. Scalability — Read load can be distributed across replicas

The Challenge Replication Creates:

Multiple copies introduce consistency challenges:

• Updates must propagate to all replicas
• Different replicas may have different data momentarily
• Clients accessing different replicas may see different values

Replication transparency hides these challenges from applications.

How Replication Transparency Works:

1. Consistent Reads

   • Clients always read the (logically) latest value
   • Reads may be routed to the most current replica
   • Or read protocols ensure consistency

2. Atomic Updates

   • Updates appear to execute once, despite multiple replicas
   • Update protocols coordinate replicas (e.g., two-phase commit)
   • Failures handled without client awareness

3. Single-Copy Semantics

   • The replicated resource behaves as if unreplicated
   • No observable difference from centralized storage
   • All consistency guarantees hold

Concurrency transparency hides that a resource may be accessed by multiple users simultaneously. Each user can access the shared resource without needing to coordinate with others or be aware that contention exists.

The Problem Concurrency Creates:

Simultaneous access to shared resources causes conflicts:

• Two users editing the same document
• Two transactions updating the same bank account
• Two services modifying the same configuration

Without protection, concurrent modifications can:

• Overwrite each other (lost updates)
• Create inconsistent states (partial updates)
• Cause observable anomalies (non-repeatable reads)

How Concurrency Transparency Works:

1. Locking Mechanisms

   • Exclusive locks prevent concurrent modification
   • Shared locks allow concurrent reads
   • Locks acquired and released automatically by system

2. Optimistic Concurrency Control

   • Operations proceed assuming no conflict
   • Conflicts detected at commit time
   • Conflicting operations automatically retried

3. Serializable Transactions

   • System guarantees transactions execute as if sequentially
   • Concurrent transactions isolated from each other
   • Result equivalent to some sequential ordering

Concurrency Transparency in Databases:

Relational databases provide excellent concurrency transparency through transactions with ACID properties:

• Atomicity: All-or-nothing execution
• Consistency: Database constraints maintained
• Isolation: Concurrent transactions don't interfere (concurrency transparency)
• Durability: Committed changes survive failures

Distributed Concurrency Challenges:

Concurrency transparency becomes harder in distributed settings:

• Locks require coordination across nodes (distributed locking)
• Deadlock detection spans multiple machines
• Two-Phase Commit (2PC) required for distributed transactions
• Performance overhead of coordination

Systems like Google Spanner achieve global concurrency transparency using synchronized clocks (TrueTime) and complex protocols, but this requires significant infrastructure investment.

Failure transparency hides faults and recovery from users. When components fail, the system continues operating (perhaps at reduced capacity) without users experiencing errors or needing to take corrective action.

Why Failure Transparency Is Critical:

Distributed systems experience frequent failures:

• Servers crash or restart
• Network links fail or become congested
• Disks fail silently or catastrophically
• Software has bugs causing unexpected behavior

With many components, failure is the norm, not the exception. Large-scale distributed systems experience component failures constantly, yet users expect reliable service.

Levels of Failure Transparency:

1. Detection Transparency

   • System detects failures internally
   • Users don't see failure indicators
   • Timeouts, health checks, heartbeats

2. Recovery Transparency

   • System recovers from failures automatically
   • Users don't need to retry requests
   • Failover, restart, retry handled internally

3. Masking Transparency

   • Failures entirely hidden from users
   • Requests succeed despite component failures
   • Full redundancy enables seamless operation

Example: Load Balancer Health Checks

A load balancer provides failure transparency by:

1. Active Monitoring: Regularly pinging backend servers with health check requests
2. Failure Detection: Marking servers as unhealthy after consecutive failed checks
3. Traffic Rerouting: Directing requests only to healthy servers
4. Recovery Detection: Returning servers to pool when health checks pass again

From the client's perspective, requests always succeed (assuming at least one healthy backend). Server failures are entirely invisible—the client doesn't know a server crashed, doesn't receive an error, doesn't need to retry. This is failure transparency in action.

Persistence transparency hides whether a resource is stored in volatile memory or persistent storage, and the mechanisms used to maintain durability. Applications interact with resources without concern for their storage characteristics or the complexity of ensuring data survives failures.

The Persistence Challenge:

Data exists in different storage tiers with different characteristics:

Applications shouldn't need to manage data movement between these tiers or explicitly handle durability.

How Persistence Transparency Works:

1. Unified Data Access

   • Same API regardless of underlying storage
   • File system abstracts device details
   • Memory-mapped files blend memory and storage

2. Automatic Durability

   • Databases ensure committed data survives crashes
   • File systems handle write-back policies
   • Replication ensures geographic durability

3. Transparent Caching

   • Frequently accessed data cached in memory
   • Cache policy handled by system, not application
   • Dirty data written back automatically

Persistence Transparency in Operating Systems:

Operating systems provide persistence transparency through:

1. Virtual Memory + Swap

   • Applications see unified address space
   • OS pages data to disk when memory scarce
   • Page-ins/page-outs invisible to applications

2. Page Cache

   • Recent file reads cached in memory
   • Subsequent reads served from cache
   • Eviction and write-back handled automatically

3. Journaling File Systems

   • Write-ahead logging ensures consistency
   • Crash recovery transparent to applications
   • fsck or journal replay automatic at boot

Transparency is the mechanism by which distributed systems achieve their defining goal: appearing as single coherent systems despite distributed implementation. Let's consolidate what we've learned:

Looking Ahead:

With transparency understood, we next examine scalability—how distributed systems grow to handle increasing load. Scalability is why we distribute in the first place, and understanding its patterns and limits is essential for distributed system design.