Why We Need Distributed Systems

If distributed systems are so complex—so prone to subtle failures, so difficult to debug, so expensive to operate—why do we build them? The answer is simple: we have no choice.

The demands of modern software exceed what any single machine can provide. A smartphone in your pocket has more computing power than the machines that sent humans to the moon, yet that power is insufficient for the services you use daily. When you search Google, you're not querying one computer—you're querying thousands, coordinated to return results in under 200 milliseconds. When Netflix serves a movie, it's not streaming from one server—it's orchestrating content delivery from edge nodes distributed across the globe.

This page examines the fundamental forces that make distribution inevitable and why understanding these forces is essential for sound engineering decisions.

For decades, the solution to performance problems was simple: buy a bigger machine. This approach—vertical scaling or "scaling up"—worked remarkably well. Moore's Law delivered exponential growth in processing power every 18-24 months. But this era has ended.

The Physical Limits of Single Machines:

CPU Limits:

• Clock speeds plateaued around 2005 (~5 GHz practical limit due to heat)
• More cores don't help single-threaded workloads
• The fastest single-core performance improves ~5-10% per year now
• Amdahl's Law limits parallelization gains: a 5% sequential portion caps speedup at 20x

Memory Limits:

• Largest commodity servers: ~12 TB RAM (as of 2024)
• Specialized machines (SGI UV): ~64 TB RAM, but cost millions
• Many workloads exceed even these limits (e.g., search indices, ML training datasets)
• Memory access is the new bottleneck (100x slower than CPU cache)

Storage Limits:

• Single-machine storage: ~1 PB practical (dense NVMe configurations)
• Storage I/O bandwidth: ~100 GB/s for enterprise SSDs
• Large datasets (10+ PB) cannot physically fit on one machine

Network Limits:

• Single NIC: 100-400 Gbps (current state-of-the-art)
• Serving millions of users simultaneously requires more bandwidth
• Connection count limits: ~1 million concurrent connections per machine with optimization

The Cost Curve Reality:

Vertical scaling follows a non-linear cost curve. Doubling capacity often costs more than double:

• A 64-core server might cost $20,000
• A 128-core server might cost $60,000 (3x for 2x capacity)
• A 256-core server might cost $200,000 (10x for 4x capacity)

Beyond certain thresholds, horizontal scaling (adding more commodity machines) becomes economically rational even before hitting physical limits. This is why hyperscalers run millions of modest machines rather than thousands of extreme ones.

Modern applications serve global audiences at scales that were unimaginable a decade ago. These scales fundamentally require distributed architectures.

The Numbers That Drive Distribution:

Scaling Dimensions:

Scale manifests across multiple dimensions, each requiring different distribution strategies:

1. User Scale (Concurrent Sessions)

• Problem: Each active user requires state, connections, and resources
• Solution: Distribute across multiple application servers, share-nothing architecture
• Example: WhatsApp serves 2 billion users with ~3 million Erlang connections per server

2. Data Scale (Storage Volume)

• Problem: Datasets exceed single-machine storage capacity
• Solution: Partition (shard) data across nodes, replicate for durability
• Example: CERN's LHC generates ~1 PB/second during experiments, stored across distributed file systems

3. Transaction Scale (Operations/Second)

• Problem: More operations than one CPU can process
• Solution: Partition workload, parallelize processing
• Example: Visa processes 65,000+ transaction messages per second during peaks

4. Compute Scale (Processing Requirements)

• Problem: Computation too large for single machine (ML training, simulations)
• Solution: Distribute computation, aggregate results
• Example: GPT-4 training required thousands of GPUs across multiple clusters

Single points of failure are unacceptable for modern services. Distribution provides redundancy—if one component fails, others continue operating. This is not optional for systems that society depends on.

Understanding Availability Math:

Why Single Machines Cannot Achieve High Availability:

Hardware fails at predictable rates:

• Hard Drives: 2-4% annual failure rate
• Memory (DIMM): ~1% annual failure rate per DIMM
• Power Supplies: ~3% annual failure rate
• Entire Server: ~5-10% annual failure rate (any component)

With a 5% annual server failure rate, a single server achieves ~99.5% availability at best—and that's only hardware. Add software bugs, deployments, security patches, and operating system updates, and realistic single-machine availability is 99-99.5%.

The Path to Higher Availability:

Redundancy at Every Level:

• Multiple servers: If one fails, others serve traffic
• Multiple data centers: If one loses power, others continue
• Multiple regions: If one has network issues, others are isolated

No Single Points of Failure (SPOF):

• Load balancers in pairs (active-passive or active-active)
• Databases with replicas (master-slave or multi-master)
• Network paths with multiple routes

Failure Domains:

• Components should fail independently
• Cascading failures must be prevented
• Blast radius of any failure should be limited

The speed of light is the universe's ultimate rate limiter—and it makes geographic distribution essential for low-latency global services.

The Physics of Latency:

Light travels at ~299,792 km/s in a vacuum, but in fiber optic cables, it's slower (~200,000 km/s due to refraction). Even at these speeds, distance creates unavoidable latency:

• New York to San Francisco: ~4,000 km → ~20ms one-way → ~40ms round-trip (RTT)
• New York to London: ~5,500 km → ~28ms one-way → ~55ms RTT
• New York to Tokyo: ~10,800 km → ~54ms one-way → ~108ms RTT
• New York to Sydney: ~16,000 km → ~80ms one-way → ~160ms RTT

These are theoretical minimums. Real-world latencies are 1.5-2x higher due to:

• Non-direct routing (cables follow geography, not straight lines)
• Network hops (routers add 0.5-5ms each)
• Congestion (queueing delays)
• Protocol overhead (TCP handshakes, TLS negotiation)

Why Latency Matters:

User Experience:

• Google research: 200ms delay → 0.22% drop in searches
• Amazon research: 100ms delay → 1% drop in sales
• Akamai research: 100ms delay → 7% drop in conversion

Application Constraints:

• Video conferencing: <150ms for natural conversation
• Online gaming: <50ms for competitive play, <100ms for casual
• Financial trading: <1ms for high-frequency (co-location essential)
• Voice calls: <150ms for acceptable quality

The Solution: Bring Computation Closer to Users

Content Delivery Networks (CDNs):

• Cache static content at edge locations
• Reduce RTT for assets from ~200ms to ~20ms
• Major CDNs (Cloudflare, Akamai) have 200+ edge locations globally

Regional Deployments:

• Deploy application instances in multiple regions
• Route users to nearest region
• Trade consistency complexity for latency

Edge Computing:

• Run application logic at edge locations
• Process data before sending to origin
• Examples: Cloudflare Workers, AWS Lambda@Edge

Beyond technical necessity, distributed systems often make economic sense—enabling scale, efficiency, and flexibility that monolithic approaches cannot match.

The Economics of Commodity Hardware:

Distributed systems pioneered by Google demonstrated that many cheap machines often outperform few expensive ones:

The Google Philosophy (circa 2003):

• Commodity hardware fails frequently, but it's cheap
• Design software to expect and handle failures
• Aggregate capacity of 1000 cheap machines > capacity of 10 expensive ones
• Total cost of 1000 cheap machines < total cost of 10 expensive ones

This philosophy—software solving hardware reliability—revolutionized infrastructure economics.

The Elasticity Advantage:

Distributed architectures enable elastic scaling—adjusting capacity to match demand:

Fixed Resources (Monolithic):

• Provision for peak load
• Pay for capacity you're not using 90% of the time
• Over-provision to avoid failures
• Typical utilization: 10-20% of peak capacity

Elastic Resources (Distributed):

• Scale up during high demand
• Scale down during low demand
• Pay for what you use
• Typical utilization: 50-70% of provisioned capacity

Example Cost Impact:

Consider a service with 10x traffic variation (normal: 10K RPS, peak: 100K RPS):

• Fixed provisioning: Buy capacity for 100K RPS = $100,000/month
• Elastic provisioning: Average 20K RPS = $20,000/month
• Savings: 80% cost reduction

Cloud platforms (AWS, GCP, Azure) are built on this premise. The entire cloud computing model—pay-per-use, infinite scalability—is enabled by distributed architectures.

Organizational Economics:

Beyond infrastructure costs, distributed systems affect organizational efficiency:

Development Velocity:

• Independent services can be deployed independently
• Teams don't block each other
• Faster iteration cycles

Talent Utilization:

• Teams can specialize in specific services
• Smaller cognitive load per developer
• Easier onboarding (understand one service, not entire monolith)

Risk Distribution:

• Bugs in one service don't crash everything
• Deployments affect limited blast radius
• Experiments can be isolated to specific user segments

Distribution provides security and isolation benefits that are difficult to achieve with monolithic systems.

Defense in Depth:

Network Segmentation:

• Sensitive systems can be isolated in private networks
• Only designated services can access databases
• API gateways provide a single, hardened entry point
• Internal services communicate over encrypted channels

Blast Radius Containment:

• A compromised service cannot directly access all data
• Each service can have minimal permissions (least privilege)
• Security breaches can be contained to affected components
• Forensic investigation is easier with isolated services

Multi-tenancy Isolation:

• Different customers can be served by different instances
• Data can be physically separated by tenant
• Noisy neighbor problems can be mitigated
• Compliance requirements can be met per-region

The Shared-Nothing Architecture:

Distributed systems often embrace "shared-nothing" architecture:

• Each node has its own CPU, memory, and storage
• No shared resources to become bottlenecks
• No shared state to become security vulnerabilities
• Failure of one node doesn't affect others

This isolation model, while complex to coordinate, provides inherent security boundaries that monolithic systems must artificially construct.

Understanding why we need distributed systems also requires understanding when we don't need them. Premature distribution is a common and costly mistake.

Signs You Might Be Distributing Too Early:

1. You Haven't Hit Single-Machine Limits

• A PostgreSQL database handles 10,000+ TPS on commodity hardware
• A single application server can serve 10,000+ concurrent users
• If you're not at these limits, distribution adds complexity without benefit

2. You're Optimizing for Imaginary Scale

• "We need to be ready for millions of users"
• But you have 100 users and no indication of growth
• Build for 10x current scale, not 1000x imaginary scale

3. Your Team Is Small

• Distributed systems require distributed expertise
• Debugging distributed issues requires specific skills
• A 3-person team shouldn't manage 30 microservices

4. Your Problem Is Actually Algorithmic

• Sometimes "slow" is an O(n²) algorithm, not a scale problem
• Fixing the algorithm is cheaper than distributing computation
• Profile before distributing; you might be solving the wrong problem

The Cost of Premature Distribution:

• Operational Overhead: Monitoring, alerting, log aggregation across services
• Debugging Complexity: Distributed traces vs simple stack traces
• Latency Addition: Network calls between services add milliseconds
• Consistency Challenges: Transactions across services require sagas/2PC
• Development Friction: Local environment requires running multiple services
• Testing Complexity: Integration tests must coordinate multiple services

The Monolith-First Approach:

Many successful companies started with monoliths and distributed later:

• Shopify: Monolith serving billions in GMV; selectively extracting services
• Amazon: Started as a monolith; decomposed over years as needed
• Etsy: Monolith supporting millions of sellers
• Stack Overflow: Serves 1 billion monthly visitors with a handful of servers

The path: Monolith → Identify bottlenecks → Extract specific services → Repeat as needed.