Fallacies of Distributed Computing
All Eight Fallacies and Their Implications
Read aloud is unavailable in this browser.
The Complete Fallacy Framework
In the previous pages, we examined four foundational fallacies: network reliability, latency, bandwidth, and security. These are the most commonly discussed because they cause the most visible failures. But Peter Deutsch's original seven fallacies (plus James Gosling's eighth) form a complete framework for understanding distributed systems pitfalls.
This page introduces the remaining four fallacies and then synthesizes all eight into a unified mental model for designing robust distributed systems.
Peter Deutsch originally identified seven fallacies in 1994. James Gosling later added the eighth. Together, they represent the accumulated wisdom of Sun Microsystems engineers who built some of the first large-scale networked systems.
| Fallacy | Reality | Coverage | |
|---|---|---|---|
| 1 | The network is reliable | Networks fail in countless ways | Page 1 (detailed) |
| 2 | Latency is zero | Data takes time to travel | Page 2 (detailed) |
| 3 | Bandwidth is infinite | Network capacity is finite and shared | Page 3 (detailed) |
| 4 | The network is secure | Traffic can be intercepted and modified | Page 4 (detailed) |
| 5 | Topology doesn't change | Network structure evolves constantly | This page |
| 6 | There is one administrator | Networks span organizational boundaries | This page |
| 7 | Transport cost is zero | Moving data has real costs | This page |
| 8 | The network is homogeneous | Networks contain diverse technologies | This page |
Fallacy 5: Topology Doesn't Change
The topology fallacy assumes that the network's physical and logical structure remains static. In reality, network topology is in constant flux.
- •Server failures and replacements — Machines die, new ones spin up with different IPs
- •Autoscaling — Container orchestrators add/remove instances based on load
- •Network equipment changes — Routers, switches replaced or reconfigured
- •DNS updates — Service endpoints resolve to different addresses
- •Cloud region changes — Services migrate between availability zones
- •Load balancer reconfigurations — Traffic routing changes dynamically
- •Failover events — Primary fails, secondary becomes primary at new address
- •Network partitions — Temporary topology changes during outages
What goes wrong when you assume static topology:
- Hardcoded IP addresses — Service moves, clients break
- Cached DNS beyond TTL — Stale resolution prevents reaching new endpoints
- Assumptions about co-location — 'These services are on the same rack' becomes false after migration
- Firewall rules by IP — Legitimate traffic blocked after IP change
- Path-dependent load balancing — Consistent hashing assumes stable node set
Architectural implications:
- Use service discovery (Consul, Kubernetes DNS) instead of hardcoded addresses
- Implement health checks to detect topology changes
- Design for graceful membership changes in distributed protocols
- Use connection pooling with refresh to adapt to endpoint changes
- Prefer relative locations (same AZ, different region) over absolute addresses
Treat servers as cattle (replaceable, numbered), not pets (named, irreplaceable). If your architecture mourns when a specific server dies, you've fallen for the topology fallacy. Design so any component can be replaced without special handling.
Fallacy 6: There Is One Administrator
The administrator fallacy assumes that a single entity controls the entire network and can coordinate changes uniformly. Modern systems span multiple administrative domains with different policies, priorities, and capabilities.
- •Your organization — Internal IT policies, security requirements, budget constraints
- •Cloud providers — AWS, GCP, Azure have their own maintenance windows and policies
- •SaaS vendors — Third-party services you depend on operate independently
- •ISPs and carriers — Network providers have their own change schedules
- •Partner organizations — Business partners with different IT practices
- •Regulatory bodies — GDPR, HIPAA, PCI impose requirements you don't control
- •End users — Client devices with varying configurations and network conditions
What goes wrong with single-admin assumptions:
- Coordinated upgrades — You can't schedule downtime across domains you don't control
- Consistent policies — Different security standards across partners
- Synchronized changes — Independent maintenance windows cause unexpected interactions
- Shared responsibility gaps — Everyone assumes someone else handles a concern
- Vendor lock-in blindness — Assuming you can change providers without friction
Architectural implications:
- Design for external dependencies failing independently
- Implement vendor abstraction layers for portability
- Plan backup strategies for critical third-party services
- Use contracts and SLAs to set expectations across domains
- Build multi-vendor resilience where possible
- Accept that coordinated global changes are often impossible
Major cloud outages happen. AWS us-east-1 has had multiple multi-hour outages. When your 'reliable' cloud provider fails, you can't call their admin and demand they fix it. Multi-region and multi-cloud architectures acknowledge this reality.
Fallacy 7: Transport Cost Is Zero
The transport cost fallacy assumes that moving data over the network is free. We touched on bandwidth costs earlier, but transport cost encompasses more than just data transfer fees.
| Cost Type | Description | Impact |
|---|---|---|
| Direct Egress Fees | Cloud provider charges for outbound data | $0.02-0.12/GB, scales to millions |
| Cross-Region Transfer | Premium for inter-region communication | 2-5x intra-region cost |
| Serialization CPU | Converting objects to/from wire format | CPU cycles per request |
| Encryption CPU | TLS encryption/decryption | ~1-5% CPU overhead |
| Memory Buffers | Copying data in/out of network stack | Memory allocation and GC pressure |
| Connection Overhead | TCP/TLS handshakes, keepalives | Latency and resource consumption |
| Protocol Overhead | Headers, framing, acknowledgments | Bandwidth waste on metadata |
| Operational Complexity | Monitoring, debugging, security for network | Engineering time and tooling |
Case study: The hidden cost of microservices
A monolith making in-process function calls has near-zero communication cost. Breaking it into microservices introduces:
- Serialization cost for every call (JSON/Protobuf encoding)
- Network latency (even same-rack has sub-millisecond cost)
- Connection management overhead
- Retry and timeout logic
- Service mesh sidecar proxies (Envoy adds ~1ms per hop, CPU overhead)
- Observability infrastructure (distributed tracing, log aggregation)
For high-frequency, low-latency operations, these costs can exceed the original processing time. The boundary between services should be chosen with transport cost in mind.
Architectural implications:
- Batch operations where possible to amortize overhead
- Choose service boundaries wisely — high-frequency communication within services, not across
- Use efficient serialization (gRPC/Protobuf vs REST/JSON)
- Consider compute-storage colocation for data-intensive workloads
- Cache aggressively to eliminate repeated transfers
- Monitor and attribute costs per service and operation
Despite abstractions like 'the cloud' that hide physical location, the physics of data transfer hasn't changed. Processing data close to where it's stored is always cheaper than shipping it across the network. Edge computing, CDNs, and data locality optimizations exist because transport has real costs.
Fallacy 8: The Network Is Homogeneous
The homogeneity fallacy assumes that all parts of the network have the same characteristics, use the same protocols, and behave the same way. Real networks are heterogeneous patchworks of different technologies, configurations, and capabilities.
- •Different physical media — Fiber, copper, wireless, satellite with different characteristics
- •Varying MTU sizes — Ethernet (1500), jumbo frames (9000), tunnels (lower)
- •Different protocol support — IPv4 vs IPv6, HTTP/1.1 vs HTTP/2 vs HTTP/3 (QUIC)
- •Inconsistent QoS — Some paths prioritize differently, throttle differently
- •NAT and firewalls — Different middleboxes with different behaviors
- •Load balancer differences — Different algorithms, health check behaviors, timeout defaults
- •Cloud provider variations — AWS networking differs from GCP differs from Azure
- •Client diversity — Mobile networks, corporate proxies, regional restrictions
What goes wrong with homogeneity assumptions:
- Path MTU discovery failures — Packets too large for some path segments get dropped
- Protocol feature assumptions — Assuming all clients support HTTP/2 push or WebSockets
- Timeout inconsistencies — Your 30s timeout vs their 10s connection limit
- Connection behavior variations — Some networks aggressively close idle connections
- Certificate compatibility — Older clients don't support newer TLS or certificates
- Regional internet restrictions — Great Firewall of China, regional geoblocking
Architectural implications:
- Test across diverse network conditions — Use tools like Toxiproxy, Charles Proxy, or real devices on various networks
- Support protocol fallbacks — HTTP/3 → HTTP/2 → HTTP/1.1 graceful degradation
- Handle MTU issues — Avoid hardcoding, let path MTU discovery work
- Plan for regional differences — Different CDN pops, local regulations, network characteristics
- Abstract network dependencies — Service mesh or abstraction layer isolates application from network details
Testing on localhost or within a single data center hides heterogeneity issues. Test with real mobile devices on cellular networks, across different ISPs, and internationally. The network doesn't behave the same everywhere—and your users are everywhere.
The Fallacies as a Complete System
The eight fallacies aren't independent—they interact and compound. Understanding them as a system helps you anticipate second-order effects in your designs.
| When This Fallacy... | ...Intersects With This | ...You Get |
|---|---|---|
| Network unreliable | Latency isn't zero | Timeouts don't mean failure—they mean unknown |
| Latency isn't zero | Bandwidth isn't infinite | Large payloads multiply latency effects |
| Network insecure | Topology changes | Security policies must be dynamic, not IP-based |
| One administrator | Transport has cost | Can't force partners to optimize their egress |
| Topology changes | Network heterogeneous | Failover might land on path with different characteristics |
| Transport cost | Bandwidth limits | Optimizing for one affects the other |
| Network unreliable | Multiple administrators | Can't coordinate on reliability improvements |
| Homogeneous network | Security assumptions | Security controls vary by network segment |
Example: A multi-region outage scenario
Consider a system deployed across three regions. When the primary region fails:
- Fallacy 1 (Reliability): Network partition isolates primary
- Fallacy 5 (Topology): Failover changes traffic routing entirely
- Fallacy 2 (Latency): Users now route through distant secondary, latency increases
- Fallacy 3 (Bandwidth): Secondary region absorbs all traffic, bandwidth saturates
- Fallacy 6 (Administrator): Cloud provider's failover, not yours to control
- Fallacy 7 (Transport cost): Cross-region traffic fees skyrocket
- Fallacy 8 (Homogeneity): Secondary region has different network characteristics
- Fallacy 4 (Security): New network paths need new firewall rules
A system designed with only one fallacy in mind would fail when the others manifest simultaneously—which they will during any significant incident.
Real outages rarely violate just one fallacy. Network issues trigger topology changes, which affect latency, which causes timeouts, which trigger retries, which saturate bandwidth. Design for the cascade, not just the initial trigger.
The Anti-Fallacy Architecture
Knowing the fallacies, we can invert them into architectural principles. A system that addresses all eight fallacies embodies network-aware design.
- •Network is reliable
- •Latency is zero
- •Bandwidth is infinite
- •Network is secure
- •Topology doesn't change
- •One administrator
- •Transport cost is zero
- •Network is homogeneous
- •Design for failure at every layer
- •Minimize round-trips, parallelize calls
- •Compress, paginate, cache aggressively
- •Encrypt everything, verify always
- •Use service discovery, expect change
- •Plan for external failures
- •Batch operations, colocate data
- •Test across diverse conditions
- •Every network call has a timeout — No unbounded waits
- •Retries have exponential backoff and jitter — Prevent thundering herds
- •Operations are idempotent where possible — Safe retries
- •Circuit breakers protect against failing dependencies — Fail fast
- •Critical paths minimize network calls — Fewer calls = less exposure
- •All traffic is encrypted with mutual authentication — Zero trust
- •Service discovery is dynamic — Endpoints can change
- •Graceful degradation is planned — Third parties will fail
- •Payload sizes are bounded and compressed — Efficient transfer
- •System tested across network conditions — Simulated failures, varied latencies
Experienced distributed systems engineers don't hope the network will behave—they plan for when it doesn't. The eight fallacies are the curriculum for developing this mindset. Every production incident reinforces these lessons.
Applying the Fallacies in System Design
Let's apply fallacy-aware thinking to a concrete system design scenario: designing a global e-commerce order processing system.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
# E-Commerce Order System: Fallacy-Aware Design ## System Requirements- Global user base (Americas, Europe, Asia-Pacific)- High availability (99.9% uptime target)- Order processing with payment and inventory management- Real-time inventory visibility ## Fallacy-Aware Design Decisions ### F1: Network Is UnreliableDecision: Asynchronous, event-driven order processing- Orders placed via sync API return immediately with orderId- Processing happens async via message queue- Payment confirmation sent via email/push (async)- Idempotency keys prevent duplicate charges on retry ### F2: Latency Isn't Zero Decision: Regional deployments with local processing- API servers in each region for low-latency order submission- Product catalog cached at edge (CDN)- Critical path: Order submission < 3 network calls- Parallel inventory check + payment authorization ### F3: Bandwidth Isn't InfiniteDecision: Optimized API payloads- Product list returns IDs + essential fields only (< 1KB per item)- Images served via CDN with responsive variants- Inventory updates batched, not per-item ### F4: Network Isn't SecureDecision: Zero Trust security model- mTLS between all services- JWT with short expiry for user sessions- Payment data never logged, tokenized immediately- All storage encrypted at rest ### F5: Topology ChangesDecision: Service mesh with dynamic discovery- Kubernetes + Istio for service mesh- No hardcoded IPs, all discovery via DNS- Blue-green deployments for zero-downtime updates ### F6: Multiple AdministratorsDecision: Graceful third-party degradation- Payment: Primary + fallback payment processor- Shipping: Queue orders when carrier API unavailable- Inventory: Sell with reservation, reconcile later ### F7: Transport Cost MattersDecision: Data locality optimization- Order data persisted in region of origin- Cross-region sync is eventual (acceptable for analytics)- Avoid cross-region DB reads on critical path ### F8: Network Is HeterogeneousDecision: Client-aware API design- Support HTTP/1.1 and HTTP/2- Progressive image loading for slow connections- Offline-capable mobile app with syncYou won't get every decision right initially. The fallacies provide a framework for asking the right questions. During design reviews, systematically ask: 'What if the network fails here? What if latency spikes? What if this service is unavailable?' The answers drive iterative improvement.
KEEP LEARNING
There’s more to understand.
You’re reading the introduction. Create a free account to read the complete lesson, save notes and keep track of your progress.