Network Failures

The network is the connective tissue of every distributed system. It's also the component most likely to behave unexpectedly in production. Unlike local function calls—which either succeed quickly or fail immediately with a clear exception—network calls exist in a probabilistic realm where packets can be delayed, duplicated, reordered, corrupted, or dropped entirely. Connections can hang indefinitely. DNS can return stale data. Routers can silently blackhole traffic.

Peter Deutsch's famous observation encapsulates this reality: "The network is reliable" is the first and perhaps most dangerous of the fallacies of distributed computing. Engineers who design systems assuming reliable networks build systems that catastrophically fail when that assumption breaks—as it inevitably does.

Network failure injection forces your system to confront the harsh reality of network behavior. By deliberately introducing the very conditions that production networks will eventually impose, you discover weaknesses before they become outages.

Latency injection adds artificial delay to network communications. This is arguably the most important network failure to test because latency is more dangerous than complete failure. A dead service fails fast—your retry logic kicks in, your circuit breaker triggers, your fallback activates. A slow service ties up resources for extended periods, consuming threads, connections, memory, and patience.

Why Latency Is Insidious:

Consider a service that normally responds in 50ms. If it dies completely, your timeout fires after (say) 5 seconds, and you fail the request or try a fallback. But if that service responds in 4.9 seconds instead—just under your timeout—you've waited 100x longer and still gotten a response. Your system spends 100x more resources on each request, and the experience for users is terrible.

Now multiply this across many concurrent requests. Threads block. Connection pools exhaust. Request queues back up. Memory fills with pending requests. The slow dependency doesn't just hurt itself—it propagates slowness upstream to every service that depends on it.

Packet loss occurs when network data fails to reach its destination. This happens constantly in real networks—router buffers overflow, cables experience interference, wireless signals degrade, network equipment fails. TCP handles most packet loss transparently through retransmission, but this has costs: increased latency, reduced throughput, and eventually, connection failures if loss is severe enough.

How Packet Loss Manifests:

At low rates (1-5%), packet loss typically manifests as increased latency because TCP retransmit timers fire and data must be resent. At moderate rates (5-20%), connections become sluggish as TCP's congestion control algorithm reduces throughput. At high rates (20%+), connections may fail entirely as retransmission limits are exceeded.

Packet loss also affects different protocols differently. TCP handles packet loss (eventually), but UDP doesn't—lost UDP packets are simply gone. Application-level protocols may have their own retry mechanisms that interact with network-level behavior in complex ways.

Observable Effects and Analysis:

Monitor these metrics during packet loss experiments:

• TCP retransmission rate: Should increase proportionally with induced loss
• Request latency distribution: Tail latency (p99, p999) will increase significantly
• Throughput: Should decrease as TCP backs off
• Connection timeouts: May increase with high loss rates
• Application error rates: Reveals whether application-level retries help or hurt

What Packet Loss Reveals:

A network partition occurs when network failures divide a system into isolated segments that cannot communicate with each other but may still be individually functional. Partitions are the most challenging network failure to handle correctly because they create situations where different parts of your system make conflicting decisions based on incomplete information.

The Fundamental Challenge:

During a partition, isolated segments cannot know whether other segments are down or simply unreachable. This ambiguity creates impossible choices:

• If a primary database becomes unreachable, should a replica promote itself to primary? If yes, you risk split-brain with two primaries accepting conflicting writes. If no, you risk unavailability while the primary might be perfectly healthy but network-isolated.

• If a service instance loses contact with its load balancer, should it continue serving requests from clients that can reach it directly? Or should it shut down to avoid inconsistent behavior?

The CAP theorem formalizes this dilemma: during partitions, you must choose between consistency (all nodes see the same data) and availability (all requests receive a response). Different systems make different tradeoffs, and partition injection reveals which tradeoff your system actually implements versus which tradeoff you think it implements.

DNS (Domain Name System) is the first step in almost every network connection—your services must resolve hostnames to IP addresses before they can connect. DNS failures are particularly dangerous because they affect all new connections while often being invisible to monitoring that tracks individual services.

The Hidden DNS Dependency:

Every time your code opens a connection to database.internal, api.partner.com, or s3.amazonaws.com, DNS resolution happens first. This resolution typically uses cached results, but caches expire. When they do, fresh DNS queries must succeed, or connections fail.

DNS failures can manifest in multiple ways:

• Complete failure: DNS queries receive no response
• Slow resolution: Queries take seconds instead of milliseconds
• Wrong answers: DNS returns stale or incorrect IP addresses
• Partial failure: Some record types fail, others succeed
• Infrastructure failure: The DNS servers themselves are unreachable

Observable Effects and Their Meaning:

Bandwidth throttling limits the rate at which data can flow between systems. While modern data centers typically have abundant bandwidth, throttling experiments are valuable for testing:

• Cross-region traffic: WAN links often have limited bandwidth compared to local networks
• Third-party API limits: Many APIs impose rate limits that effectively throttle bandwidth
• User traffic over cellular networks: Mobile users experience constrained bandwidth regularly
• Burst capacity exhaustion: Even high-bandwidth links can saturate during traffic spikes
• Cost optimization scenarios: Understanding minimum viable bandwidth helps optimize cloud networking costs

Real-world network degradation rarely involves just one type of failure. A congested network path typically exhibits increased latency, higher packet loss, and reduced effective bandwidth simultaneously. A failing network switch might partition some traffic while introducing latency to other traffic. Realistic chaos experiments combine multiple network failure types to model these compound scenarios.

Best Practices for Combined Failure Experiments:

1. Start simple, add complexity — Begin with single failure types to establish baseline behavior, then combine them.

2. Model real incidents — Base combined scenarios on past incidents you've experienced or read about in postmortems.

3. Vary intensity — Test combinations at different severity levels (e.g., 1% loss + 50ms latency vs. 10% loss + 500ms latency).

4. Test recovery — Combined failures may heal in different orders. Test how systems behave as individual components recover.

5. Monitor holistically — Combined failures may cause unexpected interactions. Monitor all system components, not just those directly affected.

6. Document precisely — Record exact parameters so experiments can be reproduced and compared over time.

While tc and iptables provide low-level control for Linux-based network failure injection, several higher-level tools offer more convenient interfaces and additional capabilities: