The Congestion Problem: Understanding Network Collapse

In October 1986, the early Internet experienced its first congestion collapse. The network between Lawrence Berkeley National Laboratory and UC Berkeley saw throughput plummet from 32 Kbps to a mere 40 bps—a devastating 99.9% reduction. Packets that should have traversed the network in milliseconds were taking minutes, and most never arrived at all.

This wasn't a hardware failure. Every router, every cable, every interface was functioning perfectly. The network had simply overwhelmed itself. Too many senders were pushing too much data too fast, and the cumulative effect was catastrophic.

This event—and the brilliant work by Van Jacobson and Michael Karels that followed—fundamentally shaped how TCP operates today. Understanding the congestion problem isn't merely academic; it's understanding why the modern Internet works at all.

Network congestion occurs when the aggregate demand for network resources—bandwidth, buffer space, processing capacity—exceeds the available supply. It is, fundamentally, a resource contention problem that emerges from the statistical nature of shared network infrastructure.

To understand congestion precisely, we must first understand what it is not:

• Congestion is not high utilization. A link running at 95% capacity with smooth, predictable traffic is efficient, not congested.
• Congestion is not packet loss. While congestion often causes packet loss, packets can be lost for many reasons (bit errors, power outages, misconfiguration), and not all packet loss implies congestion.
• Congestion is not slow connections. A low-bandwidth link serving its traffic well is not congested; congestion implies degraded performance relative to what the network should deliver.

The mathematical framing:

Consider a router with incoming traffic rate λ (lambda) and service capacity μ (mu). In queueing theory terms:

• If λ < μ: The system is stable. Packets arrive slower than they can be processed, queues remain bounded, and delays are finite.
• If λ = μ: The system is at capacity. Any burst causes temporary queuing, but the system can recover.
• If λ > μ: The system is overloaded. Queues grow unboundedly, delays increase without limit, and eventually packets must be dropped.

The critical observation is that routers have finite buffer space. When queues fill, new arriving packets are dropped—this is the router's only option. But here's where things get interesting: how senders react to those drops determines whether the network stabilizes or collapses.

Congestion manifests at specific points in the network where traffic converges and resources are constrained. Understanding these congestion points is essential for appreciating why congestion control is both necessary and challenging.

The Bottleneck Link:

In any end-to-end path, the bottleneck link is the link with the lowest available bandwidth. This is where congestion first appears and persists. Consider a path where:

• Source connects to router at 1 Gbps
• Router connects to another router at 100 Mbps (the bottleneck)
• Final router connects to destination at 1 Gbps

If the source attempts to send at 500 Mbps, the 100 Mbps link becomes overwhelmed. Packets queue at the router preceding this link, and when the queue fills, packets drop.

The Many-to-One Problem:

Congestion becomes particularly severe in many-to-one traffic patterns, where multiple senders transmit toward a single receiver. Consider a web request that triggers responses from dozens of microservices, or a distributed storage read that returns data from many storage nodes simultaneously.

In these scenarios:

1. Each sender has adequate bandwidth individually
2. The receiver link can handle each flow individually
3. But the aggregate arrival rate exceeds the receiver's ability to accept traffic
4. The converging flows compete destructively

This incast congestion pattern is especially problematic in data centers, where it can cause severe performance degradation or even TCP timeouts despite network health.

What makes congestion particularly insidious is its tendency to self-reinforce. Without proper control mechanisms, minor congestion escalates into network-wide collapse through a vicious cycle that feeds on itself.

The Deadly Spiral:

Consider what happens when a network becomes congested and TCP senders detect packet loss:

Step 1: Initial Overload

• Traffic exceeds capacity at a bottleneck
• Router buffers fill; packets are dropped
• Some packets reach their destinations, many don't

Step 2: Retransmission Storm

• Senders detect losses (timeout or duplicate ACKs)
• Senders retransmit lost packets
• These retransmissions add to the traffic load
• Network is now handling original traffic PLUS retransmissions

Step 3: Amplification

• Increased load causes more packets to be dropped
• More drops trigger more retransmissions
• Each round adds more traffic than it removes
• RTT increases, so senders have more packets "in flight"

Step 4: Collapse

• Virtually all traffic is retransmissions
• Useful throughput approaches zero
• The network is "full" but accomplishing nothing
• Recovery requires traffic to subside—which may take hours

The Mathematics of Collapse:

Let's quantify this. Suppose a network has true capacity C and current load L = 1.2C (20% overload). With no congestion control:

• 20% of packets are dropped (those exceeding capacity)
• Senders retransmit these packets, adding 0.2C more traffic
• New load: 1.2C + 0.2C = 1.4C
• Now 40% of packets drop, triggering 0.4C in retransmissions
• Load becomes 1.8C, then 2.6C, then 4.2C...

Within a few round-trip times, the network is carrying mostly retransmissions. Actual useful data—new bytes never-before-transmitted—becomes a tiny fraction of traffic. Goodput (useful throughput) collapses even as throughput (raw bytes moved) remains high.

For TCP to handle congestion effectively, it must first recognize congestion. This is complicated by the fact that packet loss—TCP's primary signal—can result from multiple causes:

Congestion-Induced Loss:

• Occurs when router buffers overflow
• Affects multiple flows traversing the congested point
• Correlated in time (bursts of drops)
• Remedied by reducing transmission rate

Non-Congestion Loss:

• Bit errors (wireless, noisy links)
• Hardware failures (transient or permanent)
• Routing changes (temporary blackholes)
• Firewall/policy drops
• Random fiber cuts, power outages

Why the distinction matters:

If TCP interprets all packet loss as congestion, it will reduce its transmission rate unnecessarily when losses are due to other causes. This is particularly problematic for wireless networks, where bit error rates can be significant but don't indicate network overload.

The Sender's Dilemma:

A TCP sender observing packet loss faces an impossible inference problem: it cannot directly observe what caused the loss. The sender only knows:

• It sent a packet
• It didn't receive an acknowledgment (either timeout or duplicate ACKs)
• The packet may have been lost, or the ACK may have been lost

Classic TCP makes a conservative assumption: treat all loss as congestion. This is safe in the sense that reducing rate never harms the network—but it causes TCP to underperform on lossy links. Modern variants and extensions (ECN, BBR, loss differentiation algorithms) attempt to distinguish congestion from random loss, but the fundamental challenge remains.

Congestion manifests in two primary symptoms: increased delay and packet loss. Understanding the relationship between these is crucial for understanding modern congestion control debates.

The Buffer Dimension:

Routers contain buffers (queues) to absorb traffic bursts. Without buffers, any momentary overload would cause immediate drops. But buffers introduce delay—packets waiting in queue aren't moving toward their destination.

Small Buffers:

• Low queuing delay
• Packets drop quickly during congestion
• Clear congestion signal to senders
• May drop packets unnecessarily during short bursts

Large Buffers:

• Can absorb longer bursts without drops
• High queuing delay during congestion
• Delayed congestion signals (sender doesn't know it's congesting)
• Bufferbloat: excessive buffering that breaks congestion feedback

The Bufferbloat Problem:

In the mid-2000s, memory became cheap enough that network equipment vendors began installing very large buffers. The intention was good: prevent packet loss. The effect was disastrous.

Consider a home router with 3 seconds of buffering at its upstream capacity:

1. User starts a large upload
2. TCP ramps up to fill the available capacity
3. Buffer fills with 3 seconds of packets
4. Interactive traffic (gaming, video calls) enters the queue
5. This interactive traffic waits 3 seconds before even starting to transmit
6. TCP sees no drops, so it doesn't slow down
7. The queue stays permanently full
8. Every packet experiences 3+ seconds of delay

The network isn't "dropping" packets—it's delaying them into uselessness. A VoIP call with 3 seconds of delay is unusable. A game with 3 seconds of latency is unplayable. Yet TCP, measuring only loss, sees nothing wrong.

Network congestion isn't just a capacity problem—it's a resource allocation problem. When demand exceeds supply, how should the scarce resource be divided among competing flows?

The Fairness Question:

Imagine a 100 Mbps link shared by 10 TCP connections. What's the "fair" allocation?

• Simple answer: 10 Mbps each
• But what if flows have different RTTs? Flows with shorter RTT can ramp up faster
• What if flows have different loss rates? Some may be penalized more heavily
• What if flows use different congestion control algorithms? Some may be more aggressive

Fairness in networking is both a technical and philosophical challenge.

The RTT Bias Problem:

Classic TCP has a significant fairness flaw: it favors flows with shorter RTT. Here's why:

• TCP increases its window by ~1 packet per RTT in congestion avoidance
• A flow with 10ms RTT gets 100 opportunities to increase per second
• A flow with 100ms RTT gets only 10 opportunities to increase per second
• Over time, the low-RTT flow captures more bandwidth

This RTT unfairness means a user close to a server (low RTT) gets better performance than a user far away (high RTT), even when both are paying for the same bandwidth. Various congestion control modifications attempt to address this bias, with varying success.

The history of TCP congestion control is a fascinating case study in engineering evolution under pressure. Understanding this history illuminates why current mechanisms work the way they do.

The Original TCP (pre-1986):

The original TCP specification (RFC 793, 1981) had no congestion control whatsoever. TCP would send packets as fast as the receiver's advertised window allowed. The implicit assumption was that network capacity would always exceed demand.

This worked fine when:

• The Internet was small (thousands of hosts)
• Traffic was mostly interactive (small bursts)
• Links were typically underutilized

But as the Internet grew, these assumptions broke down spectacularly.

Van Jacobson's Breakthrough:

In a seminal 1988 paper, Van Jacobson described the algorithms that saved the Internet. His key insights:

1. Conservation of packets: In equilibrium, a TCP connection should maintain a constant number of packets in flight. New packets enter only when old packets exit (through ACKs).

2. Equilibrium must be reached, not assumed: TCP cannot know network capacity a priori. It must probe the network by gradually increasing rate until congestion occurs.

3. Additive Increase, Multiplicative Decrease (AIMD): To be stable and fair, TCP should increase rate slowly but decrease rate rapidly when congestion is detected.

4. Self-clocking: Use ACK arrivals to pace packet transmission, naturally adapting to available bandwidth.

These principles remain the foundation of TCP congestion control today, 35+ years later.