Chunked Transfer Encoding

Consider this scenario: a server needs to send a dynamically generated report that takes 30 seconds to compute. Under HTTP/1.0's model, the server had two options—both terrible:

1. Buffer the entire response: Wait for all 30 seconds of computation, accumulate the complete report in memory, then send it with a Content-Length header. Users stare at a blank screen.

2. Close the connection: Send data as it's generated without Content-Length, relying on connection close to signal completion. This breaks persistent connections and requires a new TCP handshake for the next request.

Chunked transfer encoding solved this dilemma elegantly. Introduced in HTTP/1.1, chunked encoding allows servers to send data in discrete chunks, each prefixed with its size, without knowing the total size upfront. The response streams to clients in real-time while the connection remains open for subsequent requests.

HTTP/1.0's response model was fundamentally based on known content length. Every response was expected to include a Content-Length header specifying the exact size of the body in bytes. This requirement created several serious problems:

Problem 1: Dynamic content generation

Web applications often generate content on-the-fly. A database query might return anywhere from 0 to millions of rows. A server-side template might produce different output sizes based on data. Computing the exact byte count required either:

• Generating the entire response body first (memory-intensive)
• Generating twice: once to count bytes, once to send (CPU-intensive)

Neither approach scales for large responses or high-traffic servers.

Problem 2: Compression compatibility

HTTP compression (gzip, deflate) transforms response bodies before transmission. The compressed size differs from the original size and isn't known until compression completes. Streaming compressed content was impossible—the server had to compress entirely, then count bytes, then send.

Problem 3: Proxies and transformations

Intermediate proxies might transform responses—adding headers, modifying content, or changing encoding. Each transformation potentially changed the content length, requiring the proxy to buffer the entire response before forwarding.

Problem 4: Real-time streaming

Some applications naturally produce unbounded streams: live video, server-sent events, real-time logs. These have no inherent "end" that would allow calculating a total length.

The HTTP/1.0 workaround:

HTTP/1.0's only escape hatch was to omit Content-Length entirely and signal "end of response" by closing the TCP connection. The client would read until the connection closed, knowing the response was complete. But this approach destroyed connection reuse:

HTTP/1.0 200 OK
Content-Type: text/html
[No Content-Length header]

<html>...dynamic content...</html>

[Server closes connection to signal end]
[Next request requires new TCP handshake]

Chunked transfer encoding introduces a simple but powerful framing format. Instead of one large body with a known length, the response is divided into chunks, each prefixed with its size in hexadecimal. A final zero-length chunk signals the end of the response.

The format:

[chunk-size in hex]\r

[chunk-data]\r

[chunk-size in hex]\r

[chunk-data]\r

...
0\r

[optional trailer headers]\r

Each chunk is self-describing: the hex size tells the client exactly how many bytes to read, followed by CRLF (\r ), then the data bytes, followed by another CRLF. The terminating chunk has size 0 and no data.

Key format details:

1. Chunk size is hexadecimal: A = 10 bytes, FF = 255 bytes, 1000 = 4096 bytes
2. Case-insensitive: a, A, 0a, 0A are all valid for 10 bytes
3. Leading zeros allowed: 0000A is valid for 10 bytes
4. Size excludes CRLF delimiters: A chunk of size 5 contains exactly 5 bytes of data, plus the surrounding CRLFs
5. No Content-Length header: The Transfer-Encoding: chunked header replaces (and is mutually exclusive with) Content-Length
6. Final chunk must be zero-sized: This is the only valid termination signal

Chunked encoding unlocked entirely new patterns for web applications. Servers could now send data as it became available, fundamentally changing the relationship between content generation and transmission.

Pattern 1: Progressive HTML rendering

Servers can send HTML in stages as it's generated:

1. Send <head> section immediately (critical CSS, preloads)
2. Send navigation and header HTML
3. Stream content sections as database queries complete
4. Send footer and closing tags last

Browsers begin rendering as chunks arrive, improving perceived performance dramatically. Users see content within milliseconds instead of waiting for full generation.

Pattern 2: Streaming API responses

Modern APIs often return large result sets. Chunked encoding enables streaming JSON arrays:

// Traditional (buffered): Wait for all results, send once
{"users": [{...}, {...}, {...}, ...(10000 items)...]}

// Streamed (chunked): Send items as they're retrieved
{"users": [
{...},   // Chunk 1: First item
{...},   // Chunk 2: Second item
{...},   // Chunk N: Nth item
]}       // Final chunk

Clients can begin processing results immediately, potentially rendering items while remaining data is still in transit.

An often-overlooked feature of chunked encoding is trailer headers—HTTP headers sent after the body content. Trailers solve a specific problem: headers whose values depend on the body content itself.

The use case:

Consider sending a large file with an integrity checksum. The checksum can only be calculated after processing the entire file. Without trailers:

• Option A: Calculate checksum first, then send (requires reading file twice or buffering)
• Option B: Send checksum in a separate request/response (complicates protocol)

With trailers, the checksum header is sent after the body, computed as data streams through.

Trailer header rules:

1. Must be announced: The Trailer header in the initial response lists which headers will appear as trailers
2. Prohibited headers: Certain headers cannot be trailers:
   • Transfer-Encoding (controls message framing)
   • Content-Length (would contradict chunked encoding)
   • Trailer (recursive)
   • Host (request header)
   • Authentication headers
3. Client support required: Not all clients process trailers correctly
4. Proxy considerations: Some proxies strip trailers when forwarding

Practical trailer applications:

Chunked encoding interacts importantly with HTTP compression. When both are used, the order of encoding matters, and the relationship can be confusing.

The encoding order:

HTTP distinguishes between two types of encoding:

1. Content-Encoding: Transforms the content itself (compression)
2. Transfer-Encoding: Transforms the message for transmission (chunking)

These encodings are applied in a specific order:

Original Content
     ↓
Content-Encoding (gzip/deflate) ← Applied first, to content
     ↓  
Transfer-Encoding (chunked) ← Applied second, to message
     ↓
Transmitted bytes

The receiver reverses the process: dechunk first, then decompress.

Streaming compression:

Chunked encoding enables streaming compression, where data is compressed incrementally rather than buffered entirely:

1. Application generates content byte-by-byte
2. Content flows through compressor (gzip/brotli)
3. Compressor emits compressed bytes as available
4. Compressed bytes are chunked for transmission
5. Client dechunks and decompresses incrementally

This pipeline minimizes memory usage at every stage—neither server nor client needs to buffer the complete uncompressed or compressed content.

The Transfer-Encoding: chunked, gzip ambiguity:

RFC 7230 technically allows stacking multiple transfer encodings:

Transfer-Encoding: gzip, chunked

This means: "First gzip the message, then chunk it." However, this creates complications:

• What if the receiver supports chunked but not gzip?
• How does this interact with Content-Encoding?
• Proxies might not understand stacked transfer encodings

In practice, servers use Content-Encoding for compression and Transfer-Encoding: chunked for framing, avoiding stacked transfer encodings entirely.

Chunked encoding is the foundation for HTTP-based streaming technologies. One of the most important is Server-Sent Events (SSE)—a standardized mechanism for servers to push events to clients over HTTP.

How SSE leverages chunked encoding:

SSE uses chunked encoding to maintain a long-lived HTTP response that never truly "ends." The server sends event data as chunks whenever events occur:

1. Client initiates request: GET /events
2. Server responds with Transfer-Encoding: chunked
3. Server sends event chunks as events occur
4. Connection remains open indefinitely
5. Client processes events as they arrive
6. If connection drops, client reconnects automatically

Other streaming patterns enabled by chunked encoding:

Implementing chunked encoding correctly requires attention to several details and edge cases.

Chunk parsing correctness:

Parsing chunked responses requires careful handling:

1. Read chunk size (hex digits until CRLF)
2. Convert hex to integer
3. Read exactly that many bytes (the chunk data)
4. Read and discard following CRLF
5. If size was 0, read trailer headers until empty line
6. Otherwise, repeat from step 1

Edge cases and error handling: