HTTP Headers: The Metadata Layer

Every HTTP request and response carries a hidden payload that most users never see—HTTP headers. While the URL defines what you're requesting and the body contains the content, headers form the metadata layer that controls how the communication happens.

Headers answer questions like:

• What content formats can the client accept?
• Is this response cacheable, and for how long?
• What credentials does the client present?
• What character encoding does the content use?
• Should this content be downloaded or displayed?
• Is this request coming from a mobile or desktop browser?

Without headers, HTTP would be limited to simple document retrieval. With headers, HTTP becomes a rich, extensible protocol capable of powering the modern web—from streaming video to real-time APIs to content delivery networks operating at planetary scale.

HTTP headers follow a simple, consistent syntax defined in RFC 9110. Understanding this structure is fundamental to working with HTTP at any level.

Basic Syntax

Header-Name: header-value

Rules:

• Name: Case-insensitive (though conventionally Title-Case or lowercase)
• Colon: Immediately follows the name (no space before)
• Value: May contain spaces; leading/trailing whitespace is ignored
• Line ending: CRLF (\r )

Header Structure in Context

GET /api/users/123 HTTP/1.1\r

Host: api.example.com\r

Accept: application/json\r

Authorization: Bearer token123\r

\r

The blank line (CRLF only) separates headers from the body.

Multiple Values in Headers

Some headers accept multiple values. Two approaches exist:

1. Comma-Separated (Single Header)

Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache, no-store, must-revalidate

2. Repeated Headers (Multiple Lines)

Set-Cookie: sessionId=abc123; HttpOnly
Set-Cookie: theme=dark; Path=/

Important: Not all headers support repetition. Set-Cookie specifically requires separate headers because its values contain commas for expiration dates.

Header Size Limits

While HTTP doesn't specify maximum header sizes, practical limits exist:

Exceeding limits typically results in 431 Request Header Fields Too Large.

Request headers communicate client capabilities, preferences, and context to the server. Understanding these headers is essential for building and debugging web applications.

Host Header (Required in HTTP/1.1)

The Host header is the only required header in HTTP/1.1. It specifies the domain name of the server:

Host: api.example.com
Host: api.example.com:8080  # With non-standard port

Why required? A single IP address often hosts multiple domains (virtual hosting). Without Host, the server wouldn't know which site you're accessing.

Accept Headers (Content Negotiation)

These headers enable content negotiation—the client describing what it can accept:

Quality Values (q=): Lower values indicate lower preference:

Accept: application/json;q=1.0, text/html;q=0.9, */*;q=0.1

• Prefer JSON (q=1.0, maximum)
• HTML acceptable (q=0.9)
• Anything else as last resort (q=0.1)

User-Agent Header

Identifies the client software making the request:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Uses:

• Serve optimized content for specific browsers
• Analytics and traffic analysis
• Block known bots or scrapers

Caution: User-Agent can be easily spoofed. Don't rely on it for security decisions.

Authorization Header

Carries authentication credentials:

# Bearer Token (OAuth 2.0, JWT)
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

# Basic Authentication (Base64 encoded username:password)
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

# API Key (custom scheme)
Authorization: ApiKey sk-live-abc123

Security Note: Always use HTTPS when sending Authorization headers. Without TLS, credentials are transmitted in plain text.

Cookie Header

Sends stored cookies back to the server:

Cookie: sessionId=abc123; userId=42; theme=dark

Multiple cookies are semicolon-separated. The Cookie header provides state persistence for the stateless HTTP protocol.

Origin and Referer Headers

These headers indicate where the request came from:

Referer (yes, misspelled historically):

Referer: https://www.example.com/products/123

Full URL of the page that linked to this request. Used for analytics, logging, CSRF protection.

Origin:

Origin: https://www.example.com

Scheme + domain only (no path). Sent with cross-origin requests; fundamental for CORS.

Key Difference:

• Referer: Full URL, stripped in many privacy-focused browsers
• Origin: Domain only, always sent for cross-origin/same-origin POST

X-Request-ID / Correlation ID

X-Request-ID: 7d3f8a2b-4c5e-6f7g-8h9i-0jklmnopqrst

Client-generated unique ID for request tracking across distributed systems. Servers typically echo it back and include it in logs, enabling end-to-end tracing.

Response headers communicate server decisions, content metadata, and instructions back to the client.

Content-Type Header

The most important response header—tells the client how to interpret the body:

Content-Type: application/json; charset=utf-8
Content-Type: text/html; charset=utf-8
Content-Type: image/png
Content-Type: application/pdf
Content-Type: text/plain

Structure: type/subtype; parameter=value

Common MIME Types:

Content-Length Header

Content-Length: 15234

Exact size of the response body in bytes. Enables:

• Progress indicators for downloads
• Connection reuse (client knows when body ends)
• Validation that complete response was received

Content-Encoding Header

Content-Encoding: gzip
Content-Encoding: br
Content-Encoding: deflate

Indicates compression applied to the body. The client must decompress before use.

Brotli (br) typically achieves 20-25% better compression than gzip for text content.

Cache Control Headers

Caching headers control how responses are stored and reused:

Cache-Control (Primary):

Cache-Control: public, max-age=3600, stale-while-revalidate=86400

ETag (Validation):

ETag: "abc123def456"

Unique identifier for this specific version. Client sends back in If-None-Match to check freshness.

Last-Modified (Validation):

Last-Modified: Fri, 17 Jan 2025 18:30:00 GMT

Date of last change. Client sends back in If-Modified-Since.

Set-Cookie Header

Sends cookies to the client for storage:

Set-Cookie: sessionId=abc123; HttpOnly; Secure; SameSite=Strict; Path=/; Max-Age=86400

Vary Header

Vary: Accept-Encoding, Accept-Language

Informs caches that the response varies based on these request headers. Without Vary, a cache might serve a gzip response to a client that doesn't support gzip.

Location Header

Location: /api/users/12345
Location: https://new-domain.com/resource

Used with 3xx redirects and 201 Created to indicate where to go next.

Security headers provide defense-in-depth against common web attacks. Every production web application should implement these headers.

Strict-Transport-Security (HSTS)

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

Instructs browsers to only connect via HTTPS for the specified duration.

Attack prevented: SSL stripping (attacker downgrades HTTPS to HTTP to intercept traffic).

How it works:

1. User visits https://example.com
2. Server sends HSTS header
3. Browser remembers: "Always use HTTPS for example.com"
4. Future requests to http://example.com auto-redirect to HTTPS before any network request

Preload: Submit to the HSTS Preload List to have your domain hardcoded into browsers.

Content-Security-Policy (CSP)

Content-Security-Policy: default-src 'self'; script-src 'self' https://trusted-cdn.com; style-src 'self' 'unsafe-inline'; img-src *; connect-src 'self' https://api.example.com

CSP is the most powerful security header—it controls what resources the page can load.

Values:

• 'self' — Same origin only
• 'none' — Block completely
• https://domain.com — Specific origin
• 'unsafe-inline' — Allow inline scripts/styles (weaker security)
• 'nonce-abc123' — Allow specific inline with matching nonce

Cross-Origin Resource Sharing (CORS) headers control which external domains can access your resources. This is fundamental for modern web architectures where APIs are on different domains than frontends.

The Same-Origin Policy

Browsers enforce the Same-Origin Policy: scripts on one origin cannot read responses from another origin. Two URLs are same-origin if they share:

• Protocol (http vs https)
• Domain (api.example.com vs example.com)
• Port (80 vs 8080)

CORS headers explicitly relax this policy for trusted origins.

Key CORS Headers

Simple vs. Preflighted Requests

Simple requests (no preflight needed):

• Methods: GET, HEAD, POST
• Headers: Only Accept, Accept-Language, Content-Language, Content-Type (with limited values)
• Content-Type: only text/plain, multipart/form-data, application/x-www-form-urlencoded

Preflighted requests (browser sends OPTIONS first):

• Custom headers (Authorization, X-Custom-Header)
• Non-simple methods (PUT, DELETE, PATCH)
• Content-Type: application/json

Preflight Flow

1. Browser: "I want to PUT to api.example.com with JSON body and Authorization header"
2. Browser sends OPTIONS request first:
   OPTIONS /resource
   Origin: https://app.example.com
   Access-Control-Request-Method: PUT
   Access-Control-Request-Headers: Authorization, Content-Type

3. Server responds with permissions:
   Access-Control-Allow-Origin: https://app.example.com
   Access-Control-Allow-Methods: PUT
   Access-Control-Allow-Headers: Authorization, Content-Type
   Access-Control-Max-Age: 86400

4. Browser caches preflight result (86400 seconds)
5. Browser sends actual PUT request
6. Server processes request and includes CORS headers in response

Beyond standard headers, applications commonly define custom headers for specific needs.

X-Prefix Convention (Deprecated)

Historically, custom headers used the X- prefix:

X-Request-ID: abc123
X-Rate-Limit-Remaining: 4999

RFC 6648 deprecated this convention in 2012 because:

• Transitioning from experimental to standard is awkward
• Too many production systems use X-prefixed headers
• The prefix provides no real benefit

Current recommendation: Just name your header descriptively. If it becomes a standard, it can keep its name.

Common Custom Headers

Forwarding Headers (Proxy Context)

When requests pass through load balancers or reverse proxies, these headers preserve original request information:

X-Forwarded-For: 203.0.113.1, 198.51.100.22, 10.0.0.5
X-Forwarded-Proto: https
X-Forwarded-Host: original-domain.com
X-Forwarded-Port: 443

X-Forwarded-For chain: Each proxy appends its view of the client IP. The leftmost is (supposedly) the original client, but can be spoofed. The rightmost is the immediate upstream proxy, which is trustworthy.

Security consideration: Only trust XFF headers from known proxy IPs. Clients can forge these headers; your proxy should overwrite/append, not blindly trust.

The Forwarded Header (Standard)

RFC 7239 standardized proxy forwarding into a single header:

Forwarded: for=203.0.113.1; proto=https; host=example.com; by=10.0.0.5

This is cleaner but not as widely supported as the X-Forwarded-* family.

HTTP headers form the invisible infrastructure that enables the modern web. Let's consolidate the essential knowledge: