Encapsulating Security Payload (ESP)

Every ESP-protected packet carries within its structure the machinery for confidentiality, integrity, and authentication. Understanding this structure is not merely academic—it's essential for troubleshooting VPN issues, analyzing packet captures, optimizing performance, and implementing security systems correctly.

The ESP packet format is elegantly designed, balancing security requirements against overhead constraints. Unlike protocols where headers simply identify fields, ESP's format defines critical boundaries: where encryption starts and ends, what gets authenticated, and how padding ensures both security and block cipher alignment.

In this page, we'll dissect the ESP packet byte by byte, understanding the purpose of every field, the constraints that shaped its design, and the practical implications for network operations.

An ESP packet consists of four main components, arranged in a specific order that enables both encryption and authentication operations:

1. ESP Header (8 bytes, unencrypted)

• Security Parameters Index (SPI) — 32 bits
• Sequence Number — 32 bits

2. Payload Data (variable, encrypted)

• Original IP packet (tunnel mode) or upper-layer protocol (transport mode)
• Initialization Vector (IV) if required by algorithm

3. ESP Trailer (variable, encrypted)

• Padding — 0-255 bytes
• Pad Length — 8 bits
• Next Header — 8 bits

4. Integrity Check Value (ICV) (variable, unencrypted)

• Computed over ESP header, payload, and trailer
• Length depends on authentication algorithm (typically 12-16 bytes)

The Critical Boundaries:

Understanding what gets encrypted versus what gets authenticated is crucial:

Encrypted Portion:

• Starts immediately after ESP header (or after IV, conceptually)
• Includes: Payload data, Padding, Pad Length, Next Header
• Does NOT include: ESP header (SPI, Sequence Number)

Authenticated Portion:

• Starts at ESP header
• Includes: SPI, Sequence Number, Payload, Padding, Pad Length, Next Header
• Does NOT include: The ICV itself (it's the result of authentication)

This means an attacker can see the ESP header (knowing which Security Association is being used and the sequence number) but cannot read or modify the payload without detection.

The ESP header contains just two fields—8 bytes total—yet these carry critical information for processing the packet.

Security Parameters Index (SPI) — 32 bits

The SPI is a 32-bit identifier that, combined with the destination IP address and protocol (ESP = 50), uniquely identifies the Security Association under which this packet should be processed.

Key Characteristics:

• Assigned by the receiving side during SA establishment
• Must be unique within the receiver's SA database for the given destination
• Values 1-255 reserved by IANA for future use
• Value 0 is reserved (indicates 'no SA')
• Typically appears as 8 hexadecimal digits (e.g., 0xCAFEBABE)

SPI Selection:

• IPSec implementations automatically assign SPIs during IKE negotiation
• Common approaches: random selection, sequential allocation, hash-based generation
• The sender learns the SPI to use when the receiver proposes it during IKE Phase 2

Sequence Number — 32 bits

The Sequence Number is a 32-bit unsigned integer that provides anti-replay protection. It starts at 1 when an SA is established and increments by 1 for each packet sent.

Critical Behavior:

• Never wraps: If the sequence number reaches 2³²-1, the SA MUST be terminated and a new SA established
• Monotonically increasing: Sender never decrements or reuses sequence numbers
• Receiver verification: Receiver maintains a sliding window to detect replays

Extended Sequence Numbers (ESN):

For high-volume connections, 2³² packets (~4.3 billion) might be exhausted too quickly. RFC 4303 defines Extended Sequence Numbers:

• 64-bit sequence number space (2⁶⁴ packets before rekey)
• Only lower 32 bits transmitted in header (bandwidth conservation)
• Upper 32 bits implicitly maintained by both ends
• Negotiated during IKE SA establishment

Many encryption algorithms require an Initialization Vector (IV) to ensure that identical plaintexts produce different ciphertexts. The IV's placement and handling in ESP depends on the encryption algorithm used.

Purpose of the IV:

Without an IV, encrypting the same plaintext with the same key always produces the same ciphertext. This creates a vulnerability—an attacker observing traffic could detect repeated messages and potentially deduce patterns or content. The IV introduces randomness into each encryption operation, ensuring unique ciphertexts even for identical plaintexts.

IV Characteristics:

• Must be unpredictable for CBC mode (random or counter-based)
• Must be unique per-packet for counter modes (CTR, GCM)
• Length depends on algorithm (typically 8-16 bytes)
• Transmitted in cleartext (but this is cryptographically safe)

*Note: AES-GCM technically uses a 12-byte nonce, but ESP implementations typically derive it from an 8-byte explicit IV concatenated with a 4-byte salt from the SA.

IV Placement in ESP:

The IV conceptually falls within the encrypted payload area, but since it's needed to begin decryption, it must be accessible before decryption. In practice:

1. IV is transmitted at the beginning of the Payload Data field
2. Receiver extracts IV before decryption
3. IV is considered part of the encrypted data for protocol purposes
4. IV is included in ICV calculation (authenticated)

Explicit vs Implicit IV:

Some algorithms allow implicit IV derivation from the sequence number and SA parameters, reducing per-packet overhead. For example:

• Implicit IV with AES-GCM-ESP: 8 bytes saved per packet
• Computation: IV = SA-derived-salt || Sequence Number
• Requires Extended Sequence Number (ESN) for 64-bit uniqueness

The Payload Data field contains the actual protected content—what we're ultimately trying to secure. Its contents differ based on whether ESP operates in transport mode or tunnel mode.

Transport Mode Payload:

In transport mode, ESP protects only the upper-layer protocol (TCP, UDP, ICMP, etc.) without encapsulating the original IP header:

[Original IP Header][ESP Header][IV][TCP/UDP/etc. Header + Data][ESP Trailer][ICV]
                                \________ Encrypted _________/

• Original IP addresses remain visible (unencrypted in IP header)
• Only the upper-layer protocol data is protected
• Used for host-to-host communication

Tunnel Mode Payload:

In tunnel mode, ESP encapsulates the entire original IP packet within a new IP packet:

[New IP Header][ESP Header][IV][Original IP Header + Complete Packet][ESP Trailer][ICV]
                               \______________ Encrypted ______________/

• Original IP addresses are hidden (encrypted inside ESP)
• Entire original packet protected, including inner IP header
• Used for site-to-site VPNs (gateway-to-gateway)

Payload Size Considerations:

The payload field has no fixed size limit imposed by ESP itself, but practical constraints exist:

1. MTU Limitations: ESP overhead reduces available space for payload
2. Fragmentation Issues: Tunnel mode creates larger packets that may require fragmentation
3. Block Cipher Alignment: Payload + padding must align to cipher block size

MTU Impact Calculation:

For a typical IPSec tunnel with AES-GCM:

Original MTU: 1500 bytes
- Outer IP header: 20 bytes
- ESP header: 8 bytes
- IV: 8 bytes
- ESP trailer (padding + pad length + next header): 2-17 bytes
- ICV: 16 bytes
─────────────────
Maximum payload: 1431-1446 bytes
(Inner IP header + original data)

This MTU reduction causes fragmentation if not accounted for—a common source of VPN performance issues.

The ESP trailer follows the payload data and contains three components critical for proper packet processing: Padding, Pad Length, and Next Header. Unlike the ESP header, the trailer is encrypted—visible only after decryption.

Padding (0-255 bytes):

Padding serves multiple purposes in ESP:

1. Block Cipher Alignment: Many encryption algorithms (AES-CBC, 3DES-CBC) operate on fixed-size blocks (16 bytes for AES). The plaintext must be padded to a multiple of the block size.

2. Traffic Flow Confidentiality (TFC): Padding can obscure the actual message length, preventing traffic analysis based on packet sizes.

3. Alignment Requirements: Some hardware implementations require 4-byte or 8-byte alignment of specific fields.

Padding Content:

RFC 4303 specifies that padding bytes should contain sequential values (1, 2, 3, ... n). This pattern serves as a verification mechanism—the receiver can check that padding follows the expected pattern after decryption.

Pad Length (1 byte):

The Pad Length field indicates how many bytes of padding precede it. After decryption, the receiver:

1. Reads the Pad Length field (last byte before Next Header)
2. Removes that many bytes from the end of the decrypted data
3. What remains is the original payload

Next Header (1 byte):

The Next Header field identifies what type of data is contained in the payload:

• In Transport Mode: The upper-layer protocol (TCP=6, UDP=17, ICMP=1, etc.)
• In Tunnel Mode: Always IP (4 for IPv4, 41 for IPv6)

This field is essential because, after decryption, the receiver must know how to process the decrypted data. It occupies the same semantic position as the IP Protocol field, enabling seamless reintegration of decrypted packets into the IP stack.

The Integrity Check Value (ICV) is the cryptographic tag that provides authentication and integrity verification. It's computed over the ESP header, payload, and trailer, and appended to the end of the packet (after the ESP trailer, unencrypted).

ICV Computation:

The ICV is calculated using a cryptographic authentication algorithm, typically either:

1. HMAC (Hash-based Message Authentication Code): Uses a hash function (SHA-256, SHA-384, SHA-512) with a secret key
2. AEAD Tag: For algorithms like AES-GCM, the authentication tag is a natural output of the encryption process

Coverage:

The ICV covers (authenticates):

• ESP header (SPI + Sequence Number)
• IV (if explicit)
• Encrypted payload
• ESP trailer (Padding + Pad Length + Next Header)

The ICV does NOT cover (and cannot cover):

• Outer IP header (would break NAT traversal)
• The ICV field itself (circular dependency)

ICV Verification Process:

When a receiver processes an ESP packet:

1. Extract SPI from ESP header
2. Look up Security Association in SA database
3. Retrieve authentication algorithm and key from SA
4. Compute expected ICV over received (ESP header + encrypted payload + trailer)
5. Compare computed ICV with received ICV
6. If mismatch: Discard packet silently (no ICMP error—security principle)
7. If match: Proceed to decryption

The 'silent discard' for failed verification is deliberate—sending an error would confirm receipt and could be exploited for oracle attacks.

AEAD Algorithms:

Modern deployments increasingly use Authenticated Encryption with Associated Data (AEAD) algorithms like AES-GCM. These combine encryption and authentication in a single cryptographic operation:

• Single key for both operations
• Better performance (one pass over data)
• Mathematically stronger binding between ciphertext and authentication tag
• Tag computation uses Associated Data (AAD) = ESP header

Let's walk through a complete ESP packet to see how all components fit together. Consider an ESP packet in tunnel mode using AES-GCM-256, protecting an ICMP echo request.

Key Observations:

1. The outer IP addresses are visible: Anyone sniffing the network sees traffic between VPN gateways, not original hosts

2. SPI and Sequence Number are visible: An observer knows which SA is being used and can count packets

3. Everything else is hidden: Inner IP addresses, protocol, and data are encrypted

4. Significant overhead: A small 84-byte ICMP packet becomes 148 bytes—76% overhead. For larger packets, the percentage decreases (1500-byte payload → ~5% overhead)

5. Padding ensures alignment: AES-GCM uses 16-byte blocks; padding brings the encrypted section to a multiple of 16

The ESP packet format is a carefully engineered structure that enables comprehensive security services while maintaining reasonable efficiency. Understanding this format is essential for anyone implementing, troubleshooting, or analyzing IPSec deployments.

What's Next:

Now that we understand ESP's packet structure, we'll dive deeper into the encryption mechanisms ESP employs. You'll learn how symmetric encryption algorithms like AES work within ESP, key derivation from SA parameters, cipher mode selection and implications, and best practices for algorithm configuration in modern deployments.