IPSec's security services—confidentiality, integrity, authentication, and anti-replay—are implemented by two distinct security protocols: the Authentication Header (AH) and the Encapsulating Security Payload (ESP). These protocols define the exact packet formats, transformation rules, and processing requirements that turn a vulnerable IP packet into a protected one.

Understanding these protocols at the wire level is essential for:

• Configuring IPSec policies correctly
• Troubleshooting connection failures and authentication errors
• Analyzing packet captures during security audits
• Understanding the trade-offs in protocol selection
• Designing interoperable multi-vendor deployments

While modern deployments almost exclusively use ESP (and you'll understand why by the end of this page), AH remains part of the IPSec standard and appears in legacy configurations. Complete IPSec mastery requires understanding both protocols.

The Authentication Header (AH) protocol provides data origin authentication, connectionless integrity, and anti-replay protection for IP packets. AH does not provide confidentiality—packets protected by AH are authenticated but not encrypted.

IP Protocol Number: 51

Defined in: RFC 4302 (obsoletes RFC 2402)

Security Services Provided:

• ✅ Data origin authentication
• ✅ Connectionless integrity
• ✅ Anti-replay (optional but recommended)
• ❌ Confidentiality (no encryption)

Key Characteristic: AH authenticates the entire IP packet, including (most) IP header fields. This is more complete authentication than ESP provides, but creates significant practical problems (especially with NAT).

AH Header Format:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Next Header   |  Payload Len  |          RESERVED             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Security Parameters Index (SPI)                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Sequence Number Field                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                                 |
+               Integrity Check Value (ICV)                       +
|                         (variable length)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Minimum header size: 12 bytes + ICV (typically 12-32 bytes depending on algorithm)

AH Authentication Scope:

AH's defining characteristic is that it authenticates the outer IP header in addition to the payload. However, some IP header fields change in transit (TTL is decremented by routers, header checksum is recalculated, flags may change with fragmentation). To handle this:

Immutable Fields (included in ICV calculation):

• Version
• Header Length
• Total Length (before fragmentation)
• Identification
• Protocol (value will be 51 = AH)
• Source Address
• Destination Address
• Most IP options

Mutable Fields (set to zero for ICV calculation):

• DSCP/ECN (TOS byte)
• Flags
• Fragment Offset
• TTL
• Header Checksum

How it works:

1. Sender zeroes all mutable fields
2. Sender computes HMAC over entire packet (IP header + AH header with ICV=0 + payload)
3. Sender inserts computed ICV into AH header
4. Sender restores mutable fields to actual values
5. Packet is sent—mutable fields change in transit
6. Receiver zeroes mutable fields
7. Receiver computes HMAC over received packet
8. Receiver compares computed ICV with received ICV
9. Match → accept; Mismatch → discard

AH Transport Mode Packet Structure:

Original:
| IPv4 Header | TCP Header | Data |
  Proto=6

After AH (Transport Mode):
| IPv4 Header | AH Header | TCP Header | Data |
  Proto=51      Next=6      
  ←───────────── Authenticated (except mutable fields) ─────────────→

AH Tunnel Mode Packet Structure:

Original:
| IPv4 Header | TCP Header | Data |
  Src: Host A
  Dst: Host B

After AH (Tunnel Mode):
| New IPv4 Header | AH Header | Original IPv4 Header | TCP Header | Data |
  Src: Gateway A    Next=4      Src: Host A
  Dst: Gateway B    Proto=51    Dst: Host B
  Proto=51
  ←────────────────── Authenticated (except mutable fields) ──────────────────→

In tunnel mode, both the outer (new) header and the inner (original) header are authenticated. The original header is not encrypted, so while authenticated, it remains visible to observers.

The Encapsulating Security Payload (ESP) protocol provides confidentiality, data origin authentication, connectionless integrity, and anti-replay protection. ESP is the workhorse of modern IPSec deployments, providing all the security services organizations typically require.

IP Protocol Number: 50

Defined in: RFC 4303 (obsoletes RFC 2406)

Security Services Provided:

• ✅ Confidentiality (encryption)
• ✅ Data origin authentication
• ✅ Connectionless integrity
• ✅ Anti-replay (optional but recommended)
• ✅ Limited traffic flow confidentiality (in tunnel mode)

Key Characteristic: ESP encrypts the payload, providing confidentiality in addition to authentication. Unlike AH, ESP does NOT authenticate the outer IP header—this makes ESP compatible with NAT but means the IP header is not integrity-protected.

ESP Packet Format:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----
|               Security Parameters Index (SPI)                 |   ^
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                      Sequence Number                           |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ --+-- Authenticated
|                    Payload Data (variable)                     |   |  (and Encrypted
~                                                                 ~   |   starting here)
|                                                                 |   |
+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|               |     Padding (0-255 bytes)                      |   |
+-+-+-+-+-+-+-+-+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                               |  Pad Length   |  Next Header  |   v
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----
|         Integrity Check Value (ICV) (variable)                 |
~                                                                 ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

ESP Encryption and Authentication Boundaries:

Understanding exactly what is encrypted vs. authenticated is critical:

| IP Header | SPI | Seq# | Encrypted: (Payload + Padding + Pad Len + Next Header) | ICV |
            |_____|______|_________________________________________________|
                                    ↓
                            Authenticated portion
                                    
            |_____|______|__________________________________________________|
                  |
                  Not encrypted (needed for SA lookup)

AEAD Mode (AES-GCM):

With AEAD algorithms, encryption and authentication are combined:

• The IV/Nonce is derived from sequence number
• Associated Authenticated Data (AAD): SPI + Seq# (+ high-order ESN bits if applicable)
• Encrypted: Payload + Padding + Pad Length + Next Header
• ICV: Authentication tag from GCM operation

Non-AEAD Mode (AES-CBC + HMAC):

With traditional encrypt-then-MAC:

• IV is transmitted explicitly (typically 16 bytes before encrypted data)
• Payload + Padding + Pad Length + Next Header are encrypted with CBC
• HMAC is computed over: SPI + Seq# + IV + Encrypted data
• ICV contains truncated HMAC output

ESP Transport Mode Packet Structure:

Original:
| IPv4 Header | TCP Header | Data |
  Proto=6

After ESP (Transport Mode):
| IPv4 Header | ESP Hdr | Encrypted:(TCP Header + Data + Pad + PL + NH) | ICV |
  Proto=50     SPI+Seq   ←────────── Encrypted ──────────→
               ←────────────────── Authenticated ──────────────────→

ESP Tunnel Mode Packet Structure:

Original:
| IPv4 Header | TCP Header | Data |
  Src: Host A
  Dst: Host B

After ESP (Tunnel Mode):
| New IPv4 Hdr | ESP Hdr | Encrypted:(Orig IPv4 Hdr + TCP Hdr + Data + Pad + PL + NH) | ICV |
  Src: GW_A     SPI+Seq   ←──────────────────── Encrypted ────────────────────→
  Dst: GW_B
  Proto=50                ←─────────────────────── Authenticated ───────────────────────→

In tunnel mode, the original IP header is encrypted, providing traffic flow confidentiality—observers can't see the internal source/destination addresses.

Understanding the differences between AH and ESP is essential for protocol selection and troubleshooting. While ESP dominates modern deployments, the comparison illuminates important IPSec concepts.

Why ESP Dominates Modern Deployments:

Several factors have led to ESP's near-universal adoption:

1. NAT Ubiquity: NAT is everywhere—home routers, corporate firewalls, carrier-grade NAT, cloud load balancers. AH simply doesn't work through NAT. ESP with NAT-T (UDP encapsulation) traverses NAT seamlessly.

2. Confidentiality Expected: In an era of pervasive surveillance concerns, encryption is expected for any VPN. AH's authentication-only approach doesn't meet modern security requirements.

3. ESP-NULL Alternative: When only authentication is needed, ESP with NULL encryption provides the same security as AH without NAT problems.

4. AEAD Algorithms: Modern AEAD algorithms (AES-GCM) combine encryption and authentication efficiently. Using ESP with AEAD is simpler and faster than using AH for authentication plus another mechanism for encryption.

5. Header Protection Rarely Needed: The IP header authentication that AH provides is valuable in theory but rarely necessary in practice. Most attacks target payload content, not IP headers. When header tampering is a concern, other mechanisms (like signed routing in BGPsec) address it more appropriately.

Understanding the exact processing steps helps with troubleshooting and security analysis. Let's trace ESP processing for both sending and receiving.

Outbound ESP Processing (Sender):

Inbound ESP Processing (Receiver):

RFC 4301 permits using both AH and ESP together on the same packet. This provides defense-in-depth: ESP encrypts the payload, while AH authenticates the IP header that ESP cannot protect.

Combined Processing (ESP then AH):

The recommended order is to apply ESP first, then AH:

Original:
| IP Header | TCP Header | Data |

After ESP:
| IP Header | ESP Header | Encrypted Payload | ICV |

After AH (applied to ESP-protected packet):
| IP Header | AH Header | ESP Header | Encrypted Payload | ESP ICV |
  Proto=51   Next=50
  ←─────────── AH authenticates entire packet ───────────→

Why ESP First?

Applying ESP before AH means:

1. The ESP-protected payload is already encrypted when AH authenticates
2. AH protects the ESP SPI and sequence number from modification
3. AH's authentication covers the IP header that ESP doesn't protect

If AH were applied first, the ESP processing would occur after AH, meaning ESP couldn't benefit from AH's IP header protection.

Practical Considerations:

Combining AH and ESP is rare in practice due to:

1. NAT Incompatibility: AH breaks NAT, so the combination doesn't work in most real-world networks.

2. Diminishing Returns: The additional protection from AH (IP header authentication) is rarely worth the complexity and overhead.

3. Implementation Support: Not all IPSec implementations support combined AH+ESP bundles.

4. Performance Overhead: Two separate cryptographic operations per packet.

When Combined AH+ESP Might Be Used:

• Military/government networks with specific assurance requirements
• Controlled environments guaranteed free of NAT
• When IP header tampering is a specific threat model concern
• Some legacy configurations from the late 1990s/early 2000s

When analyzing packet captures (Wireshark, tcpdump), you'll encounter IPSec traffic frequently. Here's how to identify and interpret it:

Quick Identification:

Wireshark Display:

Frame 123: 154 bytes on wire
  Internet Protocol Version 4, Src: 203.0.113.1, Dst: 198.51.100.1
    Protocol: ESP (50)
  Encapsulating Security Payload
    ESP SPI: 0xc32a7b91 (3274668945)
    ESP Sequence: 15847
    ESP Payload (encrypted)

Without decryption keys, Wireshark shows only the ESP header (SPI, sequence number) and notes the payload is encrypted.

Decrypting ESP in Wireshark:

For troubleshooting, Wireshark can decrypt ESP if you provide the Security Association parameters:

1. Edit → Preferences → Protocols → ESP
2. Check 'Attempt to detect/decode encrypted ESP payloads'
3. Add ESP SA: SPI, encryption algorithm + key, integrity algorithm + key
4. Wireshark will decrypt and display the original payload

Example SA Configuration in Wireshark:

SPI: 0xc32a7b91
Encryption: AES-GCM-256
Key: 0x02e8a5f7c9b1d4e8a5f7c9b1d4e8a5f7c9b1d4e8a5f7c9b1d4e8a5f7c9b1d4e8
Authentication: (implicit in GCM)

NAT-T Detection:

NAT-Traversal encapsulates ESP in UDP. In captures:

Frame 456: 178 bytes on wire
  Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.1
  User Datagram Protocol, Src Port: 4500, Dst Port: 4500
    Non-ESP Marker: 0 (ESP encapsulated data)
  Encapsulating Security Payload
    ESP SPI: 0xd4b8e2c1
    ESP Sequence: 9821

The Non-ESP Marker (4 zero bytes) indicates ESP data rather than IKE control traffic.

We've examined the two protocols that implement IPSec security services. Let's consolidate the key insights:

What's Next:

With a solid understanding of IPSec protocols, we now examine the IPSec architecture—the security databases (SPD, SAD), Security Associations, and the control plane (IKE) that manages the entire system. This architectural foundation ties together everything we've learned about purposes, services, modes, and protocols.