Integrity protection is the cornerstone of the Authentication Header protocol. Every packet protected by AH carries mathematical proof that its contents haven't been altered since leaving the sender—a guarantee that enables trust in an inherently untrusted network environment.

But how does a sequence of bytes prove its own authenticity? The answer lies in the elegant application of cryptographic hash functions and message authentication codes. Understanding these mechanisms is essential for anyone implementing, configuring, or troubleshooting IPSec security.

This page explores the complete integrity protection lifecycle: from the cryptographic primitives used, through the careful handling of mutable header fields, to the precise validation procedures that determine whether a packet is accepted or rejected.

AH's integrity protection relies on Message Authentication Codes (MACs)—specifically, the HMAC (Hash-based Message Authentication Code) construction defined in RFC 2104. To understand why this choice was made, we must first understand the alternatives and their limitations.

Hash Functions Alone Are Insufficient:

A cryptographic hash function (like SHA-256) produces a fixed-size digest from arbitrary input. If the input changes, the digest changes unpredictably. This seems perfect for integrity verification:

1. Sender computes hash of packet
2. Sender appends hash to packet
3. Receiver recomputes hash and compares

The fatal flaw: anyone can compute a hash. An attacker who modifies the packet simply recomputes the hash and replaces the old one. There's no secret, so there's no authentication.

HMAC: Keyed Integrity

HMAC solves this by incorporating a secret key into the hash computation. Only parties possessing the key can compute valid HMACs. The construction is:

HMAC(K, M) = H((K ⊕ opad) || H((K ⊕ ipad) || M))

Where:

• K = Secret key (padded to hash block size)
• M = Message
• H = Hash function (SHA-256, SHA-384, etc.)
• opad = Outer padding constant (0x5c repeated)
• ipad = Inner padding constant (0x36 repeated)
• ⊕ = XOR operation
• || = Concatenation

The double-hashing with key-dependent padding provides several security properties:

HMAC in AH Context:

When AH uses HMAC-SHA-256-128:

1. The SHA-256 hash function provides cryptographic strength
2. The shared secret key (from the SA) ensures only authorized parties can authenticate
3. The "-128" indicates truncation to 128 bits (16 bytes) for the ICV

Truncation reduces overhead while maintaining adequate security margin. For HMAC-SHA-256, 128 bits of ICV provides roughly 2^64 security against collision attacks—far beyond practical attack capabilities.

One of AH's unique capabilities—and one of its operational challenges—is protection of the IP header itself. Unlike ESP, which only protects its own header and payload, AH authenticates most of the outer IP header, detecting any unauthorized modifications.

However, certain IP header fields legitimately change as packets traverse the network. AH must handle these mutable fields carefully to avoid false authentication failures.

IPv4 Header Field Treatment:

IPv6 Header Field Treatment:

IPv6's streamlined design simplifies mutable field handling:

IPv6 Extension Header Considerations:

IPv6 extension headers between the main header and AH require careful handling:

1. Hop-by-Hop Options: Fields may be mutable—zero per-option mutability flags
2. Destination Options (before routing header): Included in ICV computation
3. Routing Header: Destination address prediction required—ICV uses final destination
4. Fragment Header: AH SHOULD be applied before fragmentation (more on this later)
5. Destination Options (after AH): Not part of AH protection

The Mutable Field Algorithm:

During ICV computation, the algorithm is:

1. Create a copy of the IP header
2. For each mutable field:
   - If predictable: set to predicted final value
   - If unpredictable: set to zero
3. Compute HMAC over modified header + AH header (ICV zeroed) + payload
4. Insert resulting ICV into AH header

Receivers follow the identical process, comparing their computed ICV to the received one.

The ICV computation follows a precise sequence that ensures consistent results at sender and receiver. Let's trace through this process step by step for an IPv4 packet in transport mode.

Step 1: Prepare the IP Header

The sender creates a modified copy of the IP header for ICV computation:

Step 2: Prepare the AH Header

The AH header is constructed with the ICV field set to zero:

Step 3: Assemble and Compute

The complete input to HMAC is assembled and the ICV computed:

Step 4: Transmit

The packet is assembled with the computed ICV and transmitted:

1. Original IP header (with Protocol = 51 for AH)
2. AH header with computed ICV
3. Original payload (TCP header + data, etc.)

Note that the transmitted IP header contains the original values—not the zeroed values used for computation. The receiver will perform the same zeroing process before validating.

Computation Complexity:

For a 1500-byte packet with HMAC-SHA-256:

• SHA-256 processes data in 64-byte blocks
• 1500 bytes ≈ 24 blocks ≈ 24 × SHA-256 compression function calls
• Plus HMAC overhead: 2 additional SHA-256 operations for key mixing
• Hardware acceleration (SHA-NI, AESNI with AES-GMAC) dramatically reduces computational cost

When an AH-protected packet arrives, the receiver must validate its integrity before processing. This validation is the mirror image of computation, with additional security checks.

Validation Sequence:

Critical Validation Points:

1. SA Lookup

The triplet (destination IP, protocol 51, SPI) must map to an existing SA. If no SA exists:

• Packet is from an unknown source
• OR SA has expired
• OR SPI is forged

2. Anti-Replay Check

Before expensive cryptographic verification, check if the sequence number is:

• Within the acceptable window (not too old)
• Not already received (not a duplicate)

This check is computationally cheap and prevents replay-based denial-of-service.

3. Constant-Time Comparison

The ICV comparison MUST be constant-time—taking the same amount of time regardless of where mismatches occur. Variable-time comparison enables timing attacks where attackers measure response times to deduce correct ICV bytes.

// WRONG: Variable-time comparison
for (i = 0; i < length; i++) {
    if (received[i] != expected[i]) return false;  // Early exit leaks info!
}

// CORRECT: Constant-time comparison  
result = 0;
for (i = 0; i < length; i++) {
    result |= received[i] ^ expected[i];  // No early exit
}
return result == 0;

4. Replay Window Update

Only after ICV validation succeeds should the sequence number be marked as received. This prevents attackers from corrupting the window by sending invalid packets with specific sequence numbers.

IP fragmentation presents a significant challenge for AH integrity protection. When a packet is fragmented, the ICV computed over the original packet becomes invalid for individual fragments.

The Fragmentation Problem:

Consider a 3000-byte packet protected with AH, traversing a path with 1500-byte MTU:

1. Sender computes ICV over complete 3000-byte packet
2. Router fragments into two ~1500-byte fragments
3. Each fragment contains only part of the original payload
4. ICV is based on complete payload—fragments can't validate!

IPv4 Fragmentation Handling:

RFC 4302 specifies two approaches:

Approach 1: Transport Mode

• ICV computed over unfragmented packet
• If fragmentation occurs, receiver must reassemble before AH processing
• Fragments themselves are not individually authenticated
• Attackers could inject malicious fragments

Approach 2: Tunnel Mode

• AH protects the complete inner packet (including inner IP header)
• Outer IP packet may be fragmented
• After reassembly, AH validates the complete inner packet
• Inner packet integrity is preserved

Best Practice: Avoid Fragmentation

The recommended approach is to prevent fragmentation entirely:

1. Path MTU Discovery (PMTUD): Discover the minimum MTU along the path and size packets accordingly
2. Set DF bit: Set Don't Fragment flag and let ICMP "Fragmentation Needed" messages trigger PMTUD
3. Conservative MTU: Use conservative packet sizes (e.g., 1400 bytes) to accommodate varying paths
4. MSS Clamping: Adjust TCP MSS values to account for IPSec overhead

IPv6 Advantage:

IPv6 does not perform router fragmentation—only the source host can fragment. This simplifies AH processing:

• Source knows about AH, so sizes packets appropriately
• No unexpected fragmentation in the network
• Fragment header, if needed, is authenticated by AH

Fragment Authentication Attacks:

Without per-fragment authentication, attackers can:

• Inject malicious overlapping fragments
• Cause fragment reassembly to produce malformed packets
• Conduct fragment-based denial of service

These attacks are mitigated by fragment reassembly timeouts and anomaly detection, but avoiding fragmentation remains the robust solution.

When AH authentication fails, packets are silently discarded (with possible logging). Understanding common failure causes enables faster troubleshooting.

Failure Category 1: Configuration Mismatches

Failure Category 2: Network Issues

Failure Category 3: Implementation Bugs

Diagnostic Approach:

1. Enable IPSec debugging on both endpoints
2. Capture traffic with tcpdump/Wireshark at both ends
3. Compare captures:
   - Are IP headers identical except mutable fields?
   - Are AH headers identical?
   - Is payload byte-for-byte identical?
4. Manually compute ICV on captured packet
5. Compare manual computation with received ICV
6. Identify the divergence point

Using Wireshark for Diagnosis:

Wireshark can decode AH headers but cannot validate ICVs (it doesn't have the secret key). However, it can reveal:

• SPI values (verify they match expected SA)
• Sequence numbers (detect replay or out-of-order)
• Next Header (verify correct protocol interpretation)
• Header structure (detect malformation)

IPSec implementations maintain two databases that govern AH processing: the Security Policy Database (SPD) and the Security Association Database (SAD). Understanding their interaction clarifies the complete processing flow.

Outbound Processing (Sending):

SPD Outbound Lookup:

The SPD matches packets against policies based on:

• Source/destination IP address (ranges or specific)
• Source/destination port (for TCP/UDP)
• Protocol (TCP, UDP, ICMP, etc.)
• Direction (outbound)

Matching policy specifies:

• Action: BYPASS, DISCARD, or PROTECT
• If PROTECT: which SA (or SA parameters for dynamic creation)

Inbound Processing (Receiving):

Post-Validation SPD Check:

After successful AH validation, the SPD is consulted again to verify:

• The traffic is allowed from this source
• The protection level matches policy requirements
• The SA used is appropriate for this traffic type

This prevents "downgrade" attacks where an attacker sends unprotected traffic that should be protected.

Processing Priority:

IPSec processing must occur at the correct point in the IP stack:

• Outbound: After routing decision, before transmission
• Inbound: After IP header processing, before upper-layer delivery

For tunnel mode, this effectively happens twice (inner and outer processing).

We've thoroughly examined how AH achieves its integrity protection guarantee. Let's consolidate the essential concepts:

What's Next:

With integrity protection understood, we move to the anti-replay mechanism—how sequence numbers and sliding windows prevent attackers from capturing and retransmitting valid packets. This protection is crucial for preventing replay-based attacks on authenticated sessions.