Computer NetworksEncapsulating Security Payload

Encapsulating Security Payload (ESP)

LevelIntermediate

Duration60 mins

TopicEncapsulating Security Payload

4 / 5

Authentication

Trust but Verify: Cryptographic Authentication

Encryption alone is not enough. A packet encrypted with the strongest cipher becomes useless—or worse, dangerous—if an attacker can modify it without detection. ESP's authentication mechanisms provide the cryptographic guarantee that packets are exactly as the sender transmitted them, originating from a legitimate source, and not replayed from previous sessions.

Authentication in ESP encompasses three related but distinct services:

Data Integrity: Ensuring packets haven't been modified in transit
Data Origin Authentication: Confirming packets came from an authorized sender
Anti-Replay Protection: Preventing reuse of previously captured packets

These services work together to ensure that even across hostile networks, receivers can trust the authenticity and freshness of every packet they process.

What You Will Learn

By the end of this page, you will understand how ESP provides authentication through HMAC algorithms and AEAD constructions, the structure and verification of the Integrity Check Value (ICV), how anti-replay protection works via sequence numbers and sliding windows, the critical relationship between encryption and authentication (and why both are necessary), and best practices for configuring authentication in production deployments.

The Purpose of Authentication

To understand why authentication is essential, consider what encryption alone provides—and what it doesn't.

What Encryption Provides:

Confidentiality: Only key holders can read the plaintext
Privacy: Content is hidden from observers

What Encryption Does NOT Provide:

Integrity: Doesn't detect if ciphertext was modified
Authenticity: Doesn't prove who encrypted the data
Freshness: Doesn't prevent reuse of old encrypted packets

The Modification Problem:

Without authentication, an attacker can modify encrypted packets in ways that produce predictable changes in the decrypted output. This is especially problematic with stream ciphers and counter modes:

Attacks Enabled Without Authentication
Attack	Mechanism	Potential Impact
Bit-Flipping Attack	Flip bits in ciphertext → predictable bit changes in plaintext (CTR, stream ciphers)	Modify financial amounts, change destinations, corrupt data
Block Reordering	Rearrange encrypted blocks (ECB mode)	Shuffle data sections, corrupt structured messages
Padding Oracle	Modify padding, observe decryption errors (CBC mode)	Decrypt without key through error analysis
Replay Attack	Resend previously captured valid packets	Duplicate transactions, redo commands, amplify traffic
Cut-and-Paste	Combine encrypted fragments from different messages	Construct malicious messages from legitimate components

Example: Bit-Flipping Attack on CTR Mode

In counter mode encryption:

Ciphertext = Plaintext ⊕ Keystream
If attacker knows bit position of value "100" (ASCII)
XOR ciphertext with "100" ⊕ "999" = flip those bits
Decryption yields "999" instead of "100"
Attack works without knowing the key!

Authentication prevents this by detecting any ciphertext modification before decryption occurs.

Data Origin Authentication:

Authentication also establishes who sent the data. In symmetric key systems, this is implicit:

Only holders of the secret key can compute valid authentication tags
If a tag verifies correctly, the sender must have had key access
Keys are only shared with legitimate communicating parties
Therefore, valid tag → legitimate sender

This logical chain provides data origin authentication as a side effect of integrity verification.

Encryption Alone is Insecure

Never use encryption without authentication. This is not a theoretical concern—practical attacks like BEAST, POODLE, and Lucky13 exploited unauthenticated encryption (or improper authentication ordering). ESP without authentication (null integrity) should never be deployed, even when encryption is enabled.

HMAC - Hash-Based Message Authentication

For non-AEAD encryption modes (like AES-CBC), ESP uses HMAC (Hash-based Message Authentication Code) to provide integrity and authentication. HMAC combines a cryptographic hash function with a secret key to produce an authentication tag that verifies both integrity and authenticity.

HMAC Construction:

HMAC uses nested hash operations with inner and outer padding:

HMAC(K, M) = H((K' ⊕ opad) || H((K' ⊕ ipad) || M))

Where:

K = Secret key
K' = Key padded/hashed to block size
M = Message to authenticate
H = Hash function (SHA-256, SHA-384, SHA-512)
ipad = 0x36 repeated (inner padding)
opad = 0x5c repeated (outer padding)

Why HMAC Instead of Simple Hash?

Using a plain hash (like SHA-256(key || message)) is vulnerable to length-extension attacks. HMAC's nested structure prevents this and provides provable security properties when the underlying hash function is secure.

HMAC Algorithms for ESP Authentication
Algorithm	Hash Function	Full Output	ICV (Truncated)	Status
HMAC-SHA-256-128	SHA-256	256 bits	128 bits (16 bytes)	Recommended
HMAC-SHA-384-192	SHA-384	384 bits	192 bits (24 bytes)	Recommended
HMAC-SHA-512-256	SHA-512	512 bits	256 bits (32 bytes)	Recommended
HMAC-SHA-1-96	SHA-1	160 bits	96 bits (12 bytes)	Deprecated
HMAC-MD5-96	MD5	128 bits	96 bits (12 bytes)	Avoid

ICV Truncation:

ESP truncates HMAC output to reduce per-packet overhead. For example, HMAC-SHA-256 produces 256 bits but ESP uses only the first 128 bits (HMAC-SHA-256-128).

Is Truncation Safe?

Yes, when done correctly. Security analysis shows that truncating to half the hash output maintains strong security bounds. An attacker would need to forge approximately 2^(n/2) tags before finding a collision, where n is the truncated length. For 128 bits, this is 2^64—computationally infeasible.

HMAC Key Derivation:

The HMAC key is derived from IKE keying material, separate from the encryption key:

HMAC-SHA-256: 256-bit key (32 bytes)
HMAC-SHA-384: 384-bit key (48 bytes)
HMAC-SHA-512: 512-bit key (64 bytes)

Using separate keys for encryption and authentication is a security requirement—never reuse the same key for both operations.

SHA-1 Deprecation

While HMAC-SHA-1 is not directly vulnerable to SHA-1's collision attacks (HMAC uses SHA-1 differently than certificate signing), best practice is to migrate away from SHA-1. The cryptographic community has deprecated SHA-1 comprehensively, and modern systems should use SHA-256 or stronger. RFC 8221 downgrades HMAC-SHA-1-96 to SHOULD NOT.

AEAD Authentication

Authenticated Encryption with Associated Data (AEAD) algorithms combine encryption and authentication in a single cryptographic operation. For ESP, AEAD eliminates the need for separate HMAC computation, improving both efficiency and security.

AEAD Concept:

AEAD algorithms take four inputs:

Key: Secret key for both encryption and authentication
Nonce: Unique value per encryption (never reuse with same key)
Plaintext: Data to encrypt
Associated Data (AD): Data to authenticate but NOT encrypt

And produce two outputs:

Ciphertext: Encrypted plaintext
Authentication Tag: Proof of integrity and authenticity

Associated Data in ESP:

The Associated Data for ESP is the ESP header (SPI + Sequence Number). This data is:

Transmitted in cleartext (needed for SA lookup)
Authenticated (any modification detected)
NOT encrypted (must be accessible before decryption)

AEAD Algorithms for ESP
Algorithm	Tag Size	Nonce Size	Key Size	Notes
AES-128-GCM	128 bits (16 bytes)	96 bits (12 bytes)	128 bits	Hardware accelerated, widely deployed
AES-256-GCM	128 bits (16 bytes)	96 bits (12 bytes)	256 bits	Best choice for high security
ChaCha20-Poly1305	128 bits (16 bytes)	96 bits (12 bytes)	256 bits	Excellent without AES-NI
AES-CCM	Variable (4-16 bytes)	56-104 bits	128/256 bits	Less common in ESP

AES-GCM Deep Dive:

AES-GCM (Galois/Counter Mode) is the dominant AEAD algorithm for ESP. It combines:

AES-CTR for encryption: Counter mode provides confidentiality
GHASH for authentication: Galois field multiplication produces authentication tag

GCM Authentication Process:

1. Encrypt plaintext using AES-CTR
2. Compute GHASH over:
   - Associated Data (ESP header)
   - Ciphertext
   - Lengths of AAD and ciphertext
3. Encrypt GHASH result with AES(Key, Nonce||0³²) to produce tag

ChaCha20-Poly1305:

For environments without AES hardware acceleration, ChaCha20-Poly1305 provides comparable security with excellent software performance:

ChaCha20: Stream cipher for encryption (20-round variant of Salsa20)
Poly1305: Authentication using polynomial evaluation
Combined: Single-pass authenticated encryption

AEAD Advantages

•Single key for both operations — simpler key management
•Single pass over data — better performance
•Atomic operation — authentication failure prevents any decryption output
•Proven security — well-analyzed constructions
•Hardware optimized — dedicated CPU instructions

Encrypt-then-MAC Disadvantages

•Two keys required — encryption + authentication
•Two passes over data — encrypt then compute HMAC
•Ordering critical — must authenticate ciphertext, not plaintext
•More overhead — separate ICV field
•Implementation risk — easier to get ordering wrong

Use AEAD Algorithms

For new deployments, always choose AEAD algorithms (AES-GCM or ChaCha20-Poly1305). They provide combined encryption+authentication with fewer configuration errors, better performance, and stronger security guarantees than separate encryption and HMAC.

Integrity Check Value (ICV) Computation

The Integrity Check Value (ICV) is the authentication tag appended to ESP packets. Whether computed via HMAC or as part of AEAD, the ICV provides the cryptographic proof of integrity and authenticity.

ICV Coverage:

The ICV is computed over specific portions of the ESP packet:

For HMAC (with CBC/CTR encryption):

ICV = HMAC-SHA-256(AuthKey, ESP_Header || IV || Ciphertext || ESP_Trailer)

For AEAD (GCM/ChaCha20-Poly1305):

(Ciphertext, Tag) = AES-GCM(Key, Nonce, Plaintext, AAD)
where AAD = ESP_Header (SPI || Sequence Number)

What's Authenticated:

ESP Header (SPI + Sequence Number)
IV (if explicit)
Encrypted payload
ESP Trailer (padding, pad length, next header)

What's NOT Authenticated:

Outer IP header (would break NAT traversal)
The ICV itself (can't authenticate its own computation)

icv_computation.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def compute_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer):
    """
    Compute ICV for ESP using HMAC-SHA-256
    
    Args:
        auth_key: 256-bit authentication key from SA
        esp_header: SPI (4 bytes) + Sequence Number (4 bytes)
        iv: Initialization Vector (algorithm dependent)
        ciphertext: Encrypted payload
        esp_trailer: Encrypted padding + pad_length + next_header
    
    Returns:
        128-bit truncated ICV
    """
    import hmac
    import hashlib
    
    # Concatenate authenticated data
    authenticated_data = esp_header + iv + ciphertext + esp_trailer
    
    # Compute HMAC-SHA-256
    full_mac = hmac.new(auth_key, authenticated_data, hashlib.sha256).digest()
    
    # Truncate to 128 bits (16 bytes) for HMAC-SHA-256-128
    icv = full_mac[:16]
    
    return icv
 
 
def verify_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer, received_icv):
    """
    Verify received ICV matches computed ICV
    
    Returns:
        True if verification passes, False otherwise
    """
    import hmac
    
    expected_icv = compute_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer)
    
    # Constant-time comparison to prevent timing attacks
    return hmac.compare_digest(expected_icv, received_icv)
 
 
def aead_authenticate(key, nonce, ciphertext, aad, received_tag):
    """
    For AEAD, authentication is integrated with decryption.
    This is conceptual - actual implementation uses crypto library.
    """
    from cryptography.hazmat.primitives.ciphers.aead import AESGCM
    
    aesgcm = AESGCM(key)
    
    try:
        # Decrypt and verify in single atomic operation
        plaintext = aesgcm.decrypt(nonce, ciphertext + received_tag, aad)
        return True, plaintext
    except Exception:
        # Authentication failed - ciphertext was tampered
        return False, None

ICV Placement:

The ICV is appended to the end of the ESP packet, after the encrypted ESP trailer:

[ESP Header][IV][Encrypted Payload + Trailer][ICV]
                                              ^^^^
                                              Unencrypted

Verification Process:

Receiver extracts SPI from ESP header
Looks up SA to find authentication algorithm and key
Computes expected ICV over received data
Compares computed ICV with received ICV
If mismatch: Discard packet silently (security requirement)
If match: Proceed to decryption (for HMAC) or return plaintext (for AEAD)

Timing Attacks

ICV comparison must use constant-time operations. If comparison short-circuits on first mismatched byte, an attacker can measure response timing to determine how many bytes match—potentially forging valid ICVs byte by byte. Always use cryptographic comparison functions (hmac.compare_digest in Python, crypto.timingSafeEqual in Node.js).

Anti-Replay Protection

Even with encryption and authentication, packets can be captured and retransmitted. Anti-replay protection ensures that each packet is processed exactly once, preventing replay attacks where legitimate encrypted packets are resent to duplicate their effect.

The Replay Threat:

Consider a banking application where an encrypted, authenticated packet transfers $1000 from Account A to Account B. Without anti-replay:

Attacker captures the legitimate transfer packet
Attacker resends the packet 100 times
Each resend transfers another $1000 (packet is valid!)
Account A is drained by replay attack

Anti-replay protection detects duplicate packets and rejects them.

ESP Sequence Number:

Every ESP packet contains a 32-bit Sequence Number that:

Starts at 1 when SA is established
Increments by 1 for each packet sent
Never wraps (SA must rekey before 2^32 packets)
Is authenticated (included in ICV computation)

Anti-Replay Mechanism Components
Component	Location	Size	Purpose
Sequence Number	ESP Header	32 bits	Monotonically increasing per-packet counter
Extended SN (ESN)	Implicit (not transmitted)	32 bits upper	Extends counter to 64 bits for high-throughput
Replay Window	Receiver state	32-8192 bits	Bitmap tracking received sequence numbers
Window Position	Receiver state	32/64 bits	Highest sequence number received

Sliding Window Mechanism:

The receiver maintains a sliding window (typically 32 or 64 packets wide) representing recently received sequence numbers:

Window Example (size = 32):

Received: 42, 43, 45, 47, 48 (44, 46 not yet received)

                    ┌──── Window Right Edge (highest received = 48)
                    ▼
  ...[16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31]
  ──────────────────────────────────────────────────────────────────
      [32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47][48]
                                          ✓   ✓   ○   ✓   ○   ✓   ✓
                                              ▲
                          Window Left Edge ───┘ (48 - 32 + 1 = 17)

✓ = Received    ○ = Not yet received (acceptable if arrives later)

Reception Rules:

Sequence < (Window Right - Window Size): REJECT (too old)
Sequence already marked received: REJECT (replay)
Sequence > Window Right: Accept, slide window, mark received
Sequence in window, not received: Accept, mark received

anti_replay_window.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
class AntiReplayWindow:
    """
    ESP Anti-Replay Window Implementation
    """
    
    def __init__(self, window_size=64):
        self.window_size = window_size
        self.window_right = 0  # Highest sequence number received
        self.bitmap = 0        # Bit i set if (window_right - i) received
    
    def check_and_update(self, sequence_number):
        """
        Check if packet should be accepted and update window.
        
        Returns:
            True if packet is acceptable, False if replay/too-old
        """
        
        # First packet initializes window
        if self.window_right == 0:
            self.window_right = sequence_number
            self.bitmap = 1
            return True
        
        # Calculate position relative to window
        diff = sequence_number - self.window_right
        
        if diff > 0:
            # New packet ahead of window
            # Slide window right
            if diff >= self.window_size:
                # Way ahead - reset bitmap
                self.bitmap = 1
            else:
                # Slide and set new bit
                self.bitmap = (self.bitmap << diff) | 1
                # Mask to window size
                self.bitmap &= (1 << self.window_size) - 1
            
            self.window_right = sequence_number
            return True
        
        elif diff == 0:
            # Duplicate of most recent packet
            return False
        
        else:
            # Packet behind window_right
            index = -diff  # Position in window (1 = one behind, etc.)
            
            if index >= self.window_size:
                # Too old - outside window
                return False
            
            # Check if already received
            mask = 1 << index
            if self.bitmap & mask:
                # Already received - replay!
                return False
            
            # Not received yet - accept and mark
            self.bitmap |= mask
            return True
    
    def get_window_status(self):
        """Debug: Show window state"""
        status = []
        for i in range(self.window_size):
            seq = self.window_right - i
            received = bool(self.bitmap & (1 << i))
            status.append(f"{seq}: {'✓' if received else '○'}")
        return status[::-1]  # Oldest to newest

Window Size Considerations

Larger windows accommodate more out-of-order delivery but consume more memory and increase the risk of accepting reordered attacks. Default of 64 works for most deployments. High-latency, high-bandwidth links may benefit from larger windows (128-1024). Consider your network's reordering characteristics when tuning.

Extended Sequence Numbers (ESN)

For high-throughput connections, 32-bit sequence numbers can exhaust too quickly. Extended Sequence Numbers (ESN) extend the sequence space to 64 bits while transmitting only 32 bits per packet.

The Exhaustion Problem:

At 10 Gbps with 1000-byte packets:

~1.25 million packets per second
2^32 packets exhausted in ~57 minutes

At 100 Gbps with 1000-byte packets:

~12.5 million packets per second
2^32 exhausted in ~5.7 minutes

Rekeying this frequently is undesirable—ESN provides the solution.

ESN Mechanism:

Sequence number is conceptually 64 bits
Only low-order 32 bits transmitted in packet
High-order 32 bits maintained implicitly at both ends
High-order bits included in ICV computation (not transmitted)

ESN vs Standard Sequence Numbers
Aspect	32-bit Sequence	64-bit ESN
Transmitted bits	32 bits	32 bits (same)
Total sequence space	2^32 (~4.3 billion)	2^64 (~18 quintillion)
Time to exhaust @10Gbps	~57 minutes	~47 million years
ICV computation	Uses 32-bit SN	Uses full 64-bit ESN
Negotiation	Default	Must be negotiated in IKE

ESN ICV Computation:

When ESN is used, the full 64-bit sequence number is included in authentication:

ICV = Auth(Key, ESP_Header(SPI || Low32_Seq) || IV || Ciphertext || Trailer || High32_Seq)

Note that the high-order 32 bits are appended to the authenticated data after the trailer for ICV computation, but they're NOT transmitted—both sender and receiver compute them independently.

Receiver ESN Handling:

The receiver must infer the correct high-order bits:

Receive packet with low-order 32 bits
Based on current window position, determine if:
- Same epoch (high bits = current high bits)
- Previous epoch (high bits = current - 1, if low bits very high)
- Next epoch (high bits = current + 1, if low bits very low)
Reconstruct full 64-bit sequence number
Verify ICV with reconstructed ESN
If verification fails with one choice, may retry with adjacent epoch

Enable ESN for High-Throughput

Any deployment expecting more than ~1 billion packets per SA lifetime should use ESN. This includes most data center, cloud, and high-bandwidth VPN deployments. ESN is negotiated during IKE Child SA creation—ensure both endpoints support and request it.

Authentication Configuration Best Practices

Proper authentication configuration is critical for ESP security. Misconfigurations can completely undermine protection, even with strong algorithms.

Algorithm Selection:

Recommended Configurations

•First choice: AES-256-GCM (AEAD) — Encryption + authentication integrated, hardware accelerated
•Alternative: ChaCha20-Poly1305-ESP — Excellent without AES hardware, mobile-friendly
•Legacy acceptable: AES-256-CBC + HMAC-SHA-512-256 — Separate encryption/authentication, more overhead
•Minimum: AES-128-GCM or AES-128-CBC + HMAC-SHA-256-128 — Still secure, lower performance margin

Configurations to Avoid

•NULL authentication (AUTH_NULL): No integrity protection—any modification undetected
•HMAC-MD5-96: MD5 is cryptographically broken
•HMAC-SHA-1-96: SHA-1 deprecated, though HMAC-SHA-1 isn't directly broken
•Custom/proprietary algorithms: Unvetted, potentially vulnerable
•Short ICVs (< 12 bytes): Increased collision risk

Anti-Replay Configuration:

Always enable anti-replay: Only disable for specific testing scenarios
Use ESN for high-throughput: Negotiate ESN if expecting > 1 billion packets
Size window appropriately: Default (64) works for most; increase for high-latency links
Configure sensible SA lifetimes: Rekey before sequence exhaustion

Operational Considerations:

Monitor authentication failures: Spikes may indicate attacks or misconfiguration
Log replay rejections: Detect replay attack attempts
Verify algorithm negotiation: Ensure actual SA uses intended algorithms
Test authentication: Intentionally corrupt packets to verify rejection

Defense in Depth

Authentication protects against network-layer attacks but doesn't replace application-layer security. Continue using TLS for web services, SSH for remote access, etc. ESP authentication defends the network transport; application security defends the application logic. Both layers providing complementary protection.

Summary and Key Takeaways

ESP authentication provides the critical guarantees that encrypted data hasn't been tampered with, comes from a legitimate source, and isn't a replay of previously captured traffic. Without authentication, encryption alone is vulnerable to numerous attacks.

Key Takeaways

•Authentication is mandatory — Encryption without authentication is vulnerable to modification, replay, and oracle attacks; never use null authentication
•AEAD algorithms are preferred — AES-GCM and ChaCha20-Poly1305 combine encryption and authentication efficiently with stronger security guarantees
•HMAC provides separate authentication — When using CBC mode encryption, HMAC-SHA-256/384/512 provides integrity checking via ICV
•Anti-replay uses sequence numbers — Monotonically increasing counters with sliding windows prevent packet replay attacks
•ESN extends sequence space — 64-bit extended sequence numbers support high-throughput connections without frequent rekeying
•Configuration matters — Strong algorithms incorrectly configured provide false security; verify actual SA parameters match intentions

What's Next:

With ESP's encryption and authentication mechanisms covered, we'll examine ESP modes of operation—transport mode versus tunnel mode. You'll learn when each mode is appropriate, how they differ in their protection boundaries, their interaction with NAT, and how to choose the correct mode for different deployment scenarios.

Page Complete

You now understand ESP authentication comprehensively—from HMAC computation to AEAD integration, ICV verification to anti-replay protection. This knowledge is essential for configuring secure VPN tunnels and diagnosing authentication-related failures. Next, we'll explore how ESP operates differently in transport and tunnel modes.

4 / 5

Loading learning content...

Computer NetworksEncapsulating Security Payload

Encapsulating Security Payload (ESP)

LevelIntermediate

Duration60 mins

TopicEncapsulating Security Payload

4 / 5

Authentication

Trust but Verify: Cryptographic Authentication

Authentication in ESP encompasses three related but distinct services:

Data Integrity: Ensuring packets haven't been modified in transit
Data Origin Authentication: Confirming packets came from an authorized sender
Anti-Replay Protection: Preventing reuse of previously captured packets

These services work together to ensure that even across hostile networks, receivers can trust the authenticity and freshness of every packet they process.

What You Will Learn

The Purpose of Authentication

To understand why authentication is essential, consider what encryption alone provides—and what it doesn't.

What Encryption Provides:

Confidentiality: Only key holders can read the plaintext
Privacy: Content is hidden from observers

What Encryption Does NOT Provide:

Integrity: Doesn't detect if ciphertext was modified
Authenticity: Doesn't prove who encrypted the data
Freshness: Doesn't prevent reuse of old encrypted packets

The Modification Problem:

Attacks Enabled Without Authentication
Attack	Mechanism	Potential Impact
Bit-Flipping Attack	Flip bits in ciphertext → predictable bit changes in plaintext (CTR, stream ciphers)	Modify financial amounts, change destinations, corrupt data
Block Reordering	Rearrange encrypted blocks (ECB mode)	Shuffle data sections, corrupt structured messages
Padding Oracle	Modify padding, observe decryption errors (CBC mode)	Decrypt without key through error analysis
Replay Attack	Resend previously captured valid packets	Duplicate transactions, redo commands, amplify traffic
Cut-and-Paste	Combine encrypted fragments from different messages	Construct malicious messages from legitimate components

Example: Bit-Flipping Attack on CTR Mode

In counter mode encryption:

Ciphertext = Plaintext ⊕ Keystream
If attacker knows bit position of value "100" (ASCII)
XOR ciphertext with "100" ⊕ "999" = flip those bits
Decryption yields "999" instead of "100"
Attack works without knowing the key!

Authentication prevents this by detecting any ciphertext modification before decryption occurs.

Data Origin Authentication:

Authentication also establishes who sent the data. In symmetric key systems, this is implicit:

Only holders of the secret key can compute valid authentication tags
If a tag verifies correctly, the sender must have had key access
Keys are only shared with legitimate communicating parties
Therefore, valid tag → legitimate sender

This logical chain provides data origin authentication as a side effect of integrity verification.

Encryption Alone is Insecure

HMAC - Hash-Based Message Authentication

HMAC Construction:

HMAC uses nested hash operations with inner and outer padding:

HMAC(K, M) = H((K' ⊕ opad) || H((K' ⊕ ipad) || M))

Where:

K = Secret key
K' = Key padded/hashed to block size
M = Message to authenticate
H = Hash function (SHA-256, SHA-384, SHA-512)
ipad = 0x36 repeated (inner padding)
opad = 0x5c repeated (outer padding)

Why HMAC Instead of Simple Hash?

HMAC Algorithms for ESP Authentication
Algorithm	Hash Function	Full Output	ICV (Truncated)	Status
HMAC-SHA-256-128	SHA-256	256 bits	128 bits (16 bytes)	Recommended
HMAC-SHA-384-192	SHA-384	384 bits	192 bits (24 bytes)	Recommended
HMAC-SHA-512-256	SHA-512	512 bits	256 bits (32 bytes)	Recommended
HMAC-SHA-1-96	SHA-1	160 bits	96 bits (12 bytes)	Deprecated
HMAC-MD5-96	MD5	128 bits	96 bits (12 bytes)	Avoid

ICV Truncation:

ESP truncates HMAC output to reduce per-packet overhead. For example, HMAC-SHA-256 produces 256 bits but ESP uses only the first 128 bits (HMAC-SHA-256-128).

Is Truncation Safe?

HMAC Key Derivation:

The HMAC key is derived from IKE keying material, separate from the encryption key:

HMAC-SHA-256: 256-bit key (32 bytes)
HMAC-SHA-384: 384-bit key (48 bytes)
HMAC-SHA-512: 512-bit key (64 bytes)

Using separate keys for encryption and authentication is a security requirement—never reuse the same key for both operations.

SHA-1 Deprecation

AEAD Authentication

AEAD Concept:

AEAD algorithms take four inputs:

Key: Secret key for both encryption and authentication
Nonce: Unique value per encryption (never reuse with same key)
Plaintext: Data to encrypt
Associated Data (AD): Data to authenticate but NOT encrypt

And produce two outputs:

Ciphertext: Encrypted plaintext
Authentication Tag: Proof of integrity and authenticity

Associated Data in ESP:

The Associated Data for ESP is the ESP header (SPI + Sequence Number). This data is:

Transmitted in cleartext (needed for SA lookup)
Authenticated (any modification detected)
NOT encrypted (must be accessible before decryption)

AEAD Algorithms for ESP
Algorithm	Tag Size	Nonce Size	Key Size	Notes
AES-128-GCM	128 bits (16 bytes)	96 bits (12 bytes)	128 bits	Hardware accelerated, widely deployed
AES-256-GCM	128 bits (16 bytes)	96 bits (12 bytes)	256 bits	Best choice for high security
ChaCha20-Poly1305	128 bits (16 bytes)	96 bits (12 bytes)	256 bits	Excellent without AES-NI
AES-CCM	Variable (4-16 bytes)	56-104 bits	128/256 bits	Less common in ESP

AES-GCM Deep Dive:

AES-GCM (Galois/Counter Mode) is the dominant AEAD algorithm for ESP. It combines:

AES-CTR for encryption: Counter mode provides confidentiality
GHASH for authentication: Galois field multiplication produces authentication tag

GCM Authentication Process:

1. Encrypt plaintext using AES-CTR
2. Compute GHASH over:
   - Associated Data (ESP header)
   - Ciphertext
   - Lengths of AAD and ciphertext
3. Encrypt GHASH result with AES(Key, Nonce||0³²) to produce tag

ChaCha20-Poly1305:

For environments without AES hardware acceleration, ChaCha20-Poly1305 provides comparable security with excellent software performance:

ChaCha20: Stream cipher for encryption (20-round variant of Salsa20)
Poly1305: Authentication using polynomial evaluation
Combined: Single-pass authenticated encryption

AEAD Advantages

•Single key for both operations — simpler key management
•Single pass over data — better performance
•Atomic operation — authentication failure prevents any decryption output
•Proven security — well-analyzed constructions
•Hardware optimized — dedicated CPU instructions

Encrypt-then-MAC Disadvantages

•Two keys required — encryption + authentication
•Two passes over data — encrypt then compute HMAC
•Ordering critical — must authenticate ciphertext, not plaintext
•More overhead — separate ICV field
•Implementation risk — easier to get ordering wrong

Use AEAD Algorithms

Integrity Check Value (ICV) Computation

ICV Coverage:

The ICV is computed over specific portions of the ESP packet:

For HMAC (with CBC/CTR encryption):

ICV = HMAC-SHA-256(AuthKey, ESP_Header || IV || Ciphertext || ESP_Trailer)

For AEAD (GCM/ChaCha20-Poly1305):

(Ciphertext, Tag) = AES-GCM(Key, Nonce, Plaintext, AAD)
where AAD = ESP_Header (SPI || Sequence Number)

What's Authenticated:

ESP Header (SPI + Sequence Number)
IV (if explicit)
Encrypted payload
ESP Trailer (padding, pad length, next header)

What's NOT Authenticated:

Outer IP header (would break NAT traversal)
The ICV itself (can't authenticate its own computation)

icv_computation.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def compute_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer):
    """
    Compute ICV for ESP using HMAC-SHA-256
    
    Args:
        auth_key: 256-bit authentication key from SA
        esp_header: SPI (4 bytes) + Sequence Number (4 bytes)
        iv: Initialization Vector (algorithm dependent)
        ciphertext: Encrypted payload
        esp_trailer: Encrypted padding + pad_length + next_header
    
    Returns:
        128-bit truncated ICV
    """
    import hmac
    import hashlib
    
    # Concatenate authenticated data
    authenticated_data = esp_header + iv + ciphertext + esp_trailer
    
    # Compute HMAC-SHA-256
    full_mac = hmac.new(auth_key, authenticated_data, hashlib.sha256).digest()
    
    # Truncate to 128 bits (16 bytes) for HMAC-SHA-256-128
    icv = full_mac[:16]
    
    return icv
 
 
def verify_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer, received_icv):
    """
    Verify received ICV matches computed ICV
    
    Returns:
        True if verification passes, False otherwise
    """
    import hmac
    
    expected_icv = compute_esp_hmac(auth_key, esp_header, iv, ciphertext, esp_trailer)
    
    # Constant-time comparison to prevent timing attacks
    return hmac.compare_digest(expected_icv, received_icv)
 
 
def aead_authenticate(key, nonce, ciphertext, aad, received_tag):
    """
    For AEAD, authentication is integrated with decryption.
    This is conceptual - actual implementation uses crypto library.
    """
    from cryptography.hazmat.primitives.ciphers.aead import AESGCM
    
    aesgcm = AESGCM(key)
    
    try:
        # Decrypt and verify in single atomic operation
        plaintext = aesgcm.decrypt(nonce, ciphertext + received_tag, aad)
        return True, plaintext
    except Exception:
        # Authentication failed - ciphertext was tampered
        return False, None

ICV Placement:

The ICV is appended to the end of the ESP packet, after the encrypted ESP trailer:

[ESP Header][IV][Encrypted Payload + Trailer][ICV]
                                              ^^^^
                                              Unencrypted

Verification Process:

Receiver extracts SPI from ESP header
Looks up SA to find authentication algorithm and key
Computes expected ICV over received data
Compares computed ICV with received ICV
If mismatch: Discard packet silently (security requirement)
If match: Proceed to decryption (for HMAC) or return plaintext (for AEAD)

Timing Attacks

Anti-Replay Protection

The Replay Threat:

Consider a banking application where an encrypted, authenticated packet transfers $1000 from Account A to Account B. Without anti-replay:

Attacker captures the legitimate transfer packet
Attacker resends the packet 100 times
Each resend transfers another $1000 (packet is valid!)
Account A is drained by replay attack

Anti-replay protection detects duplicate packets and rejects them.

ESP Sequence Number:

Every ESP packet contains a 32-bit Sequence Number that:

Starts at 1 when SA is established
Increments by 1 for each packet sent
Never wraps (SA must rekey before 2^32 packets)
Is authenticated (included in ICV computation)

Anti-Replay Mechanism Components
Component	Location	Size	Purpose
Sequence Number	ESP Header	32 bits	Monotonically increasing per-packet counter
Extended SN (ESN)	Implicit (not transmitted)	32 bits upper	Extends counter to 64 bits for high-throughput
Replay Window	Receiver state	32-8192 bits	Bitmap tracking received sequence numbers
Window Position	Receiver state	32/64 bits	Highest sequence number received

Sliding Window Mechanism:

The receiver maintains a sliding window (typically 32 or 64 packets wide) representing recently received sequence numbers:

Window Example (size = 32):

Received: 42, 43, 45, 47, 48 (44, 46 not yet received)

                    ┌──── Window Right Edge (highest received = 48)
                    ▼
  ...[16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31]
  ──────────────────────────────────────────────────────────────────
      [32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47][48]
                                          ✓   ✓   ○   ✓   ○   ✓   ✓
                                              ▲
                          Window Left Edge ───┘ (48 - 32 + 1 = 17)

✓ = Received    ○ = Not yet received (acceptable if arrives later)

Reception Rules:

Sequence < (Window Right - Window Size): REJECT (too old)
Sequence already marked received: REJECT (replay)
Sequence > Window Right: Accept, slide window, mark received
Sequence in window, not received: Accept, mark received

anti_replay_window.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
class AntiReplayWindow:
    """
    ESP Anti-Replay Window Implementation
    """
    
    def __init__(self, window_size=64):
        self.window_size = window_size
        self.window_right = 0  # Highest sequence number received
        self.bitmap = 0        # Bit i set if (window_right - i) received
    
    def check_and_update(self, sequence_number):
        """
        Check if packet should be accepted and update window.
        
        Returns:
            True if packet is acceptable, False if replay/too-old
        """
        
        # First packet initializes window
        if self.window_right == 0:
            self.window_right = sequence_number
            self.bitmap = 1
            return True
        
        # Calculate position relative to window
        diff = sequence_number - self.window_right
        
        if diff > 0:
            # New packet ahead of window
            # Slide window right
            if diff >= self.window_size:
                # Way ahead - reset bitmap
                self.bitmap = 1
            else:
                # Slide and set new bit
                self.bitmap = (self.bitmap << diff) | 1
                # Mask to window size
                self.bitmap &= (1 << self.window_size) - 1
            
            self.window_right = sequence_number
            return True
        
        elif diff == 0:
            # Duplicate of most recent packet
            return False
        
        else:
            # Packet behind window_right
            index = -diff  # Position in window (1 = one behind, etc.)
            
            if index >= self.window_size:
                # Too old - outside window
                return False
            
            # Check if already received
            mask = 1 << index
            if self.bitmap & mask:
                # Already received - replay!
                return False
            
            # Not received yet - accept and mark
            self.bitmap |= mask
            return True
    
    def get_window_status(self):
        """Debug: Show window state"""
        status = []
        for i in range(self.window_size):
            seq = self.window_right - i
            received = bool(self.bitmap & (1 << i))
            status.append(f"{seq}: {'✓' if received else '○'}")
        return status[::-1]  # Oldest to newest

Window Size Considerations

Extended Sequence Numbers (ESN)

For high-throughput connections, 32-bit sequence numbers can exhaust too quickly. Extended Sequence Numbers (ESN) extend the sequence space to 64 bits while transmitting only 32 bits per packet.

The Exhaustion Problem:

At 10 Gbps with 1000-byte packets:

~1.25 million packets per second
2^32 packets exhausted in ~57 minutes

At 100 Gbps with 1000-byte packets:

~12.5 million packets per second
2^32 exhausted in ~5.7 minutes

Rekeying this frequently is undesirable—ESN provides the solution.

ESN Mechanism:

Sequence number is conceptually 64 bits
Only low-order 32 bits transmitted in packet
High-order 32 bits maintained implicitly at both ends
High-order bits included in ICV computation (not transmitted)

ESN vs Standard Sequence Numbers
Aspect	32-bit Sequence	64-bit ESN
Transmitted bits	32 bits	32 bits (same)
Total sequence space	2^32 (~4.3 billion)	2^64 (~18 quintillion)
Time to exhaust @10Gbps	~57 minutes	~47 million years
ICV computation	Uses 32-bit SN	Uses full 64-bit ESN
Negotiation	Default	Must be negotiated in IKE

ESN ICV Computation:

When ESN is used, the full 64-bit sequence number is included in authentication:

ICV = Auth(Key, ESP_Header(SPI || Low32_Seq) || IV || Ciphertext || Trailer || High32_Seq)

Note that the high-order 32 bits are appended to the authenticated data after the trailer for ICV computation, but they're NOT transmitted—both sender and receiver compute them independently.

Receiver ESN Handling:

The receiver must infer the correct high-order bits:

Receive packet with low-order 32 bits
Based on current window position, determine if:
- Same epoch (high bits = current high bits)
- Previous epoch (high bits = current - 1, if low bits very high)
- Next epoch (high bits = current + 1, if low bits very low)
Reconstruct full 64-bit sequence number
Verify ICV with reconstructed ESN
If verification fails with one choice, may retry with adjacent epoch

Enable ESN for High-Throughput

Authentication Configuration Best Practices

Proper authentication configuration is critical for ESP security. Misconfigurations can completely undermine protection, even with strong algorithms.

Algorithm Selection:

Recommended Configurations

•First choice: AES-256-GCM (AEAD) — Encryption + authentication integrated, hardware accelerated
•Alternative: ChaCha20-Poly1305-ESP — Excellent without AES hardware, mobile-friendly
•Legacy acceptable: AES-256-CBC + HMAC-SHA-512-256 — Separate encryption/authentication, more overhead
•Minimum: AES-128-GCM or AES-128-CBC + HMAC-SHA-256-128 — Still secure, lower performance margin

Configurations to Avoid

•NULL authentication (AUTH_NULL): No integrity protection—any modification undetected
•HMAC-MD5-96: MD5 is cryptographically broken
•HMAC-SHA-1-96: SHA-1 deprecated, though HMAC-SHA-1 isn't directly broken
•Custom/proprietary algorithms: Unvetted, potentially vulnerable
•Short ICVs (< 12 bytes): Increased collision risk

Anti-Replay Configuration:

Always enable anti-replay: Only disable for specific testing scenarios
Use ESN for high-throughput: Negotiate ESN if expecting > 1 billion packets
Size window appropriately: Default (64) works for most; increase for high-latency links
Configure sensible SA lifetimes: Rekey before sequence exhaustion

Operational Considerations:

Monitor authentication failures: Spikes may indicate attacks or misconfiguration
Log replay rejections: Detect replay attack attempts
Verify algorithm negotiation: Ensure actual SA uses intended algorithms
Test authentication: Intentionally corrupt packets to verify rejection

Defense in Depth

Summary and Key Takeaways

Key Takeaways

•Authentication is mandatory — Encryption without authentication is vulnerable to modification, replay, and oracle attacks; never use null authentication
•AEAD algorithms are preferred — AES-GCM and ChaCha20-Poly1305 combine encryption and authentication efficiently with stronger security guarantees
•HMAC provides separate authentication — When using CBC mode encryption, HMAC-SHA-256/384/512 provides integrity checking via ICV
•Anti-replay uses sequence numbers — Monotonically increasing counters with sliding windows prevent packet replay attacks
•ESN extends sequence space — 64-bit extended sequence numbers support high-throughput connections without frequent rekeying
•Configuration matters — Strong algorithms incorrectly configured provide false security; verify actual SA parameters match intentions

What's Next:

Page Complete

4 / 5