Computer NetworksEncapsulating Security Payload

Encapsulating Security Payload (ESP)

LevelIntermediate

Duration60 mins

TopicEncapsulating Security Payload

3 / 5

Encryption

The Science of Digital Secrecy

At the heart of ESP's confidentiality service lies symmetric key encryption—the mathematical machinery that transforms readable data into unintelligible ciphertext. Without the correct key, encrypted ESP payloads are computationally indistinguishable from random noise, protecting sensitive information even when packets traverse hostile networks.

But encryption in ESP is not merely about applying an algorithm—it's about understanding how cipher modes affect security, why key management matters more than algorithm choice, how to balance performance against protection strength, and what can go wrong when encryption is misconfigured.

This page explores ESP encryption from cryptographic foundations to practical implementation, equipping you with the knowledge to configure, evaluate, and troubleshoot encrypted VPN tunnels.

What You Will Learn

By the end of this page, you will understand symmetric encryption fundamentals and how they apply to ESP, the major cipher algorithms used in modern IPSec (AES, ChaCha20), block cipher modes and their security implications (CBC, CTR, GCM), key derivation from IKE-negotiated material, algorithm selection criteria for different deployment scenarios, and common encryption misconfigurations that compromise security.

Symmetric Encryption Fundamentals

ESP uses symmetric key encryption, where the same secret key is used for both encryption (at the sender) and decryption (at the receiver). This contrasts with asymmetric (public-key) encryption where different keys are used for each operation.

Why Symmetric Encryption for ESP?

Symmetric ciphers offer significant advantages for bulk data encryption:

Performance: Symmetric algorithms are orders of magnitude faster than asymmetric algorithms—critical when encrypting every packet at line rate
Efficiency: Key sizes are smaller (256 bits vs. 2048+ bits for equivalent security), reducing computational overhead
Streaming Capability: Symmetric ciphers can encrypt data incrementally without buffering entire messages
Hardware Acceleration: Modern CPUs include dedicated instructions (AES-NI) that accelerate symmetric encryption to near-native speeds

The Key Exchange Problem:

Symmetric encryption's challenge is key distribution—both parties must share the secret key securely before communication begins. IPSec solves this through IKE (Internet Key Exchange), which uses asymmetric cryptography to negotiate and derive symmetric keys securely over untrusted networks.

Symmetric vs. Asymmetric Encryption Trade-offs
Characteristic	Symmetric Encryption	Asymmetric Encryption
Key Relationship	Same key for encrypt/decrypt	Different keys (public/private)
Speed	Very fast (hardware accelerated)	Slow (100-1000x slower)
Key Size for 128-bit security	128 bits (AES-128)	3072 bits (RSA)
Use Case	Bulk data encryption	Key exchange, signatures
Key Distribution	Requires secure channel	Public key can be shared openly
ESP Usage	All payload encryption	IKE key negotiation only

Block Ciphers vs. Stream Ciphers:

Symmetric ciphers fall into two categories:

Block Ciphers: Encrypt fixed-size blocks of data (e.g., 128 bits for AES)

AES, 3DES, Blowfish
Require mode of operation to handle arbitrary-length data
Most ESP implementations use block ciphers

Stream Ciphers: Encrypt data bit-by-bit or byte-by-byte

ChaCha20, RC4 (deprecated)
Generate keystream that's XORed with plaintext
Simpler but require careful nonce management

In practice, block ciphers in counter mode (AES-CTR, AES-GCM) behave like stream ciphers, combining block cipher security with stream cipher flexibility.

The Hybrid Approach

Modern cryptographic systems like IPSec use a hybrid approach: asymmetric cryptography (RSA, ECDH) for key exchange and authentication during IKE, and symmetric cryptography (AES) for bulk data encryption during ESP. This combines the security properties of both approaches while mitigating their weaknesses.

ESP Cipher Algorithms

ESP supports multiple encryption algorithms, allowing deployments to select ciphers based on security requirements, performance constraints, and regulatory compliance. The choice of algorithm is negotiated during IKE Phase 2 (or IKEv2 CREATE_CHILD_SA).

The AES Family (Advanced Encryption Standard):

AES is the dominant cipher in ESP deployments, standardized by NIST after an open competition that evaluated security, performance, and implementation characteristics.

Key AES Characteristics:

Block size: 128 bits (16 bytes)
Key sizes: 128, 192, or 256 bits
Structure: Substitution-permutation network (SPN)
Rounds: 10 (AES-128), 12 (AES-192), 14 (AES-256)
Status: No practical attacks; considered secure for foreseeable future

AES alone encrypts single 128-bit blocks. For ESP (which handles variable-length packets), AES must be used with a mode of operation that extends it to arbitrary-length messages.

ESP Encryption Algorithms
Algorithm	Type	Key Size	Status	Notes
AES-GCM	AEAD	128/192/256 bits	Recommended	Best choice; hardware acceleration
AES-CBC	Block + HMAC	128/192/256 bits	Acceptable	Requires separate integrity; legacy
AES-CTR	Stream-like	128/192/256 bits	Acceptable	Requires separate integrity
ChaCha20-Poly1305	AEAD	256 bits	Recommended	Excellent without AES-NI
3DES-CBC	Block + HMAC	168 bits (112 effective)	Deprecated	Sweet32 attack; avoid
ENCR_NULL	None	N/A	Never use	No encryption; authentication only

ChaCha20-Poly1305:

ChaCha20-Poly1305 is a modern AEAD cipher gaining adoption in ESP (RFC 7634). Designed by Daniel Bernstein, it offers:

Excellent software performance: As fast as AES-GCM even without hardware acceleration
Mobile/IoT suitability: Great for devices without AES-NI
Resistance to timing attacks: Constant-time implementation is straightforward
256-bit key only: Single key size simplifies configuration

Deprecated Algorithms:

Several legacy algorithms should be avoided:

3DES: Susceptible to Sweet32 attacks; 64-bit block size leads to birthday-bound attacks at ~32GB of traffic
DES: 56-bit key; trivially breakable with modern hardware
Blowfish: 64-bit block; similar birthday-bound concerns
RC4: Multiple attacks; completely broken

Algorithm Upgrade Priority

If your IPSec deployment uses 3DES, prioritize migration to AES-GCM. The Sweet32 attack makes 3DES unsafe for connections that transfer more than ~32GB of data before rekeying. Modern systems have no reason to use 3DES except for legacy interoperability—and even then, aggressive SA lifetime limits are mandatory.

Block Cipher Modes of Operation

Block ciphers like AES encrypt fixed-size blocks (128 bits). To encrypt arbitrary-length ESP payloads, we need modes of operation that extend the block cipher to handle messages of any length. The mode choice significantly impacts security, performance, and implementation complexity.

Cipher Block Chaining (CBC) Mode:

CBC was the traditional mode for ESP encryption (AES-CBC, 3DES-CBC).

Operation:

XOR first plaintext block with Initialization Vector (IV)
Encrypt the XOR result
XOR second plaintext block with first ciphertext block
Encrypt and repeat...

Characteristics:

Each ciphertext block depends on all previous blocks (chaining)
Requires random, unpredictable IV per packet
Encryption is sequential (can't parallelize)
Decryption can be parallelized
Requires padding to block boundary

Comparison of ESP Cipher Modes
Mode	Type	Parallelizable	IV Requirement	Padding	Authentication
CBC	Block	Decrypt only	Random, unpredictable	Required	Separate HMAC needed
CTR	Stream-like	Yes (enc & dec)	Unique (never reuse)	Not required	Separate HMAC needed
GCM	AEAD	Yes (enc & dec)	Unique (never reuse)	Not required	Built-in
CCM	AEAD	Encrypt only	Unique (never reuse)	Not required	Built-in

Counter (CTR) Mode:

CTR mode transforms a block cipher into a stream cipher by encrypting sequential counter values and XORing the result with plaintext.

Operation:

Concatenate nonce with counter (starting at 0 or 1)
Encrypt nonce||counter to produce keystream block
XOR keystream with plaintext block
Increment counter, repeat

Advantages:

Fully parallelizable (both encryption and decryption)
No padding needed (encrypt exactly the message length)
Pre-compute keystream before plaintext arrives

Critical Requirement: The nonce+counter combination must NEVER repeat with the same key. Reuse reveals XOR of two plaintexts—catastrophic security failure.

Galois/Counter Mode (GCM):

GCM combines CTR mode encryption with Galois field authentication, providing Authenticated Encryption with Associated Data (AEAD) in a single operation.

Components:

CTR encryption: Provides confidentiality (as described above)
GHASH authentication: Galois field multiplication for integrity
Combined output: Ciphertext + authentication tag

Why GCM Dominates Modern ESP:

Single pass: Encrypt and authenticate simultaneously
Hardware acceleration: AES-NI includes GCM-specific instructions (PCLMULQDQ)
Strong security proofs: Well-analyzed construction
Efficiency: Minimal overhead beyond raw encryption

Choose AES-GCM

For new ESP deployments, AES-GCM-256 should be the default choice. It provides encryption and authentication in one efficient operation, benefits from widespread hardware acceleration, and is recommended by virtually all modern security guidelines (NIST, NSA CNSA, RFC 8221). Use AES-GCM-128 only when interoperability requires it.

Key Derivation and Management

ESP encryption keys are not configured directly—they're derived from keying material established during IKE negotiation. Understanding this derivation process is essential for troubleshooting key-related issues and ensuring adequate security.

The IKE Key Hierarchy:

IKE (Internet Key Exchange) establishes a hierarchy of keys:

SKEYSEED: Initial keying material derived from Diffie-Hellman exchange
SK_d: Key for deriving child SA keys
SK_ai/SK_ar: Authentication keys for IKE messages
SK_ei/SK_er: Encryption keys for IKE messages
ESP Keys: Derived from SK_d for each Child SA

ESP Key Derivation:

When an ESP SA is created, the following keys are derived:

Encryption Key: Length depends on algorithm (16/24/32 bytes for AES-128/192/256)
Authentication Key (if not AEAD): Length depends on HMAC algorithm
Salt (for some algorithms): Fixed value mixed with IV

Key derivation uses a PRF (Pseudo-Random Function) to expand SK_d into the required key material:

KEYMAT = PRF+(SK_d, Ni | Nr | SPIi | SPIr)

ESP Key Material Requirements
Algorithm	Encryption Key	Auth Key	Salt	Total KEYMAT
AES-128-CBC + HMAC-SHA-256	16 bytes	32 bytes	—	48 bytes per direction
AES-256-CBC + HMAC-SHA-256	32 bytes	32 bytes	—	64 bytes per direction
AES-128-GCM	16 bytes	— (integrated)	4 bytes	20 bytes per direction
AES-256-GCM	32 bytes	— (integrated)	4 bytes	36 bytes per direction
ChaCha20-Poly1305	32 bytes	— (integrated)	4 bytes	36 bytes per direction

Key Separation:

ESP uses separate keys for each direction of communication. The initiator and responder derive different keys for:

Initiator → Responder encryption
Responder → Initiator encryption
Initiator → Responder authentication (if applicable)
Responder → Initiator authentication (if applicable)

This separation ensures that compromising one direction's keys doesn't immediately compromise the other direction.

Perfect Forward Secrecy (PFS):

With PFS enabled, each Child SA performs a new Diffie-Hellman exchange to derive fresh keying material. Without PFS, all Child SA keys derive from the original IKE SA's DH exchange.

With PFS:

Compromise of one SA's keys doesn't reveal other SAs' keys
Higher security but additional computational overhead
Recommended for high-security deployments

Without PFS:

Slight performance improvement
If IKE SA keys are compromised, all Child SA keys are derivable
Acceptable for many deployments with strong IKE SA protection

Rekeying is Critical

ESP keys have limited safe usage. Cryptographic best practice limits the amount of data encrypted under a single key. Configure SA lifetimes (time-based and byte-based) to trigger rekeying before cryptographic limits are reached. For AES-GCM, rekey before 2^32 packets per SA (sequence number exhaustion is the practical limit with ESN).

Encryption Process in Detail

When an ESP endpoint encrypts a packet, it executes a precise sequence of operations. Understanding this process helps diagnose encryption failures and optimize implementations.

ESP Encryption Workflow (AES-GCM):

Step 1: Prepare Payload

Assemble data to encrypt: Original packet (tunnel) or upper-layer data (transport)
Append ESP trailer: Add padding, pad length, next header

Step 2: Construct Nonce

Combine salt (from SA, 4 bytes) with explicit IV (8 bytes, typically sequence number or counter)
Result: 12-byte nonce for GCM

Step 3: Assemble Associated Data (AAD)

AAD = ESP header (SPI + Sequence Number)
This data is authenticated but not encrypted

Step 4: Encrypt and Authenticate

Input to AES-GCM: Key, Nonce, Plaintext, AAD
Output: Ciphertext, Authentication Tag (ICV)

Step 5: Construct ESP Packet

Concatenate: ESP Header + IV + Ciphertext + ICV
Prepend outer IP header (tunnel mode) or original IP header (transport mode)

esp_encryption_pseudocode.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def encrypt_esp_packet(plaintext, sa):
    """
    ESP Encryption Process (AES-GCM)
    
    Args:
        plaintext: Original packet data to protect
        sa: Security Association with keys and parameters
    
    Returns:
        Complete ESP packet ready for transmission
    """
    
    # Step 1: Prepare payload with ESP trailer
    # Padding for alignment (AES-GCM doesn't strictly require, but helps)
    pad_length = calculate_padding(len(plaintext))
    padding = bytes(range(1, pad_length + 1))  # 0x01, 0x02, ...
    
    payload = plaintext + padding + bytes([pad_length, sa.next_header])
    
    # Step 2: Generate unique IV
    # Typically derived from sequence number for implicit IV
    # Or random for explicit IV modes
    sa.sequence_number += 1
    iv = sa.sequence_number.to_bytes(8, 'big')
    
    # Step 3: Construct nonce for AES-GCM
    # nonce = salt (4 bytes from SA) || IV (8 bytes)
    nonce = sa.salt + iv
    
    # Step 4: Assemble ESP header (also serves as AAD)
    esp_header = sa.spi.to_bytes(4, 'big') + sa.sequence_number.to_bytes(4, 'big')
    aad = esp_header  # Authenticated but not encrypted
    
    # Step 5: Encrypt and authenticate
    ciphertext, tag = aes_gcm_encrypt(
        key=sa.encryption_key,
        nonce=nonce,
        plaintext=payload,
        associated_data=aad
    )
    
    # Step 6: Construct final ESP packet
    esp_packet = esp_header + iv + ciphertext + tag
    
    return esp_packet
 
 
def calculate_padding(data_length):
    """
    Calculate padding needed for block alignment.
    For AES-GCM, minimal padding (at least 0 bytes).
    For AES-CBC, must align to 16-byte boundary.
    """
    # Including 2 bytes for pad_length and next_header
    trailer_fixed = 2
    total = data_length + trailer_fixed
    
    # Align to 4-byte boundary (common requirement)
    if total % 4 != 0:
        return 4 - (total % 4)
    return 0

Decryption Workflow:

The receiver performs the inverse process:

Parse ESP Header: Extract SPI and Sequence Number
Lookup SA: Find matching SA in SAD using SPI + destination IP
Verify Sequence Number: Check against anti-replay window
Reconstruct Nonce: Combine SA salt with received IV
Verify and Decrypt: AES-GCM authenticates then decrypts
- If authentication fails: Discard silently
- If authentication passes: Obtain plaintext
Remove Trailer: Read pad length, strip padding
Process Payload: Pass to upper layer (transport) or IP stack (tunnel)

Authentication Before Decryption

AEAD algorithms like AES-GCM verify authentication BEFORE revealing decrypted content. This prevents processing of tampered ciphertext—a significant security advantage over encrypt-then-MAC with separate operations. The implementation ensures atomic verification: either you get valid plaintext, or you get nothing.

Performance Optimization

ESP encryption must operate at line rate—potentially processing millions of packets per second without becoming a bottleneck. Modern implementations achieve this through hardware acceleration, parallelization, and careful algorithm selection.

Hardware Acceleration:

AES-NI (AES New Instructions): Intel and AMD processors include dedicated instructions for AES operations:

AESENC, AESDEC: Single-round encryption/decryption
AESKEYGENASSIST: Key expansion assistance
PCLMULQDQ: Carry-less multiplication (for GCM's GHASH)

Performance Impact:

AES-GCM with AES-NI: 3-10 Gbps per core (varies by processor)
AES-GCM without AES-NI: 100-500 Mbps per core
10-30x improvement with hardware acceleration

ESP Encryption Performance (Single Core, Modern CPU)
Algorithm	With AES-NI	Without AES-NI	Relative
AES-128-GCM	~6 Gbps	~300 Mbps	20x faster with HW
AES-256-GCM	~4.5 Gbps	~250 Mbps	18x faster with HW
AES-256-CBC + HMAC-SHA-256	~3 Gbps	~200 Mbps	15x faster with HW
ChaCha20-Poly1305	~3 Gbps	~2.5 Gbps	SW implementation is fast
3DES-CBC + HMAC-SHA-1	~400 Mbps	~100 Mbps	4x, but still slow

Parallelization Strategies:

Multi-core Distribution:

Distribute SA workload across multiple cores
Hash incoming packets to cores based on SA (SPI-based hashing)
Each core maintains subset of SA state
Scales linearly with core count (up to practical limits)

Within-packet Parallelization:

AES-GCM and AES-CTR allow parallel block encryption
Pipelining: Process multiple blocks simultaneously
SIMD: Wide vector instructions encrypt 4-8 blocks at once

Dedicated Hardware:

Network Interface Cards with crypto offload
FPGA-based encryption engines
Dedicated security processors (QuickAssist, NITROX)
10-100+ Gbps throughput in dedicated silicon

Software Optimization:

Avoid memory allocation in fast path
Minimize cache misses (hot SA data structures)
Use zero-copy techniques where possible
Batch small packets for processing efficiency

Algorithm Selection for Performance

When hardware AES-NI is available, choose AES-GCM unquestioningly. When it's not (embedded devices, older processors), ChaCha20-Poly1305 often outperforms software AES significantly. Always benchmark your specific hardware—never assume performance characteristics.

Common Encryption Misconfigurations

Even strong algorithms can be rendered useless by configuration errors. Recognizing common misconfigurations helps ensure that encryption provides its intended protection.

1. Null Encryption (ENCR_NULL)

Some configurations accidentally or intentionally use null encryption, providing no confidentiality:

Symptom: Packets authenticated but payload visible in captures
Cause: Misconfigured proposal, legacy compatibility settings
Fix: Explicitly require non-null encryption in proposals

2. Weak Algorithm Selection

Using deprecated algorithms that provide inadequate security:

Examples: 3DES, DES, Blowfish, RC4
Symptom: Passes interoperability tests but vulnerable to attacks
Fix: Configure only modern algorithms (AES-GCM, ChaCha20-Poly1305)

Critical Encryption Errors

•IV/Nonce Reuse: Catastrophic for CTR/GCM modes—reveals XOR of plaintexts. Cause: Poor random number generation or counter bugs. Detection: Review implementation; test with packet replay.
•Key Reuse Across SAs: Using the same keys for multiple SAs violates security assumptions. Cause: Manual keying with copied keys. Fix: Always use IKE for automatic unique key derivation.
•Inadequate Key Length: AES-128 is secure but AES-256 recommended for long-term security. Cause: Default configurations. Fix: Explicitly specify AES-256.
•No Rekeying: SAs never rekeyed, accumulating risk over time. Cause: No lifetime configured, IKE issues prevent rekey. Fix: Configure conservative lifetimes (1 hour or less).
•Encryption Without Authentication: Using null authentication with encryption. Cause: Misunderstanding that encryption implies integrity. Fix: Always pair encryption with authentication (or use AEAD).

3. Mismatched Algorithm Proposals

When initiator and responder support different algorithms, negotiation fails or falls back to weak algorithms:

Symptom: SA establishment fails or uses weaker-than-expected cipher
Cause: Asymmetric configuration between endpoints
Fix: Ensure matching proposals on both ends; prioritize strong algorithms

4. Missing Perfect Forward Secrecy

Without PFS, compromise of IKE SA enables decryption of all past traffic:

Symptom: Configuration appears correct but lacks resilience
Cause: PFS not enabled in Child SA creation
Fix: Enable PFS (DH group negotiation) for Child SAs

5. Improper Random Number Generation

Weak random numbers compromise IV generation and key derivation:

Symptom: Patterns in IVs, predictable keys
Cause: Exhausted entropy, poor PRNG seeding
Fix: Ensure strong entropy sources; use cryptographic PRNGs

Test Your Configuration

Never assume encryption is working correctly based on 'VPN is up'. Verify: (1) Capture traffic and confirm it's encrypted (not null cipher), (2) Check negotiated algorithms match expectations, (3) Verify rekeying occurs as configured, (4) Test that tampering is detected (modify packets, confirm rejection).

Summary and Key Takeaways

ESP encryption transforms readable data into protected ciphertext, enabling confidential communication across untrusted networks. Proper understanding of encryption mechanisms ensures secure, performant deployments.

Key Takeaways

•ESP uses symmetric encryption for performance—keys derived from IKE, different for each direction
•AES-GCM is the modern standard — AEAD algorithm providing encryption + authentication efficiently with hardware acceleration
•Block cipher modes matter — CBC requires separate authentication; GCM provides integrated authentication; CTR needs careful nonce management
•Key derivation follows IKE hierarchy — ESP keys derived from IKE SA keying material using PRF; PFS adds additional DH exchanges
•Hardware acceleration is critical — AES-NI provides 10-30x performance improvement; ChaCha20-Poly1305 excels without hardware support
•Misconfigurations defeat encryption — Null encryption, weak algorithms, IV reuse, missing authentication can render encryption useless

What's Next:

With encryption covered, we'll examine ESP authentication mechanisms in detail. You'll learn how HMAC and AEAD authentication work, the importance of the Integrity Check Value, the relationship between encryption and authentication, and best practices for configuring authentication in ESP deployments.

Page Complete

You now understand ESP encryption comprehensively—from symmetric cipher fundamentals to algorithm selection, modes of operation, key derivation, performance optimization, and common pitfalls. This knowledge enables you to configure encrypted VPN tunnels correctly and troubleshoot encryption-related issues. Next, we'll explore how ESP ensures data integrity and authenticity through its authentication mechanisms.

3 / 5

Loading learning content...

Computer NetworksEncapsulating Security Payload

Encapsulating Security Payload (ESP)

LevelIntermediate

Duration60 mins

TopicEncapsulating Security Payload

3 / 5

Encryption

The Science of Digital Secrecy

This page explores ESP encryption from cryptographic foundations to practical implementation, equipping you with the knowledge to configure, evaluate, and troubleshoot encrypted VPN tunnels.

What You Will Learn

Symmetric Encryption Fundamentals

Why Symmetric Encryption for ESP?

Symmetric ciphers offer significant advantages for bulk data encryption:

Performance: Symmetric algorithms are orders of magnitude faster than asymmetric algorithms—critical when encrypting every packet at line rate
Efficiency: Key sizes are smaller (256 bits vs. 2048+ bits for equivalent security), reducing computational overhead
Streaming Capability: Symmetric ciphers can encrypt data incrementally without buffering entire messages
Hardware Acceleration: Modern CPUs include dedicated instructions (AES-NI) that accelerate symmetric encryption to near-native speeds

The Key Exchange Problem:

Symmetric vs. Asymmetric Encryption Trade-offs
Characteristic	Symmetric Encryption	Asymmetric Encryption
Key Relationship	Same key for encrypt/decrypt	Different keys (public/private)
Speed	Very fast (hardware accelerated)	Slow (100-1000x slower)
Key Size for 128-bit security	128 bits (AES-128)	3072 bits (RSA)
Use Case	Bulk data encryption	Key exchange, signatures
Key Distribution	Requires secure channel	Public key can be shared openly
ESP Usage	All payload encryption	IKE key negotiation only

Block Ciphers vs. Stream Ciphers:

Symmetric ciphers fall into two categories:

Block Ciphers: Encrypt fixed-size blocks of data (e.g., 128 bits for AES)

AES, 3DES, Blowfish
Require mode of operation to handle arbitrary-length data
Most ESP implementations use block ciphers

Stream Ciphers: Encrypt data bit-by-bit or byte-by-byte

ChaCha20, RC4 (deprecated)
Generate keystream that's XORed with plaintext
Simpler but require careful nonce management

In practice, block ciphers in counter mode (AES-CTR, AES-GCM) behave like stream ciphers, combining block cipher security with stream cipher flexibility.

The Hybrid Approach

ESP Cipher Algorithms

The AES Family (Advanced Encryption Standard):

AES is the dominant cipher in ESP deployments, standardized by NIST after an open competition that evaluated security, performance, and implementation characteristics.

Key AES Characteristics:

Block size: 128 bits (16 bytes)
Key sizes: 128, 192, or 256 bits
Structure: Substitution-permutation network (SPN)
Rounds: 10 (AES-128), 12 (AES-192), 14 (AES-256)
Status: No practical attacks; considered secure for foreseeable future

AES alone encrypts single 128-bit blocks. For ESP (which handles variable-length packets), AES must be used with a mode of operation that extends it to arbitrary-length messages.

ESP Encryption Algorithms
Algorithm	Type	Key Size	Status	Notes
AES-GCM	AEAD	128/192/256 bits	Recommended	Best choice; hardware acceleration
AES-CBC	Block + HMAC	128/192/256 bits	Acceptable	Requires separate integrity; legacy
AES-CTR	Stream-like	128/192/256 bits	Acceptable	Requires separate integrity
ChaCha20-Poly1305	AEAD	256 bits	Recommended	Excellent without AES-NI
3DES-CBC	Block + HMAC	168 bits (112 effective)	Deprecated	Sweet32 attack; avoid
ENCR_NULL	None	N/A	Never use	No encryption; authentication only

ChaCha20-Poly1305:

ChaCha20-Poly1305 is a modern AEAD cipher gaining adoption in ESP (RFC 7634). Designed by Daniel Bernstein, it offers:

Excellent software performance: As fast as AES-GCM even without hardware acceleration
Mobile/IoT suitability: Great for devices without AES-NI
Resistance to timing attacks: Constant-time implementation is straightforward
256-bit key only: Single key size simplifies configuration

Deprecated Algorithms:

Several legacy algorithms should be avoided:

3DES: Susceptible to Sweet32 attacks; 64-bit block size leads to birthday-bound attacks at ~32GB of traffic
DES: 56-bit key; trivially breakable with modern hardware
Blowfish: 64-bit block; similar birthday-bound concerns
RC4: Multiple attacks; completely broken

Algorithm Upgrade Priority

Block Cipher Modes of Operation

Cipher Block Chaining (CBC) Mode:

CBC was the traditional mode for ESP encryption (AES-CBC, 3DES-CBC).

Operation:

XOR first plaintext block with Initialization Vector (IV)
Encrypt the XOR result
XOR second plaintext block with first ciphertext block
Encrypt and repeat...

Characteristics:

Each ciphertext block depends on all previous blocks (chaining)
Requires random, unpredictable IV per packet
Encryption is sequential (can't parallelize)
Decryption can be parallelized
Requires padding to block boundary

Comparison of ESP Cipher Modes
Mode	Type	Parallelizable	IV Requirement	Padding	Authentication
CBC	Block	Decrypt only	Random, unpredictable	Required	Separate HMAC needed
CTR	Stream-like	Yes (enc & dec)	Unique (never reuse)	Not required	Separate HMAC needed
GCM	AEAD	Yes (enc & dec)	Unique (never reuse)	Not required	Built-in
CCM	AEAD	Encrypt only	Unique (never reuse)	Not required	Built-in

Counter (CTR) Mode:

CTR mode transforms a block cipher into a stream cipher by encrypting sequential counter values and XORing the result with plaintext.

Operation:

Concatenate nonce with counter (starting at 0 or 1)
Encrypt nonce||counter to produce keystream block
XOR keystream with plaintext block
Increment counter, repeat

Advantages:

Fully parallelizable (both encryption and decryption)
No padding needed (encrypt exactly the message length)
Pre-compute keystream before plaintext arrives

Critical Requirement: The nonce+counter combination must NEVER repeat with the same key. Reuse reveals XOR of two plaintexts—catastrophic security failure.

Galois/Counter Mode (GCM):

GCM combines CTR mode encryption with Galois field authentication, providing Authenticated Encryption with Associated Data (AEAD) in a single operation.

Components:

CTR encryption: Provides confidentiality (as described above)
GHASH authentication: Galois field multiplication for integrity
Combined output: Ciphertext + authentication tag

Why GCM Dominates Modern ESP:

Single pass: Encrypt and authenticate simultaneously
Hardware acceleration: AES-NI includes GCM-specific instructions (PCLMULQDQ)
Strong security proofs: Well-analyzed construction
Efficiency: Minimal overhead beyond raw encryption

Choose AES-GCM

Key Derivation and Management

The IKE Key Hierarchy:

IKE (Internet Key Exchange) establishes a hierarchy of keys:

SKEYSEED: Initial keying material derived from Diffie-Hellman exchange
SK_d: Key for deriving child SA keys
SK_ai/SK_ar: Authentication keys for IKE messages
SK_ei/SK_er: Encryption keys for IKE messages
ESP Keys: Derived from SK_d for each Child SA

ESP Key Derivation:

When an ESP SA is created, the following keys are derived:

Encryption Key: Length depends on algorithm (16/24/32 bytes for AES-128/192/256)
Authentication Key (if not AEAD): Length depends on HMAC algorithm
Salt (for some algorithms): Fixed value mixed with IV

Key derivation uses a PRF (Pseudo-Random Function) to expand SK_d into the required key material:

KEYMAT = PRF+(SK_d, Ni | Nr | SPIi | SPIr)

ESP Key Material Requirements
Algorithm	Encryption Key	Auth Key	Salt	Total KEYMAT
AES-128-CBC + HMAC-SHA-256	16 bytes	32 bytes	—	48 bytes per direction
AES-256-CBC + HMAC-SHA-256	32 bytes	32 bytes	—	64 bytes per direction
AES-128-GCM	16 bytes	— (integrated)	4 bytes	20 bytes per direction
AES-256-GCM	32 bytes	— (integrated)	4 bytes	36 bytes per direction
ChaCha20-Poly1305	32 bytes	— (integrated)	4 bytes	36 bytes per direction

Key Separation:

ESP uses separate keys for each direction of communication. The initiator and responder derive different keys for:

Initiator → Responder encryption
Responder → Initiator encryption
Initiator → Responder authentication (if applicable)
Responder → Initiator authentication (if applicable)

This separation ensures that compromising one direction's keys doesn't immediately compromise the other direction.

Perfect Forward Secrecy (PFS):

With PFS enabled, each Child SA performs a new Diffie-Hellman exchange to derive fresh keying material. Without PFS, all Child SA keys derive from the original IKE SA's DH exchange.

With PFS:

Compromise of one SA's keys doesn't reveal other SAs' keys
Higher security but additional computational overhead
Recommended for high-security deployments

Without PFS:

Slight performance improvement
If IKE SA keys are compromised, all Child SA keys are derivable
Acceptable for many deployments with strong IKE SA protection

Rekeying is Critical

Encryption Process in Detail

When an ESP endpoint encrypts a packet, it executes a precise sequence of operations. Understanding this process helps diagnose encryption failures and optimize implementations.

ESP Encryption Workflow (AES-GCM):

Step 1: Prepare Payload

Assemble data to encrypt: Original packet (tunnel) or upper-layer data (transport)
Append ESP trailer: Add padding, pad length, next header

Step 2: Construct Nonce

Combine salt (from SA, 4 bytes) with explicit IV (8 bytes, typically sequence number or counter)
Result: 12-byte nonce for GCM

Step 3: Assemble Associated Data (AAD)

AAD = ESP header (SPI + Sequence Number)
This data is authenticated but not encrypted

Step 4: Encrypt and Authenticate

Input to AES-GCM: Key, Nonce, Plaintext, AAD
Output: Ciphertext, Authentication Tag (ICV)

Step 5: Construct ESP Packet

Concatenate: ESP Header + IV + Ciphertext + ICV
Prepend outer IP header (tunnel mode) or original IP header (transport mode)

esp_encryption_pseudocode.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def encrypt_esp_packet(plaintext, sa):
    """
    ESP Encryption Process (AES-GCM)
    
    Args:
        plaintext: Original packet data to protect
        sa: Security Association with keys and parameters
    
    Returns:
        Complete ESP packet ready for transmission
    """
    
    # Step 1: Prepare payload with ESP trailer
    # Padding for alignment (AES-GCM doesn't strictly require, but helps)
    pad_length = calculate_padding(len(plaintext))
    padding = bytes(range(1, pad_length + 1))  # 0x01, 0x02, ...
    
    payload = plaintext + padding + bytes([pad_length, sa.next_header])
    
    # Step 2: Generate unique IV
    # Typically derived from sequence number for implicit IV
    # Or random for explicit IV modes
    sa.sequence_number += 1
    iv = sa.sequence_number.to_bytes(8, 'big')
    
    # Step 3: Construct nonce for AES-GCM
    # nonce = salt (4 bytes from SA) || IV (8 bytes)
    nonce = sa.salt + iv
    
    # Step 4: Assemble ESP header (also serves as AAD)
    esp_header = sa.spi.to_bytes(4, 'big') + sa.sequence_number.to_bytes(4, 'big')
    aad = esp_header  # Authenticated but not encrypted
    
    # Step 5: Encrypt and authenticate
    ciphertext, tag = aes_gcm_encrypt(
        key=sa.encryption_key,
        nonce=nonce,
        plaintext=payload,
        associated_data=aad
    )
    
    # Step 6: Construct final ESP packet
    esp_packet = esp_header + iv + ciphertext + tag
    
    return esp_packet
 
 
def calculate_padding(data_length):
    """
    Calculate padding needed for block alignment.
    For AES-GCM, minimal padding (at least 0 bytes).
    For AES-CBC, must align to 16-byte boundary.
    """
    # Including 2 bytes for pad_length and next_header
    trailer_fixed = 2
    total = data_length + trailer_fixed
    
    # Align to 4-byte boundary (common requirement)
    if total % 4 != 0:
        return 4 - (total % 4)
    return 0

Decryption Workflow:

The receiver performs the inverse process:

Parse ESP Header: Extract SPI and Sequence Number
Lookup SA: Find matching SA in SAD using SPI + destination IP
Verify Sequence Number: Check against anti-replay window
Reconstruct Nonce: Combine SA salt with received IV
Verify and Decrypt: AES-GCM authenticates then decrypts
- If authentication fails: Discard silently
- If authentication passes: Obtain plaintext
Remove Trailer: Read pad length, strip padding
Process Payload: Pass to upper layer (transport) or IP stack (tunnel)

Authentication Before Decryption

Performance Optimization

Hardware Acceleration:

AES-NI (AES New Instructions): Intel and AMD processors include dedicated instructions for AES operations:

AESENC, AESDEC: Single-round encryption/decryption
AESKEYGENASSIST: Key expansion assistance
PCLMULQDQ: Carry-less multiplication (for GCM's GHASH)

Performance Impact:

AES-GCM with AES-NI: 3-10 Gbps per core (varies by processor)
AES-GCM without AES-NI: 100-500 Mbps per core
10-30x improvement with hardware acceleration

ESP Encryption Performance (Single Core, Modern CPU)
Algorithm	With AES-NI	Without AES-NI	Relative
AES-128-GCM	~6 Gbps	~300 Mbps	20x faster with HW
AES-256-GCM	~4.5 Gbps	~250 Mbps	18x faster with HW
AES-256-CBC + HMAC-SHA-256	~3 Gbps	~200 Mbps	15x faster with HW
ChaCha20-Poly1305	~3 Gbps	~2.5 Gbps	SW implementation is fast
3DES-CBC + HMAC-SHA-1	~400 Mbps	~100 Mbps	4x, but still slow

Parallelization Strategies:

Multi-core Distribution:

Distribute SA workload across multiple cores
Hash incoming packets to cores based on SA (SPI-based hashing)
Each core maintains subset of SA state
Scales linearly with core count (up to practical limits)

Within-packet Parallelization:

AES-GCM and AES-CTR allow parallel block encryption
Pipelining: Process multiple blocks simultaneously
SIMD: Wide vector instructions encrypt 4-8 blocks at once

Dedicated Hardware:

Network Interface Cards with crypto offload
FPGA-based encryption engines
Dedicated security processors (QuickAssist, NITROX)
10-100+ Gbps throughput in dedicated silicon

Software Optimization:

Avoid memory allocation in fast path
Minimize cache misses (hot SA data structures)
Use zero-copy techniques where possible
Batch small packets for processing efficiency

Algorithm Selection for Performance

Common Encryption Misconfigurations

Even strong algorithms can be rendered useless by configuration errors. Recognizing common misconfigurations helps ensure that encryption provides its intended protection.

1. Null Encryption (ENCR_NULL)

Some configurations accidentally or intentionally use null encryption, providing no confidentiality:

Symptom: Packets authenticated but payload visible in captures
Cause: Misconfigured proposal, legacy compatibility settings
Fix: Explicitly require non-null encryption in proposals

2. Weak Algorithm Selection

Using deprecated algorithms that provide inadequate security:

Examples: 3DES, DES, Blowfish, RC4
Symptom: Passes interoperability tests but vulnerable to attacks
Fix: Configure only modern algorithms (AES-GCM, ChaCha20-Poly1305)

Critical Encryption Errors

•IV/Nonce Reuse: Catastrophic for CTR/GCM modes—reveals XOR of plaintexts. Cause: Poor random number generation or counter bugs. Detection: Review implementation; test with packet replay.
•Key Reuse Across SAs: Using the same keys for multiple SAs violates security assumptions. Cause: Manual keying with copied keys. Fix: Always use IKE for automatic unique key derivation.
•Inadequate Key Length: AES-128 is secure but AES-256 recommended for long-term security. Cause: Default configurations. Fix: Explicitly specify AES-256.
•No Rekeying: SAs never rekeyed, accumulating risk over time. Cause: No lifetime configured, IKE issues prevent rekey. Fix: Configure conservative lifetimes (1 hour or less).
•Encryption Without Authentication: Using null authentication with encryption. Cause: Misunderstanding that encryption implies integrity. Fix: Always pair encryption with authentication (or use AEAD).

3. Mismatched Algorithm Proposals

When initiator and responder support different algorithms, negotiation fails or falls back to weak algorithms:

Symptom: SA establishment fails or uses weaker-than-expected cipher
Cause: Asymmetric configuration between endpoints
Fix: Ensure matching proposals on both ends; prioritize strong algorithms

4. Missing Perfect Forward Secrecy

Without PFS, compromise of IKE SA enables decryption of all past traffic:

Symptom: Configuration appears correct but lacks resilience
Cause: PFS not enabled in Child SA creation
Fix: Enable PFS (DH group negotiation) for Child SAs

5. Improper Random Number Generation

Weak random numbers compromise IV generation and key derivation:

Symptom: Patterns in IVs, predictable keys
Cause: Exhausted entropy, poor PRNG seeding
Fix: Ensure strong entropy sources; use cryptographic PRNGs

Test Your Configuration

Summary and Key Takeaways

Key Takeaways

•ESP uses symmetric encryption for performance—keys derived from IKE, different for each direction
•AES-GCM is the modern standard — AEAD algorithm providing encryption + authentication efficiently with hardware acceleration
•Block cipher modes matter — CBC requires separate authentication; GCM provides integrated authentication; CTR needs careful nonce management
•Key derivation follows IKE hierarchy — ESP keys derived from IKE SA keying material using PRF; PFS adds additional DH exchanges
•Hardware acceleration is critical — AES-NI provides 10-30x performance improvement; ChaCha20-Poly1305 excels without hardware support
•Misconfigurations defeat encryption — Null encryption, weak algorithms, IV reuse, missing authentication can render encryption useless

What's Next:

Page Complete

3 / 5