IPSec Overview

In the previous page, we established why IPSec exists—to address the fundamental security deficiencies of IP networks. Now we turn to what security properties IPSec actually provides and how it achieves them.

IPSec offers four core security services, each addressing a distinct threat vector. Understanding these services—their capabilities, limitations, and cryptographic foundations—is essential for correctly deploying and evaluating IPSec solutions.

These aren't abstract concepts. Every VPN connection you've ever used, every site-to-site tunnel protecting corporate traffic, every remote worker accessing sensitive systems—all rely on these security services working correctly. A misconfiguration or misunderstanding can leave traffic completely exposed while administrators believe it's protected.

Confidentiality ensures that packet contents cannot be read by unauthorized parties. Even if an attacker captures packets in transit—through network taps, compromised routers, or Wi-Fi interception—they cannot extract meaningful information. All they see is apparently random ciphertext.

The Cryptographic Foundation: Symmetric Encryption

IPSec achieves confidentiality through symmetric-key encryption. Both communicating parties share a secret key; the sender encrypts data with this key, and the receiver decrypts using the same key. This is fundamentally different from asymmetric (public-key) encryption used in TLS handshakes—symmetric encryption is dramatically faster, enabling wire-speed encryption on high-bandwidth links.

Key Insight: Symmetric encryption's speed is why IPSec can protect multi-gigabit links with acceptable overhead. AES encryption in hardware can process 100+ Gbps. Asymmetric encryption would be orders of magnitude slower and impractical for bulk data protection.

Understanding Encryption Modes:

Block Cipher Modes (CBC, CTR):

Block ciphers like AES encrypt fixed-size blocks (128 bits for AES). Real-world data rarely aligns perfectly with block boundaries. Modes of operation define how to apply block ciphers to arbitrary-length data:

• CBC (Cipher Block Chaining): Each plaintext block is XORed with the previous ciphertext block before encryption. Requires an Initialization Vector (IV) for the first block. Sequential encryption; cannot parallelize. Vulnerable to padding oracle attacks if implementation checks padding before MAC.

• CTR (Counter Mode): Encrypts sequential counter values; XORs the result with plaintext. Essentially converts block cipher to stream cipher. Parallelizable—critical for high-speed hardware. Never reuse (key, counter) pairs!

Authenticated Encryption with Associated Data (AEAD):

Modern best practice combines encryption and integrity verification in a single operation. AEAD modes like AES-GCM and ChaCha20-Poly1305 provide:

1. Confidentiality: Data is encrypted
2. Integrity: MAC protects against modification
3. Authenticity: The sender is implicitly authenticated (only key holders can produce valid MACs)

AEAD modes avoid subtle implementation vulnerabilities that plagued encrypt-then-MAC schemes and are now the recommended approach for all new IPSec deployments.

Key Derivation and Key Material:

Symmetric encryption requires shared key material that:

1. Is known only to the communicating parties
2. Is sufficiently random (high entropy)
3. Changes periodically (key rotation) to limit exposure from potential compromise

IPSec derives encryption keys from the IKE negotiation process. The key derivation function (typically based on HMAC-SHA2) uses:

• Diffie-Hellman shared secret
• Nonces from both parties
• Security Parameter Indexes (SPIs)

The result: fresh keys for each direction of traffic, each security association. Even if one SA's key is compromised, other SAs remain secure.

Practical Consideration: Hardware Acceleration

Modern CPUs include AES instructions (AES-NI on x86, ARMv8 Cryptography Extensions) that accelerate AES operations by 10-100x. Enterprise routers and firewalls include dedicated cryptographic accelerators. When selecting algorithms:

Data integrity ensures that packet contents have not been modified in transit. Any change—whether a single bit flip or wholesale content replacement—is detected, and the modified packet is silently discarded.

Without integrity protection, an attacker can perform man-in-the-middle attacks: intercepting packets, modifying their contents, and forwarding the modified packets. Even with encryption, an attacker might corrupt ciphertext in ways that produce meaningful changes when decrypted (bit-flipping attacks in stream ciphers and CTR mode).

The Cryptographic Foundation: Hash Functions and MACs

Integrity verification relies on two cryptographic primitives:

1. Cryptographic Hash Functions

A hash function takes arbitrary-length input and produces a fixed-size output (the 'hash' or 'digest'). Properties:

• Deterministic: Same input always produces same output
• Pre-image resistant: Given hash output, cannot find the input
• Collision resistant: Cannot find two inputs with the same hash
• Avalanche effect: Tiny input changes produce drastic hash changes

Common hash functions: SHA-256, SHA-384, SHA-512 (from the SHA-2 family)

2. Message Authentication Codes (MACs)

A hash alone isn't sufficient for integrity protection. An attacker could modify the packet and recompute the hash. We need a keyed integrity check: a verification that only parties knowing the secret key can produce.

HMAC (Hash-based Message Authentication Code) combines a hash function with a secret key:

HMAC(key, message) = Hash((key ⊕ opad) || Hash((key ⊕ ipad) || message))

Where:

• ipad = 0x36 repeated to block size
• opad = 0x5C repeated to block size
• || = concatenation
• ⊕ = XOR operation

Security Properties of HMAC:

• Without the key, an attacker cannot forge a valid MAC
• Knowing multiple (message, MAC) pairs doesn't reveal the key
• Altering any bit of the message invalidates the MAC
• The construction is provably secure if the underlying hash is secure

How Integrity Works in IPSec:

1. Sender: Computes MAC over protected packet fields using the shared integrity key
2. Sender: Appends MAC (Integrity Check Value, ICV) to the packet
3. Packet traverses network (potentially through hostile infrastructure)
4. Receiver: Recomputes MAC over received packet fields
5. Receiver: Compares computed MAC with received ICV
6. Match: Packet is authentic and unmodified → accept
7. Mismatch: Packet was modified or forged → silently discard

Authenticated Encryption: Combining Confidentiality and Integrity

Modern IPSec strongly recommends AEAD algorithms (AES-GCM, ChaCha20-Poly1305) that provide both encryption and authentication in a single operation. AEAD offers several advantages:

GCM Mode Internals:

GCM (Galois/Counter Mode) combines CTR mode encryption with Galois field multiplication for authentication:

1. CTR mode encrypts the plaintext
2. GHASH (based on multiplication in GF(2^128)) authenticates ciphertext + Additional Authenticated Data (AAD)
3. Result: single tag providing both confidentiality and authenticity

The Additional Authenticated Data (AAD) allows authenticating unencrypted header fields (like ESP header information) alongside the encrypted payload—a perfect fit for IPSec's requirements.

Data origin authentication guarantees that packets genuinely originated from the claimed sender—not from an attacker spoofing their identity. Combined with integrity, it proves both who sent the data and that the data arrived unmodified.

Why Is This Different from Integrity?

At first glance, integrity and authentication seem identical—both use the same cryptographic MAC. The distinction is conceptual:

• Integrity alone: Proves the packet wasn't modified in transit
• Authentication: Proves the packet came from an entity that knows the shared secret

In practice, if you verify integrity using a key known only to legitimate parties, you implicitly verify authenticity. An attacker who doesn't know the key cannot forge a valid MAC. However, authentication encompasses additional concerns:

The Spoofing Problem:

IP packets include source addresses, but nothing prevents an attacker from inserting arbitrary addresses. Standard IP has no mechanism to verify that the source address in a packet corresponds to the actual originator. This enables:

• Blind attacks: Send packets with forged source addresses to elicit responses to the victim (DDoS amplification)
• Man-in-the-middle: Position yourself between two parties and impersonate both
• Session hijacking: Inject packets into an existing session

IPSec's authentication service makes these attacks infeasible. Even if an attacker can inject packets with forged addresses, they cannot produce valid MACs without knowing the secret key.

Authentication in IKE vs. ESP/AH:

IPSec authentication operates at two levels:

1. IKE Authentication (Control Plane):

When establishing a Security Association, IKE authenticates the peer using:

• Pre-shared keys (PSK): Both parties configure identical secrets
• Digital certificates (X.509): Public key infrastructure validates identity
• EAP (Extensible Authentication Protocol): Username/password, smart cards, etc.

IKE authentication proves identity of the communicating parties and establishes a secure channel for negotiating session keys.

2. Per-Packet Authentication (Data Plane):

ESP and AH provide per-packet authentication using symmetric MACs derived from IKE-negotiated keys. This proves each individual packet originated from the authenticated peer and hasn't been modified.

The Subtle Difference:

IKE authentication says: "I am communicating with entity X" Per-packet authentication says: "This specific packet came from entity X and hasn't been modified"

Both are necessary. Without IKE authentication, you might negotiate keys with an attacker. Without per-packet authentication, an attacker might inject packets into an authenticated session.

Anti-replay protection prevents attackers from recording valid packets and retransmitting them later. Even though such packets would have valid authentication (the attacker didn't modify them), accepting them again would cause duplicate transactions, state corruption, or security bypasses.

The Replay Attack Scenario:

1. Attacker captures authenticated packets (can't read or modify them)
2. Attacker stores packets for later use
3. Attacker retransmits stored packets
4. Without anti-replay: receiver accepts packets as valid
5. Attack impact varies: duplicate bank transfers, duplicate authentication, duplicate commands

Example Attack:

Imagine a financial application where an authenticated IPSec-protected packet contains a "transfer $1000" instruction. An attacker captures this packet. They can't read or modify it, but they can retransmit it 100 times. Without anti-replay protection, the server would process 100 transfers—even though the legitimate user intended only one.

The IPSec Anti-Replay Mechanism:

ESP and AH include a 32-bit (or extended 64-bit) Sequence Number field that provides anti-replay protection:

Sender Behavior:

1. Initialize sequence counter to 0 when Security Association is established
2. Increment sequence counter for each packet sent
3. Include current sequence number in packet header
4. Sequence number is covered by MAC (cannot be modified)

Receiver Behavior:

1. Maintain a sliding window of acceptable sequence numbers (typically 64-128 packets)
2. Track the highest sequence number received
3. For each received packet:
   • If sequence number is ahead of window: accept (after MAC verification), advance window
   • If sequence number is within window: check if already received; if new, accept after MAC verification
   • If sequence number is before window: reject (too old; likely replay)
   • If sequence number matches previously received: reject (duplicate; replay attempt)

Extended Sequence Numbers (ESN):

The original 32-bit sequence number allows 2^32 (~4.3 billion) packets per Security Association. This seems large, but on a 10 Gbps link with 1500-byte packets, you'd exhaust the counter in about 86 minutes.

To address this, IPSec supports Extended Sequence Numbers (ESN): a 64-bit counter where only the low-order 32 bits are transmitted. The high-order 32 bits are maintained locally and implicitly inferred. This allows 2^64 packets per SA—effectively unlimited.

Performance Implications:

Anti-replay requires per-packet state at the receiver:

• Track highest sequence number
• Maintain bitmap of received packets within window
• Compare, update for each packet

For high-speed links (100+ Gbps) with millions of packets per second, this becomes significant. Hardware implementations typically store window state in fast SRAM or use bloom filters for approximate tracking.

Beyond protecting packet contents, IPSec in tunnel mode provides a fifth security service: limited traffic flow confidentiality. This partially conceals traffic patterns that could otherwise reveal sensitive information.

The Traffic Analysis Threat:

Even when packet contents are encrypted, an observer can learn valuable information from:

• Packet sizes: VoIP calls have distinct size distributions
• Timing patterns: Interactive sessions have characteristic delays
• Source/destination addresses: Who's communicating with whom
• Volume: How much data is exchanged

Traffic analysis has been used to:

• Identify websites visited over VPNs (fingerprinting)
• Detect Tor usage despite encryption
• Identify video streams by bitrate patterns
• Map organizational communication structures

How Tunnel Mode Helps:

In tunnel mode, the original IP header (containing real source/destination) is encapsulated inside the encrypted ESP payload. External observers see only:

• Tunnel endpoint addresses (gateway IPs)
• Encrypted blob containing the real packet

This provides network-level anonymity: an observer can tell that Gateway A is communicating with Gateway B, but not which internal hosts are involved.

Limitations of IPSec Traffic Flow Confidentiality:

IPSec's traffic flow confidentiality is 'limited' because it doesn't fully protect against traffic analysis:

Advanced Protection Techniques:

For higher traffic flow confidentiality, additional measures can be applied:

• Padding: Adding random padding to packets to obscure size distributions
• Traffic shaping: Transmitting at constant rate regardless of actual traffic (expensive in bandwidth)
• Dummy traffic: Injecting decoy packets that are discarded by the receiver
• Mixing/Onion routing: Using multiple hops to obscure origin (Tor-like approaches)

These are not part of standard IPSec but can be layered on top for high-security scenarios.

In practice, IPSec security services are not deployed individually—they're combined based on security requirements. Understanding common combinations and their tradeoffs is essential for security architects.

Common Deployment Profiles:

Modern Best Practice:

For nearly all deployments, the recommended configuration is:

ESP with AES-256-GCM-16
- 256-bit AES key for encryption
- 128-bit (16-byte) authentication tag
- Integrated confidentiality + integrity + authentication
- Anti-replay enabled with appropriate window size
- Perfect Forward Secrecy (PFS) with Diffie-Hellman group 19 or 20

Why AH Is Rarely Used Today:

Historically, AH was used when only authentication (not encryption) was required. AH authenticates the entire IP packet, including the outer header—which ESP cannot do because the outer header changes in transit (TTL decrement, etc.).

However:

1. NAT traversal breaks AH (NAT modifies IP addresses, invalidating authentication)
2. ESP with AEAD provides authentication that's sufficient for most purposes
3. The outer header protection AH provides is rarely necessary
4. Maintaining two protocols increases complexity

Result: Modern deployments almost universally use ESP only. AH is retained for backward compatibility and niche applications.

We've examined the security services that IPSec provides and the cryptographic mechanisms underlying each. Let's consolidate the key insights:

What's Next:

Now that we understand what security services IPSec provides, we'll examine how they're applied. The next page explores IPSec's two operational modes: Transport Mode (end-to-end host protection) and Tunnel Mode (gateway-to-gateway protection). Understanding these modes is essential for designing IPSec deployments that match your security architecture.