Loading learning content...
While GRE provides flexible encapsulation, it offers no protection against eavesdropping or tampering. IPSec tunneling addresses this fundamental gap, providing cryptographically strong security for network layer traffic. When IPSec operates in tunnel mode, it doesn't just protect packets—it creates entirely new packets with new headers, effectively building secure virtual links across untrusted networks.
IPSec tunnel mode is the backbone of modern VPN technology. Every time an employee connects to their corporate network from home, every site-to-site VPN between data centers, every secure connection between cloud providers relies on IPSec tunneling to ensure that sensitive data traverses the hostile Internet without exposure.
By the end of this page, you will understand: how IPSec tunnel mode differs from transport mode; the complete packet structure of IPSec tunneled traffic; ESP tunnel mode encapsulation and processing; the GRE over IPSec architecture; IKE's role in establishing secure tunnels; and practical deployment scenarios for IPSec tunneling.
IPSec operates in two modes: transport mode and tunnel mode. The distinction is fundamental to understanding how IPSec creates secure tunnels.
Transport Mode:
Tunnel Mode:
| Characteristic | Transport Mode | Tunnel Mode |
|---|---|---|
| Original IP Header | Preserved (visible) | Encrypted (hidden) |
| New IP Header | Not added | Added (outer header) |
| Protected Scope | Payload only | Entire original packet |
| Endpoint Types | Hosts only | Hosts or gateways |
| Primary Use Case | Host-to-host security | VPN tunnels |
| Header Overhead | Lower (no new IP header) | Higher (+20 bytes min) |
Why tunnel mode for VPNs:
Tunnel mode is essential for VPN scenarios because:
Gateway Protection: Security gateways can protect traffic on behalf of entire networks, not just themselves. The gateway encrypts traffic from any internal host, not just traffic it originates.
Address Hiding: Original source and destination addresses are encrypted, preventing traffic analysis that could reveal internal network structure.
NAT Compatibility: The outer IP header uses the gateway's public addresses, while internal private addresses remain hidden inside the encrypted payload.
Network-to-Network Connectivity: Entire subnets can communicate securely through a single tunnel between gateways.
Use transport mode when both endpoints are the actual communicating hosts and you only need to protect the communication between them. Use tunnel mode whenever a gateway (router, firewall, VPN concentrator) is protecting traffic on behalf of other hosts, or when you need to hide the original addressing information.
ESP (Encapsulating Security Payload) in tunnel mode creates a layered packet structure where the entire original packet becomes encrypted payload within a new IP packet.
ESP TUNNEL MODE PACKET STRUCTURE═══════════════════════════════════════════════════════════════════════════ ┌─────────────────────────────────────────────────────────────────────────┐│ NEW (OUTER) IP HEADER ││ ┌───────────────────────────────────────────────────────────────────┐ ││ │ Source IP: Tunnel Source (Gateway A's public IP) │ ││ │ Destination IP: Tunnel Destination (Gateway B's public IP) │ ││ │ Protocol: 50 (ESP) │ ││ │ TTL: Fresh value for transit │ ││ └───────────────────────────────────────────────────────────────────┘ │├─────────────────────────────────────────────────────────────────────────┤│ ESP HEADER (8 bytes) ││ ┌───────────────────────────────────────────────────────────────────┐ ││ │ SPI (Security Parameter Index): 4 bytes │ ││ │ Sequence Number: 4 bytes │ ││ └───────────────────────────────────────────────────────────────────┘ │├─────────────────────────────────────────────────────────────────────────┤│ ENCRYPTED PAYLOAD (variable) ◄─────────┼── ENCRYPTED│ ┌───────────────────────────────────────────────────────────────────┐ ││ │ IV (Initialization Vector): Algorithm-dependent (8-16 bytes) │ ││ │ ┌─────────────────────────────────────────────────────────────┐ │ ││ │ │ ORIGINAL IP HEADER (20+ bytes) │ │ ││ │ │ Source IP: Original source (internal host) │ │ ││ │ │ Destination IP: Original destination (remote internal host)│ │ ││ │ │ Protocol: TCP/UDP/ICMP/etc. │ │ ││ │ └─────────────────────────────────────────────────────────────┘ │ ││ │ ┌─────────────────────────────────────────────────────────────┐ │ ││ │ │ ORIGINAL TRANSPORT HEADER │ │ ││ │ │ (TCP, UDP, or other transport protocol) │ │ ││ │ └─────────────────────────────────────────────────────────────┘ │ ││ │ ┌─────────────────────────────────────────────────────────────┐ │ ││ │ │ APPLICATION DATA │ │ ││ │ └─────────────────────────────────────────────────────────────┘ │ ││ └───────────────────────────────────────────────────────────────────┘ │├─────────────────────────────────────────────────────────────────────────┤│ ESP TRAILER (variable) ◄─────────┼── ENCRYPTED│ ┌───────────────────────────────────────────────────────────────────┐ ││ │ Padding: 0-255 bytes (for block cipher alignment) │ ││ │ Pad Length: 1 byte │ ││ │ Next Header: 1 byte (identifies encrypted payload type: 4=IPv4) │ ││ └───────────────────────────────────────────────────────────────────┘ │├─────────────────────────────────────────────────────────────────────────┤│ ESP ICV (Integrity Check Value) ││ ┌───────────────────────────────────────────────────────────────────┐ ││ │ Authentication Tag: 12-16 bytes (algorithm-dependent) │ ││ │ Covers: ESP Header + Encrypted Payload + ESP Trailer │ ││ └───────────────────────────────────────────────────────────────────┘ │└─────────────────────────────────────────────────────────────────────────┘ OVERHEAD CALCULATION (AES-256-GCM example):─────────────────────────────────────────── Outer IP Header: 20 bytes ESP Header: 8 bytes IV: 16 bytes (AES-GCM) ESP Trailer: 2-17 bytes (padding + 2) ICV: 16 bytes (AES-GCM) ───────────────────────────────────────── Minimum Overhead: ~62 bytes Maximum Overhead: ~77 bytesCritical observations:
1. Complete Encapsulation The entire original IP packet—header and all—is encrypted within the ESP payload. An observer on the network sees only the outer IP header (tunnel endpoints) and the ESP header. The original addressing, ports, and data are completely hidden.
2. SPI Identifies the Security Association The Security Parameter Index (SPI) tells the receiving gateway which Security Association to use for decryption. Multiple tunnels between the same endpoints use different SPIs.
3. Next Header in Trailer (Not Outer Header) Unlike GRE where Protocol Type is in the tunnel header, ESP puts the payload identifier (Next Header field) in the encrypted trailer. This prevents observers from knowing what's inside the tunnel.
4. Significant Overhead ESP tunnel mode adds substantial overhead (50-80+ bytes) compared to GRE (24 bytes). This must be factored into MTU calculations.
With ~60+ bytes of overhead, IPSec tunnels significantly reduce the effective MTU. For a 1500-byte Ethernet frame, the maximum inner packet size drops to approximately 1400-1440 bytes. Proper MTU configuration and Path MTU Discovery are essential to prevent fragmentation issues.
Unlike GRE's stateless operation, IPSec tunnels require a negotiation phase to establish the Security Association (SA). The Internet Key Exchange (IKE) protocol handles this negotiation, authenticating peers and generating encryption keys.
Two-Phase IKE Process:
Phase 1 (IKE SA / ISAKMP SA):
Phase 2 (IPSec SA / Child SA):
IKE TUNNEL ESTABLISHMENT PROCESS═══════════════════════════════════════════════════════════════════════════ Gateway A Gateway B (Initiator) (Responder) │ │ │ PHASE 1 (IKE SA) │ │◄─────────────────────────────────────────────────────►│ │ │ │ 1. IKE_SA_INIT Request │ │ - Proposed algorithms (encryption, integrity, DH) │ │ - Key exchange material (DH public value) │ │ - Nonce (random value for freshness) │ │─────────────────────────────────────────────────────►│ │ │ │ 2. IKE_SA_INIT Response │ │ - Selected algorithms │ │ - Key exchange material (DH public value) │ │ - Nonce │ │◄─────────────────────────────────────────────────────│ │ │ │ [Both compute shared secret from DH exchange] │ │ [IKE SA now protects further messages] │ │ │ │ 3. IKE_AUTH Request (encrypted) │ │ - Identity (ID payload) │ │ - Authentication (PSK hash or certificate sig) │ │ - Traffic selectors (what traffic to protect) │ │ - Proposed IPSec transforms │ │─────────────────────────────────────────────────────►│ │ │ │ 4. IKE_AUTH Response (encrypted) │ │ - Identity │ │ - Authentication │ │ - Traffic selectors (confirmed/narrowed) │ │ - Selected IPSec transforms │ │◄─────────────────────────────────────────────────────│ │ │ │ PHASE 1 COMPLETE │ │ IKE SA established (for control channel) │ │ IPSec SA established (for data traffic) │ │ │ ╔═══════════════════════════════════════════════════════╗ ║ SECURE TUNNEL NOW ACTIVE ║ ║ All traffic matching selectors is encrypted/decrypted║ ╚═══════════════════════════════════════════════════════╝ │ │ │ ENCRYPTED DATA TRAFFIC │ │═══════════════════════════════════════════════════════│ │ (ESP-protected packets flow) │ │═══════════════════════════════════════════════════════│IKEv2 vs IKEv1:
Modern deployments use IKEv2 (RFC 7296), which offers significant improvements:
| Aspect | IKEv1 | IKEv2 |
|---|---|---|
| Messages for SA | 6-9 messages | 4 messages |
| NAT Traversal | Add-on (NAT-T) | Built-in |
| Dead Peer Detection | Optional extension | Built-in |
| EAP Authentication | Complex | Native support |
| MOBIKE | Not available | Mobile/multihoming support |
Traffic Selectors:
Traffic selectors define which traffic the tunnel protects. They specify source/destination address ranges and optionally ports and protocols. Only traffic matching the selectors is encrypted; other traffic bypasses the tunnel.
Security Associations have limited lifetimes (typically hours or gigabytes of data). Before expiration, IKE automatically negotiates new SAs (rekeying) to maintain continuous protection without traffic interruption. This limits the exposure if a key is somehow compromised.
Combining GRE and IPSec provides the best of both worlds: GRE's multicast capability and protocol flexibility with IPSec's security. This combination is the most common architecture for enterprise site-to-site VPNs that need to run dynamic routing protocols.
Why combine them?
IPSec alone limitations:
GRE alone limitations:
GRE OVER IPSEC PACKET STRUCTURE═══════════════════════════════════════════════════════════════════════════ Original Packet (to be sent through secure tunnel):┌──────────────────────────────────────────────┐│ IP: 10.1.0.100 → 10.2.0.100 | TCP | Payload │└──────────────────────────────────────────────┘ After GRE Encapsulation:┌────────────────────────────────────────────────────────────────────────┐│ Outer IP: 203.0.113.1 → 198.51.100.1 | GRE | Inner IP | TCP | Payload │└────────────────────────────────────────────────────────────────────────┘ After IPSec Protection (Transport Mode ESP on GRE):┌─────────────────────────────────────────────────────────────────────────┐│ Outer IP: 203.0.113.1 → 198.51.100.1 (proto 50) │├─────────────────────────────────────────────────────────────────────────┤│ ESP Header │├─────────────────────────────────────────────────────────────────────────┤│ ┌─────────────────────────────────────────────────────────────────────┐ ││ │ GRE Header ◄─────┼─┤ ENCRYPTED│ ├─────────────────────────────────────────────────────────────────────┤ ││ │ Inner IP: 10.1.0.100 → 10.2.0.100 ◄─────┼─┤ AND│ ├─────────────────────────────────────────────────────────────────────┤ ││ │ TCP Header ◄─────┼─┤ AUTHENTICATED│ ├─────────────────────────────────────────────────────────────────────┤ ││ │ Application Payload ◄─────┼─┤│ └─────────────────────────────────────────────────────────────────────┘ │├─────────────────────────────────────────────────────────────────────────┤│ ESP Trailer (encrypted) │├─────────────────────────────────────────────────────────────────────────┤│ ESP ICV (authentication tag) │└─────────────────────────────────────────────────────────────────────────┘ Overhead Breakdown: GRE IP Header: 20 bytes GRE Header: 4 bytes ESP Header: 8 bytes IV: 16 bytes ESP Trailer: ~10 bytes ESP ICV: 16 bytes ────────────────────────── Total: ~74 bytesConfiguration approach:
This layered approach keeps concerns separated and simplifies troubleshooting—you can verify GRE connectivity independently of IPSec, then add security.
Virtual Tunnel Interfaces (VTI) provide route-based IPSec without GRE. They offer native IPSec tunnel interfaces that work like GRE interfaces but with direct IPSec encapsulation. VTI reduces overhead and complexity while supporting multicast through special handling. Many modern deployments use VTI instead of GRE over IPSec.
Understanding how packets are processed at IPSec tunnel endpoints is essential for troubleshooting and performance optimization.
IPSEC TUNNEL PROCESSING FLOW═══════════════════════════════════════════════════════════════════════════ OUTBOUND PROCESSING (Sending through tunnel)──────────────────────────────────────────── ┌─────────────────┐ │ Original Packet │ (e.g., from internal host 10.1.0.100) │ Arrives │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ SPD Lookup │ Policy check: Does this traffic match │ (Security │ a tunnel policy? Look at src/dst IP, │ Policy DB) │ ports, protocol └────────┬────────┘ │ Match found: PROTECT ▼ ┌─────────────────┐ │ SAD Lookup │ Find the SA for this traffic flow │ (Security │ Retrieve encryption key, algorithm, │ Association DB)│ sequence counter └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Fragment if │ If packet + overhead > Path MTU, │ Necessary │ fragment BEFORE encryption └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Encapsulate │ Tunnel mode: Prepend new outer IP header │ │ Add ESP header with SPI and seq number └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Encrypt │ Encrypt: Original IP header + payload │ │ Add padding as needed for block cipher └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Generate ICV │ Calculate authentication tag over │ │ ESP header + ciphertext + trailer └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Transmit │ Send encrypted packet toward │ │ tunnel destination └─────────────────┘ INBOUND PROCESSING (Receiving from tunnel)────────────────────────────────────────── ┌─────────────────┐ │ Encrypted Packet│ Arrives from tunnel peer │ Arrives │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Check Protocol │ Is outer IP protocol = 50 (ESP)? │ │ Extract SPI from ESP header └────────┬────────┘ │ ▼ ┌─────────────────┐ │ SAD Lookup │ Find SA matching SPI │ by SPI │ Get decryption key, algorithm └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Verify ICV │ Verify authentication tag │ │ If invalid: DROP packet, log event └────────┬────────┘ │ Valid ▼ ┌─────────────────┐ │ Anti-Replay │ Check sequence number against window │ Check │ If replay: DROP packet └────────┬────────┘ │ Within window ▼ ┌─────────────────┐ │ Decrypt │ Decrypt payload using SA parameters │ │ Remove padding, extract original packet └────────┬────────┘ │ ▼ ┌─────────────────┐ │ SPD Check │ Verify decrypted packet matches │ (Post-decrypt) │ expected traffic selectors └────────┬────────┘ │ Matches ▼ ┌─────────────────┐ │ Forward │ Route original packet to │ │ final destination └─────────────────┘IPSec encryption/decryption is computationally intensive. Hardware acceleration (AES-NI on CPUs, dedicated crypto ASICs on routers) is essential for high-throughput deployments. Without hardware acceleration, a software IPSec implementation might handle only hundreds of Mbps; with acceleration, multi-gigabit speeds are possible.
A significant challenge for IPSec is traversing NAT (Network Address Translation) devices. NAT modifies IP headers and TCP/UDP port numbers—actions that break IPSec's integrity verification. NAT Traversal (NAT-T) solves this problem.
The problem:
The solution: UDP encapsulation
NAT-T encapsulates ESP packets inside UDP (port 4500). This gives NAT devices something to work with—they see UDP packets they can track normally.
NAT TRAVERSAL (NAT-T) ENCAPSULATION═══════════════════════════════════════════════════════════════════════════ Standard ESP (without NAT):┌─────────────────────────────────────────────────────────────────┐│ IP Header (proto 50) │ ESP Header │ Encrypted Payload │ ICV │└─────────────────────────────────────────────────────────────────┘ ESP with NAT-T (UDP encapsulation):┌────────────────────────────────────────────────────────────────────────┐│ IP Header (proto 17) │ UDP Header │ ESP Header │ Encrypted │ ICV ││ │ Src: 4500 │ │ Payload │ ││ │ Dst: 4500 │ │ │ │└────────────────────────────────────────────────────────────────────────┘ ▲ │ NAT can modify source port and track this like any UDP flow NAT-T DETECTION AND NEGOTIATION:──────────────────────────────── During IKE Phase 1:1. Both peers send NAT-D (NAT Detection) payloads containing: - Hash of their source IP:port - Hash of peer's destination IP:port 2. Each peer compares received hashes to expected values: - If hash matches: No NAT detected - If hash differs: NAT is present on this path 3. If NAT detected, both peers switch to port 4500: - IKE continues on UDP 4500 (instead of 500) - ESP is encapsulated in UDP 4500 4. Keepalives prevent NAT binding expiration: - NAT-T keepalives sent every 20 seconds (configurable) - Maintains UDP binding in NAT device - Ensures tunnel remains reachable| Traffic Type | Protocol/Port | NAT Behavior |
|---|---|---|
| IKE (no NAT) | UDP 500 | Works through NAT |
| IKE (NAT detected) | UDP 4500 | Better NAT compatibility |
| ESP (no NAT) | Protocol 50 | May fail through NAT |
| ESP with NAT-T | UDP 4500 | Full NAT compatibility |
In IKEv2, NAT-T is integrated into the protocol. Detection happens automatically during IKE_SA_INIT, and if NAT is present, the endpoints switch to UDP encapsulation seamlessly. No special configuration is typically required—it just works.
IPSec tunnel mode provides the cryptographic foundation for secure network-layer tunneling. Let's consolidate the key concepts:
What's next:
We've seen how GRE provides encapsulation and IPSec adds security. The next page examines Tunnel Mode vs Transport Mode in depth, clarifying when each is appropriate and exploring the hybrid scenarios where both might be used in a single deployment.
You now understand IPSec tunnel mode—its packet structure, IKE negotiation, GRE integration, and NAT traversal. This knowledge enables you to design, deploy, and troubleshoot secure VPN solutions using IPSec as the security foundation.