IPSec provides two fundamentally different modes of operation, each designed for distinct network architectures and security requirements. The choice between Transport Mode and Tunnel Mode determines which parts of the IP packet are protected, how the packet is transformed, and what kind of security deployment is possible.

This isn't merely a configuration option—it's an architectural decision that shapes your entire security deployment. Choose incorrectly, and you'll end up with security gaps, routing problems, or unnecessary complexity. Choose correctly, and IPSec seamlessly integrates into your network, providing exactly the protection you need.

The distinction between these modes is one of the most important concepts in IPSec, yet it's frequently misunderstood. We'll examine each mode in depth, trace the exact packet transformations involved, and develop clear guidelines for mode selection.

Transport Mode provides security for the payload of an IP packet while leaving the original IP header mostly intact. It's designed for end-to-end protection between two communicating hosts—the IPSec endpoints are the same as the communication endpoints.

Key Characteristics:

• The original IP header is preserved (with minor modifications)
• Only the transport layer segment (TCP/UDP payload) and above are protected
• Both endpoints must implement IPSec
• Used for host-to-host scenarios
• Lower overhead than tunnel mode (no additional IP header)
• Original source and destination addresses remain visible

When to Use Transport Mode:

1. Direct server-to-server communication requiring encryption
2. Remote management protocols (SSH/RDP over IPSec for double encryption)
3. Database replication between servers
4. Any scenario where both communicating hosts are IPSec-capable

ESP Transport Mode Packet Structure:

When ESP is applied in transport mode, the packet undergoes this transformation:

Original Packet:

| IP Header (20 bytes) | TCP Header (20 bytes) | Application Data |
  Protocol = 6 (TCP)

After ESP Transport Mode:

| IP Header (20 bytes) | ESP Header (8 bytes) | TCP Header (20 bytes) | Application Data | ESP Trailer | Auth Tag |
  Protocol = 50 (ESP)    SPI + Seq#             ←────────────── Encrypted ─────────────→
                                                ←──────────────────── Authenticated ───────────────────→

Detailed Field Description:

The Protocol Field Transform:

The original IP header's Protocol field indicated the next layer protocol (6 for TCP, 17 for UDP). After ESP encapsulation, this changes to 50 (ESP). The original protocol value moves to the ESP Trailer's 'Next Header' field.

This allows the receiver to:

1. See Protocol=50 → process as ESP
2. After decryption, read ESP Trailer's Next Header=6 → pass to TCP stack

AH Transport Mode Packet Structure:

When AH is used in transport mode (authentication only, no encryption):

After AH Transport Mode:

| IP Header (20 bytes) | AH Header (24 bytes) | TCP Header (20 bytes) | Application Data |
  Protocol = 51 (AH)    SPI + Seq# + ICV       ←───────── Authenticated but NOT encrypted ─────────→

AH authenticates:

• Immutable IP header fields (addresses, etc.)
• The AH header itself
• The entire original payload

Mutable fields (TTL, header checksum) are zeroed before authentication calculation and excluded from verification—otherwise, every router decrementing TTL would invalidate the authentication.

Security Implications of Transport Mode:

Tunnel Mode encapsulates the entire original IP packet inside a new IP packet with new headers. It's designed for gateway-to-gateway VPN architectures where security gateways protect traffic between networks—the IPSec endpoints differ from the communication endpoints.

Key Characteristics:

• The entire original packet becomes the payload of a new packet
• A new outer IP header is created with gateway addresses
• Original source/destination addresses are hidden inside the encrypted payload
• Endpoints don't need to implement IPSec—gateways handle everything
• Higher overhead (additional IP header) but greater flexibility
• Provides limited traffic flow confidentiality (hides internal topology)

When to Use Tunnel Mode:

1. Site-to-site VPN between corporate locations
2. Remote access VPN (mobile users connecting to corporate network)
3. Cloud connectivity (on-premises to cloud VPC)
4. Protecting legacy systems that can't implement IPSec
5. Any scenario where network gateways rather than endpoints implement security

ESP Tunnel Mode Packet Structure:

Original Packet (from Host A to Host B):

| Original IP Header | TCP Header | Application Data |
  Src: 10.0.1.5
  Dst: 10.0.2.7
  Protocol: 6 (TCP)

After ESP Tunnel Mode (on the wire between gateways):

| New IP Header | ESP Header | Original IP Header | TCP Header | Data | ESP Trailer | Auth Tag |
  Src: 203.0.113.1 (GW_A)     ←───────────────────── Encrypted ─────────────────────→
  Dst: 198.51.100.1 (GW_B)
  Protocol: 50 (ESP)          ←────────────────────────── Authenticated ─────────────────────────→

The Double Header Structure:

Traffic Flow:

1. Host A (10.0.1.5) sends packet to Host B (10.0.2.7)
2. Packet arrives at Gateway A (default route or policy routing)
3. Gateway A looks up SPD → PROTECT policy matches
4. Gateway A encapsulates in ESP tunnel mode:
   • Creates new IP header with Gateway A → Gateway B addresses
   • Encrypts original packet as ESP payload
   • Computes authentication tag
5. Packet traverses Internet with gateway addresses visible
6. Gateway B receives packet (its own IP address as destination)
7. Gateway B looks up SAD by SPI → finds matching SA
8. Gateway B verifies authentication, decrypts, extracts original packet
9. Gateway B routes original packet to Host B based on inner IP header
10. Host B (10.0.2.7) receives original packet—unaware IPSec was involved

Tunnel Mode and NAT:

Tunnel mode handles NAT more gracefully than transport mode:

• The outer IP header can be NATed without affecting the encrypted interior
• With NAT-T (NAT Traversal), ESP is encapsulated in UDP port 4500
• The inner packet remains pristine—NAT never touches it

This is why tunnel mode is used for remote access VPNs where users connect from behind NAT (home routers, hotel networks, etc.).

AH Tunnel Mode:

While rarely used, AH can also operate in tunnel mode:

| New IP Header | AH Header | Original IP Header | TCP Header | Data |
  Src: GW_A       SPI+Seq+ICV  Src: 10.0.1.5
  Dst: GW_B       Protocol: 51 Dst: 10.0.2.7
  Protocol: 51    ←─────────────────── Authenticated ─────────────────────→

AH provides authentication and integrity but no encryption—the original packet is visible. Still, interior addresses are contained in authenticated (not encrypted) data, providing some topology hiding.

Understanding when to use each mode requires examining multiple factors: overhead, security properties, deployment requirements, and operational considerations.

Overhead Analysis:

The overhead difference between modes impacts network performance, especially for small packets.

Transport Mode Overhead:

ESP Header:           8 bytes (SPI + Seq#)
ESP Trailer:          2-16 bytes (padding to block boundary + 2 byte trailer)
Authentication Tag:   16 bytes (typical for GCM-128)
Total:                26-40 bytes

Tunnel Mode Overhead:

Outer IP Header:      20 bytes (IPv4) or 40 bytes (IPv6)
ESP Header:           8 bytes
ESP Trailer:          2-16 bytes
Authentication Tag:   16 bytes
Total:                46-60 bytes (IPv4) or 66-80 bytes (IPv6)

MTU Considerations:

With 1500-byte Ethernet MTU:

• Transport mode: ~1460 bytes available for original payload
• Tunnel mode: ~1440 bytes available for original packet

For applications generating 1400+ byte packets, tunnel mode may trigger fragmentation or require Path MTU Discovery adjustments. This is particularly relevant for:

• Database traffic (large query results)
• File transfers (NFS, SMB)
• Backup applications

Performance Trade-off: Tunnel mode's additional overhead is frequently worth accepting for:

• Deployment simplicity (no endpoint changes)
• Security benefits (address hiding)
• Operational flexibility (policy changes at gateway only)

Let's examine specific deployment scenarios and the mode selection logic for each:

Understanding the exact byte-level structure of IPSec packets is essential for troubleshooting, packet analysis, and understanding what firewalls and NAT devices see.

ESP Header Format:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|               Security Parameters Index (SPI)                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sequence Number                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Payload Data (variable)                     |
~                                                                 ~
|                                                                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Padding (0-255 bytes)                                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Pad Length   |  Next Header  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Integrity Check Value (ICV) (variable)                 |
~                                                                 ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Encryption Boundaries:

| IP Header | ESP Header | Payload + Padding + Pad Len + Next Hdr | ICV |
            ↑___________|_________________________________________|
                     Not encrypted (ESP header is plaintext)
            
| IP Header | ESP Header |           Encrypted Data                | ICV |
                         ↑________________________________________↑
                                    Encrypted portion

| IP Header | ESP Header | Encrypted: Payload + Pad + PL + NH     | ICV |
            ↑_____________|_________________________________________↑
                              Authenticated portion (includes ESP header)

Key Points:

• ESP Header (SPI + Seq#) is transmitted in cleartext—necessary for receiver to identify SA
• Payload, padding, pad length, and next header are encrypted
• Entire ESP packet (header + encrypted data, excluding ICV) is authenticated
• ICV placement varies: typically at end, but always calculated over preceding data

NAT-Traversal (NAT-T) Encapsulation:

When NAT is detected during IKE negotiation, ESP packets are encapsulated in UDP:

| IP Header | UDP Header | Non-ESP Marker | ESP Packet |
            port 4500     (zeros if ESP)

The UDP encapsulation allows NAT devices to translate addresses without breaking ESP authentication (they only modify the outer UDP/IP headers, not the ESP content inside).

Selecting the appropriate IPSec mode should follow a systematic decision process. Here's a framework for common scenarios:

Decision Criteria Summary:

The Default Choice:

When in doubt, Tunnel Mode is the safer default:

• Works in more scenarios
• Doesn't require endpoint modification
• Provides better traffic flow confidentiality
• Modern hardware easily handles the overhead

Choose Transport Mode only when:

• Both endpoints are under your control AND
• IPSec can be installed on both AND
• No NAT is involved (or NAT-T is acceptable) AND
• Overhead reduction justifies deployment complexity

We've examined IPSec's two operational modes in depth. Let's consolidate the key insights:

What's Next:

With a solid understanding of IPSec modes, we now turn to the specific protocols that implement IPSec security services. The next page examines the Authentication Header (AH) and Encapsulating Security Payload (ESP) protocols—their formats, capabilities, and the trade-offs between them.