Virtual Private Networks (VPN)

VPN is not a single technology but an architecture built on various protocols that provide encryption, authentication, and encapsulation. The choice of VPN protocol affects everything: security strength, performance characteristics, firewall traversal capability, and operational complexity.

Networking professionals encounter a alphabet soup of VPN protocols: IPSec, IKE, ESP, AH, L2TP, PPTP, SSTP, OpenVPN, WireGuard, GRE. Understanding these protocols—what each does, how they relate to each other, and when to use which—is essential for designing secure, performant VPN deployments.

This page dissects the major VPN protocols in depth. We'll examine the IPSec protocol suite that dominates enterprise site-to-site VPN, the SSL/TLS-based solutions that excel at remote access, the modern WireGuard protocol that challenges established approaches, and legacy protocols you may encounter in existing deployments. By the end, you'll understand not just what these protocols do, but why they were designed as they were and how to choose appropriately among them.

IP Security (IPSec) is a framework of protocols that provides security services at the IP layer (Layer 3). Standardized by the IETF, IPSec is the dominant protocol for site-to-site VPN and a major option for remote access.

Why IPSec Operates at Layer 3

Operating at the network layer provides fundamental advantages:

• Protocol independence: IPSec can protect any IP-layer traffic—TCP, UDP, ICMP, GRE, etc.—without modification to higher-layer protocols
• Transparency: Applications don't need modification; they're unaware IPSec is providing security
• Efficiency: Encryption happens once per packet, regardless of application protocol
• Flexibility: Can protect host-to-host, host-to-gateway, or gateway-to-gateway communication

IPSec Suite Components

IPSec is not a single protocol but a suite of protocols that work together:

1. IKE (Internet Key Exchange): Negotiates security parameters and establishes shared keys. Uses UDP port 500 (or 4500 with NAT-T).

2. ESP (Encapsulating Security Payload): Provides encryption, authentication, and integrity of payloads. IP Protocol 50.

3. AH (Authentication Header): Provides authentication and integrity (no encryption). IP Protocol 51. Rarely used today.

4. SA (Security Association): A logical connection defining security parameters between peers. Not a protocol but a critical concept.

5. NAT-T (NAT Traversal): Encapsulates ESP in UDP to traverse NAT devices. UDP port 4500.

These components work in concert: IKE negotiates SAs that define how ESP (or AH) will protect traffic.

Security Associations (SA)

A Security Association is the foundation of IPSec—a set of parameters that define how to protect traffic between peers:

SA Components:

• Security Parameters Index (SPI): 32-bit identifier for the SA
• IP destination address: Peer endpoint
• Security protocol: ESP or AH
• Encryption algorithm and key: AES-256 with specific key
• Integrity algorithm and key: SHA-384 HMAC with specific key
• Lifetime: Time or data limit before rekeying
• Mode: Tunnel or transport

SA Databases:

• SAD (Security Association Database): Stores active SAs with their parameters
• SPD (Security Policy Database): Defines which traffic should be protected and how

SA Direction:

• SAs are unidirectional—separate SAs for inbound and outbound
• A bidirectional IPSec tunnel requires minimum 2 SAs (one each direction)
• With IKE, you have 4+ SAs: IKE SA pair + IPSec SA pair(s)

IPSec Modes: Tunnel vs. Transport

Tunnel Mode:

• Entire original IP packet is encapsulated
• New IP header prepended (with gateway addresses)
• Original packet's source/destination hidden
• Used for site-to-site and remote access VPN

Transport Mode:

• Only IP payload is protected
• Original IP header preserved (with source/destination visible)
• Used for host-to-host communication
• Lighter overhead but reveals endpoints

Internet Key Exchange (IKE) is the protocol that negotiates and establishes Security Associations for IPSec. It performs authenticated key exchange so that both peers share cryptographic keys without transmitting them over the network.

IKE Phases

Phase 1: IKE SA Establishment

Phase 1 creates a secure, authenticated channel between peers that protects Phase 2 negotiations:

1. Peers propose security parameters (encryption, integrity, DH group)
2. Peers exchange Diffie-Hellman public values
3. Peers authenticate each other (PSK or certificates)
4. Shared secret is derived from DH exchange
5. IKE SA is established

The IKE SA has its own encryption and integrity keys, used to protect subsequent negotiations.

Phase 2: IPSec SA Establishment

Phase 2 negotiates the actual IPSec SAs that will protect user traffic:

1. Runs over the secure IKE SA from Phase 1
2. Peers propose IPSec security parameters (ESP with AES-256-GCM, etc.)
3. Optional additional DH exchange (Perfect Forward Secrecy)
4. Peers agree on traffic selectors (which traffic is protected)
5. IPSec SAs established (one per direction, per selector)

IKEv1 vs. IKEv2

IKEv2 (RFC 7296) was developed to address IKEv1's complexity and limitations:

IKEv2 Exchange Flow

Initiator                                   Responder
    |                                           |
    |---------- IKE_SA_INIT Request ----------->| (DH, nonces, proposals)
    |<--------- IKE_SA_INIT Response -----------| (DH, nonces, selected proposal)
    |                                           |
    |    [IKE SA established - encrypted from here]
    |                                           |
    |---------- IKE_AUTH Request -------------->| (identity, auth, traffic selectors)
    |<--------- IKE_AUTH Response --------------| (identity, auth, traffic selectors)
    |                                           |
    |    [IPSec CHILD_SA established]
    |                                           |
    |<======== Encrypted User Traffic =========>|

Four messages complete initial setup (vs. 6-9 for IKEv1).

IKE Authentication Methods

Pre-Shared Key (PSK):

• Identical secret configured on both peers
• Simple but key distribution is challenging
• Compromise of one peer's configuration exposes key
• Suitable for static site-to-site with few peers

RSA/ECDSA Certificates:

• Each peer has private key; trusts partner's CA
• Scalable to many peers
• Revocation handling is possible
• More complex PKI infrastructure required

EAP (Extensible Authentication Protocol) - IKEv2 only:

• Allows username/password, token-based, or other authentication
• Useful for remote access (user identity, not just machine)
• Gateway typically uses certificate; client uses EAP

Perfect Forward Secrecy (PFS)

PFS performs additional Diffie-Hellman exchange during Phase 2:

• Each IPSec SA gets unique keys not derivable from IKE SA keys
• Compromise of long-term authentication keys doesn't reveal past session traffic
• Adds computational overhead (DH calculation per SA)
• Strongly recommended for high-security environments

Encapsulating Security Payload (ESP) (RFC 4303) is the IPSec protocol that actually protects user data. It provides confidentiality (encryption), data-origin authentication, and integrity.

ESP Packet Structure

ESP encapsulates the protected data with header and trailer fields:

| IP Header (new in tunnel mode) | ESP Header | Encrypted Payload | ESP Trailer | ESP Auth |
                                 |   (8 bytes)|                   |  (variable) | (variable)|

ESP Header:
├─ Security Parameters Index (SPI): 32 bits - Identifies the SA
└─ Sequence Number: 32 bits - Anti-replay protection

ESP Trailer (encrypted with payload):
├─ Padding: 0-255 bytes - Block cipher alignment
├─ Pad Length: 8 bits
└─ Next Header: 8 bits - Protocol of encapsulated packet

ESP Authentication Data:
└─ ICV (Integrity Check Value): Variable (typically 96-256 bits)

What Gets Encrypted and Authenticated

                    |<-------- Encrypted -------->|
| IP Header | ESP Header | Payload | ESP Trailer | ICV |
            |<----------- Authenticated ---------->|

• Encrypted: Original packet (in tunnel mode) or payload (transport mode), plus ESP trailer
• Authenticated: ESP header, encrypted payload, and ESP trailer
• Not encrypted: Outer IP header, ESP header (SPI and sequence number visible)
• Not authenticated: Outer IP header (can be modified by NAT)

ESP Encryption Algorithms

Recommended (Modern):

• AES-GCM (128, 192, or 256-bit): Authenticated encryption - provides both confidentiality and integrity in single operation. High performance with hardware acceleration. Strongly preferred.
• AES-CCM: Alternative authenticated encryption mode.
• ChaCha20-Poly1305: Software-optimized alternative, excellent for devices without AES hardware acceleration.

Legacy (Still Supported):

• AES-CBC (with separate HMAC): Older but still secure when properly implemented.
• 3DES: Deprecated due to small block size vulnerability, slow performance.
• DES: Obsolete - never use.
• NULL: No encryption (authentication only) - special use cases only.

ESP Integrity Algorithms

When using AES-GCM or other AEAD:

• Integrity is built into the algorithm

When using non-AEAD encryption:

• HMAC-SHA-256 / HMAC-SHA-384 / HMAC-SHA-512: Recommended
• HMAC-SHA-1: Deprecated but still common in legacy systems
• AES-XCBC-MAC: Alternative to HMAC

Anti-Replay Protection

ESP's sequence number field enables anti-replay:

1. Sender increments sequence number for each packet
2. Receiver maintains "replay window" (typically 64-1024 packets)
3. If received sequence number is:
   • Below window left edge: Reject (too old)
   • Already seen within window: Reject (replay)
   • Within window, not seen: Accept, mark seen
   • Beyond window right edge: Accept, advance window

Extended Sequence Numbers (ESN) provide 64-bit sequence numbers for high-speed links where 32-bit (4 billion packets) might wrap.

Authentication Header (AH) (RFC 4302) provides data-origin authentication and integrity protection without encryption. While largely superseded by ESP (which can provide authentication without encryption using NULL cipher), understanding AH is important for completeness and legacy systems.

AH Packet Structure

| IP Header | AH | Payload |

AH Fields:
├─ Next Header: 8 bits - Protocol of following header
├─ Payload Length: 8 bits - Length of AH in 32-bit words minus 2
├─ Reserved: 16 bits
├─ Security Parameters Index (SPI): 32 bits
├─ Sequence Number: 32 bits
└─ Integrity Check Value (ICV): Variable length

What AH Authenticates

AH authenticates more than ESP:

• The entire IP header (except mutable fields)
• The AH header itself
• The entire payload

Mutable fields excluded:

• TTL/Hop Limit (changes at each router)
• Header checksum (recalculated when TTL changes)
• TOS/DSCP (may be modified by QoS policies)

AH vs. ESP Comparison

Why AH is Rarely Used Today

AH's primary distinguishing feature—IP header authentication—is also its fatal flaw in modern networks:

1. NAT Incompatibility: NAT devices modify IP addresses, breaking AH authentication. Since NAT is ubiquitous, AH is impractical for most deployments.

2. ESP Can Do Everything AH Can (Almost): ESP with NULL encryption provides integrity and authentication without confidentiality. The minor gap (IP header coverage) isn't worth NAT incompatibility.

3. Overhead Without Encryption Benefit: If you're going to add header overhead, you might as well encrypt.

When AH Might Still Be Seen

• Legacy systems configured before ESP-NULL was common
• Regulatory requirements mandating header integrity (rare)
• Internal networks with no NAT (uncommon)
• Combined with ESP (ESP-AH bundles) for defense-in-depth (very rare, adds significant complexity)

Practical Recommendation

Don't use AH. Use ESP with AES-GCM for encryption + authentication, or ESP with NULL cipher and HMAC-SHA-256 if you somehow need integrity without confidentiality. AH's NAT incompatibility makes it impractical for virtually all modern deployments.

SSL/TLS VPN uses the same Transport Layer Security (TLS) protocol that secures HTTPS websites to create VPN tunnels. This approach offers significant advantages for remote access, particularly firewall traversal.

TLS VPN Architecture

SSL/TLS VPNs typically operate in one of two modes:

Full Tunnel Mode:

• Client software creates virtual network interface
• IP traffic is encapsulated and encrypted over TLS
• Functions like traditional VPN with full network access
• Requires client software installation

Clientless (Portal) Mode:

• User accesses web portal via browser
• Gateway intercepts requests and proxies to internal resources
• Usually limited to HTTP/HTTPS applications
• No client installation required
• Increasingly supports RDP, SSH, and other protocols via HTML5

OpenVPN

OpenVPN is the most widely used open-source VPN solution:

Architecture:

• Uses TLS for key exchange and control channel
• Custom protocol for data channel (can use TLS or custom encryption)
• Single UDP or TCP port (commonly UDP 1194 or TCP 443)
• Cross-platform client support (Windows, macOS, Linux, iOS, Android)

Protocol Details:

• Control channel: TLS 1.2/1.3 for authentication and key exchange
• Data channel: Configurable encryption (typically AES-256-GCM)
• Supports tun (IP tunnel) and tap (Ethernet bridge) modes
• HMAC for packet authentication
• Static-key or certificate-based authentication

Cisco AnyConnect

Cisco's enterprise SSL VPN solution:

Architecture:

• DTLS (Datagram TLS) or TLS for transport
• DTLS preferred (UDP-based) for better performance
• Falls back to TLS (TCP) when DTLS blocked
• Tight integration with Cisco security ecosystem

Unique Features:

• Always-on VPN capabilities
• Posture assessment integration
• Roaming between networks with session preservation
• Web security module, network access control
• Extensive enterprise management

SSTP (Secure Socket Tunneling Protocol)

Microsoft's SSL VPN protocol built into Windows:

Architecture:

• PPP frames encapsulated in HTTPS (TLS)
• Uses TCP port 443 exclusively
• Native Windows client support
• Server support on Windows Server RRAS

Characteristics:

• Excellent firewall traversal (looks like HTTPS)
• TCP-only means performance limitations
• Windows-centric, limited cross-platform support
• Good integration with Windows authentication (AD, NPS)

SSL/TLS VPN vs. IPSec Comparison

WireGuard is a modern VPN protocol that has gained significant traction due to its simplicity, performance, and cryptographic elegance. Designed by Jason Donenfeld, WireGuard takes a fundamentally different approach from IPSec and OpenVPN.

Design Philosophy

WireGuard was designed with several principles:

1. Simplicity: ~4,000 lines of code (vs. 100,000+ for OpenVPN or IPSec implementations). Small codebase is easier to audit.

2. Modern Cryptography: Uses only state-of-the-art primitives—no algorithm negotiation or legacy support.

3. Protocol Silence: Doesn't respond to unauthenticated packets, making it invisible to network scanners.

4. Roaming: Handles endpoint address changes seamlessly.

5. Minimal State: Stateless operation where possible, reducing attack surface.

Cryptographic Primitives

WireGuard uses a fixed cryptographic suite (no negotiation):

• Key Exchange: Curve25519 Elliptic Curve Diffie-Hellman
• Symmetric Encryption: ChaCha20-Poly1305 (AEAD)
• Hashing: BLAKE2s
• Key Derivation: HKDF

No cipher suite negotiation means:

• Simpler protocol, fewer attack vectors
• If cryptographic break discovered, upgrade entire protocol
• No downgrade attacks possible

WireGuard Configuration Model

WireGuard configuration is remarkably simple:

[Interface]
PrivateKey = <base64-encoded private key>
Address = 10.200.200.1/24
ListenPort = 51820

[Peer]
PublicKey = <base64-encoded public key of peer>
AllowedIPs = 10.200.200.2/32
Endpoint = peer.example.com:51820

Key Concepts:

• Interface: Local WireGuard tunnel interface with private key
• Peer: Each remote endpoint, identified by public key
• AllowedIPs: What source IPs can come from this peer; also serves as routing table
• Endpoint: Where to send packets for this peer (optional for initiator-only peers)

Cryptokey Routing

WireGuard's "AllowedIPs" serves dual purpose:

1. Inbound filtering: Only accept packets from peer if source IP is in AllowedIPs
2. Routing: When sending packet, find peer whose AllowedIPs includes destination

This elegant design replaces complex routing configuration with simple association of IP ranges to public keys.

Performance Characteristics

• Extremely fast handshake (1-RTT)
• Very low CPU usage due to efficient cryptography
• ChaCha20-Poly1305 excellent for devices without AES hardware
• Built into Linux kernel (since 5.6) for maximum performance
• Typically faster than OpenVPN, competitive with highly optimized IPSec

Enterprise Considerations

WireGuard was designed for simplicity, which means some enterprise features are external:

• User authentication: WireGuard uses key pairs, not usernames. Enterprise solutions (like Tailscale, NetBird) add user identity layer.
• Logging: Minimal logging by design. Enterprise wrappers add auditing.
• Dynamic IP assignment: Basic WireGuard uses static IPs. Management layers add DHCP-like functionality.
• Centralized management: Third-party tools required for fleet management.

Several VPN protocols, while still encountered in some environments, are considered legacy or deprecated. Understanding these is important for migration planning and security assessments.

PPTP (Point-to-Point Tunneling Protocol)

PPTP was developed by Microsoft and widely deployed in the 1990s-2000s:

Architecture:

• Control channel: TCP port 1723
• Data channel: GRE (IP Protocol 47)
• Encryption: MPPE (Microsoft Point-to-Point Encryption) based on RC4
• Authentication: MS-CHAPv2 (most common)

Security Problems:

• MS-CHAPv2 is broken—can be cracked to recover password
• MPPE/RC4 encryption has significant weaknesses
• No server authentication in typical configurations
• Man-in-the-middle attacks practical

PPTP should never be used. It is fundamentally insecure, with practical attacks publicly demonstrated. Any remaining PPTP deployments should be migrated immediately.

L2TP/IPSec

Layer 2 Tunneling Protocol (L2TP) combined with IPSec:

Architecture:

• L2TP creates tunnel (encapsulates PPP)
• IPSec (ESP) encrypts L2TP traffic
• UDP port 1701 for L2TP, ports 500/4500 for IPSec

Characteristics:

• L2TP itself has no encryption—relies entirely on IPSec
• Double encapsulation (PPP in L2TP in IPSec) adds overhead
• Built into Windows, macOS, iOS, Android
• Security depends on IPSec configuration

L2TP/IPSec Security Considerations

L2TP/IPSec security varies significantly by configuration:

Weak Configuration:

• IKEv1 Aggressive Mode with PSK
• Weak DH groups (1, 2, 5)
• 3DES or DES encryption
• Vulnerable to offline attack against PSK

Acceptable Configuration:

• IKEv2 with certificates
• Strong DH groups (14+, or ECC)
• AES-256-GCM encryption
• Modern integrity algorithms

The protocol itself isn't broken, but legacy deployments often use weak parameters. Certificate-based IKEv2 with modern crypto is secure but has significant configuration complexity and overhead compared to alternatives.

GRE (Generic Routing Encapsulation)

GRE is a tunneling protocol without inherent security:

• Encapsulates IP packets with simple 4-8 byte header
• No encryption, authentication, or integrity
• Often combined with IPSec for security (IPSec/GRE)
• Used for carrying non-IP protocols or multicast
• Still useful for specific routing scenarios (DMVPN uses GRE)

GRE itself is not a VPN security protocol—it's an encapsulation mechanism.

Migration Recommendations

From PPTP:

• Immediate priority; any replacement is better
• For Windows environments, IKEv2 is built-in
• For cross-platform, OpenVPN or WireGuard

From L2TP/IPSec:

• If using weak configuration, urgent migration
• If using strong IKEv2, can keep but consider simplifying to pure IKEv2
• Benefit from reduced overhead without L2TP layer

VPN protocols are the foundation upon which secure network connectivity is built. Understanding their characteristics enables informed technology selection and secure implementation. Let's consolidate the key knowledge from this page:

What's Next:

The final page in this module examines VPN Security—the threats VPNs face, vulnerabilities in implementations, secure configuration practices, and the security considerations that complement protocol selection. You'll learn about VPN-specific attacks, hardening measures, and how VPN fits within a comprehensive security architecture.