IPSec Overview

Everything we've studied—security services, modes, and protocols—comes together in the IPSec architecture. This architecture defines how policies are specified, how cryptographic parameters are negotiated and stored, and how packets are matched to appropriate security treatments.

The IPSec architecture is fundamentally policy-driven. Rather than applying security uniformly to all traffic, IPSec allows administrators to define granular policies: different security levels for different traffic types, sources, destinations, and applications. This flexibility is both IPSec's greatest strength and its primary complexity factor.

Understanding this architecture is essential for:

• Configuring IPSec correctly (policies must be consistent between peers)
• Troubleshooting connection failures (often policy mismatches)
• Designing secure network architectures
• Understanding how IKE establishes and maintains security relationships

The IPSec architecture, defined in RFC 4301, consists of several interconnected components that work together to provide security services. At the highest level:

Core Components:

Conceptual Flow:

1. Administrator defines security policies in SPD
2. When traffic matches a PROTECT policy, IPSec engine checks for existing SA
3. If no SA exists, IKE negotiates one with the peer
4. Negotiated SA parameters are stored in SAD
5. IPSec engine uses SA parameters to protect/verify packets

Planes of Operation:

Management Plane: Administrator configuration—defining policies, trust anchors (certificates/PSKs), and IKE parameters. This is typically done through configuration files, CLI commands, or management interfaces.

Control Plane: IKE daemon handles automatic key negotiation, SA establishment, and SA lifecycle management (rekey, expiration). IKE uses UDP port 500 (or 4500 for NAT-T) to communicate with peers.

Data Plane: The IPSec engine (kernel module or hardware accelerator) performs the actual packet processing—SPD lookup, SA lookup, encryption/decryption, authentication. This must be extremely fast for line-rate operation.

Separation of Concerns:

This separation is critical for both performance and security:

• Control plane can be complex (asymmetric crypto, certificate validation) because it runs infrequently
• Data plane must be simple and fast (symmetric crypto only) because it runs on every packet
• Management plane can be offline or protected separately from data flows

The Security Policy Database (SPD) is the heart of IPSec's policy-driven architecture. It contains ordered rules that specify what security treatment to apply to what traffic. Every packet processed by an IPSec implementation is evaluated against the SPD.

SPD Entry Components:

Each SPD entry (policy rule) contains:

1. Selectors: Match criteria identifying traffic
2. Action: PROTECT, BYPASS, or DISCARD
3. Security Protocols: For PROTECT entries, which protocols (ESP, AH, or both)
4. Mode: Transport or Tunnel
5. Tunnel Endpoints: For tunnel mode, the gateway addresses
6. Algorithm Suite: Required cryptographic algorithms
7. SA Parameters: Lifetime, PFS requirements, etc.

SPD Actions:

Example SPD Configuration (Conceptual):

# Site-to-site VPN policy
Rule 1: src=10.0.1.0/24, dst=10.0.2.0/24, protocol=any
        Action: PROTECT
        Mode: tunnel
        Endpoints: 203.0.113.1 <-> 198.51.100.1
        Protocol: ESP (AES-256-GCM, SHA-256)
        PFS: Group 19 (256-bit ECP)

# Allow outbound HTTPS without IPSec (TLS provides security)
Rule 2: src=10.0.1.0/24, dst=any, protocol=TCP, dport=443
        Action: BYPASS

# Block SSH from external to internal
Rule 3: src=any, dst=10.0.1.0/24, protocol=TCP, dport=22
        Action: DISCARD

# Default: allow all other traffic
Rule 4: src=any, dst=any, protocol=any
        Action: BYPASS

Inbound vs. Outbound SPD:

IPSec maintains separate SPD entries (or conceptually separate databases) for inbound and outbound traffic:

Outbound SPD (SPD-O): Evaluated when local system generates or forwards a packet. Determines whether to protect, bypass, or discard outgoing traffic.

Inbound SPD (SPD-I): Evaluated when a packet arrives and after any IPSec processing. Ensures that traffic that SHOULD have been protected actually WAS protected. Prevents downgrade attacks where an attacker removes IPSec protection.

Inbound SPD Verification Example:

Policy states: "Traffic from 10.0.2.0/24 to 10.0.1.0/24 MUST be IPSec-protected."

If an unprotected packet arrives with source 10.0.2.5 and destination 10.0.1.10, the inbound SPD check fails—an attacker might have injected this packet bypassing IPSec. The packet is discarded.

SPD Caching (SPD-S):

RFC 4301 introduces SPD-S (SPD cache), which caches the SPD lookup result along with the corresponding SA. This speeds up subsequent packets in the same flow, avoiding repeated SPD searches.

A Security Association (SA) is a unidirectional agreement between two IPSec peers specifying the security parameters for traffic in one direction. SAs are the runtime manifestation of security policies—they contain all the cryptographic material and parameters needed to protect traffic.

Key Characteristics of Security Associations:

• Unidirectional: Separate SAs for each direction of traffic. A bidirectional flow requires two SAs.
• Uniquely Identified: By the tuple (SPI, Destination IP, Protocol)
• Contains Complete Parameters: All cryptographic keys, algorithms, modes, and counters
• Has Lifetime: SAs expire after time or byte limits; must be rekeyed
• Managed by IKE: Typically negotiated and maintained automatically

SA Identification:

An SA is uniquely identified by three values:

1. SPI (Security Parameters Index): 32-bit identifier chosen by the receiving peer during SA negotiation. When the receiver gets a packet, it uses the SPI to find the corresponding SA.

2. Destination IP Address: The destination in the outer (on-the-wire) IP header. For tunnel mode, this is the receiving gateway.

3. Security Protocol: Either ESP (50) or AH (51). An ESP SA and AH SA can have the same SPI without conflict.

Why SPI Alone Isn't Enough:

Multiple peers can (coincidentally) choose the same SPI. Including destination IP disambiguates:

• Peer A chooses SPI 0x12345678 for SA to you
• Peer B also chooses SPI 0x12345678 for SA to you
• Without destination IP, you couldn't tell them apart
• With destination IP, (0x12345678, PeerA, ESP) ≠ (0x12345678, PeerB, ESP)

Actually, in practice, since the SPI is chosen by the receiver, and both SAs terminate at your IP, the destination IP is the same. The disambiguation comes from the combination being unique per negotiation—IKE won't assign the same SPI to two different SAs.

SA Lifecycle:

        ┌─────────────────────────────────────────────┐
        │                                             │
        ▼                                             │
    [No SA] ──► IKE Negotiation ──► [Active SA] ──► [Expire] ──► Rekey
                                          │              │
                                          │              ▼
                                          └───────► [Deleted]

Lifecycle Events:

1. Creation: IKE negotiates SA parameters with peer, exchanges keying material, both sides create matching SAs.

2. Active Use: Packets are encrypted/decrypted using SA parameters. Sequence counter increments.

3. Soft Lifetime Expiration: Configurable threshold (e.g., 80% of hard lifetime). Triggers rekey—IKE negotiates replacement SA before old one expires.

4. Rekey: New SA created with fresh keys. Traffic transitions to new SA. Old SA deleted after grace period.

5. Hard Lifetime Expiration: SA must not be used after hard limit (time or bytes). If not rekeyed, traffic drops.

6. Deletion: SA removed from SAD when no longer needed (explicit delete, expiration, or error).

The Security Association Database (SAD) stores all active Security Associations. It's the runtime repository of security relationships, containing the keying material and parameters needed for packet processing.

SAD Operations:

SAD Lookup Performance:

For high-speed networks processing millions of packets per second, SAD lookup must be extremely fast. Implementations use:

• Hash tables: SPI as primary hash key for O(1) average lookup
• Hardware acceleration: TCAM (Ternary Content-Addressable Memory) in network ASICs
• Caching: SPD-S caches recent (packet flow → SA) mappings

Typical SAD capacity:

• Small router: Hundreds of SAs
• Enterprise VPN concentrator: Thousands of SAs
• Large service provider: Tens of thousands of SAs

SAD Entry Structure (RFC 4301):

For each SA, the SAD contains:

SAD Entry {
    // Identification
    SPI: 32-bit identifier
    Protocol: ESP or AH
    Destination: IP address (outer header destination)
    Source: IP address (for validation, optional)
    
    // Cryptographic Parameters
    EncryptionAlgorithm: e.g., AES-256-GCM
    EncryptionKey: byte array
    IntegrityAlgorithm: e.g., HMAC-SHA-256 (if not AEAD)
    IntegrityKey: byte array (if not AEAD)
    
    // Mode and Tunnel
    Mode: transport or tunnel
    TunnelSource: IP (for tunnel mode)
    TunnelDestination: IP (for tunnel mode)
    
    // Anti-Replay
    SequenceCounter: 32 or 64 bits
    SequenceCounterOverflow: bool (error if true and counters exhausted)
    AntiReplayWindow: bitmap + highest seen
    
    // Lifetime
    LifetimeSeconds: remaining time
    LifetimeBytes: remaining bytes
    
    // Traffic Selectors
    LocalSelector: what local traffic this SA protects
    RemoteSelector: what remote traffic this SA protects
    
    // State
    State: active, expiring, larval, dying
}

SAD Security Considerations:

Key Protection: SAD contains highly sensitive cryptographic keys. Implementations must:

• Store keys in protected memory (not swappable to disk)
• Securely erase keys when SA is deleted (not just unmarked)
• Protect SAD from unauthorized access
• Consider memory encryption for keys at rest

Side-Channel Resistance: SAD access patterns could leak information:

• Which SPIs are active (traffic analysis)
• When SAs are rekeyed (timing analysis)
• SAD implementation must resist cache timing attacks

Concurrent Access: In multi-core systems, multiple threads may access SAD simultaneously:

• Read (lookup) operations can be parallel
• Write (create, update, delete) requires synchronization
• High-performance implementations use lock-free data structures

IKE (Internet Key Exchange) is the control-plane protocol that manages IPSec Security Associations. IKE handles the complex cryptographic negotiations—authentication, key agreement, algorithm selection—so that the data-plane (ESP/AH) can use simple, fast symmetric cryptography.

IKE Versions:

IKEv2 is the modern standard and should be used for all new deployments. IKEv1 exists only for legacy interoperability.

What IKE Does:

1. Mutual Authentication: Verifies each peer's identity using pre-shared keys, certificates, or EAP
2. Key Agreement: Performs Diffie-Hellman exchange to establish shared secret material
3. Parameter Negotiation: Agrees on encryption algorithms, integrity algorithms, and other parameters
4. SA Management: Creates, rekeys, and deletes Security Associations
5. NAT Detection: Determines if NAT exists in path, enabling NAT-T
6. Dead Peer Detection: Detects peer failures for failover scenarios

IKE SA vs. Child SA:

IKE operates with two levels of Security Associations:

IKE SA (Parent SA):

• Protects IKE messages between peers
• Established first in IKE_SA_INIT + IKE_AUTH exchanges
• Contains keys for encrypting/authenticating IKE packets
• Has longer lifetime than Child SAs
• A single IKE SA can create multiple Child SAs

Child SA (IPSec SA):

• The actual ESP/AH SAs that protect traffic
• Created within established IKE SA using CREATE_CHILD_SA
• What we've been calling 'SAs' throughout this module
• Shorter lifetime; rekeyed more frequently
• Traffic selectors define what packets use this Child SA

Initial Exchange: IKE_SA_INIT + IKE_AUTH

A complete IKE session establishment requires 4 messages (2 round trips):

IKE_SA_INIT (Messages 1-2):

Initiator → Responder:
  - SA payload: Proposed algorithms for IKE SA
  - KE payload: Diffie-Hellman public value
  - Ni payload: Initiator nonce

Responder → Initiator:
  - SA payload: Selected algorithms
  - KE payload: Responder's DH public value
  - Nr payload: Responder nonce
  - [CERTREQ]: Optional request for certificates

After this exchange, both parties compute:

• DH shared secret
• SKEYSEED = prf(Ni | Nr, DH_shared)
• SK_d, SK_ai, SK_ar, SK_ei, SK_er, SK_pi, SK_pr derived from SKEYSEED

The IKE SA is now established and subsequent messages are encrypted.

IKE_AUTH (Messages 3-4):

Initiator → Responder (encrypted):
  - IDi: Initiator identity
  - [CERT]: Optional certificate chain
  - AUTH: Authentication payload (signature or PSK-based MAC)
  - SAi2, TSi, TSr: First Child SA proposal + traffic selectors

Responder → Initiator (encrypted):
  - IDr: Responder identity
  - [CERT]: Optional certificate chain
  - AUTH: Response authentication
  - SAr2, TSi, TSr: Selected Child SA + traffic selectors

After IKE_AUTH:

• Both peers are mutually authenticated
• First Child SA is established (ESP/AH ready for traffic)
• Total: 4 messages, 2 round trips

IKE supports multiple authentication methods, each with different security properties, deployment requirements, and operational characteristics.

Pre-Shared Keys (PSK):

Both peers are configured with an identical secret string. The AUTH payload contains a MAC computed over the IKE messages using a key derived from the PSK.

Advantages:

• Simple to configure
• No PKI infrastructure required
• Fast authentication

Disadvantages:

• PSK must be securely distributed to all peers (out of band)
• Same PSK often used for many peers (poor security)
• PSK compromise requires reconfiguration everywhere
• No identity binding (anyone with PSK can impersonate)

Best Practice: Use strong, unique PSKs per peer. Minimum 20 characters random. Never share PSK across multiple sites.

Digital Certificates (RSA/ECDSA):

Peers authenticate using X.509 certificates and corresponding private keys. The AUTH payload contains a digital signature over the IKE messages.

How It Works:

1. During IKE_SA_INIT, peers may request certificates (CERTREQ)
2. In IKE_AUTH, each peer sends:
   • IDi/IDr: Identity (typically certificate's Subject or SAN)
   • CERT: Full certificate chain (or just end-entity cert if chain known)
   • AUTH: Signature over (IKE messages | ID payload) using private key
3. Peer validates:
   • Certificate chain to trusted CA
   • Certificate not expired/revoked
   • Certificate identity matches expected peer
   • Signature verifies correctly

PKI Requirements:

• Certificate Authority (CA) infrastructure
• Certificate issuance process
• Revocation mechanism (CRL, OCSP)
• Certificate distribution to peers

EAP (Extensible Authentication Protocol):

EAP allows integration with external authentication systems:

• EAP-TLS: Client certificate authentication
• EAP-MSCHAPv2: Username/password (for Windows integration)
• EAP-PEAP: Protected EAP (tunneled authentication)
• EAP-RADIUS: Forward authentication to RADIUS server

EAP is asymmetric in IKEv2: the responder (server) authenticates to initiator (client) using certificates, then initiator authenticates using EAP method.

Let's trace a complete IPSec session from initial configuration through active traffic protection:

Scenario: Site-to-site VPN between two corporate locations.

Phase 1: Configuration (Management Plane)

Site A Gateway (203.0.113.1):
  SPD:
    - src=10.0.1.0/24, dst=10.0.2.0/24 → PROTECT
    - ESP tunnel mode to 198.51.100.1
    - AES-256-GCM, SHA-384 PRF, Group 19 DH
  IKE:
    - Peer: 198.51.100.1
    - Auth: PSK "very-strong-random-psk-here"
    - Lifetime: 86400s (IKE), 28800s (Child)

Site B Gateway (198.51.100.1):
  SPD:
    - src=10.0.2.0/24, dst=10.0.1.0/24 → PROTECT
    - ESP tunnel mode to 203.0.113.1
    - AES-256-GCM, SHA-384 PRF, Group 19 DH
  IKE:
    - Peer: 203.0.113.1
    - Auth: PSK "very-strong-random-psk-here"

Phase 2: First Packet Triggers IKE (Control Plane)

1. Host 10.0.1.5 sends packet to 10.0.2.7
2. Packet arrives at Site A Gateway
3. Gateway checks SPD → matches PROTECT rule
4. Gateway checks SAD → no existing SA to Site B
5. Gateway triggers IKE to 198.51.100.1:500

Phase 3: IKE Negotiation

Site A → Site B: IKE_SA_INIT
  - Propose: ENCR_AES_GCM_256, PRF_HMAC_SHA2_384, DH_ECP256
  - DH public value (256-bit ECC point)
  - Initiator nonce (32 bytes)

Site B → Site A: IKE_SA_INIT Response
  - Accept: ENCR_AES_GCM_256, PRF_HMAC_SHA2_384, DH_ECP256
  - DH public value
  - Responder nonce

[Both compute shared secret, derive IKE SA keys]

Site A → Site B: IKE_AUTH (encrypted)
  - ID: 203.0.113.1
  - AUTH: prf(prf(PSK, "Key Pad"), signed octets)
  - Child SA proposal: ESP, ENCR_AES_GCM_256
  - Traffic Selectors: 10.0.1.0/24 ↔ 10.0.2.0/24

Site B → Site A: IKE_AUTH Response (encrypted)
  - ID: 198.51.100.1
  - AUTH: verification
  - Child SA accept: SPI = 0xaabbccdd (Site A's inbound)
  - Traffic Selectors confirmed

Phase 4: SA Installation

Both gateways create matching SAD entries:

Site A SAD Entry (Outbound to Site B):
  SPI: 0x11223344 (Site B's choice)
  Encryption: AES-256-GCM
  Key: [derived from IKE keying material]
  Sequence: 0
  Mode: tunnel
  Tunnel: 203.0.113.1 → 198.51.100.1

Site A SAD Entry (Inbound from Site B):
  SPI: 0xaabbccdd (Site A's choice)
  Encryption: AES-256-GCM
  Key: [derived from IKE keying material]
  Anti-replay window: initialized
  Mode: tunnel

[Mirror entries created at Site B]

Phase 5: Traffic Protection (Data Plane)

The original packet (10.0.1.5 → 10.0.2.7) is now processed:

1. SPD lookup → PROTECT
2. SAD lookup → found SA (SPI 0x11223344)
3. ESP encapsulation:
   • New outer IP: 203.0.113.1 → 198.51.100.1
   • ESP header: SPI=0x11223344, Seq=1
   • Encrypted: original IP packet + padding
   • GCM auth tag
4. Packet sent to 198.51.100.1

At Site B:

1. Receive ESP packet to own IP
2. SAD lookup by SPI 0x11223344 → found
3. Verify auth tag + decrypt
4. Extract original packet (10.0.1.5 → 10.0.2.7)
5. Route to 10.0.2.7 on internal network

We've completed our exploration of IPSec architecture. Let's consolidate the key insights:

Module Complete:

You have now completed the IPSec Overview module. You understand:

• Why IPSec exists and what problems it solves
• The security services IPSec provides
• Transport vs. Tunnel modes and when to use each
• AH and ESP protocols and their trade-offs
• The complete IPSec architecture and how components interact

This foundation prepares you for deeper study of IPSec components in subsequent modules, including Authentication Header details, ESP internals, Security Association management, and VPN design patterns.