Network Virtualization: Abstracting Physical Infrastructure

Of all overlay encapsulation protocols developed for network virtualization, VXLAN (Virtual eXtensible LAN) has emerged as the undisputed industry standard. Originally developed by VMware, Cisco, and other vendors and standardized in RFC 7348, VXLAN provides the encapsulation mechanism that enables Layer 2 networks to span across Layer 3 infrastructure at massive scale.

VXLAN's dominance stems from several key advantages:

• Simplicity: Straightforward UDP encapsulation without complex state machines
• Hardware offload: Widely supported by NIC and switch ASICs for line-rate performance
• ECMP compatibility: UDP encapsulation enables seamless ECMP load balancing
• 24-bit VNI space: Over 16 million isolated networks vs. VLAN's 4,094
• Industry adoption: Universal support from VMware, Microsoft, Linux, all major switch vendors

This page will take you through every aspect of VXLAN—from packet format to control plane options to production deployment patterns. By the end, you'll have the complete knowledge needed to design, deploy, and troubleshoot VXLAN-based overlay networks.

VXLAN uses UDP encapsulation to carry Layer 2 Ethernet frames across Layer 3 networks. Understanding the exact packet structure is essential for troubleshooting and understanding protocol behavior.

Complete VXLAN Packet Structure

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Outer Ethernet Header (14 bytes)             |
|      Destination MAC      |       Source MAC      | Type      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Outer IP Header (20 bytes)                 |
| Ver | IHL | ToS |     Total Length    | Identification        |
| Flags | Fragment Offset | TTL | Proto=17 | Header Checksum    |
|                    Source IP Address                          |
|                  Destination IP Address                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Outer UDP Header (8 bytes)                 |
|       Source Port (hash)      |   Dest Port = 4789            |
|         UDP Length            |      UDP Checksum             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    VXLAN Header (8 bytes)                     |
|R|R|R|R|I|R|R|R|         Reserved (24 bits)                    |
|               VXLAN Network Identifier (VNI) 24 bits          | Reserved
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Inner Ethernet Frame                         |
|  (Original L2 frame with Dest MAC, Src MAC, Type, Payload)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Header Field Details

Outer Ethernet Header (14 bytes)

• Standard Ethernet header with underlay MACs
• Destination MAC: Next-hop router MAC (not the remote VTEP)
• Source MAC: Local NIC MAC
• EtherType: 0x0800 (IPv4) or 0x86DD (IPv6)

Outer IP Header (20 bytes for IPv4)

• Source IP: Local VTEP IP address
• Destination IP: Remote VTEP IP address
• Protocol: 17 (UDP)
• TTL: Typically 64; should be sufficient for underlay path

Outer UDP Header (8 bytes)

• Source Port: Hash of inner frame fields (provides ECMP entropy)
• Destination Port: 4789 (IANA-assigned VXLAN port)
• Checksum: Often set to 0 for performance (valid per RFC 768)

VXLAN Header (8 bytes)

• Flags (8 bits): Bit 4 (I flag) must be 1 to indicate valid VNI
• Reserved: 24 bits, must be 0
• VNI (24 bits): VXLAN Network Identifier (0 to 16,777,215)
• Reserved: 8 bits, must be 0

The VXLAN Network Identifier (VNI) is the core segmentation mechanism in VXLAN. Each VNI defines an isolated Layer 2 broadcast domain, allowing millions of independent virtual networks to coexist on the same physical infrastructure.

VNI Address Space

With 24 bits, VNIs range from 0 to 16,777,215:

• VNI 0 is typically reserved/unused
• VNIs 1-4095 may be mapped 1:1 to VLANs for simplified management
• VNIs 4096+ are available for overlay-only networks
• Maximum usable VNIs: 16,777,214

This represents a 4,096x increase over VLAN capacity, enabling true multi-tenant cloud scale.

VNI-to-VLAN Mapping

In hybrid environments with both overlay networks and traditional VLANs, VNIs are often mapped to VLANs at the network edge:

Ingress (Physical to Overlay):

• Frame arrives on physical port with VLAN tag 100
• VTEP strips VLAN tag, encapsulates with VNI 100
• Frame enters overlay network in VNI 100

Egress (Overlay to Physical):

• VXLAN frame with VNI 100 arrives at gateway VTEP
• VTEP decapsulates, adds VLAN tag 100
• Frame sent to physical network on VLAN 100

This mapping is configurable—VNI 5000 could map to VLAN 100 if needed.

VNI Isolation Properties

Complete Layer 2 Isolation: VMs in different VNIs cannot communicate at Layer 2, even if they have MAC address collisions. Each VNI is a completely independent Ethernet domain.

Overlapping IP Spaces: Different VNIs can use identical IP subnets. Tenant A's 10.0.0.0/24 in VNI 1000 is completely isolated from Tenant B's 10.0.0.0/24 in VNI 2000.

Independent Policies: Security policies, QoS, and forwarding rules can differ per VNI.

VXLAN as specified in RFC 7348 only defines the data plane—how packets are encapsulated. It leaves the control plane (how VTEPs discover each other and learn VM locations) undefined. This flexibility has led to multiple control plane approaches:

Option 1: Multicast-Based Control Plane

The original VXLAN approach uses IP multicast for BUM (Broadcast, Unknown unicast, Multicast) traffic:

Configuration:

• Each VNI is mapped to an IP multicast group (e.g., VNI 5000 → 239.1.1.5)
• VTEPs join the multicast group for VNIs they participate in
• Underlay routers perform multicast routing (PIM-SM typically)

BUM Traffic Handling:

• VTEP sends BUM traffic to the VNI's multicast group
• Underlay multicast infrastructure delivers to all member VTEPs
• Recipient VTEPs learn source MAC → source VTEP from flooded frames

Advantages:

• Efficient multicast delivery (underlay replicates, not source VTEP)
• Follows traditional Ethernet learning semantics

Disadvantages:

• Requires multicast-capable underlay (not all networks support it)
• Multicast adds operational complexity
• Multicast groups consume TCAM resources on switches

Option 2: Ingress Replication (Head-End Replication)

When multicast isn't available, the source VTEP can unicast BUM traffic to each remote VTEP:

Configuration:

• Each VTEP maintains a list of peer VTEPs per VNI
• List is statically configured or provided by a controller

BUM Traffic Handling:

• VTEP creates copies of BUM frame for each peer
• Sends unicast encapsulated frame to each peer VTEP
• N copies for N peer VTEPs

Advantages:

• Works on any IP network (no multicast needed)
• Simpler underlay requirements

Disadvantages:

• N×M replication load on source VTEP for large VNIs
• Increased underlay bandwidth consumption
• Scales poorly beyond a few hundred VTEPs per VNI

Option 3: BGP EVPN Control Plane

The gold standard for production VXLAN deployments is EVPN (Ethernet VPN) with BGP as the control plane:

How It Works:

• VTEPs peer with BGP route reflectors (or each other in small deployments)
• When a VM is learned locally, VTEP advertises EVPN Type-2 route:
  • MAC address
  • Optionally: IP address (for ARP suppression)
  • VNI
  • Next-hop (VTEP IP)
• Remote VTEPs install the route in their forwarding tables
• BUM traffic uses selective ingress replication or EVPN multicast

EVPN Route Types for VXLAN:

• Type-2 (MAC/IP Advertisement): Advertises VM MAC and IP
• Type-3 (Inclusive Multicast Ethernet Tag): Joins VTEP to VNI flood domain
• Type-5 (IP Prefix): Advertises routed IP prefixes (inter-VNI routing)

Advantages:

• No flooding for known unicast: Eliminates broadcast storms from MAC learning
• ARP suppression: EVPN Type-2 routes with IP enable local ARP response
• Multi-vendor interoperability: IETF standard (RFC 8365)
• Scales massively: BGP route reflector hierarchy handles millions of routes
• Integrated routing: Same control plane for L2 and L3

BGP EVPN provides a sophisticated, standards-based control plane for VXLAN. Let's examine how EVPN routes enable efficient VXLAN operation.

EVPN Route Type 2: MAC/IP Advertisement

This is the workhorse route type for VXLAN. When a VM powers on or is detected by a VTEP:

Route Type 2 - MAC/IP Advertisement
├── Route Distinguisher: 192.168.1.10:100 (VTEP + VNI combo)
├── Ethernet Segment Identifier: 0 (single-homed)
├── Ethernet Tag ID: 0
├── MAC Address Length: 48
├── MAC Address: 00:50:56:01:02:03
├── IP Address Length: 32 (or 0 if MAC-only)
├── IP Address: 10.0.1.50
├── Extended Communities:
│   ├── Route Target: 64512:5000 (imported by VNI 5000 members)
│   └── Encapsulation: VXLAN
└── Next Hop: 192.168.1.10 (VTEP IP, used as tunnel destination)

The Route Target (RT) determines which VTEPs import the route. VTEPs configured for VNI 5000 import routes with RT 64512:5000.

EVPN Route Type 3: Inclusive Multicast Ethernet Tag (IMET)

Type-3 routes allow VTEPs to discover each other for BUM traffic replication:

Route Type 3 - Inclusive Multicast
├── Route Distinguisher: 192.168.1.10:100
├── Ethernet Tag ID: 0
├── IP Address Length: 32
├── Originating Router's IP: 192.168.1.10 (VTEP IP)
├── Extended Communities:
│   ├── Route Target: 64512:5000
│   └── PMSI Tunnel Attribute:
│       ├── Tunnel Type: Ingress Replication
│       └── Tunnel Endpoint: 192.168.1.10
└── Importing VTEPs add this VTEP to VNI 5000 flood list

When a VTEP needs to flood BUM traffic, it sends copies to all VTEPs that advertised Type-3 routes for that VNI.

EVPN Route Type 5: IP Prefix Route

Type-5 routes enable inter-VNI (inter-subnet) routing at the VTEP:

Route Type 5 - IP Prefix
├── Route Distinguisher: 192.168.1.10:100
├── Ethernet Tag ID: 0
├── IP Prefix Length: 24
├── IP Prefix: 10.0.1.0/24
├── Gateway IP: 0.0.0.0 (or specific gateway)
├── MPLS Label: VNI for L3 routing (L3VNI)
├── Extended Communities:
│   ├── Route Target: 64512:3000 (L3 VRF RT)
│   └── Router MAC: 00:00:00:11:22:33 (for inter-subnet routing)
└── Enables distributed routing between subnets

Let's examine practical VXLAN configuration on common platforms. We'll cover Open vSwitch, Linux native bridge, and summarize switch configurations.

Open vSwitch VXLAN Configuration

OVS is the most common software VTEP, used extensively in OpenStack, Kubernetes, and other cloud platforms.

Linux Native VXLAN Configuration

Linux kernel includes native VXLAN support via the ip command, useful for lightweight deployments without OVS.

Software VXLAN processing consumes CPU cycles—encapsulation, decapsulation, and checksum calculations add up at high packet rates. Hardware offload moves these operations to NIC ASICs, dramatically reducing CPU load and increasing throughput.

Offload Capabilities

Stateless Offloads (widely supported)

TX Offloads (transmit path):

• TSO (TCP Segmentation Offload): NIC segments large TCP payloads, adding VXLAN headers to each segment
• TX Checksum Offload: NIC calculates inner and outer checksums
• VXLAN Encapsulation: Some NICs can add VXLAN headers in hardware

RX Offloads (receive path):

• LRO/GRO (Large Receive Offload): NIC aggregates multiple VXLAN packets into large payloads
• RX Checksum Verification: NIC validates inner and outer checksums
• VXLAN Decapsulation: NIC strips VXLAN headers, presents inner frame to host
• RSS (Receive Side Scaling): NIC distributes flows across multiple CPU queues based on inner flow hash

Stateful/Flow Offloads (SmartNICs)

Advanced NICs (NVIDIA Mellanox ConnectX, Intel IPU, Broadcom Stingray) support TC flower offload or OVS-DPDK offload:

• NIC ASIC stores flow tables (match patterns + actions)
• Matching packets are processed entirely in NIC, never reaching host CPU
• Can achieve line-rate forwarding (100Gbps+) on commodity servers

VXLAN troubleshooting requires systematic approach, working from physical connectivity up through the overlay stack. Here's a comprehensive troubleshooting methodology.

Troubleshooting Layers

Layer 1: Underlay Connectivity

1. Verify IP reachability between VTEP IP addresses
2. Check for MTU issues (ICMP fragmentation needed errors)
3. Ensure UDP port 4789 is not blocked by firewalls

Layer 2: VXLAN Tunnel Status

1. Verify tunnel interface is up
2. Check for encapsulation/decapsulation errors
3. Validate VNI configuration consistency

Layer 3: Control Plane

1. Verify MAC learning (FDB entries exist)
2. Check EVPN route exchange (if applicable)
3. Validate ARP suppression cache

Layer 4: Application Connectivity

1. Test end-to-end VM connectivity
2. Check for MTU-related application failures
3. Verify security group/ACL rules

VXLAN as specified in RFC 7348 provides no inherent security—frames are encapsulated in plain UDP without encryption or authentication. Understanding these security implications is critical for production deployments.

Security Concerns

1. No Encryption VXLAN traffic is transmitted in cleartext. Anyone with access to the underlay network (physical switch port mirror, compromised router) can:

• See all encapsulated frame contents
• Extract sensitive data (credentials, personal information)
• Monitor VM-to-VM communication patterns

2. No Authentication VTEPs don't authenticate each other. A rogue VTEP can:

• Inject frames into any VNI (VNI is just a number in the header)
• Impersonate legitimate VMs by sending frames with spoofed source MACs
• Flood VNIs with spurious traffic (DoS)

3. VNI Space Vulnerability VNIs are not a security boundary—they're a logical segmentation mechanism. An attacker who can send VXLAN packets can target any VNI simply by changing the VNI field.

Mitigation Strategies

Underlay Segmentation

• Run VXLAN only on dedicated underlay networks isolated from untrusted traffic
• Use infrastructure VLANs accessible only to trusted hosts
• Implement strict access control on physical switch ports

IPsec/WireGuard Encapsulation For highly sensitive traffic, encrypt VXLAN at the underlay:

VM → VTEP → IPsec Tunnel → Underlay → IPsec Tunnel → VTEP → VM

This adds encryption and authentication but increases overhead and complexity.

MACsec (802.1AE) Hardware-based encryption at Layer 2, providing line-rate encryption between adjacent devices. Effective for securing underlay links.

Firewall and Security Groups Even if VXLAN itself isn't secure, apply rigorous firewall rules:

• Drop VXLAN traffic from unexpected sources (non-VTEP IPs)
• Implement VM-level security groups regardless of VNI isolation
• Use network IDS/IPS to detect anomalous VXLAN patterns

VXLAN has become the universal language of overlay networking—the protocol that enables massive-scale network virtualization across datacenter and cloud environments. Its combination of simplicity, scalability, and broad industry support has made it the dominant choice for production deployments.

Next Up: We'll explore Network Segmentation—the design patterns and technologies for dividing networks into isolated security and administrative domains, using the overlay and virtual switching capabilities we've covered to implement robust multi-tier architectures.