Hardware Virtualization Support

CPU and memory virtualization create the foundation for running isolated virtual machines, but real workloads need I/O—network packets, disk blocks, GPU computations. Traditional virtualization emulates devices: the guest talks to a virtual NIC, the hypervisor translates requests to real hardware. This works but adds overhead and complexity.

VT-d (Virtualization Technology for Directed I/O) enables a radically different approach: device passthrough. With VT-d, a physical device can be assigned directly to a virtual machine. The guest OS talks to real hardware, achieving near-native performance. The hypervisor doesn't intercept every I/O operation—it just sets up the assignment and lets hardware enforce isolation.

But how do you safely let a VM control real hardware? What stops a malicious guest from using DMA to read arbitrary host memory? VT-d's answer: an IOMMU (I/O Memory Management Unit) that applies address translation and access control to all device memory operations.

I/O virtualization presents unique challenges that differ fundamentally from CPU or memory virtualization.

Device Emulation: The Traditional Approach:

In traditional virtualization, devices are emulated in software:

1. Guest OS has drivers for virtual devices (e.g., virtio-net, IDE disk)
2. Guest I/O operations cause VM exits
3. Hypervisor decodes I/O and performs real operations
4. Results are passed back to guest

Guest Application
     ↓ write()
Guest Kernel Driver (e.g., virtio-net)
     ↓ I/O port write or MMIO
[VM EXIT]
Hypervisor - Device Model (e.g., QEMU)
     ↓ Real network operation
Host Kernel NIC Driver
     ↓
Physical NIC

This approach is flexible—guest doesn't need real hardware drivers—but introduces overhead at every layer.

Why Device Emulation Is Slow:

1. VM Exit Overhead: Each I/O operation causes expensive context switches
2. Memory Copying: Data often copied between guest, hypervisor, and host buffers
3. Interrupt Delivery: Virtual interrupts require hypervisor intervention
4. Latency: Multiple software layers add microseconds to every operation

For a 10 Gbps NIC processing 14 million packets per second, even 1 microsecond per packet consumes all CPU time just for overhead!

The DMA Security Problem:

Direct device access seems ideal but raises a critical security issue: DMA (Direct Memory Access). Devices perform DMA to transfer data without CPU involvement—they read/write physical memory directly. If a guest controls a device, that device can DMA to any physical address the guest programs.

Without protection, a malicious guest could:

• Read hypervisor memory (steal secrets, break isolation)
• Write to hypervisor memory (code injection, privilege escalation)
• Access other VMs' memory (lateral movement)

VT-d's Solution: The IOMMU

The IOMMU (I/O Memory Management Unit) sits between devices and memory, translating device-initiated addresses and enforcing access control. Think of it as a page table unit for DMA—just as the MMU translates CPU addresses, the IOMMU translates device addresses.

Intel VT-d IOMMU:

VT-d introduces DMA Remapping (DMAR) with these components:

1. Root Table: Indexed by bus number, points to context tables
2. Context Table: Indexed by device/function, points to address translation structure
3. Second-Level Page Tables: Similar to EPT, translates device addresses to physical

DMA Address Translation:

Device wants to DMA to address 0x1000_0000
      ↓
IOMMU looks up device identity (Bus:Device:Function)
      ↓
Root Table[Bus] → Context Entry[DevFn]
      ↓
Context Entry contains: Domain ID + Page Table Pointer
      ↓
Page Table Walk: 0x1000_0000 → Translated Physical Address
      ↓
DMA proceeds to translated address (or faults if unmapped)

Context Entry Structure:

Each device (identified by Bus:Device:Function) has a context entry:

struct context_entry {
    uint64_t lo;   /* Pointer to page tables, flags */
    uint64_t hi;   /* Domain ID, address width */
};

/* Context entry bit definitions */
#define CTX_PRESENT    (1 << 0)
#define CTX_FPD        (1 << 1)   /* Fault processing disable */
#define CTX_TRANS_TYPE(t) ((t) << 2)  /* Translation type */
#define CTX_ADDR_WIDTH(w) ((w) << 0)  /* In hi word */
#define CTX_DOMAIN_ID(d) ((uint64_t)(d) << 8)  /* In hi word */

void setup_context_entry(struct context_entry *ctx,
                          uint64_t page_table_root,
                          uint16_t domain_id) {
    ctx->lo = (page_table_root & PAGE_MASK) | 
              CTX_PRESENT | 
              CTX_TRANS_TYPE(0);  /* 0 = Second-level only */
    ctx->hi = CTX_ADDR_WIDTH(2) |  /* 48-bit */
              CTX_DOMAIN_ID(domain_id);
}

Second-Level Page Tables:

VT-d page tables are similar to EPT—4-level structure translating device addresses:

/* VT-d page table entry (similar to EPT) */
#define DMA_PTE_READ   (1 << 0)   /* Read access */
#define DMA_PTE_WRITE  (1 << 1)   /* Write access */
#define DMA_PTE_SNP    (1 << 11)  /* Snoop bit for cache coherency */
#define DMA_PTE_ADDR(a) ((a) & 0x000FFFFFFFFFF000ULL)

/* Build IOMMU page table for a VM */
void build_iommu_page_table(struct vm *vm, struct iommu_domain *domain) {
    /* For passthrough, map guest physical space */
    for (each guest_page gpfn in vm->memory) {
        uint64_t gpa = gpfn << PAGE_SHIFT;
        uint64_t hpa = ept_translate(vm->ept, gpa);
        
        /* Device sees VM's physical addresses, IOMMU translates to host */
        iommu_map_page(domain, gpa, hpa, DMA_PTE_READ | DMA_PTE_WRITE);
    }
}

Key Insight: Double Translation

With device passthrough:

• CPU accesses: GVA → (guest PT) → GPA → (EPT) → HPA
• Device DMA: GPA → (IOMMU PT) → HPA

The guest programs the device with GPAs. The IOMMU translates these to HPAs. The IOMMU page table typically mirrors the EPT, ensuring devices and CPUs see consistent memory mapping.

DMA remapping protects memory, but devices also generate interrupts. In x86, MSI (Message Signaled Interrupts) work by having the device write to a special memory address that the interrupt controller interprets as an interrupt request. Without protection, a malicious device could forge arbitrary interrupts.

The Interrupt Security Problem:

MSI interrupts contain:

• Destination APIC ID (which CPU to interrupt)
• Vector (which interrupt handler to invoke)
• Delivery mode (how to deliver)

A compromised device could:

1. Interrupt arbitrary CPUs (denial of service)
2. Specify vectors that invoke unexpected handlers
3. Inject interrupts into other VMs or the hypervisor

VT-d Interrupt Remapping:

VT-d introduces an Interrupt Remapping Table (IRT) that validates and translates device interrupts:

1. Device issues MSI with a handle instead of raw interrupt info
2. IOMMU looks up handle in Interrupt Remapping Table
3. IRT entry contains validated destination/vector
4. IOMMU generates actual interrupt with authorized parameters

Interrupt Remapping Table Entry:

struct irte {           /* Interrupt Remapping Table Entry */
    uint64_t lo;
    uint64_t hi;
};

/* Low quadword fields */
#define IRTE_PRESENT     (1 << 0)
#define IRTE_FPD         (1 << 1)   /* Fault Processing Disable */
#define IRTE_DM(m)       ((m) << 2)  /* Destination Mode */
#define IRTE_RH          (1 << 3)   /* Redirection Hint */
#define IRTE_TM          (1 << 4)   /* Trigger Mode */
#define IRTE_DLV(d)      ((d) << 5)  /* Delivery Mode */
#define IRTE_AVAIL       (0xF << 8)  /* Available for software */
#define IRTE_VECTOR(v)   ((uint64_t)(v) << 16)
#define IRTE_DEST(d)     ((uint64_t)(d) << 32)

/* High quadword fields */
#define IRTE_SID(s)      ((s) & 0xFFFF)   /* Source ID validation */
#define IRTE_SQ(q)       ((q) << 16)      /* Source ID qualifier */
#define IRTE_SVT(t)      ((t) << 18)      /* Source validation type */

void setup_irte(struct irte *entry, uint8_t vector, uint32_t dest_apic,
                uint16_t source_id) {
    entry->lo = IRTE_PRESENT |
                IRTE_DM(0) |        /* Physical destination mode */
                IRTE_DLV(0) |       /* Fixed delivery */
                IRTE_VECTOR(vector) |
                IRTE_DEST(dest_apic);
    
    entry->hi = IRTE_SVT(1) |       /* Verify source ID */
                IRTE_SID(source_id); /* Expected BDF */
}

Source ID Validation:

Critically, interrupt remapping can validate the source of interrupts. Each IRT entry specifies which device (by Bus:Device:Function) is allowed to use that entry. A device can't use another device's interrupt handles.

Device 0:3:0 tries to use handle belonging to device 0:4:0
  → IOMMU compares source ID
  → Mismatch detected
  → Interrupt blocked, fault logged

With VT-d infrastructure in place, we can implement device passthrough—assigning a physical device exclusively to a virtual machine.

Passthrough Setup Steps:

1. Unbind from host driver: Remove device from host OS control
2. Configure IOMMU domain: Create isolation domain for the VM
3. Set up DMA remapping: Map guest physical space for this device
4. Configure interrupts: Set up interrupt remapping entries
5. Expose to guest: Present device in guest's virtual PCI bus

# Linux VFIO device passthrough example

# 1. Identify device
lspci -nn | grep NVIDIA
# 01:00.0 VGA compatible controller [0300]: NVIDIA ... [10de:1b80]

# 2. Unbind from current driver
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

# 3. Bind to vfio-pci driver
echo "10de 1b80" > /sys/bus/pci/drivers/vfio-pci/new_id

# 4. Verify IOMMU group
ls /sys/kernel/iommu_groups/*/devices/
# Group contains: 0000:01:00.0 0000:01:00.1 (GPU + GPU audio)

# 5. Start VM with device passthrough (QEMU example)
qemu-system-x86_64 
    -device vfio-pci,host=01:00.0 
    -device vfio-pci,host=01:00.1 
    ...

IOMMU Groups and ACS:

Devices that can communicate without going through the IOMMU (e.g., devices behind the same PCIe switch) must be in the same IOMMU group. If devices can peer-to-peer DMA, they can't be independently isolated.

ACS (Access Control Services) is a PCIe feature that enforces routing through the root complex, enabling finer-grained IOMMU groups:

Without ACS:
  PCIe Switch
  ├── Device A ─┐
  └── Device B ─┤── Same IOMMU group
                     (can peer-to-peer)

With ACS:
  PCIe Switch (ACS enabled)
  ├── Device A ── Separate IOMMU group
  └── Device B ── Separate IOMMU group
                  (traffic forced through root)

For passthrough, you often need to pass through entire IOMMU groups. Consumer motherboards often have poor IOMMU group isolation; server hardware typically has better ACS support.

Device passthrough gives one VM exclusive access to one physical device. But what if you have 100 VMs and only 4 network ports? SR-IOV (Single Root I/O Virtualization) solves this by making one physical device appear as multiple independent devices.

SR-IOV Architecture:

An SR-IOV device presents:

1. Physical Function (PF): The actual device, managed by the host/hypervisor. Has full device capabilities including configuration.

2. Virtual Functions (VFs): Lightweight device instances, each appearing as a separate PCIe function. Each VF can be assigned to a different VM.

SR-IOV NIC Configuration:

Physical NIC (e.g., Intel X540)
├── PF0 (Physical Function) - Managed by host
├── VF0 → Assigned to VM1
├── VF1 → Assigned to VM2
├── VF2 → Assigned to VM3
├── VF3 → Assigned to VM4
└── ... (up to 64+ VFs)

Each VF has its own:
- PCIe configuration space
- Memory-mapped registers
- TX/RX queues
- Interrupts (MSI-X vectors)

Creating Virtual Functions:

# Enable SR-IOV on a network interface
# Check maximum VFs supported
cat /sys/class/net/eth0/device/sriov_totalvfs
# 63

# Create 4 virtual functions
echo 4 > /sys/class/net/eth0/device/sriov_numvfs

# Verify VFs appeared
lspci | grep Virtual
# 03:10.0 Ethernet controller: Intel ... Virtual Function
# 03:10.2 Ethernet controller: Intel ... Virtual Function
# 03:10.4 Ethernet controller: Intel ... Virtual Function
# 03:10.6 Ethernet controller: Intel ... Virtual Function

# Each VF gets its own IOMMU group and can be passed through
ls /sys/kernel/iommu_groups/*/devices/ | grep 03:10

SR-IOV Benefits:

1. Near-native performance: VFs are real hardware, not emulated
2. Scalability: One physical device serves many VMs
3. Hardware isolation: Switching/queuing in device hardware
4. Reduced host CPU usage: No hypervisor in data path

SR-IOV Mechanism:

The hardware includes an internal switch fabric that routes traffic:

• Ingress: Incoming packets are classified (by MAC, VLAN, etc.) and directed to the appropriate VF's RX queue
• Egress: Each VF's TX queue is scheduled onto the physical port
• Isolation: VFs cannot see each other's traffic; hardware enforces separation

VT-d provides essential security for device passthrough, but correct configuration is critical. Misconfiguration can break isolation guarantees entirely.

Security Guarantees of Properly Configured VT-d:

1. DMA Isolation: Device can only access memory explicitly mapped for it
2. Interrupt Isolation: Device can only trigger authorized interrupts
3. Device Isolation: Devices in different IOMMU domains cannot interfere
4. Guest Containment: Compromised guest with device access cannot escape VM

Common Misconfigurations and Attacks:

Verifying VT-d Configuration:

# Check IOMMU is enabled in kernel
dmesg | grep -i iommu
# [    0.000000] DMAR: IOMMU enabled
# [    0.123456] DMAR-IR: Enabled IRQ remapping

# Verify interrupt remapping
dmesg | grep "IRQ remapping"
# Should show "Enabled"

# Check for IOMMU faults (should be empty normally)
cat /sys/kernel/debug/iommu/intel/dmar_table_errors

# List IOMMU domains and devices
cat /sys/kernel/iommu_groups/*/type
ls -la /sys/kernel/iommu_groups/*/devices/

Pre-Boot DMA Protection:

Modern systems support blocking DMA before the OS loads:

• Intel: Kernel DMA Protection via BIOS
• Purpose: Prevent Thunderbolt DMA attacks, evil maid scenarios
• Mechanism: IOMMU blocks all DMA until OS takes control

Boot Sequence with DMA Protection:

1. Platform powers on
2. IOMMU initialized in blocking mode
3. All device DMA rejected
4. OS loads, takes IOMMU ownership
5. OS enables DMA only for trusted devices

TOCTOU and Malicious Devices:

A sophisticated attacker with physical access could use a malicious PCIe device (e.g., via Thunderbolt) to attempt:

1. DMA attacks: Attempt to read memory before IOMMU protection
2. Confusion attacks: Trick IOMMU into incorrect mappings
3. Denial of service: Flood IOMMU with invalid transactions

VT-d fault handling addresses some concerns:

/* VT-d fault handler */
void dmar_fault_handler(uint64_t source_id, uint64_t fault_addr,
                         uint32_t fault_reason) {
    /* Log the fault */
    log_security_event("IOMMU fault: device %04x addr %016lx reason %d",
                       source_id, fault_addr, fault_reason);
    
    /* Common fault reasons:
     * 1 = Page not present
     * 2 = Write to read-only page  
     * 5 = Access width violation
     */
    
    /* Consider disabling device if repeated faults */
    if (++fault_count[source_id] > FAULT_THRESHOLD) {
        disable_device(source_id);
        alert_admin("Device %04x disabled due to repeated IOMMU faults",
                    source_id);
    }
}

While VT-d enables near-native I/O performance, several optimizations can further reduce overhead.

IOTLB (I/O Translation Lookaside Buffer):

Like CPU TLBs cache address translations, IOTLBs cache IOMMU translations:

DMA without IOTLB hit:
  Device address → Root Table → Context Table → Page Walk → Physical
  (Multiple memory accesses)

DMA with IOTLB hit:
  Device address → IOTLB → Physical
  (Single lookup)

IOTLB Invalidation:

When IOMMU mappings change, IOTLB entries must be invalidated:

void invalidate_iotlb(struct iommu_domain *domain) {
    /* Global invalidation - flush all entries for domain */
    struct qi_desc desc = {
        .qw0 = QI_IOTLB_GRAN(QI_IOTLB_DOMAIN) | 
               QI_IOTLB_DID(domain->id) |
               QI_IOTLB_TYPE,
        .qw1 = 0,
    };
    qi_submit_sync(&desc);
}

void invalidate_iotlb_addr(struct iommu_domain *domain, 
                           uint64_t addr, uint64_t size) {
    /* Page-selective invalidation - more efficient */
    struct qi_desc desc = {
        .qw0 = QI_IOTLB_GRAN(QI_IOTLB_PAGE) |
               QI_IOTLB_DID(domain->id) |
               QI_IOTLB_TYPE,
        .qw1 = addr | QI_IOTLB_AM(size),
    };
    qi_submit_sync(&desc);
}

ATS (Address Translation Services):

ATS allows PCIe devices to cache address translations locally:

1. Device requests translation from IOMMU
2. IOMMU returns translated address + permissions
3. Device caches translation in its ATC (Address Translation Cache)
4. Future DMAs use cached translation directly
5. IOMMU invalidations must also invalidate device ATCs

Without ATS:
  Every DMA → IOMMU page walk

With ATS:
  First DMA → IOMMU translation → Device caches
  Subsequent DMAs → Device ATC hit → Direct access

Posted Interrupts for Devices:

Intel's Posted Interrupts feature can deliver device interrupts directly to a running vCPU without VM exit:

1. Device generates MSI → Interrupt Remapping
2. IRT entry configured for posted interrupt
3. Interrupt posted to notification vector
4. If vCPU running, delivered without exit
5. If vCPU blocked, hypervisor notified to wake it

/* Configure posted interrupt for device */
void setup_posted_interrupt_irte(struct irte *entry, 
                                  struct vcpu *vcpu,
                                  uint8_t vector) {
    entry->lo = IRTE_PRESENT |
                IRTE_MODE_POSTED |         /* Posted interrupt mode */
                IRTE_VECTOR(vector);
    entry->hi = IRTE_POSTED_ADDR(vcpu->posted_intr_desc) |
                IRTE_URGENT_BIT;           /* Wake blocked vCPU */
}

VT-d completes the hardware virtualization picture by enabling secure, high-performance I/O for virtual machines. Through DMA remapping, interrupt remapping, and device passthrough, VMs can achieve near-native I/O performance while maintaining isolation guarantees.

What's Next:

In the next page, we'll explore Performance Acceleration techniques—how all the hardware virtualization features (VT-x, EPT, VT-d) work together to minimize overhead, and practical techniques for tuning virtual machine performance. We'll examine real-world benchmarks and optimization strategies.