Virtual Machines

For decades, virtualization on x86 processors required complex software workarounds. The x86 architecture was never designed with virtualization in mind, and certain instructions simply didn't behave correctly when a hypervisor tried to intercept them. This changed dramatically in 2005-2006 when Intel and AMD introduced hardware virtualization extensions—dedicated silicon features specifically designed to make virtualization efficient, secure, and correct.

Today's virtualization story is fundamentally a hardware story. Without VT-x, AMD-V, EPT, NPT, VT-d, and AMD-Vi, the cloud computing revolution would have been far slower and more expensive. Understanding these hardware features is essential for anyone working with virtual machines, whether deploying cloud infrastructure, developing hypervisors, or debugging virtualization-related issues.

Before we appreciate hardware virtualization, let's understand what problem it solved.

The Popek and Goldberg Criteria (1974):

Gerald Popek and Robert Goldberg formalized requirements for efficient virtualization. An architecture is classically virtualizable if:

1. All sensitive instructions are privileged: Any instruction that could observe or modify the machine's true state must trap when executed in user mode.

2. Complete isolation: The VMM has complete control over system resources.

3. Efficiency: Innocuous instructions execute directly on hardware without VMM intervention.

x86's Violation:

The x86 architecture violated these requirements. Several instructions behaved differently in user mode vs kernel mode without trapping, making it impossible for a hypervisor to intercept and emulate them:

Software Workarounds:

Binary Translation (VMware): VMware's solution was to scan guest code for problematic instructions and rewrite them at runtime. Sensitive instructions were replaced with calls to VMM handlers. This approach worked but was complex (millions of lines of code) and added overhead.

Paravirtualization (Xen): Xen's approach was to modify the guest operating system to never execute problematic instructions. Guests would use hypercalls (explicit calls to the hypervisor) instead of sensitive instructions. This required guest modifications but offered good performance.

Neither was ideal:

• Binary translation was complex, had performance overhead, and couldn't handle all edge cases elegantly
• Paravirtualization required guest modifications, ruling out unmodified operating systems

The industry needed a hardware solution.

Intel VT-x (Virtual Technology for x86), codenamed "Vanderpool," was introduced in 2005 with the Pentium 4 662/672 processors. It fundamentally changed how virtualization works on x86.

Core Concepts:

VMX Operation Modes: VT-x introduces two new CPU operation modes:

1. VMX Root Mode: Where the hypervisor runs. The VMM has full control over the processor and can configure virtualization settings.

2. VMX Non-Root Mode: Where guest VMs run. Instructions and events that require hypervisor intervention automatically cause VM exits.

Transitions:

• VM Entry: Transition from VMX root (hypervisor) to VMX non-root (guest)
• VM Exit: Transition from VMX non-root (guest) back to VMX root (hypervisor)

The Virtual Machine Control Structure (VMCS):

The VMCS is a hardware-managed data structure that controls VMX operation. It contains:

Guest-State Area:

• All guest CPU registers (general-purpose, segment, control)
• Guest page table pointer (CR3)
• Guest interrupt descriptor table (IDTR)
• Guest execution state (interruptibility, activity state)

Host-State Area:

• Host CPU registers to restore on VM exit
• Host page table pointer
• Host interrupt handlers
• Host stack pointer for exit handling

VM-Execution Control Fields:

• Which operations should cause VM exits
• Exception handling configuration
• I/O permissions (which ports cause exits)
• MSR (Model-Specific Register) access controls

VM-Exit Information Fields:

• Reason for the VM exit (exit reason code)
• Exit qualification (additional information)
• Instruction information (for instruction-specific exits)

AMD-V (AMD Virtualization), codenamed "Pacifica," was introduced in 2006 with AMD's Athlon 64. While conceptually similar to Intel VT-x, AMD-V uses different terminology and data structures.

Key Concepts:

Secure Virtual Machine (SVM): AMD-V is also known as SVM. Like Intel's VMX, it provides separate execution modes for hypervisor and guest.

Virtual Machine Control Block (VMCB): AMD's equivalent to Intel's VMCS. A 4KB structure containing guest state, control bits, and exit information. Unlike VMCS (which uses special VMREAD/VMWRITE instructions), VMCB fields are accessed directly via normal memory operations.

Key Instructions:

• VMRUN: Enter guest mode (equivalent to VMLAUNCH/VMRESUME)
• VMSAVE/VMLOAD: Save/restore additional guest state
• STGI/CLGI: Set/clear global interrupt flag (control interrupt handling)

Comparison with Intel VT-x:

AMD-V Unique Features:

ASID (Address Space ID): AMD-V includes ASID tags in TLB entries, allowing multiple VMs' translations to coexist in the TLB without flushing on every VM switch. This reduces the performance cost of context switches between VMs.

Clean Bits: The VMCB includes "clean bits" that indicate which state areas have been modified since the last VM exit. The processor can skip loading unmodified state, speeding up VM entries.

Decode Assists: For some VM exits (like string I/O operations), AMD-V provides decoded instruction information in the VMCB, reducing the need for the hypervisor to decode instructions itself.

Practical Differences:

For hypervisor developers, the choice between VT-x and AMD-V is usually academic—you support both. The concepts are parallel, and hypervisors like KVM, Xen, and even VMware abstract these differences behind common interfaces. The key architectural insight is the same: dedicated CPU modes for hypervisor and guest, with automatic transitions on sensitive operations.

VM exits and entries are the fundamental transitions between guest and hypervisor execution. Understanding their causes and costs is essential for virtualization performance.

What Causes VM Exits:

VM exits are configured by the hypervisor and triggered by specific guest operations:

The Anatomy of a VM Exit:

Guest executing in VMX non-root mode
            │
            ▼
    ┌───────────────┐
    │ Trigger event │ (e.g., I/O instruction, CPUID)
    └───────────────┘
            │
            ▼
    ┌───────────────┐
    │ Save guest    │ CPU state → VMCS guest-state area
    │ state         │ (RIP, RSP, RFLAGS, segments, etc.)
    └───────────────┘
            │
            ▼
    ┌───────────────┐
    │ Record exit   │ Exit reason, qualification, instruction info
    │ information   │
    └───────────────┘
            │
            ▼
    ┌───────────────┐
    │ Load host     │ VMCS host-state area → CPU state
    │ state         │ (switches to host page tables, stack)
    └───────────────┘
            │
            ▼
    ┌───────────────┐
    │ Execute at    │ Hypervisor VM exit handler runs
    │ HOST_RIP      │ (in VMX root mode)
    └───────────────┘

VM Exit Cost:

VM exits are expensive—typically hundreds to thousands of CPU cycles:

VM Entry Process:

VM entry (VMLAUNCH or VMRESUME) performs the reverse:

1. Consistency checks on VMCS fields (invalid configuration causes entry failure)
2. Load guest state from VMCS guest-state area
3. Switch to guest page tables (CR3)
4. Transfer execution to guest RIP in VMX non-root mode

Minimizing VM Exits:

Hypervisor optimization often focuses on reducing VM exit frequency:

• Batch operations: Combine multiple I/O operations into one paravirtual batch
• Use EPT: Eliminates most memory-related exits
• Exit-less interrupts: Posted interrupts deliver to guest without exit
• MSR bitmaps: Only exit on necessary MSR accesses
• I/O bitmaps: Only exit on necessary port accesses
• Paravirtual devices: Use virtio to minimize device emulation exits

Memory virtualization was historically one of the most expensive aspects of virtualization. Each guest memory access required hypervisor intervention. Extended Page Tables (EPT) (Intel) and Nested Page Tables (NPT) (AMD) solved this by adding a second level of address translation directly in hardware.

The Two-Dimensional Address Translation:

Without EPT/NPT:

• Guest page tables translate GVA → GPA
• Hypervisor maintains shadow page tables translating GVA → HPA
• Any guest page table modification requires a VM exit
• VMM updates shadow tables, incurring significant overhead

With EPT/NPT:

• Guest page tables translate GVA → GPA (guest-controlled)
• EPT/NPT tables translate GPA → HPA (hypervisor-controlled)
• Hardware performs both translations automatically
• No VM exit for guest page table modifications

Page Walk Overhead:

A 4-level page table walk (standard for 64-bit) normally requires up to 4 memory accesses (plus the final data access). With EPT/NPT, each of those 4 accesses itself goes through the EPT/NPT tables:

Worst case: 4 × 4 + 4 = 20 memory accesses per translation

Why it's still fast:

Despite the theoretical overhead, EPT/NPT is extremely fast in practice because:

1. TLB Caching: Successful translations are cached in the TLB. Most accesses hit the TLB, skipping the page walk entirely.

2. Page Walk Caches: Modern CPUs cache intermediate page walk results. A translation of a nearby address may already have cached paging structure entries.

3. Large Pages: Using 2MB or 1GB pages reduces page table depth, cutting the number of accesses.

4. No VM Exits: Even a full 20-access walk is faster than a VM exit (1000+ cycles) that shadow paging would require.

EPT Pointer (EPTP):

The hypervisor sets the EPT pointer in the VMCS, pointing to the root of the EPT page tables for each guest. When the guest runs, the CPU uses this EPT for all GPA → HPA translations.

EPT Violations:

If the EPT walk fails (mapping doesn't exist, or access rights violated), an EPT violation VM exit occurs. The hypervisor can then:

• Establish a new mapping (for lazy allocation)
• Emulate MMIO (for memory-mapped device access)
• Inject an exception into the guest (for actual access violations)

EPT Features:

CPU and memory virtualization enable efficient computation, but I/O devices present unique challenges. Devices use DMA (Direct Memory Access) to read and write memory independently of the CPU. Without hardware support, a device could access any physical memory location—including the hypervisor's memory or other VMs' memory.

Intel VT-d (Virtualization Technology for Directed I/O) and AMD-Vi (AMD I/O Virtualization, also known as IOMMU) solve this by providing:

1. DMA Remapping: Direct device DMA through page tables, ensuring devices can only access authorized memory
2. Interrupt Remapping: Control which VM receives device interrupts
3. Device Assignment: Safely assign physical devices to individual VMs

DMA Remapping:

The IOMMU maintains page tables similar to CPU page tables, but for device memory access:

Device DMA Address → IOMMU → Physical Address
                      ↓
              Access Control
              (allowed/denied)

Each device (or device group) can have its own I/O page table, restricting its DMA to specific physical pages. A device assigned to VM1 can only DMA to memory pages allocated to VM1.

Interrupt Remapping:

Without interrupt remapping, devices could send interrupts to arbitrary CPUs or vectors, potentially disrupting the hypervisor or other VMs. Interrupt remapping validates and redirects device interrupts:

1. Device generates interrupt
2. Interrupt remapping unit validates the interrupt against a table
3. Valid interrupts are delivered to the configured destination
4. Invalid interrupts are logged/dropped

Device Assignment (Pass-Through):

With VT-d/AMD-Vi, physical devices can be safely assigned to VMs:

1. Hypervisor programs IOMMU to restrict device DMA to guest memory
2. Hypervisor maps device MMIO regions in EPT
3. Guest accesses device directly (no hypervisor intervention)
4. Device interrupts route to guest (via interrupt remapping)

Result: Near-native I/O performance with virtualization security.

Efficient interrupt handling is critical for I/O-intensive workloads. Traditional virtualization requires a VM exit for every interrupt, adding significant latency. Hardware features now enable interrupt delivery directly to guests.

Traditional Interrupt Handling:

1. Hardware interrupt occurs
2. VM exit to hypervisor (expensive)
3. Hypervisor determines which VM should receive it
4. Hypervisor injects virtual interrupt into guest
5. VM resume (expensive)
6. Guest handles interrupt

Advanced Interrupt Features:

Posted Interrupts in Detail:

┌─────────────────────────────────────────────────┐
│ Posted Interrupt Descriptor (in memory)         │
├─────────────────────────────────────────────────┤
│ PIR (Posted Interrupt Requests) - 256 bits      │
│   Each bit represents an interrupt vector       │
├─────────────────────────────────────────────────┤
│ ON (Outstanding Notification) - 1 bit           │
│   Set when new interrupts need notification     │
├─────────────────────────────────────────────────┤
│ Notification Vector                             │
│   IPI vector to notify vCPU                     │
├─────────────────────────────────────────────────┤
│ Notification Destination (APIC ID)              │
│   Which physical CPU to notify                  │
└─────────────────────────────────────────────────┘

Flow with Posted Interrupts:

1. Device generates interrupt (via MSI/MSI-X)
2. Interrupt remapping identifies destination VM
3. Hardware posts interrupt to PIR (sets bit in descriptor)
4. If vCPU is running, notification IPI wakes it to check PIR
5. Guest processes interrupt without VM exit

Result: Interrupt latency drops from thousands of cycles (VM exit path) to hundreds (posted interrupt path).

Nested virtualization allows running a hypervisor inside a virtual machine. A guest VM runs a hypervisor (L1), which manages its own nested guests (L2). This enables fascinating use cases:

Why Nested Virtualization:

• Test hypervisors: Develop and debug hypervisor code without dedicated hardware
• Cloud flexibility: Run VMware inside AWS, or Hyper-V inside Azure
• Training environments: Practice with enterprise hypervisors in VMs
• Security research: Analyze hypervisor attacks in controlled environments

Implementation Challenges:

VMCS Shadowing: L1 hypervisor manipulates VMCS for L2 guests. But only L0 can actually use hardware VMCS. L0 must:

1. Intercept L1's VMCS operations
2. Maintain shadow copies of L1's VMCS
3. Merge L1's settings with L0's requirements for actual L2 execution

EPT Translation Chain: With nested virtualization:

• L2 page tables: L2VA → L2PA (managed by L1)
• L1 EPT: L2PA → L1PA (managed by L1, but actually shadows)
• L0 EPT: L1PA → HPA (managed by L0)

L0 often merges these into a single effective EPT for L2, avoiding triple translation overhead.

Hardware Support:

Modern CPUs include features to accelerate nested virtualization:

• VMCS Shadowing (Intel): Hardware assist for L1 VMCS manipulation
• Virtual VMCS (Intel): Allows L1 to use VMREAD/VMWRITE without exits
• Nested Paging (AMD): NPT designed with nested virtualization in mind

Performance:

Nested virtualization adds overhead:

• Each L2 VM exit may cause L1 exit + L0 handling
• Memory translation has additional layers
• Typical overhead: 10-30% compared to flat L1 virtualization

For development and testing, this overhead is acceptable. For production, prefer flat virtualization.

We've explored the hardware technologies that make modern virtualization efficient and practical. Let's consolidate these critical concepts:

What's Next:

We've covered traditional virtual machines with hypervisors. Next, we'll explore OS-level virtualization (containers)—a complementary approach that virtualizes at the operating system level rather than the hardware level, offering even lighter weight isolation for many use cases.