Hardware Virtualization Support

Memory virtualization is one of the most challenging aspects of building a hypervisor. Every memory access a guest makes uses a guest virtual address (GVA). The guest's operating system translates this to a guest physical address (GPA) using its page tables. But the guest's 'physical' addresses aren't actually physical—they're abstracted by the hypervisor. The hypervisor must then translate GPAs to host physical addresses (HPA) that map to real RAM.

Before hardware-assisted nested paging, hypervisors maintained shadow page tables—complex software structures that collapsed both translation levels into direct GVA→HPA mappings. Shadow page tables worked, but they were expensive to maintain, triggered frequent VM exits on guest page table updates, and added significant hypervisor complexity.

Extended Page Tables (EPT) from Intel and Nested Page Tables (NPT) from AMD changed everything by moving the second-level translation into hardware.

To understand EPT/NPT, we must first understand why memory virtualization is hard, and what shadow page tables attempted to solve.

Address Spaces in Virtualization:

A guest application's memory access requires two translations:

1. GVA → GPA: Guest page tables (managed by guest OS)
2. GPA → HPA: Nested page tables (managed by hypervisor)

Without hardware support, the CPU only knows about one page table hierarchy. It can do GVA → HPA directly (if given the right page tables), but not a two-step translation.

The Shadow Page Table Approach (Pre-EPT/NPT):

Without hardware nested paging, hypervisors created shadow page tables that combined both translations:

1. Guest OS maintains its own page tables (GVA → GPA)
2. Hypervisor intercepts guest page table modifications
3. Hypervisor creates shadow tables translating GVA → HPA directly
4. CPU uses shadow tables for actual memory access

Problems with Shadow Page Tables:

• Write Protection: Guest page tables must be write-protected to catch modifications
• Frequent Exits: Every guest page table write causes a VM exit
• TLB Flushes: Changes require flushing TLB entries
• Memory Overhead: Shadow tables duplicate guest table structure
• Complexity: Handling all guest paging modes (32-bit, PAE, 64-bit) is error-prone

Shadow Page Table Maintenance:

1. Guest writes to its page table
2. Write causes exit (page protected)
3. VMM reads guest page table entry
4. VMM looks up GPA→HPA translation
5. VMM creates/updates shadow entry: GVA→HPA
6. VMM unprotects page temporarily
7. VMM re-enters guest

This happens on EVERY guest page table modification!

Extended Page Tables (Intel) and Nested Page Tables (AMD) solve memory virtualization by adding a second-level address translation performed entirely in hardware. The guest page tables remain unmodified, and the CPU handles both translation levels automatically.

The Two-Dimensional Walk:

When a guest accesses memory, the CPU performs a coordinated walk of both page table hierarchies:

1. First dimension (Guest tables): Walk guest CR3 → PML4 → PDPT → PD → PT to translate GVA to GPA

2. Second dimension (Nested tables): For every guest physical address encountered during the walk (including page table pointers), translate GPA to HPA using the nested page tables

The guest page table walk alone might access 4-5 memory locations (in 4-level paging). Each of those locations is a GPA that must be translated via the nested tables. A single guest memory access can trigger up to 24 memory references in the worst case!

Walk Complexity Analysis:

For 4-level paging:

• Guest walk: 4 table levels + 1 final access = 5 GPAs to translate
• Each GPA translation: 4 nested table levels = 4 memory accesses
• Worst case: 5 × 4 = 20 memory accesses (24 if counting CR3 translations)

This sounds expensive, but:

1. TLB caching dramatically reduces actual memory accesses
2. Large pages reduce table depth
3. The translations happen in hardware, no VM exits needed
4. Modern CPUs have dedicated nested page table caches

Enabling EPT (Intel):

void enable_ept(struct vmcs *vmcs, uint64_t eptp) {
    /* Set EPT pointer in VMCS */
    /* Format: [Page walk length (3)] [Memory type] [Root table address] */
    vmwrite(VMCS_EPT_POINTER, 
            (eptp & PAGE_MASK) |    /* Root table physical addr */
            (3 << 3) |               /* 4-level walk (encoded as 3) */
            (6));                    /* Write-back memory type */
    
    /* Enable EPT in secondary processor controls */
    uint32_t secondary = vmread(VMCS_SECONDARY_PROC_CONTROLS);
    secondary |= SECONDARY_EXEC_ENABLE_EPT;
    vmwrite(VMCS_SECONDARY_PROC_CONTROLS, secondary);
}

Enabling NPT (AMD):

void enable_npt(struct vmcb *vmcb, uint64_t ncr3) {
    /* Set nested CR3 (root of nested page tables) */
    vmcb->control.nested_cr3 = ncr3;
    
    /* Enable NPT */
    vmcb->control.nested_ctl |= SVM_NESTED_CTL_NP_ENABLE;
}

EPT and NPT use page table structures similar to the standard x86-64 page tables, but with different entry formats designed for virtualization needs.

EPT Entry Format (Intel):

Each EPT entry is 64 bits with the following layout:

Key Differences from Standard x86 Page Tables:

1. Separate R/W/X bits: EPT has independent read, write, and execute permissions (standard x86 only has W and NX). This enables fine-grained memory protection.

2. No User/Supervisor distinction: EPT doesn't distinguish privilege levels—it applies to all guest accesses. Guest user/supervisor is handled by guest page tables.

3. Memory Type control: EPT can specify caching behavior (uncacheable, write-combining, write-back, etc.) replacing or complementing MTRR and PAT.

4. Accessed/Dirty bits: Optional (requires CPU support). When enabled, hardware tracks page access for demand paging and dirty tracking.

/* EPT Entry definitions */
#define EPT_READ      (1ULL << 0)
#define EPT_WRITE     (1ULL << 1)
#define EPT_EXECUTE   (1ULL << 2)
#define EPT_MT_MASK   (7ULL << 3)      /* Memory type */
#define EPT_MT_UC     (0ULL << 3)      /* Uncacheable */
#define EPT_MT_WC     (1ULL << 3)      /* Write-combining */
#define EPT_MT_WB     (6ULL << 3)      /* Write-back */
#define EPT_IGNORE_PAT (1ULL << 6)
#define EPT_LARGE_PAGE (1ULL << 7)
#define EPT_ACCESSED  (1ULL << 8)
#define EPT_DIRTY     (1ULL << 9)

/* Create EPT entry for 4KB page */
uint64_t create_ept_pte(uint64_t hpa, uint64_t flags) {
    return (hpa & PAGE_MASK) | flags | EPT_READ | EPT_WRITE | EPT_EXECUTE;
}

/* Create EPT entry for 2MB large page */
uint64_t create_ept_pde_2mb(uint64_t hpa, uint64_t flags) {
    return (hpa & LARGE_PAGE_MASK) | flags | EPT_LARGE_PAGE | 
           EPT_READ | EPT_WRITE | EPT_EXECUTE | EPT_MT_WB;
}

NPT Entry Format (AMD):

AMD's NPT uses format closer to standard x86 page tables:

NPT uses the standard x86 Present/Read-Write model, making it simpler for developers familiar with x86 paging. The explicit read/write/execute granularity of EPT requires slightly more complex handling but enables additional security features.

When a guest memory access cannot be translated by the nested page tables, an EPT violation (Intel) or NPT page fault (AMD) occurs. This is distinct from a regular guest page fault—the guest's translation succeeded, but the hypervisor's translation failed.

Causes of EPT/NPT Violations:

1. Unmapped GPA: Guest accessed a physical address not configured in EPT/NPT
2. Permission violation: Access type (read/write/execute) not permitted
3. MMIO Access: Guest trying to access device memory that needs emulation
4. Lazy allocation: Hypervisor populates EPT on-demand

EPT Violation Exit Information (Intel):

When an EPT violation occurs, the VMCS contains detailed information:

• Exit Qualification: Bits indicating access type and EPT state

  • Bit 0: Data read
  • Bit 1: Data write
  • Bit 2: Instruction fetch
  • Bit 3: EPT entry readable
  • Bit 4: EPT entry writable
  • Bit 5: EPT entry executable
  • Bit 7: GVA valid
  • Bit 8: Caused by GPA access (vs. page table walk)

• Guest Physical Address: The GPA that caused the violation

• Guest Linear Address: The GVA that caused the access (if valid)

NPT Page Fault Handling (AMD):

AMD's NPT faults are handled similarly but with different exit information format:

void handle_npt_fault(struct vcpu *vcpu) {
    struct vmcb *vmcb = vcpu->vmcb;
    uint64_t error_code = vmcb->control.exitinfo1;
    uint64_t gpa = vmcb->control.exitinfo2;
    
    bool present = error_code & (1 << 0);
    bool write = error_code & (1 << 1);
    bool user = error_code & (1 << 2);
    bool reserved = error_code & (1 << 3);
    bool fetch = error_code & (1 << 4);
    
    /* Similar handling logic as EPT violations */
    if (!present) {
        /* Page not mapped - demand allocation */
        allocate_and_map_page(vcpu->vm, gpa);
    } else if (write) {
        /* Write to read-only */
        handle_write_protection(vcpu->vm, gpa);
    }
}

EPT Misconfiguration vs. Violation:

Intel distinguishes between:

• EPT Violation: Access denied by EPT permissions (normal case)
• EPT Misconfiguration: Invalid EPT entry (e.g., reserved bits set, invalid memory type)

Misconfigurations indicate hypervisor bugs and should never occur in correct operation. They result in immediate VM exit with a different exit reason.

The Translation Lookaside Buffer (TLB) is critical for paging performance—it caches virtual-to-physical translations to avoid expensive page walks. With nested paging, TLB management becomes more complex because translations now span both guest and nested tables.

What's Cached in the TLB:

With EPT/NPT, the TLB caches the combined translation: GVA → HPA. This means a single TLB entry subsumes both translation levels. This is efficient for hits but complicates invalidation—changes to either guest or nested tables can invalidate cached entries.

VPID and ASID: Tagging TLB Entries:

Without tagging, every VM entry/exit would require a TLB flush because guest and host translations might conflict. VPID (Intel) and ASID (AMD) tag TLB entries with an identifier:

TLB Entry: [VPID/ASID | GVA | HPA | attributes]

VM 1: VPID=1, GVA 0x1000 → HPA 0x5000
VM 2: VPID=2, GVA 0x1000 → HPA 0x8000  (different VM, same GVA)
Hypervisor: VPID=0, VA 0x1000 → PA 0x2000

All three can coexist in TLB!

INVEPT and INVVPID (Intel):

Intel provides specific instructions for EPT/VPID invalidation:

/* Invalidate all EPT translations for a specific EPTP */
void invept_single_context(uint64_t eptp) {
    struct {
        uint64_t eptp;
        uint64_t reserved;
    } descriptor = { eptp, 0 };
    
    asm volatile("invept %0, %1"
                 : : "m"(descriptor), "r"(1ULL)  /* Type 1 = single context */
                 : "memory");
}

/* Invalidate all EPT translations globally */
void invept_all_contexts(void) {
    struct {
        uint64_t eptp;
        uint64_t reserved;
    } descriptor = { 0, 0 };
    
    asm volatile("invept %0, %1"
                 : : "m"(descriptor), "r"(2ULL)  /* Type 2 = all contexts */
                 : "memory");
}

/* Invalidate single virtual address for a VPID */
void invvpid_individual_address(uint16_t vpid, uint64_t gva) {
    struct {
        uint64_t vpid;
        uint64_t gva;
    } descriptor = { vpid, gva };
    
    asm volatile("invvpid %0, %1"
                 : : "m"(descriptor), "r"(0ULL)  /* Type 0 = individual */
                 : "memory");
}

ASID and TLB Control (AMD):

AMD uses the TLB_CONTROL field in VMCB:

/* TLB control values for VMRUN */
#define TLB_CONTROL_DO_NOTHING    0  /* Preserve TLB entries */
#define TLB_CONTROL_FLUSH_ASID    1  /* Flush this ASID only */
#define TLB_CONTROL_FLUSH_ALL     3  /* Flush all TLB entries */

void set_guest_asid(struct vmcb *vmcb, uint32_t asid) {
    vmcb->control.guest_asid = asid;
    /* Typically flush on first use of new ASID */
    vmcb->control.tlb_control = TLB_CONTROL_FLUSH_ASID;
}

void invalidate_guest_page(struct vmcb *vmcb, uint64_t gva) {
    /* AMD doesn't have per-address invalidation in hardware */
    /* Options: flush ASID, or let guest INVLPG handle it */
    vmcb->control.tlb_control = TLB_CONTROL_FLUSH_ASID;
}

EPT/NPT fundamentally changes the performance profile of virtualized memory access. Understanding these characteristics helps optimize VM configurations.

Comparison: Shadow Tables vs. EPT/NPT:

The TLB Miss Trade-off:

On a TLB miss with EPT/NPT, the 2-dimensional page walk accesses more memory than shadow tables would. Analysis:

Shadow Tables TLB Miss:
  4 memory accesses (standard 4-level walk)

EPT/NPT TLB Miss:
  Up to 24 memory accesses (5 GPAs × 4-5 levels + overhead)

But consider:
  - Modern CPUs have nested page table caches
  - Large pages reduce both dimensions
  - 24 accesses × 100ns = 2.4μs worst case
  - One VM exit = 1,000-10,000 cycles ≈ 0.3-3μs
  - Shadow update might trigger multiple exits

Optimization: Large Pages:

Using large pages (2MB or 1GB) in EPT/NPT reduces walk depth:

Using 2MB pages throughout reduces worst-case from 24 to 15 accesses—a 37% improvement.

Workload-Dependent Performance:

Different workloads see different benefits from EPT/NPT:

Best for EPT/NPT:

• Heavy memory allocation (malloc/mmap intensive)
• Process creation (fork/exec)
• Databases with large working sets
• Applications with high page table churn

Neutral:

• Steady-state CPU computation
• I/O-bound workloads (exits dominated by I/O)
• Applications with excellent TLB hit rates

Potentially Worse:

• Extremely high TLB miss rates with small working sets
• Memory-intensive benchmarks measuring raw access latency
• (Rare in practice)

Real-World Impact:

Benchmark: Linux kernel compile in VM

Shadow Tables:
  - 847,000 VM exits for CR3 loads
  - 2.1 million exits for page table writes
  - Total: 3.2 million exits
  - Build time: 142 seconds

With EPT:
  - 0 exits for memory operations
  - ~12,000 exits (I/O, interrupts only)
  - Build time: 98 seconds

Improvement: 31% faster

Modern processors include advanced EPT/NPT features that enable sophisticated virtualization scenarios beyond basic memory translation.

Accessed and Dirty Bits:

When CPU support is present, EPT/NPT can set accessed and dirty bits automatically:

• Accessed bit: Set by hardware on read or write to page
• Dirty bit: Set by hardware on write to page

These bits are essential for:

• Live migration: Tracking which pages changed during migration iterations
• Memory overcommit: Identifying cold pages for swapping
• Checkpointing: Knowing which pages to save for incremental snapshots

/* Enable accessed/dirty bit support (Intel) */
void enable_ept_ad_bits(struct vm *vm) {
    /* Check CPU capability first */
    if (!cpu_has_ept_ad_bits())
        return;
    
    uint64_t eptp = vmread(VMCS_EPT_POINTER);
    eptp |= EPT_POINTER_AD_ENABLE;  /* Set bit 6 */
    vmwrite(VMCS_EPT_POINTER, eptp);
}

/* Scan for dirty pages (for live migration) */
void scan_dirty_pages(struct vm *vm, uint64_t *dirty_bitmap) {
    for (each ept_page_table_entry(vm->ept, &entry, &gpa)) {
        if (entry & EPT_DIRTY) {
            set_bit(dirty_bitmap, gpa >> PAGE_SHIFT);
            /* Clear dirty bit for next iteration */
            entry &= ~EPT_DIRTY;
        }
    }
    invept_single_context(vm->eptp);  /* Flush TLB after clearing */
}

Page Modification Logging (PML):

PML is particularly powerful for live migration and checkpointing. Instead of scanning the entire EPT for dirty bits, hardware maintains a log:

/* PML setup */
void enable_pml(struct vm *vm) {
    /* Allocate 512-entry PML buffer (4KB page) */
    vm->pml_buffer = alloc_page();
    vmwrite(VMCS_PML_ADDRESS, virt_to_phys(vm->pml_buffer));
    
    /* Set initial PML index to 511 (grows downward) */
    vmwrite(VMCS_PML_INDEX, 511);
    
    /* Enable PML in secondary controls */
    uint32_t ctrl = vmread(VMCS_SECONDARY_PROC_CONTROLS);
    ctrl |= SECONDARY_EXEC_ENABLE_PML;
    vmwrite(VMCS_SECONDARY_PROC_CONTROLS, ctrl);
}

/* Process dirty pages from PML buffer */
void process_pml_buffer(struct vm *vm) {
    uint16_t pml_index = vmread(VMCS_PML_INDEX);
    
    /* Entries from pml_index+1 to 511 are dirty GPAs */
    for (int i = pml_index + 1; i < 512; i++) {
        uint64_t dirty_gpa = vm->pml_buffer[i];
        mark_page_dirty(vm, dirty_gpa);
    }
    
    /* Reset PML index */
    vmwrite(VMCS_PML_INDEX, 511);
}

When the PML buffer fills, a VM exit occurs. The hypervisor drains the buffer and resumes the guest. This is far more efficient than page-table scanning for workloads with scattered dirty pages.

Extended Page Tables (Intel EPT) and Nested Page Tables (AMD NPT) represent a major advancement in virtualization technology. By moving the second-level address translation into hardware, they eliminate the complexity and overhead of shadow page tables while enabling new capabilities impossible with software-only approaches.

What's Next:

In the next page, we'll explore I/O Virtualization (VT-d)—how hardware enables direct device assignment to VMs, DMA remapping for security, and interrupt remapping for isolation. I/O virtualization completes the hardware support picture, enabling near-native I/O performance for virtualized workloads.