Hardware Virtualization Support

The holy grail of virtualization is making virtual machines indistinguishable from bare metal—not just in behavior, but in performance. With modern hardware virtualization features, this goal is remarkably achievable. Well-tuned VMs regularly achieve 95-99% of native performance for most workloads.

But getting there requires understanding how the various hardware and software components interact. A single misconfiguration—missing EPT enablement, incorrect NUMA placement, or excessive VM exits—can devastate performance. Conversely, understanding the optimization landscape lets you make informed trade-offs between isolation, flexibility, and speed.

This page synthesizes everything we've learned about hardware virtualization into a coherent performance optimization strategy.

Before optimizing, we must understand what makes VMs slower than bare metal. Virtualization overhead comes from several distinct sources:

1. VM Exits (CPU Overhead):

Every VM exit saves full CPU state, transitions to the hypervisor, processes the exit reason, and transitions back. Even with hardware acceleration, this costs 1,000-10,000 CPU cycles.

2. Address Translation (Memory Overhead):

Two-dimensional page walks (guest PT + EPT/NPT) access more memory than native paging on TLB misses.

3. I/O Emulation:

Emulated devices require hypervisor intervention for every operation, copying data through multiple layers.

4. Interrupt Delivery:

Virtual interrupts traditionally require exits for injection and acknowledgment.

5. Resource Contention:

Multiple VMs compete for physical resources—CPU time, memory bandwidth, cache space, I/O bandwidth.

Exit Frequency Analysis:

The number of VM exits per second is a key performance indicator:

# Monitor VM exit rate on Linux/KVM
perf kvm stat live

# Sample output:
#   VM-Exit    Samples    %time    Avg time
#   HLT           3421    45.2%      2.1us
#   IO             892    12.3%     15.4us  
#   EPT_VIOLATION  234     3.1%      1.8us
#   EXTERNAL_INT   178     2.4%      0.9us
#   CPUID          156     0.8%      0.4us

Interpreting Exit Data:

CPU performance in VMs depends on minimizing exits and ensuring predictable scheduling.

vCPU Pinning:

By default, vCPUs may be scheduled on any physical CPU, migrating as the host scheduler decides. This causes:

• Cache cold starts after migration
• NUMA locality violations
• Increased scheduling overhead

Pinning assigns vCPUs to specific physical CPUs:

# QEMU/KVM: Pin vCPU 0 to physical CPU 2
qemu-system-x86_64 ... \
    -smp 4 \
    -binding 0:2,1:3,2:4,3:5

# libvirt: In domain XML
<vcpu placement='static'>4</vcpu>
<cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
</cputune>

CPU Feature Exposure:

The hypervisor controls which CPU features are visible to guests via CPUID interception:

<!-- libvirt: Use host CPU model with specified features -->
<cpu mode='host-passthrough' check='none'>
    <feature policy='disable' name='vmx'/>  <!-- Hide nested virt -->
    <feature policy='require' name='avx2'/> <!-- Require AVX2 -->
</cpu>

CPU Models:

Performance Impact of Feature Hiding:

Disabling features impacts workloads that use them:

• Missing AVX: 2-4x slower for vectorized code
• Missing AES-NI: 5-10x slower for encryption
• Missing TSC deadline: Higher timer overhead

Memory performance is critical because every CPU operation ultimately involves memory. EPT/NPT are essential, but there's more to optimize.

Huge Pages:

Using 2MB or 1GB pages instead of 4KB pages:

1. Reduces EPT/NPT walk depth
2. Improves TLB coverage (one TLB entry covers more memory)
3. Reduces page table memory overhead

# Configure huge pages on host
echo 1024 > /proc/sys/vm/nr_hugepages  # Allocate 2GB (1024 * 2MB)

# Verify available huge pages
cat /proc/meminfo | grep Huge
# HugePages_Total:    1024
# HugePages_Free:     1024
# Hugepagesize:       2048 kB

# libvirt: Enable huge pages for VM
<memoryBacking>
    <hugepages>
        <page size='2048' unit='KiB'/>
    </hugepages>
</memoryBacking>

NUMA Topology:

Non-Uniform Memory Access (NUMA) means memory access speed depends on which CPU socket makes the request. Local memory access is faster than remote:

NUMA System:
┌─────────────────┐    ┌─────────────────┐
│    Socket 0     │    │    Socket 1     │
│  ┌───┐ ┌───┐   │    │  ┌───┐ ┌───┐   │
│  │CPU│ │CPU│   │    │  │CPU│ │CPU│   │
│  └───┘ └───┘   │    │  └───┘ └───┘   │
│       │        │    │       │        │
│   Memory 0     │←──→│   Memory 1     │
│   (64GB)       │QPI │   (64GB)       │
└─────────────────┘    └─────────────────┘

Local access: ~80ns
Remote access: ~130ns (1.6x slower)

NUMA-Aware VM Configuration:

<!-- libvirt: Pin VM memory to specific NUMA node -->
<numatune>
    <memory mode='strict' nodeset='0'/>
</numatune>

<!-- Topology: Show guest its own NUMA structure -->
<cpu>
    <numa>
        <cell id='0' cpus='0-3' memory='4' unit='GiB'/>
        <cell id='1' cpus='4-7' memory='4' unit='GiB'/>
    </numa>
</cpu>

Memory Pre-Allocation:

By default, VM memory is allocated on-demand (overcommit). This saves RAM but adds latency:

First access to VM page:
1. Guest accesses GPA
2. EPT violation (page not mapped)
3. Exit to hypervisor
4. Hypervisor allocates host page
5. Maps in EPT
6. Re-enter guest

With pre-allocation:
1. Guest accesses GPA  
2. EPT hit (already mapped)
3. Access completes

<!-- libvirt: Pre-allocate all memory -->
<memoryBacking>
    <allocation mode='immediate'/>
    <locked/>  <!-- Prevent swapping -->
</memoryBacking>

Memory Overcommit Considerations:

I/O is often the most impactful area for virtualization performance. The difference between emulated and passthrough devices can be 10x or more.

I/O Virtualization Hierarchy (Best to Worst):

1. SR-IOV / Hardware Passthrough: Near-native, hardware isolation
2. vhost-user: Userspace bypass, kernel not in data path
3. vhost-net: Kernel acceleration, still crosses boundaries
4. virtio: Paravirtualized, reasonable overhead
5. Emulated (e1000, IDE): Full emulation, high overhead

virtio Optimization:

When passthrough isn't available, virtio is the next best option. Optimize it:

<!-- libvirt: Optimized virtio-net -->
<interface type='network'>
    <source network='default'/>
    <model type='virtio'/>
    <driver name='vhost' queues='4'/> <!-- Multi-queue -->
    <tune>
        <sndbuf>1048576</sndbuf>  <!-- 1MB send buffer -->
    </tune>
</interface>

<!-- Optimized virtio-blk disk -->
<disk type='file' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native' 
            discard='unmap' iothread='1'/>
    <source file='/var/lib/libvirt/images/vm.raw'/>
    <target dev='vda' bus='virtio'/>
</disk>

Key virtio Settings:

Disk I/O Optimization:

# Disk image format impact
# Raw: No overhead, immediate allocation
# QCOW2: Copy-on-write, slight overhead but flexible

# Create pre-allocated raw image (best performance)
qemu-img create -f raw /var/lib/libvirt/images/vm.raw 100G

# Pre-allocate to eliminate fragmentation
fallocate -l 100G /var/lib/libvirt/images/vm.raw

# For QCOW2, use preallocation
qemu-img create -f qcow2 -o preallocation=metadata \
    /var/lib/libvirt/images/vm.qcow2 100G

I/O Scheduling for KVM:

# For SSD storage backing VMs
echo 'none' > /sys/block/nvme0n1/queue/scheduler

# For HDD storage
echo 'mq-deadline' > /sys/block/sda/queue/scheduler

# Increase queue depth for NVMe
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

Interrupts and timers cause exits. Reducing their frequency and enabling hardware acceleration dramatically improves performance.

Posted Interrupts (Intel):

Without posted interrupts:

1. Device generates interrupt
2. VM exit to hypervisor
3. Hypervisor injects interrupt to guest
4. VM entry

With posted interrupts:

1. Device generates interrupt
2. Hardware deposits interrupt directly into guest
3. No exit required (if vCPU is running)

# Verify posted interrupts enabled
cat /sys/module/kvm_intel/parameters/enable_apicv
# Y

# Monitor interrupt delivery
perf kvm stat live | grep -E 'INTERRUPT|POSTED'

Timer Virtualization:

Guests need accurate time for scheduling, timestamps, and application logic. Timer sources:

TSC Virtualization:

The TSC (Time Stamp Counter) is the fastest timer—a simple register read. VT-x provides TSC offsetting:

Guest RDTSC result = Host TSC + TSC Offset

This allows:
1. Each VM sees consistent TSC from boot
2. No exit required for RDTSC
3. Live migration can adjust offset

Clock Sources for Linux Guests:

# Check guest clock source
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
# tsc (best) or kvm-clock (paravirt) or hpet (legacy)

# Force TSC if stable
echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

Optimization without measurement is guessing. Proper benchmarking requires understanding what you're measuring and avoiding common pitfalls.

Benchmarking Principles:

1. Measure what matters: Choose benchmarks relevant to your workload
2. Control variables: Change one thing at a time
3. Multiple runs: Take averages, understand variance
4. Warmup: Let caches and JIT compilers stabilize
5. Compare fairly: Same hardware, same OS version, same benchmark version

Measuring Virtualization Overhead:

# Compare VM vs bare metal
# 1. Run benchmark on bare metal, record results
# 2. Run same benchmark in VM, record results
# 3. Calculate overhead: (native - vm) / native * 100

# CPU overhead example with sysbench
# Bare metal:
sysbench cpu --threads=4 run
# events per second: 10000

# VM:
sysbench cpu --threads=4 run  
# events per second: 9850

# Overhead: (10000 - 9850) / 10000 = 1.5%

VM Exit Analysis:

# Detailed exit analysis with perf
perf kvm stat record -p $(pgrep qemu) -- sleep 60
perf kvm stat report

# Output shows:
# - Exit counts by type
# - Time spent in each exit handler
# - Min/max/avg exit handling time

# Identify exit hotspots
perf kvm stat live
# Watch for unexpected high-frequency exits

Common Measurement Mistakes:

Performance Monitoring During Production:

# Continuous monitoring stack
# 1. Host metrics: node_exporter + Prometheus
# 2. VM metrics: libvirt exporter
# 3. Guest metrics: Guest agent + node_exporter
# 4. Storage metrics: iostat, iotop
# 5. Network metrics: iperf tests, packet drops

# Real-time VM performance
virt-top --connect qemu:///system

# Quick health check
virsh domstats myvm | grep -E 'cpu|memory|balloon|net|block'

Let's examine how these optimization techniques apply to real scenarios.

Case Study 1: Database Server (PostgreSQL)

Initial Configuration:

• 16 vCPU, 64GB RAM
• virtio-blk disk, QCOW2 image, cache=writeback
• virtio-net, single queue
• No CPU pinning, no NUMA awareness
• TPS: 15,000 (baseline)

Optimization Steps:

1. CPU Pinning + NUMA: Pin vCPUs and memory to same NUMA node

   • TPS: 18,500 (+23%)

2. Huge Pages: Enable 2MB huge pages for VM memory

   • TPS: 20,000 (+8% additional)

3. Disk: Raw image, cache=none, io=native, NVMe passthrough

   • TPS: 32,000 (+60% additional)

4. Network: Multi-queue virtio, vhost

   • TPS: 33,500 (+5% additional)

Final Result: 33,500 TPS vs 15,000 baseline = 2.2x improvement

Case Study 2: Low-Latency Trading Application

Requirements:

• Sub-100μs network latency
• Predictable performance (< 10μs jitter)
• 99th percentile matters more than average

Configuration:

<!-- Extreme low-latency configuration -->
<domain type='kvm'>
    <!-- CPU isolation -->
    <vcpu placement='static'>8</vcpu>
    <cputune>
        <vcpupin vcpu='0' cpuset='4'/>
        <vcpupin vcpu='1' cpuset='5'/>
        <!-- ... -->
        <emulatorpin cpuset='0-3'/> <!-- Emulator off perf CPUs -->
    </cputune>
    
    <!-- Memory: Huge pages, locked, NUMA-pinned -->
    <numatune>
        <memory mode='strict' nodeset='0'/>
    </numatune>
    <memoryBacking>
        <hugepages>
            <page size='1048576' unit='KiB'/> <!-- 1GB pages -->
        </hugepages>
        <locked/>
        <nosharepages/>
    </memoryBacking>
    
    <!-- Network: SR-IOV passthrough -->
    <hostdev mode='subsystem' type='pci' managed='yes'>
        <source>
            <address domain='0x0000' bus='0x82' slot='0x00' function='0x1'/>
        </source>
    </hostdev>
</domain>

Host Tuning:

# Isolate CPUs from scheduler
# Kernel cmdline: isolcpus=4-11 nohz_full=4-11 rcu_nocbs=4-11

# Disable frequency scaling
for cpu in 4 5 6 7 8 9 10 11; do
    cpupower -c $cpu frequency-set -g performance
done

# Disable C-states
for cpu in 4 5 6 7 8 9 10 11; do
    for state in /sys/devices/system/cpu/cpu$cpu/cpuidle/state*/disable; do
        echo 1 > $state
    done
done

Results:

• Average network latency: 35μs (vs 300μs with virtio emulation)
• 99th percentile: 52μs
• Jitter: < 8μs

Case Study 3: Cloud Provider Compute Instance

Challenge: Balance density, isolation, and performance for thousands of VMs.

Approach:

Trade-offs Accepted:

• Not pinning CPUs (allows overcommit, migration)
• Not using 1GB pages (too inflexible)
• Using shared storage (enables live migration)

Mitigations:

• CPU steal monitoring to detect contention
• Automatic rebalancing across hosts
• Tiered offerings (dedicated vs shared)

Performance Expectation:

• "Shared" tier: 85-95% of native (variable)
• "Dedicated" tier: 95-99% of native (consistent)

Achieving near-native performance in virtual machines requires understanding the full stack of hardware and software optimizations. With proper configuration, the overhead of virtualization can be reduced to single-digit percentages for most workloads.

Module Complete:

You've now completed the Hardware Virtualization Support module. You understand:

• Intel VT-x: VMX modes, VMCS, VM entry/exit mechanics
• AMD-V: SVM, VMCB, comparison with VT-x
• EPT/NPT: Two-dimensional page walks, TLB management, performance characteristics
• VT-d: IOMMU, DMA remapping, interrupt remapping, SR-IOV
• Performance: Optimization techniques for CPU, memory, I/O, and timing

This knowledge is foundational for building, configuring, and operating virtualized infrastructure—from single-host development environments to massive cloud deployments.