Virtual Machines

Before virtualization, operating system development was a painstaking endeavor. Every kernel modification required rebooting a physical machine. A bug in memory management could corrupt the disk. Hardware debugging required serial consoles, physical switches, and hours of downtime. Building an OS was as much about patience and physical endurance as it was about programming skill.

Virtual machines changed everything. Today, OS developers can boot a modified kernel in seconds, attach source-level debuggers, take snapshots before risky changes, and run multiple OS versions simultaneously. What once required a dedicated lab now runs on a laptop. This transformation hasn't just made OS development faster—it's made it accessible, opening the field to students, hobbyists, and researchers who would never have had access to specialized hardware.

To appreciate virtualization's impact, we must understand what OS development looked like before it became mainstream.

The Traditional OS Development Workflow (circa 1990s):

1. Edit source code on a development machine
2. Cross-compile the kernel (if not developing on the target)
3. Transfer the kernel to target hardware (floppy disk, network boot, serial cable)
4. Reboot the target machine
5. Wait for boot (potentially minutes)
6. Test the changes
7. Crash (likely, during development)
8. Inspect via console output, LED codes, or logic analyzer
9. Reboot and repeat from step 1

The Pain Points:

Virtual machines compress the edit-compile-test cycle from minutes to seconds, fundamentally changing how developers approach OS problems.

The Modern VM-Enabled Workflow:

1. Edit source code in your favorite IDE
2. Compile the kernel (incremental builds: seconds to minutes)
3. Launch VM with new kernel (seconds)
4. Test changes
5. VM crash? Restart instantly (seconds)
6. Attach debugger if needed (seamless)
7. Rollback to snapshot if changes corrupted state (instant)
8. Iterate rapidly, potentially dozens of times per hour

Practical Impact:

This speed difference isn't just convenient—it changes how developers think about experimentation. When each test costs minutes, you think carefully before trying something. When each test costs seconds, you try ten variations. This leads to:

• More experimentation: Developers try speculative changes they wouldn't risk otherwise
• Better testing: Running tests frequently catches regressions early
• Faster learning: Students can make mistakes and recover instantly
• Higher quality: Rapid iteration allows more polish and edge case handling

QEMU + KVM kernel development example:

# Compile kernel with debugging symbols
make -j$(nproc)

# Boot directly without creating a full image
qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -append "console=ttyS0 nokaslr" \
  -initrd initramfs.cpio.gz \
  -nographic \
  -enable-kvm \
  -m 2G

This command boots a newly compiled kernel in approximately 2-3 seconds. Iterating on kernel code becomes as fast as iterating on user-space applications.

Perhaps the most transformative benefit of virtual machines for OS development is the ability to debug kernel code with familiar source-level debuggers.

The GDB Stub Integration:

QEMU (and other emulators) can expose a GDB debugging interface. When the guest hits a breakpoint or stops for any reason, control transfers to GDB on the host, providing full visibility into kernel state.

What You Can Do with Kernel Debugging:

Advanced Debugging Features:

Reverse Debugging: Some VM-based debug setups support record and replay. You can "rewind" execution to understand how you reached a particular state. This is invaluable for investigating non-deterministic bugs.

KGDB (Kernel GDB): Linux includes a kernel debugger agent (KGDB) that allows debugging a running kernel via serial or network connection. Combined with a VM's virtual serial port (easily connected to the host), this provides interactive debugging without QEMU-specific GDB stubs.

Hardware Watchpoints: Using the CPU's debug registers, you can break on memory read, write, or execute for specific addresses. VMs expose this functionality through GDB, allowing you to track down buffer overflows, use-after-free bugs, and wild pointers.

Comparison with Physical Debugging:

VM snapshots provide something impossible on physical hardware: instant, complete state preservation and restoration.

What Snapshots Capture:

• Complete memory contents (every byte of RAM)
• CPU state (all registers, including hidden state)
• Device state (virtual disk positions, network buffers, pending interrupts)
• Timer state (exact time offsets, pending timers)

Essentially, a snapshot freezes the VM at a precise moment. Restoring that snapshot puts the VM back in exactly that state—mid-instruction if necessary.

Use Cases for OS Development:

Snapshot Implementation:

Snapshots are typically stored as:

1. Memory dump: Entire RAM contents (can be compressed, often significantly)
2. Device state files: Serialized state of each virtual device
3. Disk state: Either a separate copy or, more commonly, a copy-on-write overlay

Copy-on-Write Disks:

To avoid duplicating gigabytes of disk data for each snapshot, hypervisors use copy-on-write (CoW) overlays:

┌─────────────────────────────────┐
│ Current Disk State              │
│ (Overlay: tracks changes)       │
├─────────────────────────────────┤
│ Snapshot 2 Overlay              │
├─────────────────────────────────┤
│ Snapshot 1 Overlay              │
├─────────────────────────────────┤
│ Base Image (shared by all)      │
└─────────────────────────────────┘

Each snapshot only stores the blocks that changed since the previous snapshot (or base image). This allows maintaining many snapshots with minimal disk space—the base image dominates.

Live vs Offline Snapshots:

• Offline: VM paused, state captured, VM resumed (brief pause)
• Live/Online: State captured without pausing (more complex, may have consistency implications)

For OS development, offline snapshots with a brief pause are typically fine and simpler.

Virtual machines provide emulated hardware that behaves consistently and predictably—ideal for device driver development and testing.

Advantages for Driver Development:

Example: Developing a Block Device Driver

# QEMU with detailed virtio-blk tracing
qemu-system-x86_64 \
  -drive file=disk.img,if=virtio \
  -trace enable=virtio_blk_* \
  -trace file=virtio-trace.log \
  --enable-kvm \
  ...

# The trace log shows every buffer submission,
# completion, and device state change:
# virtio_blk_req_submit: sector=1024 nsectors=8 type=read
# virtio_blk_req_complete: sector=1024 status=0

With this visibility, you can verify that your driver correctly formats requests, handles completions, and manages queues—all without instrumenting the driver itself.

Developing for Non-Existent Hardware:

Researchers and hardware designers often develop software before hardware exists. QEMU's modular device architecture allows creating a software model of future hardware:

1. Write a QEMU device model implementing the proposed hardware interface
2. Develop and debug drivers against this model
3. When physical hardware arrives, drivers work immediately (or nearly so)

This approach was used for NVMe, RISC-V, and many other technologies.

Operating systems must support diverse configurations. VMs enable testing across this diversity without maintaining hardware zoos.

Multi-Version Testing:

Maintaining a complex software project means supporting multiple OS versions. VMs allow running multiple versions simultaneously:

Multi-Architecture Testing:

QEMU's full-system emulation supports dozens of architectures, enabling cross-platform development without owning exotic hardware:

# Test on 32-bit x86
qemu-system-i386 -kernel bzImage-i386 ...

# Test on ARM64
qemu-system-aarch64 -M virt -cpu cortex-a72 -kernel Image ...

# Test on RISC-V
qemu-system-riscv64 -M virt -kernel Image-riscv64 ...

# Test on PowerPC
qemu-system-ppc64 -kernel vmlinux-ppc64 ...

Performance Note: Cross-architecture emulation (x86 host running ARM guest) is much slower than native virtualization (x86 host, x86 guest). But for functional testing and debugging, the speed is usually acceptable.

Use Cases:

Configuration Permutation Testing:

Kernel configuration options create exponential test matrices. VMs enable automated testing across many configurations:

# Example: Test a file system across configurations
for config in "EXT4_FS=y" "BTRFS_FS=y" "XFS_FS=y"; do
  make defconfig
  echo "CONFIG_$config" >> .config
  make olddefconfig
  make -j$(nproc)
  run_test_in_vm "fs-stress-test"
done

Automated CI systems run thousands of such configurations nightly, catching regressions before they reach users.

Virtual machines create a safe sandboxed environment for experiments that would be dangerous or impossible on production systems.

Research Enabled by VMs:

Case Study: Kernel Fuzzing with syzkaller

syzkaller, a leading kernel fuzzer, leverages VMs extensively:

1. Spawn multiple VMs in parallel for maximum fuzzing throughput
2. Each VM runs a fuzzing agent that generates random system calls
3. On crash, syzkaller captures the crash log, extracts a reproducer, and restarts the VM instantly
4. Automated bisection finds which kernel commit introduced the bug
5. Report generation creates a bug report suitable for submission

This approach has found thousands of kernel bugs. Without VMs, such aggressive fuzzing would require an impractical amount of physical hardware and generate significant downtime for each crash.

Academic Research Acceleration:

Published OS research papers routinely describe experiments run on VMs. The reproducibility is invaluable:

• Share a VM image alongside the paper for reproducibility
• Collaborators can run experiments without replicating hardware setups
• Reviewers can verify results independently
• Students can learn from experiments by running them hands-on

VMs have democratized OS education. Activities that once required dedicated hardware labs are now possible on any student's laptop.

Educational Benefits:

Popular Educational OS Projects:

The xv6 Example:

xv6, developed by MIT for their 6.S081 operating systems course, is explicitly designed for QEMU:

# Clone, build, and run xv6 in minutes
git clone https://github.com/mit-pdos/xv6-riscv.git
cd xv6-riscv
make qemu

# You're now in a running UNIX-like OS
# Modify kernel.c, run 'make qemu' again
# Full development cycle in seconds

This accessibility has made xv6 the foundation for OS courses worldwide. Thousands of students have had their first kernel development experience through this VM-based approach.

We've explored how virtual machines have revolutionized operating system development. Let's consolidate the key insights:

What's Next:

We've seen how VMs benefit OS development. Next, we'll explore the hardware virtualization support that makes efficient virtualization possible—the CPU extensions (Intel VT-x, AMD-V), memory virtualization features (EPT, NPT), and I/O virtualization technologies that transform software virtualization into near-native performance.