MicroVMs From The Bottom Up

Chapter 1: The Virtualization Stack Map

Running lsmod | grep kvm on a Linux host usually shows two KVM modules: the common kvm core, plus either kvm_intel or kvm_amd. Those lines, the silicon beneath them, and one userspace process above them are a complete hypervisor. The whole microVM stack is four layers, and most of the confusion around it comes from collapsing them -- calling KVM "the hypervisor" when it is one layer of four, or calling Firecracker "lightweight QEMU" when the two share only the bottom two layers and diverge completely above.

flowchart TB vmm["VMM process\n(Firecracker, QEMU, Cloud Hypervisor, crosvm)"] kvm["KVM modules\n/dev/kvm"] hw["VT-x / AMD-V + EPT / NPT"] guest["Guest kernel + workload\n(non-root mode)"] vmm -->|ioctls| kvm vmm -->|virtio + guest RAM| guest kvm --> hw hw --> guest

Read it bottom to top: the hardware enforces a guest boundary, the KVM module makes that hardware programmable, the VMM is the userspace process that drives it, and the guest is an ordinary operating system running inside the machine those layers create. The rest of the chapter walks those four layers, then places the VMMs that live on the top one.

The Hardware: A Second Mode the Silicon Enforces

Classic x86 could not be virtualized cleanly. A guest operating system expects to own ring 0 -- the most privileged CPU level, the one a kernel runs in -- and two kernels cannot both own ring 0 on the same machine. Worse, on the original instruction set some privileged operations, when a guest ran them from a lower privilege level, neither took effect nor trapped -- handed control to supervising code that could stand in for them. They simply failed quietly, so a hypervisor had no way to intercept and emulate them. (Chapter 2 tells that story; it is why early VMware had to rewrite guest instructions on the fly.)

Intel VT-x and AMD-V removed the problem by adding a second dimension alongside the rings rather than competing for them. The guest kernel runs at ring 0 of a new non-root mode; the host kernel and KVM run at ring 0 of root mode; and the CPU itself enforces the wall between the two modes. When the guest does something the host must mediate, the hardware performs a VM exit and hands control to KVM. A second extension -- EPT on Intel, NPT on AMD -- adds host-controlled page tables that translate the guest's idea of physical memory into real host memory without the host having to inspect every access. Those two mechanisms, the exit and the second translation, are what make a guest both isolated and fast. Chapter 4 opens the control structures and the exit machinery; Chapter 6 covers the paging.

The KVM Module: The Hardware, Made Programmable

The hardware extensions are a mechanism, not an interface. KVM is what makes them usable from software: a kernel module that exposes the device file /dev/kvm, through which a userspace process drives an entire virtual machine with ioctl calls -- the catch-all system call for controlling a device. Through it the process asks KVM to create a VM, give it memory, create virtual CPUs, and run them.

KVM keeps exactly the parts that must live in the kernel: saving and restoring CPU context across VM exits, delivering interrupts to the guest, managing the EPT/NPT page tables, and emulating the interrupt controllers the guest expects to find. Everything above that line -- which devices the guest sees, how its memory is laid out, what happens on each exit KVM hands back to userspace -- is left to the VMM. That split is the most important boundary on the map; Chapter 5 walks the ioctl API that draws it.

The VMM: The Userspace Process That Owns Policy

The Virtual Machine Monitor is the userspace program that opens /dev/kvm and turns it into a running machine. It carves the guest's RAM out of its own address space (with mmap, the system call that maps a region of memory into a process), loads a kernel into it, creates one host thread per virtual CPU, and runs each thread in a loop: enter the guest, and when the hardware exits back -- because the guest wrote to an I/O port, touched a memory-mapped device register (an MMIO access), or halted -- service that exit and re-enter. Around the loop the VMM supplies the guest's world: its disk, its network interface, its serial console, almost always as virtio devices -- a standard family of paravirtualized devices, meaning virtual hardware designed for VMs rather than emulated copies of real network cards and disks.

The VMM is where the design choices live, which is why there is more than one. A VMM can model every device a real PC has or just the few a Linux guest truly needs; it can chase compatibility, or boot time, or a small attack surface. Part III builds a minimal VMM to show the five jobs every one of them performs; Part IV is Firecracker, start to finish.

The Guest: A Real Kernel Inside the Boundary

The guest is an ordinary operating system -- a stock Linux kernel and its userspace -- running in non-root mode. Ordinary guest code runs as if it owns a machine: privileged instructions either trap to KVM or are handled transparently by the hardware, and the few devices it sees announce themselves as virtio. Linux can still be told that it is virtualized through CPUID leaves, paravirtual clocks, and device drivers. That is not a violation of the model; it is how modern guests avoid pretending that a virtual disk is an IDE controller from 1998. What a microVM guest needs is a kernel compiled with virtio drivers and little beyond that.

What Counts As A "Hypervisor"

The four layers also explain why "hypervisor" is a slippery word. The classic taxonomy splits hypervisors into type-1, running directly on the hardware (Xen, VMware ESXi), and type-2, running as a program on top of a normal host OS (VirtualBox). KVM fits neither cleanly. The KVM module runs at the host kernel's own privilege level, which is the type-1 condition -- but a full host OS runs right beside it, scheduling guests as ordinary processes, which is the type-2 picture. Both are true at once, because the hardware's root/non-root split is independent of the "above or below the OS" question the taxonomy was built on.

KVM maintainer Paolo Bonzini's advice on the distinction was "I would just ignore it." This book calls KVM a hybrid and moves on.

The VMM Landscape

Every VMM does the same four-layer job: open /dev/kvm, back the guest's memory, model some devices, and loop on the guest. What they differ in is what they model on top, and that one choice sorts the field.

VMM Language Runs on Device model Built for
QEMU C KVM, or its own software emulation Broad PC and system emulation across many CPU architectures General-purpose emulation and virtualization
Firecracker Rust KVM only A small virtio device set over MMIO or optional PCI transport Serverless functions and containers
Cloud Hypervisor Rust KVM, or Microsoft's Hyper-V Wider: device hotplug, PCI passthrough, shared host filesystem Cloud VMs
crosvm Rust KVM Broad, each device in its own sandbox ChromeOS and Android

The four are not unrelated. Firecracker began as a fork of crosvm, and Cloud Hypervisor borrowed from both. Firecracker and Cloud Hypervisor also use components maintained under the rust-vmm umbrella, including KVM wrappers, guest-memory abstractions, and device models. Chapter 23 compares the projects in depth. For the map, the point is that "VMM" is a role, not a single program, and this book fills that role with Firecracker.

Where Firecracker Sits

Firecracker is purpose-built for serverless functions and containers. AWS open-sourced it in November 2018; it runs only on KVM, models only the devices a modern Linux guest needs for that job, and drops the rest:

Each omission buys back boot time, memory, or attack surface -- usually all three. For the configuration defined in SPECIFICATION.md, Firecracker requires at most 125 milliseconds from InstanceStart to the start of guest /sbin/init, and at most 5 MiB of VMM memory overhead. Both bounds are exercised by integration tests. Those guarantees are possible because the device model is small. Firecracker is this book's worked example precisely because it is small enough to understand completely.

The next chapter goes underneath the bottom layer of this map: what a virtual machine actually is, why classic x86 could not provide one, and how the hardware boundary differs -- concretely, in its threat model and its cost -- from the shared-kernel boundary a container draws.

Sources And Further Reading