MicroVMs From The Bottom Up

Chapter 2: What A Virtual Machine Actually Is

Run a Linux container and a Linux VM on the same host and both look isolated: each can have its own filesystem, process table, and IP address. The difference is where a system call lands. A process in a container enters the host kernel. A process in a VM enters a guest kernel, and the host sees only the virtual machine around it. That one split determines the threat model, the boot cost, and what the VMM must build.

One Kernel or Two

The mechanical difference between a container and a VM reduces to one question: how many kernel instances are running?

A container is a process or process tree whose system calls go directly into the host kernel's dispatch table. Linux namespaces partition the kernel's exported views - mount points, PIDs, network stacks, hostnames, users, and cgroup roots - while cgroups constrain resource use. Those are strong mechanisms, and the containerd book spends its container chapters on them. They do not create a second kernel. A vulnerability reachable through a system call available inside the container is still a vulnerability in the kernel all containers share.

A virtual machine carries a second kernel binary. The VMM loads that kernel into guest memory, configures a virtual CPU, and asks KVM to enter the guest. When a process inside the VM calls open(2), read(2), or clone(2), the instruction enters the guest kernel, not the host kernel. The host kernel sees the guest only when hardware virtualization or KVM needs host help: an I/O port access, a memory-mapped device register access, an interrupt event, or another exit from guest execution.

flowchart TB subgraph container["Container: one kernel"] cp["Container process"] hk["Host kernel syscall table"] cp -- "syscall" --> hk end subgraph vm["VM: two kernels"] gp["Guest process"] gk["Guest kernel"] vmm["VMM userspace"] kvm["KVM in host kernel"] gp -- "syscall" --> gk vmm -- "ioctl(KVM_RUN)" --> kvm kvm -- "VM entry" --> gk gk -- "VM exit" --> kvm kvm -- "userspace exit when needed" --> vmm end

Guest system calls therefore stay inside the guest kernel unless their effects eventually require virtual I/O. That is why a guest kernel bug is normally contained inside the VM: it corrupts the kernel instance the guest owns. Escaping requires a flaw in the VMM, KVM, a host-kernel path they use, or the CPU's virtualization machinery. The price is that the VM has to boot and carry that second kernel.

The Popek-Goldberg Theorem

In July 1974, Gerald Popek and Robert Goldberg published "Formal Requirements for Virtualizable Third Generation Architectures." The paper gave virtualization its cleanest test: what does a CPU architecture need to guarantee so that a Virtual Machine Monitor (VMM) can run an operating system safely and efficiently? They wanted necessary and sufficient conditions, not a design sketch. A VMM must satisfy three properties:

Efficiency is the property that constrains the design. If the VMM must inspect every instruction, the guest is no longer a practical machine. The useful pattern is trap-and-emulate: run ordinary guest instructions directly on the CPU, but trap when the guest touches privileged state so the VMM can emulate the effect.

For trap-and-emulate to work, the architecture must cooperate. Privileged instructions trap when executed without enough privilege. Sensitive instructions are the ones that alter or depend on privileged resource state. The theorem follows directly:

"For any conventional third-generation computer, an effective VMM may be constructed if the set of sensitive instructions for that computer is a subset of the set of privileged instructions."

If every sensitive instruction is also privileged, then the VMM can run the guest kernel with less real privilege than it thinks it has. Any attempt to touch privileged state traps. The VMM inspects the operation, updates the guest's virtual machine state, and returns control. Guest user-mode code runs directly because it cannot touch privileged state in the first place.

The 1974 paper analyzed the IBM 360, Honeywell 6000, and PDP-10. It predates x86 by several years, which turned out to matter enormously.

Why x86 Broke the Theorem

The original IA-32 instruction set failed that test. It contained sensitive instructions that did not trap when a VMM ran the guest kernel outside true ring 0. Bugnion, Devine, Govil, and Rosenblum's VMware retrospective is the primary source for the full instruction catalog; the teaching example is enough here.

At ring 0, POPF pops a value from the stack into EFLAGS, including the Interrupt Flag (IF), which controls whether maskable interrupts are enabled. At ring 1 with insufficient I/O privilege, POPF completes but does not modify IF, and it does not trap. A guest kernel can execute the instruction it normally uses before a critical section, believe interrupts are disabled, and be wrong. The VMM never learns that it needs to emulate anything.

PUSHF creates the mirror problem by exposing flag state the guest should not see. Other instructions exposed descriptor-table or control-register state. Later features such as UMIP close some user-mode information leaks, but they arrived long after the virtualization problem had already shaped the VMM designs that followed.

The consequence is that IA-32 is, in Popek and Goldberg's formal sense, not virtualizable. Building a VMM for x86 required working around the architecture.

Three Approaches to an Unvirtualizable Architecture

Binary Translation

VMware's solution, first shipped as VMware Workstation in 1999, was dynamic binary translation. The VMM occupied ring 0. The guest OS kernel ran at ring 1, a technique called ring compression because the full gap between user mode and supervisor mode was compressed into a smaller privilege range. Guest applications continued at ring 3.

Before any basic block of guest kernel code executes, the VMM scans it for sensitive-but-not-privileged instructions and rewrites them with safe equivalents -- typically calls into the VMM itself. Translated blocks are stored in a code cache so that frequently executed kernel paths are rewritten only once; the amortized overhead on a warm cache is small. Guest user-mode code runs natively without scanning, because user-mode code cannot issue privileged instructions even on bare hardware.

This achieves what Popek and Goldberg call full virtualization: the guest OS runs unmodified. The same kernel binary that boots on bare hardware boots inside the VM. The cost is translation overhead on guest kernel paths, particularly for kernel code that exercises the problematic instructions frequently.

Paravirtualization

The Xen hypervisor, presented at SOSP 2003, took a different position: tell the guest OS that it is virtualized and let it cooperate. Problematic instructions become explicit hypercalls, direct calls into the hypervisor's interface. A guest that wants to update page tables asks the hypervisor to validate and perform the update.

The trade-off is that the guest OS must be ported. Xen shipped with modified Linux and NetBSD kernels; running an unmodified proprietary guest required a different path. Paravirtualization did not disappear after hardware support arrived. It survives where cooperation beats imitation: paravirtual clocks, steal-time accounting, and virtio devices all tell the guest enough truth to avoid emulating old hardware.

Hardware-Assisted Virtualization

The cleanest solution was to fix the architecture. Intel shipped the first VT-x processors in 2005, and AMD followed with AMD-V in 2006. Both extensions solve the Popek-Goldberg problem by adding a second execution mode for guests. The guest kernel can run at its own ring 0, but not in the host's root execution mode.

VMX: The Mechanics of Hardware-Assisted Virtualization

Intel's implementation is called VMX (Virtual Machine Extensions). The CPU gains two orthogonal modes, each with its own ring hierarchy:

flowchart TB subgraph root["VMX Root Operation"] vmm["VMM / KVM (ring 0)"] hostuser["Host userspace (ring 3)"] end subgraph nonroot["VMX Non-Root Operation"] guestkernel["Guest kernel (ring 0)"] guestuser["Guest userspace (ring 3)"] end vmm -- "VMLAUNCH / VMRESUME" --> guestkernel guestkernel -- "VM exit" --> vmm guestuser -- "syscall" --> guestkernel

The transition from VMX non-root to VMX root is a hardware VM exit. It happens when the guest executes an instruction or triggers an event that the host has configured for interception. The CPU saves guest state, loads host state, and resumes the host's VM-exit handler. The opposite transition is a VM entry, which resumes guest execution.

The control block for this is the VMCS (Virtual Machine Control Structure), one per virtual CPU. It contains the guest state to load on entry, the host state to load on exit, the controls that decide which events exit, and the information the CPU writes when an exit happens. Chapter 4 opens that structure. For this chapter, the important idea is simpler: the guest kernel can run at ring 0 without being host ring 0, because the CPU added another axis of privilege.

Memory Virtualization: The Second Translation

The VMCS and VM exits handle CPU state. Memory needs a second mechanism. A guest OS manages its own page tables, mapping guest-virtual addresses to what it believes are physical addresses. The host cannot trust those "physical" addresses directly, because the guest does not own host DRAM.

Intel's answer is EPT (Extended Page Tables). AMD's equivalent is NPT (Nested Page Tables). Both add a second hardware translation. First, the guest page tables translate guest virtual address to guest physical address. Then the hardware walks host-controlled tables that translate guest physical address to host physical address. Most memory accesses complete with both translations in hardware, without the VMM inspecting the access.

This is the other half of the VM boundary. VMX/SVM prevents guest ring 0 from becoming host ring 0. EPT/NPT prevents guest physical memory from becoming arbitrary host memory. Chapter 6 is the full treatment.

KVM: The Kernel Interface

KVM (Kernel-based Virtual Machine) was merged into Linux 2.6.20. It exposes hardware virtualization through /dev/kvm, a character device that a userspace VMM controls with ioctl calls. KVM is not the whole VMM. It is the kernel interface that lets a VMM create a VM, create virtual CPUs, register guest memory, and enter the guest.

Host requirement: Opening /dev/kvm and issuing KVM ioctls requires a bare-metal Linux host or a VM with nested virtualization, plus read/write permission on the device. Keep experiments that touch it on an isolated machine prepared for that purpose.

The KVM API is organized as a three-level file descriptor hierarchy:

open("/dev/kvm") -> system fd KVM_CREATE_VM -> VM fd KVM_CREATE_VCPU -> vCPU fd

Chapter 5 walks the actual ioctl sequence. The map here is enough: the system fd represents KVM itself, the VM fd represents one virtual machine, and each vCPU fd represents one virtual CPU that can enter guest execution.

Native Execution and Emulation Inside a VM

Within a running VM, two execution regimes alternate. Most of the time the guest runs in native execution: the CPU is in non-root mode, guest ring 0 and ring 3 instructions execute directly on physical silicon, and neither KVM nor the VMM interprets them.

Emulation happens when guest execution reaches something the virtual machine does not own directly. A guest serial driver might execute OUT to an I/O port. The CPU performs a hardware VM exit; KVM classifies it; if KVM cannot finish it in the kernel, KVM_RUN returns to the userspace VMM with a KVM_EXIT_IO reason. The VMM updates its emulated UART state and enters the guest again. The guest sees a serial port. The host sees a data structure in the VMM's heap.

sequenceDiagram
  participant G as "Guest (VMX non-root)"
  participant K as "KVM (kernel)"
  participant V as "VMM userspace (firecracker)"

  G->>K: OUT to serial port (hardware VM exit)
  K->>V: KVM_RUN returns, exit_reason = KVM_EXIT_IO
  V->>V: Update 16550A device state
  V->>K: ioctl(vcpu_fd, KVM_RUN, 0)
  K->>G: VM entry

Some hardware VM exits never become userspace exits. KVM handles many events internally: interrupt-controller accesses, page faults it can resolve from existing memory slots, and selected model-specific register operations. The exits that do reach userspace are the ones that constitute the VMM's device model and policy surface. A traditional VMM like qemu-system-x86_64 exposes many devices; Firecracker exposes only the few a serverless Linux guest needs. Devices that are not present cannot be attacked, cannot stall boot probing, and cannot generate userspace exits. Chapter 3 turns that into the microVM argument.

Sources And Further Reading