Chapter 23: The VMM Landscape

Calling one VMM "small" and another "heavy" explains almost nothing. Small in which dimension: binary size, device count, process count, control surface, or lifecycle support? A VMM built for an interactive desktop needs graphics and audio. One built for a long-lived cloud server needs hotplug and migration. One built for a short-lived function can reject all four.

QEMU, Cloud Hypervisor, crosvm, and Firecracker make different promises. The useful comparison is not a leaderboard. It is the set of work each project accepts, the interfaces that work makes reachable, and the containment around those interfaces.

VMM	Primary scope	CPU backend	Defining choice
QEMU	machine emulation and virtualization	TCG plus hardware accelerators	compatibility across machines and devices
Cloud Hypervisor	modern 64-bit cloud VMs	KVM or MSHV	virtio-PCI, hotplug, migration
crosvm	ChromeOS, Android, and client VMs	several platform hypervisors	sandboxed device processes
Firecracker	dense Linux microVMs	KVM	narrow workload and device contract

QEMU: Compatibility First

QEMU solves two jobs that the other three mostly decline. Its Tiny Code Generator, or TCG, translates guest instructions in software, so one architecture can emulate another without hardware virtualization. With an accelerator such as KVM on Linux or Hypervisor.framework on macOS, QEMU instead lets the host CPU execute guest code while retaining QEMU's machine and device model.

That distinction is also a security boundary. QEMU's current security policy covers selected machine types used with a virtualization accelerator. It explicitly gives TCG's non-virtualization use case no guest-isolation guarantee. TCG is invaluable for firmware work, cross-architecture development, and testing; those uses are not the same as hosting an untrusted tenant.

Why QEMU Feels Heavy

QEMU models decades of hardware: PCI and ISA machines, storage controllers, network adapters, displays, USB, audio, TPMs, firmware interfaces, VFIO passthrough, and the virtio family. It also supplies block-image parsers, migration protocols, VNC and SPICE front ends, and the QMP monitor. QEMU's own security document consequently treats the guest, user-facing connections, network protocols, disk images, kernels, device trees, and passthrough devices as untrusted inputs.

The repository's breadth is not itself the attack surface of every QEMU VM. A device omitted from a machine configuration is not automatically guest reachable, and target-specific builds exclude code. The configured device model, enabled parsers, control channels, and file descriptors are what matter. QEMU is "heavy" because it can satisfy a broad compatibility contract, not because every invocation exposes every source file.

QEMU expects a management layer to apply much of its containment. Its security guidance calls for an unprivileged process and describes SELinux or AppArmor, cgroups, namespaces, and fd passing. Linux seccomp is available through --sandbox; it is not the universal default. QEMU can also place devices in remote processes, but process separation does not automatically create a trust boundary: QEMU's policy explicitly says its vhost-user and vfio-user backends share memory with QEMU and are not considered separate security domains.

QEMU's `microvm` Machine

The microvm x86 machine type proves that QEMU can expose a narrow guest machine. Inspired by Firecracker, it omits PCI and ACPI, supports up to eight virtio-MMIO devices, and can disable its optional PIC, PIT, RTC, and ISA serial port. It directly boots a host-supplied kernel because its firmware path cannot boot a virtio-MMIO block device. It has no hotplug and no live migration across QEMU versions.

This narrows guest reachability, boot work, and footprint while preserving QEMU's mature tooling and device implementations. It does not turn QEMU into Firecracker: the process still participates in QEMU's configuration, monitor, block, and build ecosystem. That can be an advantage when the surrounding system already speaks QMP and uses QEMU image tooling.

Cloud Hypervisor: A Cloud VM, Not A Historical PC

Cloud Hypervisor starts from a narrower compatibility target: modern 64-bit Linux and Windows guests on x86-64 or AArch64, with experimental RISC-V support. It runs on KVM and Microsoft's MSHV interface. There is no TCG cross-architecture emulator and no obligation to reproduce old desktop hardware.

Its device model is nevertheless much wider than the original Firecracker model. Cloud Hypervisor uses virtio-PCI exclusively. It implements block, network, console, entropy, vsock, persistent-memory, IOMMU, memory, balloon, and watchdog devices; supports virtiofs and other vhost-user backends; and can assign devices through VFIO. Legacy emulation is limited to the platform pieces needed for boot, interrupts, time, serial output, and shutdown.

PCI is the load-bearing choice. It permits device placement, VFIO assignment, and hotplug of CPUs, memory, passthrough devices, and several virtio device types. In return, the guest requires PCI discovery and the VMM must implement PCI configuration, ACPI notification, and a more elaborate resource allocator.

Long-Lived VM Operations

Cloud Hypervisor supports local and remote live migration, precopy and postcopy memory transfer, parallel TCP connections, and optional mutual TLS. Its versioned migration protocol accepts the current and immediately preceding protocol versions; state compatibility remains a separate constraint.

Snapshot and restore record VM configuration, memory ranges, and component state. An on-demand mode registers userfaultfd and faults snapshot pages into memory on first access. VFIO devices can participate when they implement the kernel's VFIO migration v2 protocol. These are not decorative features: they are the operational contract of persistent cloud VMs.

Cloud Hypervisor remains a multithreaded VMM, with dedicated vCPU, virtio, HTTP, and other worker threads. It installs different seccomp filters per thread by default. Vhost-user moves selected backends into external processes, adding an IPC boundary and allowing independent implementations, but the security value still depends on shared-memory access and how the backend itself is confined.

crosvm: Devices Behind Process Boundaries

Google created crosvm for ChromeOS Linux and Android guests. Its scope now also includes Android's Terminal app, the Cuttlefish virtual-device platform, and Windows hosts. That client workload explains a device set Firecracker has no reason to carry: virtio-gpu with several rendering backends, virtio-snd, virtio-input, virtiofs and 9p, video codecs, TPM, USB, and VFIO, alongside the usual block, network, balloon, console, entropy, and vsock devices.

crosvm also targets more hypervisor interfaces. Its current source documents KVM, Qualcomm Gunyah, MediaTek GenieZone, and Exynos Halla on Linux or Android, plus WHPX and HAXM on Windows. Portability here means adapting both to different host operating systems and to mobile hardware that may not expose KVM.

Process Per Device

On Linux, crosvm normally forks virtual devices into separate processes and places them in Minijail sandboxes. A device jail can combine a changed root, mount, PID, user, and network namespaces, capability removal, and a device- specific seccomp policy. The processes communicate through control channels and shared guest memory.

This changes the consequence of a device bug. Compromising a sandboxed network backend first yields that child process, not the main VMM address space. The attacker still has an IPC interface and whatever shared memory the device was given, so the boundary must be audited rather than assumed. The cost is more process lifecycle, fd passing, shared-memory coordination, and policy files.

Firecracker borrowed from crosvm when its codebase began, and both projects helped seed rust-vmm. That history does not mean the current VMMs share one implementation. Cloud Hypervisor and Firecracker consume several rust-vmm crates, while the checked crosvm workspace uses its own KVM, memory, and device abstractions. Lineage and dependency are different claims.

Firecracker: A Narrow Contract That Can Grow

Firecracker accepts the tightest workload contract of the four: one Linux microVM per process, on a Linux KVM host, with direct kernel boot and a small paravirtualized device set. It omits graphics, audio, USB, arbitrary PCI passthrough, firmware-rich desktop machines, and live migration. Those omissions remove code paths, configuration combinations, and test matrices that a serverless or container-isolation platform does not need.

The familiar claim that Firecracker has exactly six devices is no longer a sound description of current source. Its VirtioDeviceType enum now contains network, block, entropy, balloon, vsock, memory, and persistent-memory devices. The platform also includes interrupt and timer machinery, serial, reset, and newer ACPI-related devices. Counting them depends on whether a transport, platform component, or optional endpoint counts as a "device."

The more useful invariant is scope. MMIO remains the default virtio transport, but PCI can be enabled for all virtio devices. With PCI enabled, developer- preview hotplug supports block, pmem, and network devices; the guest must rescan or remove the PCI function manually because Firecracker does not yet send an automatic hotplug notification. virtio-mem adds cooperative memory resize. These features broaden the lifecycle without introducing a GPU, USB stack, or general passthrough framework.

Containment And Measured Bounds

Each Firecracker process owns one VM. Its API, VMM, and vCPU threads share an address space but install category-specific seccomp filters before guest code runs. The Jailer can add a changed root, reduced identity, namespaces, and cgroups around the process. This is a different choice from crosvm's process-per-device model: fewer IPC boundaries, but no address-space boundary between a device implementation and the VMM.

Firecracker turns its narrow workload into testable bounds. On the named bare- metal instances and reference guest in SPECIFICATION.md, CI requires the API socket within 8 CPU milliseconds, /sbin/init within 125 milliseconds of InstanceStart, and no more than 5 MiB of VMM memory overhead for one vCPU and 128 MiB of guest RAM. The document qualifies the memory number by workload and configuration; several throughput claims still carry an "integration test pending" marker. A specification is useful because its conditions are part of the claim.

Comparing Attack Surface

Device count is a useful prompt, not a security metric. A comparison must ask at least four questions:

Which guest-visible devices and transports are enabled?
Which host parsers, control APIs, and network protocols accept untrusted input?
What does a compromised component share with the rest of the VMM?
Which host kernel interfaces and external processes remain reachable after confinement?

QEMU can be configured narrowly, but its reason to exist is compatibility and its management surface reflects that. Cloud Hypervisor keeps legacy emulation small while accepting PCI, VFIO, hotplug, and migration for cloud operations. crosvm accepts rich client devices and spends processes and policies to isolate them. Firecracker rejects those devices and operations unless its target workload demonstrates a need.

Rust changes one important failure class: safe Rust prevents many spatial and temporal memory errors in safe code. It does not validate a virtqueue, make an unsafe block correct, confine a process, or repair a vulnerable KVM ioctl. Language, reachable interfaces, internal boundaries, and deployment controls are separate parts of the threat model.

The right VMM is therefore the smallest one that satisfies the workload's real contract. For a cross-architecture firmware lab that may be QEMU. For a migratable Windows cloud server it may be Cloud Hypervisor. For an Android desktop it may be crosvm. For thousands of direct-boot Linux sandboxes, the features Firecracker refuses are still its most important features.