MicroVMs From The Bottom Up

Chapter 3: Why MicroVMs Exist

The serverless billing model promises that a user pays for exactly the CPU cycles their function consumed, nothing more. Delivering that promise at scale means packing thousands of independent workloads onto a single host while giving each one a security boundary it cannot escape. Containers are the natural first answer, and for most uses they are sufficient. But in a multi-tenant serverless platform where every customer's code runs without review on shared hardware, the container model has a structural weakness: every tenant shares one kernel. That one fact, and the effort to work around it without sacrificing density, is where microVMs come from.

The Isolation-vs-Density Tradeoff

A Linux container is a set of constrained views over one running kernel. Namespaces change what a process can see, and cgroups constrain what it can consume; the containerd book owns those mechanisms in detail. The host kernel itself is still shared. A container process that reaches a vulnerable syscall path reaches the same kernel that schedules every other container on the machine.

A traditional VM draws a different boundary. The guest has its own kernel, and the CPU enforces the split between guest execution and host execution. An attacker inside the guest has to escape through the VMM, KVM, the host kernel paths KVM uses, or the CPU's virtualization machinery. That is a harder boundary, but historically it came with the wrong shape for serverless density: a large VMM process, a full PC device model, firmware, a bootloader, and boot times measured in seconds.

Firecracker exists to keep the VM boundary and remove the parts of the VM that Lambda and Fargate did not need.

flowchart LR A["Linux container\n(namespaces + seccomp)"] B["Traditional VM\n(QEMU/KVM)"] C["MicroVM\n(Firecracker/KVM)"] A -->|"+ density\n- isolation surface"| D["Shared kernel\nsyscall table"] B -->|"+ isolation\n- density"| E["Hardware VMX/SVM\nlarge VMM overhead\nboot in seconds"] C -->|"+ isolation\n+ density"| F["Hardware VMX/SVM\n<=5 MiB overhead\n<=125 ms specified boot"]

The AWS Lambda Origin

Firecracker was built at Amazon to solve this problem for Lambda and Fargate. The architecture that preceded it used Linux containers to isolate individual functions within a customer's account, and separate EC2 VMs to isolate between customers. That two-tier arrangement imposed a structural inefficiency: each outer VM had to be sized before anyone knew which tenant mix would fill it. The platform needed a boundary closer to a VM, with a footprint closer to a container.

Lambda's production constraints dictated the design requirements precisely. Small functions make fixed overhead painful: if a slot has only 128 MiB of guest memory, a large VMM process wastes the same order of memory as the workload itself. Burst traffic also makes creation time visible. A pool can absorb some cold starts, but it must be refilled quickly enough that the pool does not become the bottleneck. Firecracker's boot-time and memory-overhead targets are direct responses to those constraints.

Firecracker entered internal Lambda production in 2018. The open-source release was announced on 2018-11-27 on the AWS open-source blog under the Apache 2.0 license. The NSDI 2020 paper reports that Firecracker handles "trillions of requests per month" across Lambda and Fargate (section 1, section 4.1). The paper's authors are Alexandru Agache and colleagues at Amazon; it appeared at USENIX NSDI 2020.

Firecracker's Six Design Goals

The NSDI 2020 paper states six explicit design criteria. They are worth keeping together because they explain why the machine looks so small.

Isolation. The primary tenant boundary is hardware virtualization, then host hardening around the VMM. Chapters 18 through 20 cover the jailer, seccomp, and the threat model; here the important point is that Firecracker treats the guest and the vCPU threads that run it as hostile.

Overhead and density. SPECIFICATION.md requires the Firecracker VMM process to stay under 5 MiB of memory overhead for a single-vCPU guest with 128 MiB of RAM and the Firecracker-tuned kernel. The NSDI 2020 paper reports about 3 MiB in the evaluated configuration. That number is the difference between "a VM per function" as an idea and as a platform.

Performance. The guest compute target is better than 95 percent of bare-metal throughput. I/O does not match raw hardware, but the targets are high enough for the serverless workloads Firecracker was built around.

Soft allocation. Lambda and Fargate rely on memory and CPU oversubscription. The VMM must let the platform reclaim and multiplex resources instead of pinning every possible guest byte to a host resource forever.

Fast switching. The headline boot target is under 125 ms from InstanceStart to the guest forking /sbin/init under a minimal configuration. The process must also bring up its API socket quickly, before a guest is configured.

Compatibility. Firecracker must run an unmodified Linux guest kernel and standard ELF binaries. No kernel patches. No guest agent. This is what makes Lambda transparently support existing code without recompilation.

Why Rust

Firecracker began life as a fork of Google's crosvm, the ChromeOS VMM, which is itself written in Rust. The team deleted USB support, GPU passthrough, the 9p filesystem driver, and other components. At the time of the NSDI 2020 paper, Firecracker contained approximately 50,000 lines of Rust -- the team had added more than 20,000 new lines and changed 30,000 lines since the fork, and the codebase was fewer than half the size of crosvm at that point (NSDI 2020 section 2.1). QEMU 4.2, for comparison, has more than 1.4 million lines of C.

That ratio is not a vanity metric. A smaller codebase has fewer devices, fewer parser paths, fewer syscalls, and fewer branches reachable from hostile guest input. Rust was chosen because device emulators handle attacker-controlled virtio queues and need memory safety without a garbage collector in the vCPU path. Chapter 19 gives the syscall-level version of this argument.

What You Give Up

Firecracker is fast and dense because it refused to implement what Lambda and Fargate do not need.

The Minimal Device Model

Firecracker gives the guest a small set of paravirtual devices: network, block storage, vsock, entropy, ballooning, a serial console, and the few legacy stubs needed to make Linux boot and shut down cleanly. Chapter 14 gives the exact device model. The orientation point is that every omitted device is both a boot-time saving and an attack-surface saving.

What is absent relative to a general-purpose VMM matters as much as what is present: no USB, no display, no audio, no physical-device passthrough, and no attempt to boot arbitrary PC hardware. MMIO is the default virtio transport, and current Firecracker can expose the same curated devices over PCI; that optional transport does not add a general-purpose device model. These gaps are features for Lambda; they are missing dependencies for workloads that need them.

No BIOS, No General PC

Firecracker does not start a BIOS, wait for firmware, run GRUB, and then discover a PC. The VMM direct-boots a Linux kernel with a root filesystem and a kernel command line supplied through the API. That removes a large compatibility layer and the slow hardware probes that come with it. The price is that Firecracker is not a drop-in replacement for a traditional VM that expects firmware, PCI enumeration, and arbitrary boot media.

A Curated Guest Kernel

The compatibility goal is unmodified Linux userspace, not arbitrary guest kernels. A Firecracker guest kernel must include the drivers for Firecracker's small machine: virtio over MMIO, the serial path, the interrupt and clock paths, and the root filesystem it will mount. Appendix B is the reference for kernel config and rootfs construction. Here the key tradeoff is narrower: a curated guest kernel boots faster because it does not spend time probing devices Firecracker never exposes.

The GPU Question

Physical device passthrough is not part of the Firecracker model. That is clearest with GPUs. A GPU workload wants PCI passthrough, DMA, device reset semantics, and often memory pinning. Firecracker's soft-allocation goal wants reclaimable guest memory and a short device list. Those goals pull in different directions. For GPU compute workloads, a container or a traditional VM is usually the right tool.

The I/O Ceiling

The simple virtio device paths are predictable and small, but they are not the fastest possible I/O architecture. A platform that needs maximum storage throughput, GPU passthrough, or the broad device ecosystem around virtio-PCI has different requirements from Lambda's function slots. Firecracker optimizes for dense, isolated Linux guests with enough network and block I/O for serverless workloads, not for exposing every hardware feature a host can provide.

The Boot Time in Detail

The 125 ms target measures wall-clock time from InstanceStart to the guest forking /sbin/init under a deliberately small configuration: tuned kernel, minimal rootfs, no serial console, and no unnecessary device probing. The point is not that every VM starts in 125 ms under every host load. The point is that a VM can be cheap enough to sit behind an on-demand container-like platform.

A microVM is not a smaller compatibility VM. It is a deliberately incomplete machine whose omissions pay for one-tenant-per-kernel isolation.

Sources And Further Reading