MicroVMs From The Bottom Up

Chapter 13: Firecracker Architecture

What does one firecracker process contain, and how are its parts connected? The process must serve a REST API, run one or more guest vCPU loops, emulate a minimal device set, and do all of it while keeping the guest's influence over the host kernel as narrow as a handful of ioctls. This chapter traces the architecture that satisfies those constraints.

One Process, One MicroVM

Firecracker's design document states the invariant plainly: "Each Firecracker process encapsulates one and only one microVM." There is no multiplexing or in-process VM table. Running N microVMs means N firecracker processes. The host kernel's process isolation is therefore part of the security boundary, alongside hardware virtualization and the jailer's chroot.

Inside that one process, three categories of OS thread do distinct work.

flowchart TD proc["firecracker process"] api["fc_api thread\n(API server, control plane)"] vmm["VMM thread\n(event loop, device emulation)"] vcpu0["fc_vcpu 0\n(KVM_RUN loop)"] vcpu1["fc_vcpu 1\n(KVM_RUN loop)"] kvm["/dev/kvm"] proc --> api proc --> vmm proc --> vcpu0 proc --> vcpu1 api -->|"mpsc + EventFd"| vmm vmm -->|"mpsc VcpuEvent"| vcpu0 vmm -->|"mpsc VcpuEvent"| vcpu1 vcpu0 -->|"KVM_RUN ioctl"| kvm vcpu1 -->|"KVM_RUN ioctl"| kvm

The API thread is named fc_api. It runs the HTTP server over a Unix domain socket and translates requests into VMM actions. The VMM thread owns the in-memory Vmm, its KVM VM object, device backends, MMDS, and the EventManager epoll loop. One thread per guest CPU, named fc_vcpu 0, fc_vcpu 1, and so on, calls KVM_RUN and handles the exits it returns. Firecracker treats these vCPU threads as hostile once they start and gives them the narrowest seccomp-BPF filter.

The REST API Over a Unix Socket

Every Firecracker API operation arrives through a Unix stream socket, never TCP. The --api-sock argument sets its path; /run/firecracker.socket is a common choice. With the jailer, the socket is inside the chroot at <chroot_base>/<exec_file_name>/<id>/root/<api-sock>. A host using the standard chroot base would therefore see /srv/jailer/firecracker/<id>/root/run/firecracker.socket for that example path.

Firecracker uses its micro_http crate rather than a general-purpose HTTP stack. The crate implements the HTTP/1.x subset Firecracker needs over Unix sockets and is pulled from the firecracker-microvm/micro-http Git repository. The server enforces Firecracker's configured API payload limit. SPECIFICATION.md promises API socket availability within 8 CPU ms of process start; CPU time is used because wall-clock startup varies with host scheduling.

The API contract is the OpenAPI document at src/firecracker/swagger/firecracker.yaml. It defines the resources and request bodies; the Rust request parsers and VmmAction variants implement that contract.

How the API Thread Talks to the VMM Thread

The two threads share two std::sync::mpsc channels and one EventFd:

API thread                             VMM thread
   |                                       |
   |--- Sender<ApiRequest> ------------->  |   (boxed VmmAction)
   |--- EventFd (api_event_fd).write() --> |   (wake-up signal)
   |<-- Sender<ApiResponse> -------------- |   (reply)

ApiServer::new() takes an mpsc::Sender<ApiRequest>, an mpsc::Receiver<ApiResponse>, and the eventfd used to notify the VMM thread. When a request arrives, the API thread sends one boxed action, writes 1 to the semaphore-mode api_event_fd, and blocks for the response. Before boot, the VMM thread blocks directly on the request channel and consumes one eventfd count with each message. After boot, the eventfd is an EventManager subscriber: epoll wakes the VMM thread, the adapter receives one action, and RuntimeApiController handles it before sending the response. The channels separate transport parsing from VM mutation, but API actions still execute synchronously on the VMM thread.

The vCPU threads can continue guest execution while the VMM thread handles a runtime request, but an exit that needs userspace work can still wait for the relevant lock or handler. The separation is a control-plane boundary, not a guarantee that slow VMM work has no guest-visible latency.

The VMM Thread: Events and Devices

The VMM thread owns the in-memory Vmm struct and the KvmVm it contains. Its central primitive is EventManager, an epoll-based event loop. Every device backend and the VMM itself register as event subscribers. When a vCPU exits with KVM_EXIT_MMIO -- the exit reason is 6 in <linux/kvm.h> -- the exit delivers the MMIO address and data through the kvm_run struct; the vCPU thread that took the exit dispatches directly to the device handler via mmio_bus.read() or mmio_bus.write() on the Peripherals struct attached to that vCPU, without waking the VMM thread.

Current Firecracker backends include block, network, vsock, balloon, entropy, pmem, and virtio-mem devices; vhost-user support is used for block. Devices use the MMIO transport by default, with optional PCI support on x86-64. The legacy model is deliberately small. The serial device wraps vm_superio::Serial, while Firecracker's x86 i8042 implementation is in-tree and handles keyboard-controller traffic such as Ctrl-Alt-Del and reset commands. AArch64 also wraps the vm-superio PL031 RTC.

When a vCPU state machine reaches its exited state after a stop or emulation error, it writes to the shared exit_evt: EventFd cloned into each Vcpu. The VMM's EventManager subscribes to this fd through vcpus_exit_evt(). That is how the VMM detects vCPU termination without polling.

vCPU Threads: The KVM_RUN Loop

KvmVm::start_vcpus() calls vcpu.start_threaded(...) for each vCPU. Inside start_threaded, the spawn looks like:

thread::Builder::new() .name(format!("fc_vcpu {}", self.kvm_vcpu.index)) .spawn(move || { ... })

vCPU threads start in a paused state. Vmm::resume_vm() must be called to release them into guest execution. That call is the hard boundary between the pre-boot phase and the running phase; before it, configuration is still mutable.

Inside the vCPU thread's loop, the call is self.kvm_vcpu.fd.run(), which issues the KVM_RUN vcpu ioctl -- _IO(KVMIO, 0x80) -- on the vcpu fd. The ioctl does not return until the guest causes a VM exit. The exit reason lives in the kvm_run struct that KVM maps into the VMM's address space via mmap on the vcpu fd; the mapping size comes from KVM_GET_VCPU_MMAP_SIZE (_IO(KVMIO, 0x04)). Firecracker reads this through kvm-ioctls's KvmRunWrapper, which exposes the struct fields safely from Rust.

The common handle_kvm_exit() and the architecture-specific peripheral handler dispatch these important VcpuExit variants:

Exit kvm.h constant Action
MmioRead(addr, data) KVM_EXIT_MMIO (6) Read device register, fill data buffer
MmioWrite(addr, data) KVM_EXIT_MMIO (6) Write device register
IoIn(port, data) KVM_EXIT_IO (2) x86-64: read the PIO bus, filling zeros first
IoOut(port, data) KVM_EXIT_IO (2) x86-64: write the PIO bus
SystemEvent(RESET) KVM_EXIT_SYSTEM_EVENT (24) Return VcpuEmulation::Stopped
SystemEvent(SHUTDOWN) KVM_EXIT_SYSTEM_EVENT (24) Return VcpuEmulation::Stopped
FailEntry KVM_EXIT_FAIL_ENTRY (9) Return FaultyKvmExit
InternalError KVM_EXIT_INTERNAL_ERROR (17) Return FaultyKvmExit
Any other architecture-specific exit varies Return UnhandledKvmExit

On x86-64, Hlt and Shutdown fall through to the unexpected-exit branch and become UnhandledKvmExit; clean termination follows the explicit reset and shutdown paths described in Chapter 8.

To interrupt a vCPU that may already be inside KVM_RUN, VcpuHandle::send_event() first sends a VcpuEvent, sets immediate_exit in the shared kvm_run mapping, executes a release fence, and signals the vCPU thread with Firecracker's real-time kick signal. The signal ejects a currently running vCPU; immediate_exit also prevents a race from re-entering guest mode before the command is observed. The vCPU clears the field after EINTR and handles events such as Pause, Resume, and SaveState in its state machine.

The KVM Ioctl Hierarchy

Firecracker owns the usual KVM object hierarchy: the system fd from /dev/kvm, one VM fd for the microVM, one vCPU fd per guest CPU, and any KVM device fds created for architecture-specific devices. The architecture point is ownership, not ioctl encoding. The VMM thread owns the VM fd and VM-wide device registrations; each vCPU thread owns its own KVM_RUN loop on a vCPU fd. Chapter 5 and Appendix C are the ioctl reference; this chapter uses the hierarchy only to explain which Firecracker thread can touch which kernel object.

The rust-vmm Crates

Firecracker assembles several shared rust-vmm crates with its own device model and control plane. Exact dependency versions change; the architectural roles are more durable:

Crate Role
kvm-bindings Rust FFI types corresponding to structures in <linux/kvm.h>
kvm-ioctls Wrappers such as Kvm, VmFd, VcpuFd, and DeviceFd
vm-memory Guest addresses, mmap-backed guest regions, and dirty bitmaps
vmm-sys-util EventFd, ioctl helpers, and other Linux utilities
linux-loader Kernel image parsing and loading support
vm-allocator Guest address and resource allocators
vm-superio The 16550A serial model and AArch64 PL031 RTC model
vhost The vhost-user frontend used by the block backend
vm-fdt AArch64 flattened device tree construction
micro_http Firecracker's HTTP/1.x server over a Unix socket

kvm-ioctls and kvm-bindings

kvm-bindings is what you use when you need to pass a struct into a KVM ioctl: it provides the C types from <linux/kvm.h> as Rust FFI structs, generated by rust-bindgen from the kernel headers. kvm-ioctls wraps those into safe Rust: the Kvm struct wraps /dev/kvm; VmFd wraps the VM fd; VcpuFd wraps the vcpu fd. VcpuFd::run() is the KVM_RUN call. VcpuFd::get_regs() and set_regs() read and write kvm_regs; get_sregs() and set_sregs() handle kvm_sregs. VmFd::register_irqfd() and register_ioeventfd() wire up the kernel-side interrupt and I/O notification mechanisms without needing a VM exit.

vm-memory

GuestAddress is a newtype over u64 representing a guest physical address (GPA). The separation matters: the host never accidentally uses a GPA as a host virtual address (HVA). GuestMemoryMmap is the concrete backed-by-mmap type; it holds a collection of GuestRegionMmap objects, each mapping a contiguous GPA range to a MmapRegion on the host. Cross-region reads and writes -- a buffer that straddles two memory regions -- are handled transparently by the GuestMemory trait. The backend-bitmap Cargo feature (enabled by Firecracker) adds per-region dirty tracking; this is the userspace side of the dirty-page mechanism that KVM_MEM_LOG_DIRTY_PAGES enables in the kernel.

The Virtio Queue Firecracker Does Not Borrow

One notable absence from the crate list: Firecracker does not use the virtio-queue crate from rust-vmm. It keeps its own VIRTIO 1.2 split-virtqueue implementation in src/vmm/src/devices/virtio/queue.rs, with Kani model-checker coverage and Firecracker-specific notification helpers. Chapter 11 owns the ring layout. The architectural fact here is that Firecracker shares many rust-vmm building blocks while deliberately keeping its virtqueue implementation internal.

The Pre-Boot / Running State Machine

A Firecracker process has a pre-boot controller and a runtime controller. The first accumulates configuration and constructs or restores a VM; the second admits the operations supported after that transition. This division is enforced by distinct request handlers, not by client convention.

The REST endpoint GET / returns an InstanceInfo struct whose state field is one of three strings: "Not started", "Running", or "Paused". These states determine which operations are legal.

stateDiagram-v2 [*] --> NotStarted : process starts NotStarted --> Running : "PUT /actions InstanceStart" NotStarted --> Paused : "PUT /snapshot/load (resume_vm=false)" NotStarted --> Running : "PUT /snapshot/load (resume_vm=true)" Running --> Paused : "PATCH /vm {state: Paused}" Paused --> Running : "PATCH /vm {state: Resumed}" Paused --> [*] : process exit Running --> [*] : guest shutdown / process exit

PrebootApiController

Before InstanceStart, PrebootApiController handles boot source, machine, CPU, device, MMDS, logger, metrics, and serial configuration. It also accepts LoadSnapshot. A successful StartMicroVm or snapshot load returns a built Vmm; the caller then replaces the pre-boot loop with RuntimeApiController.

The MachineConfig that the pre-boot controller accepts has these fields:

Field Type Default Constraint
vcpu_count u8 1 1-32; must be 1 or even if SMT enabled; SMT unsupported on aarch64
mem_size_mib usize 128 > 0; must be a multiple of 2 if using 2 MiB huge pages; must be >= balloon target
smt bool false x86-64 only
cpu_template optional enum none Static or custom CPU feature template
track_dirty_pages bool false Required for diff snapshots
huge_pages enum None None or Hugetlbfs2M

track_dirty_pages: true causes Firecracker to pass KVM_MEM_LOG_DIRTY_PAGES in the flags field of kvm_userspace_memory_region when registering DRAM slots. It cannot be toggled after boot.

RuntimeApiController

After the transition, RuntimeApiController handles pause, resume, snapshots, device updates, memory resizing, balloon hinting, MMDS access, and configuration reads. When PCI transport is enabled, its preview hotplug interface also accepts block, network, and pmem insertion plus hot-unplug. Pre-boot-only actions return VmmActionError::OperationNotSupportedPostBoot.

CreateSnapshot is only valid in Paused state. After handling a pause request, ApiServerAdapter deliberately stops returning to EventManager::run() and blocks on API requests until it receives Resume; device events and periodic metric events are therefore frozen along with the vCPUs. The vCPU controller uses a 30-second response timeout while coordinating state changes.

Boot Handoff

sequenceDiagram
  participant API as "API thread (fc_api)"
  participant VMM as "VMM thread"
  participant KVM as "/dev/kvm"
  participant vCPU as "fc_vcpu 0"
  participant Guest as "Guest kernel"

  API->>VMM: InstanceStart (via mpsc + EventFd)
  VMM->>KVM: "KVM_CREATE_VM -> VmFd"
  VMM->>KVM: "KVM_CREATE_VCPU -> VcpuFd"
  VMM->>KVM: "KVM_SET_USER_MEMORY_REGION (DRAM slots)"
  VMM->>VMM: load kernel, attach devices, configure system
  VMM->>vCPU: spawn fc_vcpu 0 (paused, filter installed)
  VMM->>vCPU: resume_vm() (VcpuEvent::Resume)
  vCPU->>KVM: "KVM_RUN"
  KVM-->>Guest: guest executes at kernel entry
  KVM-->>vCPU: "VM exit: KVM_EXIT_MMIO (virtio probe)"
  vCPU->>vCPU: "mmio_bus.read/write (inline, no VMM wake)"
  vCPU->>KVM: "KVM_RUN (re-enter)"
  Guest->>Guest: "kernel_init execve /sbin/init"

The source-level boot sequence lives in src/vmm/src/builder.rs, but its shape is simple at architecture level: an API action wakes the VMM thread, the VMM builds guest memory and devices from pre-boot configuration, starts paused vCPU threads with their filters, then resumes the vCPUs into KVM_RUN. The caller installs the VMM filter before entering the runtime event loop. Chapter 15 owns the operator-facing configuration calls that feed this builder.

The Seccomp Boundary

Each thread category receives a distinct seccomp-BPF filter. The API and vCPU filters are installed inside their new threads. The VMM filter is installed on the main VMM thread after the builder returns and before it enters the event loop. The vCPU threads have the narrowest filter because they are treated as hostile as soon as guest execution starts; the API and VMM threads need wider host interfaces for sockets, files, and epoll. The important architectural property is per-thread containment. Chapter 18 covers the jailer around the process, and Chapter 19 covers the filters themselves.

What One Firecracker Process Holds Open

After a guest is started, a single firecracker process holds:

The exact fd count depends on queue count, transport, optional devices, logging, metrics, and snapshot configuration. SPECIFICATION.md sets a VMM-thread memory-overhead target of at most 5 MiB for its tested one-vCPU, 128 MiB guest, while noting that workload and VMM configuration can exceed it. This small control plane and device model are the foundation for the device choices in the next chapter.

Sources and Further Reading