MicroVMs From The Bottom Up

Chapter 9: Anatomy Of A VMM

An x86 KVM VMM needs five kinds of work: create the VM, map guest memory, create and configure vCPUs, run them, and service the device operations that return to userspace. Setup order varies, and the fifth job begins only after guest execution. The common core is still small: system, VM, and vCPU fds; optional device fds; a shared kvm_run mapping per vCPU; and a host thread that calls KVM_RUN for each vCPU that can execute concurrently.

The KVM Fd Hierarchy

The minimal KVM control hierarchy has three fd classes. /dev/kvm is a character device; VM and vCPU fds are anonymous-inode objects returned by ioctls. Modern KVM can also return device fds for objects such as an Arm VGIC. Each class accepts its own ioctls -- using an ioctl on the wrong fd normally fails with ENOTTY.

The root is /dev/kvm. Opening it gives you a system fd whose most important job is creating VMs. Before doing that, a VMM calls KVM_GET_API_VERSION (_IO(0xAE, 0x00)) and verifies that the result is KVM_API_VERSION, currently 12. Any other result is an unsupported userspace ABI for this VMM.

flowchart TD A["/dev/kvm (system fd)"] A -->|"KVM_GET_API_VERSION -> 12"| A A -->|"KVM_CREATE_VM"| B["VM fd"] B -->|"KVM_SET_USER_MEMORY_REGION"| B B -->|"KVM_CREATE_VCPU"| C["vCPU fd"] B -->|"KVM_CREATE_DEVICE"| D["device fd (optional)"] C -->|"KVM_RUN"| C C -->|"KVM_GET_REGS / KVM_SET_REGS"| C C -->|"KVM_GET_SREGS / KVM_SET_SREGS"| C

KVM_CREATE_VM (_IO(0xAE, 0x01)) on the system fd yields a VM fd that represents the guest address space and its interrupt state. KVM_CREATE_VCPU (_IO(0xAE, 0x41)) on the VM fd yields a vCPU fd -- one per guest core -- that exposes the per-CPU register file and the interface for entering and exiting guest mode. The type byte 0xAE is the KVMIO constant; it appears in every KVM ioctl number by convention.

The fd type expresses ioctl scope, not a complete security boundary. KVM does not define fd transfer between processes as its privilege model, and a vCPU fd still refers to state owned by its VM. Firecracker's jailer constrains the process with namespaces, a changed root, cgroups, dropped privileges, and seccomp; Chapter 18 covers the actual startup sequence.

Job 1: Allocate Guest Memory

A userspace-backed memory slot associates a guest-physical range with a host-virtual range in the VMM. The addresses are not interchangeable: KVM uses the slot to translate a GPA to an HVA when it must access guest memory, while EPT or NPT maps the GPA to host physical pages during guest execution. The VMCS stores an EPT pointer and the VMCB stores nested-paging control state; the translation tables themselves live in host memory and are maintained by KVM.

The registration call is KVM_SET_USER_MEMORY_REGION (_IOW(0xAE, 0x46, struct kvm_userspace_memory_region)), a VM ioctl. The struct is compact:

struct kvm_userspace_memory_region {
    __u32 slot;             /* memory slot index */
    __u32 flags;            /* KVM_MEM_LOG_DIRTY_PAGES | KVM_MEM_READONLY */
    __u64 guest_phys_addr;  /* base GPA of this region */
    __u64 memory_size;      /* size in bytes; 0 deletes the slot */
    __u64 userspace_addr;   /* HVA: address of the mmap'd backing buffer */
};

The slot field is an index into the kernel's memory-slot table. Each slot describes one contiguous GPA-to-HVA mapping. Multiple slots let a VMM represent discontiguous RAM and special memory regions; an MMIO hole is normally the absence of a RAM slot, not a slot of its own. KVM_MEM_LOG_DIRTY_PAGES enables dirty tracking for migration or incremental snapshots, and KVM_MEM_READONLY makes guest writes leave the normal RAM path.

Host requirement: Code that opens /dev/kvm must run on an isolated bare-metal Linux host or in a VM with nested virtualization enabled. The snippets below omit error cleanup to expose the KVM sequence, but every open, mmap, and ioctl result must be checked in real code.

The LWN reference VMM reduces this to its minimum: one mmap call, one slot:

void *mem = mmap(NULL, 0x1000,
    PROT_READ | PROT_WRITE,
    MAP_SHARED | MAP_ANONYMOUS, -1, 0);

struct kvm_userspace_memory_region region = {
    .slot            = 0,
    .guest_phys_addr = 0x1000,
    .memory_size     = 0x1000,
    .userspace_addr  = (uint64_t)mem,
};
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

GPA 0x1000 is chosen so the guest starts executing at CS:IP = 0x0000:0x1000 (flat physical 0x1000) in 16-bit real mode. One page, one slot, one shot.

In Firecracker source checked on July 10, 2026, build_microvm_for_boot() allocates the guest-memory mappings, creates the Kvm and KvmVm objects and vCPUs, then calls register_dram_memory_regions(). That method assigns KVM slots while registering each discontiguous DRAM region. On x86_64 the bzImage load base is HIMEM_START = 0x100000, the zero page is 0x7000, the command line begins at 0x20000, and the I/O APIC window is 0xFEC00000.

Job 2: Create vCPUs And Map kvm_run

KVM_CREATE_VCPU takes a single integer argument -- the vCPU ID, which on x86 becomes the APIC ID. It returns a vCPU fd:

int vcpufd = ioctl(vmfd, KVM_CREATE_VCPU, 0);

The VMM next maps the kvm_run communication area. The kernel writes userspace-exit state directly into memory shared with the VMM:

int mmap_size = ioctl(kvmfd, KVM_GET_VCPU_MMAP_SIZE, 0);
struct kvm_run *run = mmap(NULL, mmap_size,
    PROT_READ | PROT_WRITE, MAP_SHARED, vcpufd, 0);

KVM_GET_VCPU_MMAP_SIZE (_IO(0xAE, 0x04)) is called on the system fd, not the vCPU fd. Userspace must map exactly the ABI size the kernel returns rather than assuming sizeof(struct kvm_run). MAP_SHARED is required because KVM and the VMM exchange fields through the mapping; the guest never writes exit_reason.

struct kvm_run contains an input half and an output half, separated by the exit_reason field:

struct kvm_run {
    /* inputs: VMM writes these before KVM_RUN */
    __u8  request_interrupt_window;
    __u8  immediate_exit;
    __u8  padding1[6];

    /* output: kernel writes this after each VM-exit */
    __u32 exit_reason;

    __u8  ready_for_interrupt_injection;
    __u8  if_flag;
    __u16 flags;
    __u64 cr8;
    __u64 apic_base;

    union {
        struct { /* KVM_EXIT_IO */
            __u8  direction;   /* 0 = IN, 1 = OUT */
            __u8  size;        /* 1, 2, or 4 bytes */
            __u16 port;
            __u32 count;
            __u64 data_offset; /* offset from kvm_run* to the data buffer */
        } io;
        struct { /* KVM_EXIT_MMIO */
            __u64 phys_addr;
            __u8  data[8];
            __u32 len;
            __u8  is_write;
        } mmio;
        struct { /* KVM_EXIT_FAIL_ENTRY */
            __u64 hardware_entry_failure_reason;
            __u32 cpu;
        } fail_entry;
        struct { /* KVM_EXIT_HYPERCALL */
            __u64 nr;
            __u64 args[6];
            __u64 ret;
        } hypercall;
        char padding[256];
    };
};

KVM checks immediate_exit once when KVM_RUN begins and returns -EINTR if it is nonzero. To stop a vCPU already in guest mode, userspace sends a signal to the vCPU thread and sets the byte from that signal handler; writing it from an unrelated thread is not sufficient to cause a hardware exit. request_interrupt_window asks KVM to return when an external interrupt can be injected into the guest.

Job 3: Load The Guest

With memory registered and a vCPU created, the VMM has to put executable code into guest memory and configure the CPU state to match. This job splits naturally into two parts: writing the guest image into the GPA range and setting the register file via KVM_SET_REGS and KVM_SET_SREGS.

The register setup is architecture-specific and fiddly. The LWN minimal VMM boots a tiny flat binary in real mode and sets registers directly:

struct kvm_sregs sregs;
ioctl(vcpufd, KVM_GET_SREGS, &sregs);
sregs.cs.base     = 0;
sregs.cs.selector = 0;
ioctl(vcpufd, KVM_SET_SREGS, &sregs);

struct kvm_regs regs = {
    .rip    = 0x1000,
    .rax    = 2,
    .rbx    = 2,
    .rflags = 0x2,     /* bit 1 is architecturally reserved-set */
};
ioctl(vcpufd, KVM_SET_REGS, &regs);

rflags = 0x2 establishes the architectural reserved-one bit while leaving the other flags clear. A VMM must provide a self-consistent entry state; invalid combinations can produce KVM_EXIT_FAIL_ENTRY before the first instruction retires.

A production VMM targeting a Linux guest does not just poke registers and hope. It implements a boot contract. For Linux on x86, that means placing the kernel, command line, optional initrd, memory map, and boot-parameter structure at addresses the kernel expects, then setting the vCPU state to the selected entry mode. Chapter 10 owns the field-level details; the VMM job here is simply to make the guest's first instruction well-defined.

kvmtool chooses 16-bit real mode. Its x86/kvm-cpu.c sets:

kvm_regs.rip    = arch.boot_ip;   /* must be <= 65535 */
kvm_regs.rsp    = arch.boot_sp;
kvm_regs.rbp    = arch.boot_sp;
kvm_regs.rflags = 0x0000000000000002ULL;

All segment registers (CS, SS, DS, ES, FS, GS) receive the same boot_selector, and the base for each is computed with the standard real-mode left-shift:

static inline uint32_t selector_to_base(uint16_t sel) {
    return (uint32_t)sel << 4;
}

kvmtool also initializes MSRs via KVM_SET_MSRS (_IOW(0xAE, 0x89, struct kvm_msrs)): MSR_IA32_SYSENTER_CS/ESP/EIP are all zeroed, MSR_IA32_TSC is zeroed, and MSR_IA32_MISC_ENABLE has the FAST_STRING bit enabled. FPU state via KVM_SET_FPU (_IOW(0xAE, 0x8d, struct kvm_fpu)) is set to fcw = 0x37f, mxcsr = 0x1f80 -- the x87 control word and SSE control register their reset values.

Firecracker goes further still: it skips real mode, prepares the long-mode or PVH entry state, configures the guest-visible CPUID model, and enters the kernel directly. Those choices are Firecracker-specific, but they still satisfy the same job: load a guest image and make the first KVM_RUN land at a valid kernel entry.

Job 4: Run The Loop

KVM_RUN (_IO(0xAE, 0x80)) is a no-argument vCPU ioctl. It can cross many hardware VM exits that KVM handles internally before returning. A successful return of 0 means KVM prepared a userspace exit in kvm_run->exit_reason; a signal or nonzero immediate_exit produces -1 with errno = EINTR.

A minimal userspace run loop has this shape:

while (1) {
    int rc = ioctl(vcpufd, KVM_RUN, NULL);
    if (rc < 0) {
        if (errno == EINTR) {
            run->immediate_exit = 0;
            continue;
        }
        err(1, "KVM_RUN");
    }
    switch (run->exit_reason) {
    case KVM_EXIT_HLT:
        return 0;                    /* clean shutdown */
    case KVM_EXIT_IO:
        if (run->io.direction == KVM_EXIT_IO_OUT
            && run->io.size == 1
            && run->io.port == 0x3f8
            && run->io.count == 1)
            putchar(*(((char *)run) + run->io.data_offset));
        break;
    case KVM_EXIT_FAIL_ENTRY:
        errx(1, "KVM_EXIT_FAIL_ENTRY: 0x%llx\n",
             run->fail_entry.hardware_entry_failure_reason);
    case KVM_EXIT_INTERNAL_ERROR:
        errx(1, "KVM_EXIT_INTERNAL_ERROR: suberror = 0x%x\n",
             run->internal.suberror);
    default:
        errx(1, "unhandled KVM exit: %u", run->exit_reason);
    }
}

KVM_EXIT_HLT (value 5) is useful as termination in a toy VMM with a userspace-managed LAPIC. With an in-kernel LAPIC, KVM normally handles HLT and blocks the vCPU thread itself. KVM_EXIT_FAIL_ENTRY (value 9) means hardware rejected the VM-entry state; fail_entry.hardware_entry_failure_reason carries the architecture-specific failure reason. KVM_EXIT_INTERNAL_ERROR (value 17) reports a KVM failure that the VMM must inspect and handle according to its policy.

data_offset is a byte offset from the start of the kvm_run struct, not a pointer. The data buffer sits inside the same mmap'd region as the struct, past the fixed header. Casting to (char *)run + run->io.data_offset is correct; dereferencing run->io.data_offset as a pointer is a common first-time mistake that produces a segfault at address ~80.

The full KVM_EXIT_* vocabulary used in production is broader. A selection of the values in include/uapi/linux/kvm.h worth knowing:

Constant Value Meaning
KVM_EXIT_UNKNOWN 0 Hardware exit reason unrecognized
KVM_EXIT_IO 2 Guest PIO (IN/OUT instruction)
KVM_EXIT_HYPERCALL 3 Hypercall configured or defined to exit to userspace
KVM_EXIT_HLT 5 Guest HLT with a userspace-managed LAPIC
KVM_EXIT_MMIO 6 MMIO access KVM could not satisfy
KVM_EXIT_SHUTDOWN 8 Triple fault
KVM_EXIT_FAIL_ENTRY 9 Hardware refused VM-entry; see fail_entry.hardware_entry_failure_reason
KVM_EXIT_INTR 10 Legacy constant; signal interruption is reported as KVM_RUN returning -EINTR
KVM_EXIT_INTERNAL_ERROR 17 KVM internal error; inspect internal.suberror
KVM_EXIT_X86_RDMSR 29 Denied RDMSR delegated with KVM_CAP_X86_USER_SPACE_MSR
KVM_EXIT_X86_WRMSR 30 Denied WRMSR delegated with KVM_CAP_X86_USER_SPACE_MSR
KVM_EXIT_MEMORY_FAULT 39 Fault KVM cannot resolve, with -EFAULT or -EHWPOISON

KVM_EXIT_MEMORY_FAULT is the exception to the normal successful-return rule: userspace may trust that exit reason only with the documented -1 return and error numbers. kvmtool also supports coalesced MMIO through the ring at KVM_COALESCED_MMIO_PAGE_OFFSET. KVM still handles the trapped writes, but batches their records so fewer of them require individual userspace returns.

Job 5: Emulate Devices

Most VM-exits are the guest asking the VMM to do something on its behalf. The two most common forms are PIO (KVM_EXIT_IO) and MMIO (KVM_EXIT_MMIO).

PIO exits arrive when the guest executes an IN or OUT instruction. The io sub-struct in kvm_run carries everything needed:

For KVM_EXIT_IO_OUT, the data is already in the buffer when the VMM reads it; the VMM routes the write to the appropriate emulated device. For KVM_EXIT_IO_IN, the VMM writes the device's response into the buffer and then re-enters KVM_RUN -- the kernel delivers those bytes to the IN instruction as if they came from real hardware.

MMIO exits report a memory-mapped access that KVM could not satisfy. A GPA outside every RAM slot is a common starting point, after KVM decodes the guest instruction and classifies the access as MMIO. The mmio sub-struct carries:

The VMM dispatches phys_addr to the emulated device that owns that GPA range. Firecracker uses virtio-mmio by default and can enable virtio-pci on x86_64. Configuration-register accesses can return through its MMIO bus, while registered queue notifications hit ioeventfds in KVM and bypass the vCPU exit dispatcher.

Device completion may require a guest interrupt, but it need not occur in the MMIO exit handler. Firecracker binds device eventfds to legacy GSI or MSI routes with KVM_IRQFD; writing an enabled eventfd asks KVM to inject the interrupt without another vCPU-thread userspace round trip. The minimal VMM has no interrupt controller or asynchronous device model.

The vCPU Thread Model

KVM_RUN occupies its calling thread while that vCPU is executing or blocked inside KVM. Concurrent execution of multiple vCPUs therefore requires multiple host threads, and production VMMs normally dedicate one long-lived thread to each vCPU. The kernel can schedule those threads on different physical CPUs, allowing guest vCPUs to execute in parallel.

The sequence from process start to a running multi-vCPU guest follows this shape:

sequenceDiagram participant M as "VMM main thread" participant T1 as "vCPU thread 0" participant T2 as "vCPU thread 1" participant K as "KVM kernel module" M->>K: open /dev/kvm, KVM_CREATE_VM M->>K: KVM_SET_USER_MEMORY_REGION M->>K: KVM_CREATE_VCPU (id=0) M->>K: KVM_CREATE_VCPU (id=1) M->>K: KVM_SET_REGS / KVM_SET_SREGS (both vCPUs) M->>T1: spawn thread, pass vcpufd 0 M->>T2: spawn thread, pass vcpufd 1 T1->>K: KVM_RUN (blocks) T2->>K: KVM_RUN (blocks) K-->>T1: VM-exit -> exit_reason T1->>T1: handle exit, re-enter K-->>T2: VM-exit -> exit_reason T2->>T2: handle exit, re-enter

KVM recommends that all ioctls for a given vCPU run from one userspace thread. That thread need not be the thread that called KVM_CREATE_VCPU; VMMs commonly create the fds during setup and then hand each vCPU to its long-lived run thread. Concurrent ioctls on the same vCPU fd can race over shared state and are the pattern to avoid.

Interrupting A Running vCPU

When the VMM needs to pull a vCPU out of guest mode -- to inject an interrupt, to handle an API request, to stop the VM -- it cannot simply call a function, because the vCPU thread is blocked inside the kernel. The mechanism has three parts:

  1. Publish the control event, then arrange for kvm_run->immediate_exit to become 1. The KVM documentation's simple pattern stores it in the signal handler; Firecracker stores it before sending the signal and pairs release/acquire fences across the handler.
  2. Send the signal to the vCPU thread. If it is in guest mode, signal delivery causes KVM to leave guest execution; if the signal arrives first, the nonzero byte makes the next KVM_RUN return immediately.
  3. On -EINTR, process the pending control event and clear immediate_exit before a later run.

The vCPU thread on the other side clears immediate_exit back to 0 and checks for pending events before re-entering the loop.

An alternative is KVM_SET_SIGNAL_MASK (_IOW(0xAE, 0x8b, struct kvm_signal_mask)), which lets the VMM declare exactly which signals interrupt KVM_RUN. Any unmasked signal that arrives during the ioctl causes it to return -EINTR. This is useful when the process has other signal handlers that should not disturb the run loop.

The kernel also has an internal wake path, kvm_vcpu_kick(), used when one component of the KVM code needs to stop a vCPU that is executing inside the guest. kvm_vcpu_kick() sends an inter-processor interrupt (IPI) to the physical CPU running the guest, causing a VM-exit, after which the kernel checks the vcpu->requests bitmap (set by kvm_make_request()) before allowing re-entry.

Firecracker's Thread Categories

Firecracker follows the same one-host-thread-per-vCPU rule, then adds structure around it: an API thread accepts configuration over a Unix socket, a VMM thread owns device emulation and lifecycle control, and one vCPU thread per guest CPU spends most of its life inside KVM_RUN. Chapter 13 opens the exact source-level thread model. For a generic VMM, the lesson is that the vCPU thread should stay single-purpose and reactive; complexity belongs in the VMM/device side and crosses to the vCPU thread through explicit events.

Firecracker uses rust-vmm's kvm-bindings for kernel ABI types and kvm-ioctls for the Kvm, VmFd, VcpuFd, and device-fd wrappers. VcpuFd::run() maps successful KVM_EXIT_* values to VcpuExit variants. KVM_EXIT_MEMORY_FAULT is exceptional: KVM pairs it with a -1 return and EFAULT or EHWPOISON, so the wrapper checks that exact combination before returning VcpuExit::MemoryFault; for other errors, exit_reason may be stale.

One Minimal Order

The VM fd must exist before memory slots, vCPUs, or emulated interrupt devices can be attached. Memory registration and vCPU creation do not have a universal order -- current Firecracker creates its vCPUs before registering base DRAM -- but guest memory and entry state must be ready before the first KVM_RUN. A teaching VMM can use this order:

flowchart LR A["1. Allocate memory\n(mmap + KVM_SET_USER_MEMORY_REGION)"] B["2. Create vCPUs\n(KVM_CREATE_VCPU + mmap kvm_run)"] C["3. Load guest\n(write image + KVM_SET_REGS / KVM_SET_SREGS)"] D["4. Run the loop\n(KVM_RUN per vCPU thread)"] E["5. Emulate devices\n(KVM_EXIT_IO / KVM_EXIT_MMIO handlers)"] A --> B --> C --> D --> E E -->|"re-enter"| D

Jobs 1 through 3 are setup. The vCPU then spends most of its time inside KVM_RUN; job 5 runs only for operations KVM returns to userspace. ioeventfd, irqfd, in-kernel irqchips, and hardware virtualization all exist to keep common operations out of that boundary.

Sources And Further Reading