MicroVMs From The Bottom Up

Chapter 8: VM Exits Up Close

Every intercepted operation a guest performs -- reading a byte from a UART register, writing a trapped virtio queue notifier, asking CPUID what the CPU is -- requires the hardware to stop guest execution and enter the host's virtualization handler. That hardware transition is a VM exit. Some exits are handled entirely inside KVM; others become userspace exits, where KVM_RUN returns and the VMM must act. The distinction matters because a hardware exit can be expensive without also waking the userspace vCPU loop.

Why Exits Exist

On a physical machine, the CPU is allowed to complete most operations without asking software above the kernel. A load from RAM, a branch, a user-to-kernel syscall, or a page fault all follow ordinary architectural rules. Device access is also part of that hardware model: an OUT to an I/O port or a store to an MMIO address reaches whatever device the platform decodes at that address. The kernel is trusted to program those mappings because it owns the machine.

In a VM, the guest kernel is not allowed to own the real machine. It can write what it believes is an APIC register, load what it believes is its own CR3, or touch what it believes is a device register, but the host has to decide whether that operation changes virtual state, real hardware state, or no state at all. A VM exit is the CPU's way to stop at exactly those boundaries. The exit hands control to KVM with enough context to answer the question: should this operation be emulated, completed in-kernel, reflected to userspace, injected back as a guest exception, or treated as fatal?

That makes an exit different from an interrupt delivered to the guest. A guest interrupt is part of the machine the guest sees. A VM exit is hidden from the guest unless KVM chooses to reflect a visible event back into it. The guest's RIP, flags, and pending event state are preserved so execution can resume as if the detour never happened.

What The Hardware Does

On Intel, the exit transfers the CPU from VMX non-root operation to VMX root operation. At the hardware level, this means the processor writes the cause to a 32-bit field in the VMCS called VM_EXIT_REASON. Bits 15:0 carry the basic exit reason -- a numeric code identifying what caused the exit. Bit 31 is set only on VM-entry failures, not on exits from a running guest. Bits 28 and 29 carry special meanings for the Monitor Trap Flag and exits from VMX-root mode; they are zero in the common case.

For exit reasons that define it, the VMCS VM_EXIT_INSTRUCTION_LEN field holds the byte length of the intercepted instruction -- 2 for CPUID, for instance. EXIT_QUALIFICATION also has reason-specific meaning. For a PIO exit, it encodes the port number, access size, direction, and whether the instruction was INS or OUTS; for an EPT violation, it encodes the access type. KVM reads the fields valid for that reason before deciding what to do.

On AMD, the equivalent structure is the VMCB, and the exit writes a 64-bit exit code to the EXITCODE field, with additional context in EXITINFO1 and EXITINFO2. The numeric values differ -- PIO exits are SVM_EXIT_IOIO = 0x07b, CPUID is SVM_EXIT_CPUID = 0x072, MSR access is SVM_EXIT_MSR = 0x07c, triple fault is SVM_EXIT_SHUTDOWN = 0x07f -- but the conceptual roles are identical. For an IOIO exit, EXITINFO1 holds a bitmask: bit 0 is direction (0=OUT, 1=IN), bit 2 is a string instruction, bit 3 is a REP prefix, bits 6:4 encode the access size, and bits 31:16 hold the port number. KVM's io_interception() handler in arch/x86/kvm/svm/svm.c reads these masks and populates the same generic kvm_vcpu_io structure that the VMX path uses, so both architectures surface to userspace through an identical kvm_run.io struct.

A detail that surprises people who expect the hardware to be comprehensive: neither VMX nor SVM saves general-purpose registers automatically on exit. The hypervisor exit stub must save them before using them. VMX does save CR0, CR3, CR4, RSP, RIP, RFLAGS, segment descriptors, GDTR, IDTR, TR, and a configurable set of MSRs -- but RAX through R15 are the VMM's responsibility. The host register state is restored from the VMCS host-state area, not from general-purpose register saves.

How KVM Routes Exits

KVM's kvm_arch_vcpu_ioctl_run() in arch/x86/kvm/x86.c calls vcpu_enter_guest(), which reaches the architecture-specific VM-entry path. After an exit, KVM dispatches the hardware reason to a handler. In the common x86 handler convention, 1 means that KVM handled the event and can continue its kernel run loop, while 0 means that KVM prepared a userspace exit. Fast paths and errors use additional return conventions, so this is a useful model rather than a universal signature for every exit path.

flowchart TD A["Guest instruction / event"] --> B["VM exit: hardware writes\nVM_EXIT_REASON to VMCS"] B --> C["KVM exit handler\n(arch/x86/kvm/vmx/vmx.c)"] C --> D{return value?} D -- "1: in-kernel" --> E["VMRESUME -- guest\ncontinues immediately"] D -- "0: userspace" --> F["KVM_RUN ioctl returns\nto VMM process"] F --> G["VMM reads kvm_run.exit_reason\nservices the exit"] G --> H["VMM calls ioctl(vcpu_fd, KVM_RUN)\nagain"] H --> E

Several exit categories never reach userspace. EPT violations on RAM -- a guest touching a mapped guest-physical address whose EPT entry does not yet exist -- are resolved by kvm_mmu_page_fault(), which installs the mapping and resumes. The VMM never sees these; they are the hardware page-fault mechanism working normally.

CPUID causes an unconditional VMX exit (basic exit reason 10). KVM handles it entirely in-kernel by consulting the per-vCPU table set by KVM_SET_CPUID2. There is no KVM_EXIT_CPUID in the KVM uAPI.

MSR accesses are governed by a 4 KiB bitmap whose address is stored in the VMCS. It covers MSR addresses 0x0-0x1fff and 0xc0000000-0xc0001fff, with separate read and write bits. A clear bit permits that access without a VM exit; an intercepted access enters KVM, which handles the guest-visible MSRs it owns, including IA32_TSC, IA32_APIC_BASE, and the KVM paravirtual range.

HLT, when the "HLT exiting" bit (bit 7 of the processor-based VM-execution controls) is set, sends the vCPU thread to kvm_vcpu_block() in virt/kvm/kvm_main.c. With halt polling active, the thread spins for up to halt_poll_ns nanoseconds checking for a pending wakeup; an interrupt arriving in that window resumes the guest with no scheduler round-trip.

In-kernel irqchip accesses -- after KVM_CREATE_IRQCHIP -- route APIC and interrupt-controller operations through KVM's interrupt model. Depending on APIC virtualization support, hardware may avoid the exit entirely or KVM may service it without a KVM_RUN return. Creating the PIT is a separate choice.

The exits that do reach userspace are defined by constants in include/uapi/linux/kvm.h, part of the stable KVM ABI. The x86-relevant subset:

Value Constant When it fires
0 KVM_EXIT_UNKNOWN Hardware exit reason KVM did not recognize
2 KVM_EXIT_IO PIO IN/OUT to a port with no in-kernel owner
5 KVM_EXIT_HLT Guest HLT when the local APIC is managed in userspace
6 KVM_EXIT_MMIO MMIO to a GPA not backed by RAM or an in-kernel device
8 KVM_EXIT_SHUTDOWN Triple fault
9 KVM_EXIT_FAIL_ENTRY Hardware refused VM entry
17 KVM_EXIT_INTERNAL_ERROR KVM subsystem error
24 KVM_EXIT_SYSTEM_EVENT Architecture-specific shutdown, reset, crash, or related event
29 KVM_EXIT_X86_RDMSR RDMSR delegated to userspace (requires KVM_CAP_X86_USER_SPACE_MSR)
30 KVM_EXIT_X86_WRMSR WRMSR delegated to userspace (requires KVM_CAP_X86_USER_SPACE_MSR)

The kvm_run Exit Fields

Chapter 5 introduced the struct kvm_run shared page as part of the API. This chapter needs only the fields that carry exit information. The VMM maps the page once, before the first KVM_RUN:

int mmap_size = ioctl(kvm_fd, KVM_GET_VCPU_MMAP_SIZE, 0);
struct kvm_run *run = mmap(NULL, mmap_size, PROT_READ|PROT_WRITE,
                           MAP_SHARED, vcpu_fd, 0);

This page persists across KVM_RUN calls. The VMM reads it after each ioctl returns. The abbreviated layout, from include/uapi/linux/kvm.h:

struct kvm_run {
    /* VMM writes before KVM_RUN */
    __u8  request_interrupt_window;  /* exit when guest IF opens */
    __u8  immediate_exit;            /* nonzero: make the next KVM_RUN return -EINTR */
    __u8  padding1[6];

    /* KVM writes after exit */
    __u32 exit_reason;               /* KVM_EXIT_* */
    __u8  ready_for_interrupt_injection;
    __u8  if_flag;
    __u16 flags;

    /* in/out */
    __u64 cr8;
    __u64 apic_base;

    union {
        struct {                     /* KVM_EXIT_IO */
            __u8  direction;         /* 0 = IN, 1 = OUT */
            __u8  size;              /* 1, 2, or 4 bytes */
            __u16 port;
            __u32 count;             /* repetition count for REP INS/OUTS */
            __u64 data_offset;       /* offset from start of kvm_run to data */
        } io;

        struct {                     /* KVM_EXIT_MMIO */
            __u64 phys_addr;         /* guest physical address */
            __u8  data[8];           /* fill on read, read on write */
            __u32 len;
            __u8  is_write;
        } mmio;

        struct {                     /* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */
            __u8  error;             /* out: 0 = ok, 1 = inject #GP */
            __u8  pad[7];
            __u32 reason;            /* KVM_MSR_EXIT_REASON_* bitmask */
            __u32 index;             /* MSR address (RCX) */
            __u64 data;              /* RDMSR fill / WRMSR value */
        } msr;

        struct {                     /* KVM_EXIT_FAIL_ENTRY */
            __u64 hardware_entry_failure_reason;
            __u32 cpu;
        } fail_entry;

        struct {                     /* KVM_EXIT_INTERNAL_ERROR */
            __u32 suberror;
            __u32 ndata;
            __u64 data[16];
        } internal;

        /* KVM_EXIT_HLT: exit_reason alone is the signal; no union member */
        /* KVM_EXIT_SHUTDOWN: no union member */

        char padding[256];
    };

    __u64 kvm_valid_regs;
    __u64 kvm_dirty_regs;
};

One detail that trips people: the KVM_EXIT_IO data is not stored inside the struct. The io.data_offset field is a byte offset from the start of the kvm_run page. The VMM accesses it as (char *)run + run->io.data_offset. The separation exists because REP string I/O (INS, OUTS) can move more data than fits in eight bytes, and inline storage would overflow other struct fields.

immediate_exit is checked once when KVM_RUN starts. Setting it does not by itself eject a vCPU that is already running; the usual protocol sets it from a signal handler while the signal interrupts the vCPU thread. KVM then returns -EINTR. Chapter 5 covers that kick sequence in full.

Servicing Each Exit

PIO: KVM_EXIT_IO

When a guest executes an IN or OUT instruction to a port that is not in-kernel owned, KVM fills run->io and returns. The VMM reads run->io.port, run->io.direction, run->io.size, and run->io.count. For an OUT (guest write, direction == 1), the data is already in the buffer at data_offset. For an IN (guest read, direction == 0), the VMM must fill the buffer before returning to KVM_RUN. The canonical pattern, from the LWN KVM tutorial:

case KVM_EXIT_IO:
    if (run->io.direction == KVM_EXIT_IO_OUT && run->io.port == 0x3f8)
        putchar(*((char *)run + run->io.data_offset));
    break;

Port 0x3f8 is the conventional first COM-port base. A Linux guest configured with a ttyS0 console can send boot messages there. The one-byte example is enough to print simple UART output in a toy VMM, but it is not a complete 16550A model and does not handle access widths or repeated string I/O.

In Firecracker, the dispatch is more structured. In src/vmm/src/arch/x86_64/vcpu.rs, VcpuExit::IoIn(addr, data) and VcpuExit::IoOut(addr, data) are delivered to pio_bus.read() and pio_bus.write(). The pio_bus is a sorted map of address ranges to registered device handlers. If the port has no registered handler, the read data is zero-filled with a warn! log and the exit returns Handled -- the guest sees zeros rather than a fault.

MMIO: KVM_EXIT_MMIO

KVM_EXIT_MMIO means KVM could not satisfy an emulated memory-mapped access itself. A common path starts with an Intel EPT violation (basic reason 48) or AMD nested page fault (SVM_EXIT_NPF = 0x400) to a GPA outside RAM. KVM decodes the instruction, recognizes the access as MMIO, and exposes the address, width, direction, and data to userspace.

The VMM reads run->mmio.phys_addr, run->mmio.len, and run->mmio.is_write. For a guest write, run->mmio.data already contains the value. For a guest read, the VMM fills run->mmio.data before returning.

Firecracker handles this in src/vmm/src/vstate/vcpu.rs:

// Simplified from the current exit handler.
VcpuExit::MmioRead(addr, data) => {
    data.fill(0);
    if let Some(mmio_bus) = &peripherals.mmio_bus {
        if let Err(err) = mmio_bus.read(addr, data) {
            warn!("Invalid MMIO read: {err}");
        }
        METRICS.vcpu.exit_mmio_read.inc();
    }
    Ok(VcpuEmulation::Handled)
}
VcpuExit::MmioWrite(addr, data) => {
    if let Some(mmio_bus) = &peripherals.mmio_bus {
        if let Err(err) = mmio_bus.write(addr, data) {
            warn!("Invalid MMIO write: {err}");
        }
        METRICS.vcpu.exit_mmio_write.inc();
    }
    Ok(VcpuEmulation::Handled)
}

Firecracker maintains separate Bus instances for MMIO devices and, on x86_64, PIO devices such as the UART and i8042 controller. Each uses a BTreeMap of address ranges and resolves the range immediately before the requested address. Virtio transport configuration accesses can reach this path. Registered queue notifications do not: virtio-mmio QueueNotify writes and virtio-pci notification-BAR writes hit their ioeventfds in KVM instead of returning as KVM_EXIT_MMIO.

HLT: KVM_EXIT_HLT

KVM_EXIT_HLT (value 5) carries no data fields; the exit reason is the complete signal. With an in-kernel local APIC, KVM changes the vCPU's MP state and its run loop calls kvm_vcpu_halt() instead of returning to userspace. The userspace exit is for a userspace-managed local APIC, whose VMM must also manage wake events.

Current Firecracker creates an in-kernel irqchip on x86_64, so ordinary guest HLT instructions remain in KVM. Its run loop has no successful VcpuExit::Hlt arm; if such an exit reaches it, the architecture-specific fallback reports UnhandledKvmExit("Hlt").

CPUID: No Userspace Exit

CPUID causes an unconditional VM exit on both Intel (basic exit reason 10) and AMD (SVM_EXIT_CPUID = 0x072), but KVM handles it entirely in-kernel. There is no KVM_EXIT_CPUID in the uAPI. The VMM calls KVM_SET_CPUID2 once per vCPU before the first KVM_RUN to pre-load the CPUID table KVM consults on each exit.

The KVM paravirt signature, placed at synthetic leaf 0x40000000, is the 12-byte value "KVMKVMKVM\0\0\0" across EBX, ECX, and EDX. Leaf 0x40000001 carries feature bits: bit 0 is KVM_FEATURE_CLOCKSOURCE, bit 3 is KVM_FEATURE_CLOCKSOURCE2, bit 5 is KVM_FEATURE_STEAL_TIME, bit 6 is KVM_FEATURE_PV_EOI, and bit 24 is KVM_FEATURE_CLOCKSOURCE_STABLE_BIT. A guest that finds this signature can opt into the advertised paravirtual interfaces.

Firecracker does not simply pass host CPUID through to the guest. It applies CPU-template bitmask operations and a normalization pass that makes topology fields and selected feature leaves coherent with the configured vCPU count. Templates can narrow host variation, but they do not make arbitrary hosts interchangeable: Firecracker snapshot compatibility still requires the exposed CPU features to remain invariant, and cross-model restore is constrained.

MSR Exits: KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR

Most MSR accesses never reach userspace, but that does not mean they all bypass KVM. A clear MSR-bitmap bit permits direct hardware execution for the limited MSRs KVM chooses to pass through. Intercepted accesses enter KVM, which emulates guest-visible state such as IA32_TSC, IA32_APIC_BASE, and the KVM paravirtual MSRs. If KVM denies an access, its default response is to inject #GP into the guest.

Userspace enables KVM_CAP_X86_USER_SPACE_MSR on the VM with an args[0] mask of the denial reasons it wants reported. Matching accesses then produce KVM_EXIT_X86_RDMSR (29) or KVM_EXIT_X86_WRMSR (30) instead of an immediate guest #GP. KVM_CAP_X86_MSR_FILTER is separate: it lets userspace install allow/deny filters, while the KVM_MSR_EXIT_REASON_FILTER bit decides whether a filtered denial exits. The VMM reads run->msr.index, supplies or consumes run->msr.data, and sets run->msr.error to 0 for success or 1 to request guest #GP.

The KVM_MSR_EXIT_REASON_* flags describe why the exit occurred:

KVM_MSR_EXIT_REASON_INVAL   (1 << 0)  /* invalid MSR or reserved bits */
KVM_MSR_EXIT_REASON_UNKNOWN (1 << 1)  /* KVM has no handler for this MSR */
KVM_MSR_EXIT_REASON_FILTER  (1 << 2)  /* blocked by KVM_X86_SET_MSR_FILTER */

Firecracker does not use KVM_CAP_X86_USER_SPACE_MSR and does not handle KVM_EXIT_X86_RDMSR or KVM_EXIT_X86_WRMSR in its run loop. MSRs are configured at boot via KVM_SET_MSRS using CPU template RegisterValueFilter entries, and KVM handles them from there.

The KVM paravirt MSR range 0x4b564d00-0x4b564d08 is handled in-kernel. The addresses and their purposes:

MSR address Purpose
0x4b564d00 MSR_KVM_WALL_CLOCK_NEW -- guest wallclock GPA
0x4b564d01 MSR_KVM_SYSTEM_TIME_NEW -- per-vCPU monotonic time GPA
0x4b564d02 MSR_KVM_ASYNC_PF_EN -- async page fault control
0x4b564d03 MSR_KVM_STEAL_TIME -- vCPU steal-time GPA
0x4b564d04 MSR_KVM_EOI_EN -- PV EOI control
0x4b564d05 MSR_KVM_POLL_CONTROL -- host halt-polling enable/disable
0x4b564d06 MSR_KVM_ASYNC_PF_INT -- APF "page ready" interrupt vector
0x4b564d07 MSR_KVM_ASYNC_PF_ACK -- APF acknowledgement
0x4b564d08 MSR_KVM_MIGRATION_CONTROL -- live migration permission

The older addresses 0x11 (MSR_KVM_WALL_CLOCK) and 0x12 (MSR_KVM_SYSTEM_TIME) are deprecated; guests should use the 0x4b564d00 variants when bit 3 of leaf 0x40000001 is set.

Shutdown: KVM_EXIT_SHUTDOWN

A guest triple fault delivers KVM_EXIT_SHUTDOWN (value 8). No fields in the kvm_run union carry detail; the exit reason alone is the signal. A triple fault occurs when exception delivery itself fails and the processor cannot invoke a handler. On AMD, the corresponding hardware exit is SVM_EXIT_SHUTDOWN = 0x07f.

A VMM normally stops or resets the virtual machine rather than treating this as an orderly power-off. Current Firecracker has no successful VcpuExit::Shutdown arm, so the exit reaches its unexpected-exit path. By contrast, its generic handler accepts KVM_EXIT_SYSTEM_EVENT types KVM_SYSTEM_EVENT_SHUTDOWN and KVM_SYSTEM_EVENT_RESET and stops the vCPU cleanly.

Internal Error: KVM_EXIT_INTERNAL_ERROR

KVM_EXIT_INTERNAL_ERROR (value 17) signals a KVM failure and carries a suberror. With KVM_CAP_EXIT_ON_EMULATION_FAILURE enabled, an instruction-emulation failure returns KVM_INTERNAL_ERROR_EMULATION. The stable emulation_failure overlay provides a validity flag and, when KVM_INTERNAL_ERROR_EMULATION_FLAG_INSTRUCTION_BYTES is set, the decoded instruction length and up to 15 bytes. Additional words may contain architecture debug data, but their format is explicitly not ABI.

The Canonical Run Loop

The exit dispatch structure is the same across all VMMs. Here is the loop stripped to its essentials:

for (;;) {
    int rc = ioctl(vcpu_fd, KVM_RUN, NULL); /* _IO(0xAE, 0x80) */
    if (rc < 0) {
        if (errno == EINTR) {
            run->immediate_exit = 0;
            continue;
        }
        return -1;
    }
    switch (run->exit_reason) {
    case KVM_EXIT_HLT:
        return 0;
    case KVM_EXIT_IO:
        /* direction, size, port, count: run->io.*                   */
        /* data: (char *)run + run->io.data_offset                   */
        break;
    case KVM_EXIT_MMIO:
        /* phys_addr, len, is_write: run->mmio.*                     */
        /* for reads: fill run->mmio.data before returning           */
        break;
    case KVM_EXIT_SHUTDOWN:
        abort();
    case KVM_EXIT_INTERNAL_ERROR:
        /* run->internal.suberror for detail */
        abort();
    default:
        return -1;
    }
}

The KVM_RUN ioctl encodes as _IO(KVMIO, 0x80) where KVMIO = 0xAE; it takes no parameter. For I/O, MMIO, and userspace MSR exits, the operation is not complete until userspace re-enters KVM. A VMM that needs to pause or migrate at that boundary must first call KVM_RUN again, using a pending signal or immediate_exit if it must finish the operation without executing another guest instruction.

What An Exit Costs

The architecture manuals do not promise a cycle count for exit and entry. A useful measurement must name the processor, microcode, host kernel, mitigations, trigger, vCPU placement, and whether it measures only the hardware transition or the full KVM/userspace round trip. Historical results are not a current-host performance contract.

The cost has several layers. Hardware records guest state and loads host state. KVM dispatches the reason and may decode an instruction, walk guest memory, or update interrupt state. A userspace exit crosses the ioctl boundary, runs the VMM handler, and re-enters the kernel. Cache residency, host scheduling, security mitigations, and contention can dominate the bare transition. This is why exit counts alone are useful but insufficient: latency must be measured with the workload and host configuration that will run it.

Intel APICv can eliminate hardware exits for selected APIC operations through controls including APIC-register virtualization and virtual-interrupt delivery. AMD AVIC provides analogous acceleration. These features do not make all interrupts free; they remove particular transitions from the path when the processor, KVM configuration, and guest APIC mode line up.

Avoiding Userspace Exits: ioeventfd and irqfd

Virtio queue notifications and device interrupt injection are hot enough that Firecracker configures both outside its vCPU exit dispatcher. The hardware may still exit for a trapped notification write; the narrower guarantee is that KVM consumes a matching ioeventfd access without returning from KVM_RUN.

KVM_IOEVENTFD (capability KVM_CAP_IOEVENTFD) registers an eventfd file descriptor against a guest MMIO or PIO address. When the guest writes to that address, KVM signals the eventfd and resumes the guest -- the KVM_RUN ioctl never returns to userspace for that write. The registration struct:

struct kvm_ioeventfd {
    __u64 datamatch;    /* optional: match only this write value */
    __u64 addr;         /* MMIO or PIO address */
    __u32 len;          /* access width */
    __s32 fd;           /* eventfd to signal */
    __u32 flags;
    /* KVM_IOEVENTFD_FLAG_PIO       -- PIO (default: MMIO) */
    /* KVM_IOEVENTFD_FLAG_DATAMATCH -- filter by datamatch value */
    /* KVM_IOEVENTFD_FLAG_DEASSIGN  -- remove this registration */
};

Note: KVM_IOEVENTFD and KVM_IRQFD are VM-level ioctls on the VM fd. They are typically called during device setup, before the guest has a chance to write to the target address; they do not require the vCPU threads to be stopped.

For virtio-mmio, Firecracker registers the shared QueueNotify address at offset 0x50 once per queue with the queue index as datamatch. For virtio-pci, it registers one address per queue in the notification BAR without datamatch. A matching guest write signals the eventfd in KVM. Other transport-register accesses can still return as KVM_EXIT_MMIO.

KVM_IRQFD (capability KVM_CAP_IRQFD) registers an eventfd paired with a guest GSI route. When device emulation writes the eventfd, KVM injects the corresponding virtual interrupt without returning the vCPU through exit dispatch. Userspace still produced the device result and performed the eventfd write.

Together, ioeventfd and irqfd keep the virtio notification path out of the vCPU run-loop dispatch. The guest writes the queue notifier, KVM signals an eventfd, Firecracker's event manager dispatches the device subscriber, and completion writes the irqfd eventfd. Userspace device code still runs; the avoided work is an extra KVM_RUN return for the queue kick and another vCPU-thread round trip for interrupt injection.

Halt Polling

When a guest HLTs with an in-kernel local APIC, KVM's block path may poll before yielding the vCPU thread to the Linux scheduler. If a wakeup arrives during that interval, KVM can resume the guest without a scheduler round trip. Polling stops early when another host task on the CPU is runnable.

The polling interval adapts from observed block time. A wakeup received inside the current interval leaves that interval unchanged. If polling misses but total block time is still below the maximum, KVM grows the next interval from halt_poll_ns_grow_start or multiplies it by halt_poll_ns_grow. If total block time exceeds the maximum, KVM divides the interval by halt_poll_ns_shrink. The module parameter halt_poll_ns supplies the system-wide ceiling unless userspace overrides it for a VM with KVM_CAP_HALT_POLL.

The trade-off is host CPU time for wakeup latency. A short wait that completes during polling avoids scheduling latency but converts idle time into host kernel time. A long wait can burn the entire interval before the vCPU yields. In Linux source checked on July 10, 2026, the x86 default ceiling is 200,000 ns, but the effective per-vCPU interval adapts and operators can change the ceiling, so it is not a fixed per-halt cost.

Instruction Emulation

Some exits require KVM to reproduce the semantics of the intercepted instruction. MMIO is the common example: the memory access did not complete in hardware, so KVM decodes enough of the instruction to construct the device access and later commit the result to guest state. The x86 software emulator lives in arch/x86/kvm/emulate.c.

The emulator:

  1. Decodes the instruction at GUEST_RIP from guest memory.
  2. Runs the emulated operation against KVM's model of guest register state.
  3. Updates guest GPRs, flags, and RIP.

The decoder handles the x86 encodings and operations needed by KVM's emulation paths, including prefixes, operand sizes, and memory addressing modes. It is broad but not a promise that every architecturally valid instruction can be emulated in every guest mode. For a userspace MMIO or MSR exit, final guest-state updates are deferred until userspace re-enters KVM_RUN.

With KVM_CAP_EXIT_ON_EMULATION_FAILURE enabled, a failure returns KVM_EXIT_INTERNAL_ERROR with suberror KVM_INTERNAL_ERROR_EMULATION. The stable overlay tells userspace whether the instruction bytes are valid and can carry up to 15 of them. The remaining debug words are not a stable ABI, so a VMM may log them but must not build control flow around their layout.

Observability

The observability tools in this section read host KVM state. Use them only on an isolated bare-metal Linux host or a VM with nested virtualization enabled, and expect elevated privileges for debugfs, tracepoints, or perf.

The kvm_stat tool in the kernel tree reads KVM statistics and prints a rolling summary by event. The exact dominant events depend on the guest, devices, transport, and workload; the useful comparison is between an expected baseline and the same microVM after a configuration or code change.

For per-exit profiling, the kvm_exit and kvm_mmio kernel tracepoints feed perf kvm stat, which aggregates exit counts and average latencies by exit type across all VMs on the host. A trace that shows thousands of MMIO exits per second to addresses that should be ioeventfd-registered suggests a device setup problem worth investigating.

Firecracker exposes its own counters. METRICS.vcpu.exit_mmio_read, exit_mmio_write, exit_io_in, and exit_io_out increment in the exit dispatch paths and are surfaced via Firecracker's metrics endpoint. These are the first numbers to check when a Firecracker guest shows unexpectedly high vCPU CPU consumption.

Sources And Further Reading