Chapter 7: Virtual Interrupts And Time

A guest operating system expects to receive interrupts. The NIC driver expects the hardware to signal that a packet arrived. The block driver expects a completion notice. The timer subsystem expects the interrupt controller to fire on schedule. A microVM must reproduce those contracts with virtual devices and virtual interrupt controllers. Its clock must also advance at a stable rate and remain coherent across a long pause, snapshot restore, or migration to a host with a different TSC frequency.

Interrupts and timekeeping share a design constraint: their common paths cannot afford a return to the userspace exit dispatcher on every operation. KVM therefore emulates interrupt controllers in the kernel, uses eventfds to connect device notifications to kernel interrupt injection, applies hardware TSC offset and scaling, and shares paravirtual clock data with the guest. Mistakes in the first path produce missed interrupts, hangs, and stalled virtqueues. Mistakes in the second produce drifting logs, expired credentials, and broken timeout calculations.

The Hardware Interrupt Model

A CPU normally runs the instruction stream in front of it. Devices need a way to interrupt that stream without waiting for the kernel to poll them. A disk controller has completed a request, a NIC has received a packet, a timer has expired, or another CPU needs a TLB shootdown. The hardware answer is an interrupt: a device or timer raises an event, the platform routes it to a CPU, and the CPU vectors into a kernel handler when interrupts are unmasked.

The interrupt controller is the routing and bookkeeping hardware between devices and CPUs. It records pending events, assigns or carries an interrupt vector, chooses the target CPU, tracks whether an interrupt is still in service, and knows whether a device signal is edge-triggered or level-triggered. When the CPU accepts an external interrupt, it saves enough execution state to resume later and dispatches through the Interrupt Descriptor Table. When the handler has finished the device work, it sends EOI (end of interrupt) so the controller can clear in-service state and, for level-triggered lines, allow the device to raise the line again if work is still pending.

On x86, modern interrupt delivery is split across two APIC components. The local APIC (LAPIC) is per logical processor. It is the CPU's interrupt inbox: it receives vectors, tracks pending and in-service interrupts, provides a local timer, and sends inter-processor interrupts. The I/O APIC is the platform router for external device pins. It has a redirection table that says "pin N delivers vector V to APIC ID X." The old 8259 PIC pair remains for compatibility with legacy software, but microVMs generally keep it only because PC-compatible firmware and kernels still expect it to exist.

Not every interrupt is a physical wire. With MSI (Message Signaled Interrupts), a PCI device raises an interrupt by writing a small address/data message instead of toggling an input pin. The platform still has to route that message to a CPU vector, but the device-facing mechanism is a write transaction, not a line. KVM's GSI numbers are the virtual machine's neutral interrupt line identifiers before routing maps them to a PIC pin, an I/O APIC pin, or an MSI route.

The Interrupt Architecture Problem

On real hardware, an interrupt follows a path that software rarely thinks about: a device asserts a line, the I/O APIC records it, the local APIC on the target CPU raises an interrupt request, and the CPU saves state and dispatches the handler. This path crosses several pieces of hardware the guest believes it owns -- the local APIC register page at 0xFEE00000, the I/O APIC at 0xFEC00000, the legacy 8259 PIC pair at I/O ports 0x20 and 0xA0 -- none of which exists in the guest's physical address space unless the hypervisor puts it there.

The hypervisor has three options. It can emulate the entire interrupt controller stack in userspace -- every register write from the guest triggers a VM exit, the VMM updates its software model, and the VMM injects the interrupt on the next KVM_RUN. It can emulate the controllers inside the kernel, handling register accesses without returning to userspace. Or it can split the work: keep the local APIC in-kernel where injection is fast, but handle the legacy PIC and I/O APIC in userspace where the VMM can apply its own routing policy. Each choice makes different tradeoffs between latency, flexibility, and complexity.

In-Kernel Irqchip vs. Userspace

The canonical path for x86 microVMs is fully in-kernel. A single VM-level ioctl, KVM_CREATE_IRQCHIP (_IO(KVMIO, 0x60)), requires KVM_CAP_IRQCHIP (capability value 0) and creates three emulated controllers in one call: a master 8259 PIC (KVM_IRQCHIP_PIC_MASTER = 0), a slave 8259 PIC (KVM_IRQCHIP_PIC_SLAVE = 1), and an I/O APIC (KVM_IRQCHIP_IOAPIC = 2). Every vCPU created after this call gets an in-kernel local APIC. The state of any chip can be read or written later with KVM_GET_IRQCHIP and KVM_SET_IRQCHIP, which reference chips by these same ID constants.

On arm64, KVM_CREATE_IRQCHIP creates a GICv2 only. For a GICv3 -- the interrupt controller on any recent Arm server or embedded SoC -- userspace must instead call KVM_CREATE_DEVICE with KVM_DEV_TYPE_ARM_VGIC_V3. The kernel enforces that only one VGIC instance may exist per VM; GICv2 and GICv3 cannot coexist.

Split-irqchip mode, enabled by KVM_CAP_SPLIT_IRQCHIP (value 121), keeps the in-kernel local APIC but moves the legacy PIC, I/O APIC, and PIT to userspace. When a guest performs an EOI (end of interrupt, the write that tells the interrupt controller the current interrupt handler is done) that would normally notify the I/O APIC, KVM surfaces this to userspace as KVM_EXIT_IOAPIC_EOI rather than handling it in-kernel. This is the configuration chosen by VMMs that want fine-grained control over routing while still keeping interrupt injection fast.

Without either KVM_CREATE_IRQCHIP or split mode, the VMM emulates every controller in userspace and injects interrupts via the vCPU ioctl KVM_INTERRUPT, which queues a single interrupt vector for delivery at the next VM entry. This path adds a userspace round trip to interrupt delivery and is not the usual microVM configuration.

Firecracker calls KVM_CREATE_IRQCHIP at VM creation on x86_64, using the fully in-kernel path for all three chips. On aarch64, it provisions a GICv3 via KVM_CREATE_DEVICE with KVM_DEV_TYPE_ARM_VGIC_V3, falling back to KVM_DEV_TYPE_ARM_VGIC_V2 if the host kernel or hardware does not support GICv3.

The Local APIC In Detail

After KVM_CREATE_IRQCHIP, each vCPU's local APIC is emulated by KVM in arch/x86/kvm/lapic.c. The APIC register page is 4 KiB (LAPIC_MMIO_LENGTH = 4096), mapped at 0xFEE00000 in the guest's physical address space. Register accesses go through KVM's MMIO handler rather than to real hardware, so they are resolved in-kernel without a userspace round-trip.

The key registers and their offsets within the APIC page:

The Interrupt Request Register (IRR, pending interrupts), In-Service Register (ISR, interrupts currently being handled), and Trigger-Mode Register (TMR, edge- versus level-triggered delivery) are each 256-bit bitmaps stored as eight 32-bit registers inside the kvm_lapic_state.regs page. When the guest writes the EOI register, KVM's handler calls apic_find_highest_isr(), clears the ISR bit for the current interrupt, recomputes the Processor Priority Register, and notifies the in-kernel I/O APIC via kvm_ioapic_send_eoi(). All of this happens inside the APIC MMIO handler without returning to userspace.

The LAPIC timer complicates things. Emulating a timer accurately requires the kernel to know when the next deadline fires, then absorb the jitter introduced by VM exit and entry. Current KVM initializes its adaptive timer advance to 1,000 ns and limits it to 5,000 ns. It arms the host timer early and, when necessary, delays for the remaining interval before exposing the expiration to the guest. These constants and the adaptive calculation are implementation details, not a fixed latency guarantee.

The full LAPIC state can be saved and restored across migration with KVM_GET_LAPIC (ioctl 0x8e) and KVM_SET_LAPIC (ioctl 0x8f). Both operate on struct kvm_lapic_state { char regs[KVM_APIC_REG_SIZE]; } where KVM_APIC_REG_SIZE = 0x400 (1024 bytes).

x2APIC

The original xAPIC mode uses 8-bit APIC IDs stored in MMIO registers. x2APIC extends this to 32-bit IDs and replaces the MMIO interface with MSR accesses in the range 0x800-0x8FF -- each MSR maps to an APIC register at offset (msr - APIC_BASE_MSR) << 4 where APIC_BASE_MSR = 0x800. KVM emulates the full x2APIC MSR range, but doing so requires KVM_CREATE_IRQCHIP to have been called first; KVM does not support forwarding x2APIC MSR accesses to userspace.

Enabling the extended API requires KVM_CAP_X2APIC_API (value 129). When the KVM_X2APIC_API_USE_32BIT_IDS flag is set within this capability, KVM stores the full 32-bit x2APIC ID in bytes 32-35 of kvm_lapic_state.regs; xAPIC stores only an 8-bit ID in byte 35 (bits 31-24 of that word).

The In-Kernel I/O APIC

The KVM I/O APIC (arch/x86/kvm/ioapic.c) emulates exactly 24 input pins (KVM_IOAPIC_NUM_PINS = 24), matching the Intel 82093AA specification. The MMIO window is 256 bytes (0x100) at default base 0xFEC00000. Like real hardware, the I/O APIC uses an indirect addressing scheme: a write to IOAPIC_REG_SELECT at offset 0x00 sets the internal register index; a subsequent read or write to IOAPIC_REG_WINDOW at offset 0x10 accesses the selected register.

Indirect register 0x00 is the ID (IOAPICID), 0x01 the version (IOAPICVER; KVM reports IOAPIC_VERSION_ID = 0x11), 0x02 the arbitration register (IOAPICARB). Redirection table entries start at index 0x10 (pin 0) and occupy two 32-bit words each, running through 0x3F (pin 23). Each 64-bit entry (union kvm_ioapic_redirect_entry) encodes the destination vector, delivery mode, destination APIC ID, trigger mode (edge or level), mask bit, and remote IRR flag.

GSI routing determines which controller receives each interrupt. A GSI is KVM's global system interrupt number, the VM-wide interrupt line a device targets before KVM maps it to a PIC pin, I/O APIC pin, or message-signaled interrupt. GSIs 0-15 route to both the PIC and the I/O APIC for compatibility with legacy software; GSIs 16-23 go to the I/O APIC only. The RTC IRQ, RTC_GSI = 8, routes through both.

The GSI Routing Table

Higher-level interrupt routing -- from a device's logical signal to the right controller and pin -- lives in a table the VMM manages with KVM_SET_GSI_ROUTING (_IOW(KVMIO, 0x6a, struct kvm_irq_routing)), gated by KVM_CAP_IRQ_ROUTING (value 25). Each call atomically replaces the entire table; there is no incremental-update path. A VMM that needs to add one route must rebuild and resubmit the full table.

Each entry in the table is a struct kvm_irq_routing_entry:

struct kvm_irq_routing_entry {
    __u32 gsi;
    __u32 type;   /* KVM_IRQ_ROUTING_IRQCHIP=1, KVM_IRQ_ROUTING_MSI=2,
                     KVM_IRQ_ROUTING_S390_ADAPTER=3, KVM_IRQ_ROUTING_HV_SINT=4,
                     KVM_IRQ_ROUTING_XEN_EVTCHN=5 */
    __u32 flags;
    __u32 pad;
    union {
        struct kvm_irq_routing_irqchip irqchip;
        struct kvm_irq_routing_msi     msi;
        /* ... */
    } u;
};

For MSI devices, KVM_IRQ_ROUTING_MSI = 2 entries carry address_lo, address_hi, and data - the three fields in a Message Signaled Interrupt (MSI), where the device raises an interrupt by writing a small message instead of toggling a pin. Setting KVM_MSI_VALID_DEVID (bit 0 in struct kvm_msi.flags) passes a PCIe Requester ID via devid, which enables interrupt remapping hardware to associate the interrupt with a specific device (requires KVM_CAP_MSI_DEVID = 131).

On arm64, GSI routing applies to KVM_IRQFD bindings but does not apply to KVM_IRQ_LINE.

In the Firecracker source checked on July 10, 2026, a legacy interrupt registration adds one KVM_IRQ_ROUTING_IRQCHIP entry: it targets KVM_IRQCHIP_IOAPIC on x86_64 and chip index 0 on aarch64. Virtio-pci uses one KVM_IRQ_ROUTING_MSI entry per MSI-X vector rather than one per device. set_gsi_routes() collects the unmasked entries and replaces KVM's table with one set_gsi_routing() call.

irqfd: Interrupt Injection Without a Userspace Round Trip

The fast path for interrupt delivery avoids returning the vCPU from KVM_RUN to the userspace exit dispatcher. The mechanism that enables this is irqfd, introduced in Linux 2.6.32 and requiring KVM_CAP_IRQFD (value 32).

The underlying primitive is eventfd(2) (available since Linux 2.6.22): a file description backed by a 64-bit kernel counter. Writing 8 bytes adds to the counter; reading 8 bytes returns and resets it. The fd becomes EPOLLIN-readable when the counter is nonzero. During irqfd registration, KVM installs a custom waitqueue entry whose irqfd_wakeup() function runs when device emulation writes to the eventfd.

struct kvm_irqfd {
    __u32 fd;          /* eventfd file descriptor */
    __u32 gsi;         /* irqchip GSI / pin number */
    __u32 flags;
    __u32 resamplefd;  /* used only with KVM_IRQFD_FLAG_RESAMPLE */
    __u8  pad[16];
};

KVM_IRQFD is a VM ioctl: _IOW(KVMIO, 0x76, struct kvm_irqfd). Setting KVM_IRQFD_FLAG_DEASSIGN in flags removes the binding (both fd and gsi must be provided). Setting KVM_IRQFD_FLAG_RESAMPLE (requires KVM_CAP_IRQFD_RESAMPLE = 82) switches to level-triggered mode: when the guest performs an EOI, KVM de-asserts the GSI and writes resamplefd, allowing the VMM to re-inject if the device still has work pending.

The path through the kernel (in virt/kvm/eventfd.c) avoids acquiring irqfds.lock during the fast path; SRCU protects the cached routing entry instead. When irqfd_wakeup() sees EPOLLIN, it drains the eventfd counter, reads that route, and calls kvm_arch_set_irq_inatomic(). If the architecture cannot complete the operation in atomic context and returns -EWOULDBLOCK, KVM schedules the irqfd_inject work item for deferred delivery. On EPOLLHUP, KVM deactivates the registration and queues its cleanup.

The result: device emulation writes 8 bytes to a file descriptor, and the guest receives an interrupt without a KVM_RUN return on the vCPU thread. The delivery path is a kernel callback and virtual interrupt update, not another trip through userspace exit dispatch.

ioeventfd: Eliminating the Outbound Round-Trip

irqfd handles the host-to-guest direction. The guest-to-host direction -- a guest writing to a virtqueue notify register to tell the host that work is ready -- needs a mirror primitive. Without one, the write can cause a KVM_EXIT_MMIO return from KVM_RUN; the VMM then reads kvm_run.mmio, dispatches the operation, and re-enters the guest. The cost depends on the host, VMM, and workload rather than being a fixed property of the KVM ABI.

KVM_IOEVENTFD (_IOW(KVMIO, 0x89, struct kvm_ioeventfd), requiring KVM_CAP_IOEVENTFD = 36) was introduced in Linux 2.6.32 (commit d34e6b17, Gregory Haskins, August 2009) to eliminate that round-trip:

struct kvm_ioeventfd {
    __u64 datamatch;
    __u64 addr;    /* legal pio/mmio address */
    __u32 len;     /* 0, 1, 2, 4, or 8 bytes */
    __s32 fd;
    __u32 flags;
    __u8  pad[36];
};

The flags field controls matching behavior:

KVM_CAP_IOEVENTFD_ANY_LENGTH permits len = 0 registrations that match regardless of write size.

The kernel fast-path in virt/kvm/eventfd.c: kvm_assign_ioeventfd_idx() registers the ioeventfd on KVM's MMIO, PIO, or VIRTIO_CCW bus via kvm_io_bus_register_dev(). When the guest executes a write to the registered address, KVM's exit handler calls ioeventfd_write(), which checks the address, the write length, and (if KVM_IOEVENTFD_FLAG_DATAMATCH) the written value via ioeventfd_in_range(). On a hit, it calls eventfd_signal() in-kernel and returns 0 -- preventing the exit from propagating to userspace. A hardware-level VM exit still occurs (VMX must trap the write to unmapped MMIO), but the kernel services it without returning to the VMM process.

The patch commit message from August 2009 reported the performance effect:

These are measurements from the original 2009 patch, not current Firecracker or KVM performance numbers. They establish why eliminating the userspace exit mattered; the saving on a current host must be measured there.

How Firecracker Uses irqfd and ioeventfd

With Firecracker's virtio-mmio transport, each virtqueue has an ioeventfd at device_base + NOTIFY_REG_OFFSET, where NOTIFY_REG_OFFSET = 0x50. The datamatch value is the queue index i, so KVM signals the queue-i eventfd only when the guest writes i to the shared QueueNotify register:

// src/vmm/src/device_manager/mmio.rs (simplified)
for (i, queue_evt) in locked_device.queue_events().iter().enumerate() {
    let io_addr = IoEventAddress::Mmio(
        device.resources.addr + u64::from(NOTIFY_REG_OFFSET),
    );
    vm.fd()
        .register_ioevent(queue_evt, &io_addr, u32::try_from(i).unwrap())
        .map_err(MmioError::RegisterIoEvent)?;
}

Each virtio-mmio device also gets one legacy GSI and one irqfd binding:

vm.register_irq(&mmio_device.interrupt.irq_evt, gsi)
    .map_err(MmioError::RegisterIrqFd)?;

Firecracker's event manager polls the queue eventfds and dispatches work to the registered device subscriber. When the device signals completion, it writes the interrupt eventfd bound to the irqfd. Each virtio-mmio transport occupies a 4 KiB slot.

Virtio-pci uses the same two KVM mechanisms with different addresses and interrupt routing. Firecracker registers one ioeventfd per queue at notification_bar_base + queue_index * 4, with no datamatch because each queue has its own address. It allocates an MSI-X vector per virtqueue plus one configuration vector, and enabled vectors receive their own MSI route and irqfd.

The two mechanisms are complements:

The VMM process still performs the device work. The optimization is narrower and more important: the vCPU does not return through the generic MMIO exit path, and completion injection does not require the vCPU thread to re-enter KVM merely to deliver the interrupt.

Intel Posted Interrupt Processing

Even with irqfd, interrupt injection has a cost: KVM must make the virtual interrupt pending and ensure that a running or sleeping vCPU notices it. On supported Intel VT-x systems, APIC virtualization and posted-interrupt processing move the running-vCPU case into hardware.

Two VMCS fields identify the posted-interrupt notification vector and the address of a 64-byte Posted-Interrupt Descriptor (PID). The PID is shared between software and the processor, so updates to concurrently modified fields require the atomic protocol defined by the architecture.

The PID layout:

When the notification interrupt arrives while the target vCPU is in VMX non-root mode, the processor handles it without a VM exit: it clears ON, merges posted requests from PIR into virtual-APIC pending state, and evaluates the next deliverable vector.

When the target vCPU is not running, KVM kicks or wakes it as needed and consumes the posted requests on a later entry. Linux also changes the descriptor's notification vector when a vCPU blocks so device-posted interrupts can wake it.

KVM configures the VM-execution control, notification vector, descriptor address, and virtual-APIC state when the hardware and kernel configuration support the feature. A VMM using KVM's in-kernel LAPIC does not manipulate the PID directly.

Paravirtual Interrupt Optimizations

Even with in-kernel APIC emulation, certain interrupt operations are expensive. An EOI write to the APIC at 0xFEE000B0 is an MMIO write that KVM must intercept and handle. On a guest processing thousands of interrupts per second, those EOI exits accumulate.

PV-EOI eliminates most of them. The guest writes MSR_KVM_PV_EOI_EN = 0x4b564d04 with the low bit set and bits 63-2 holding a 4-byte-aligned guest physical address. KVM then sets bit 0 of the word at that address before injecting each interrupt. The guest's interrupt return path tests and clears that bit atomically; if the bit was set, the EOI is complete without any APIC MMIO write. Only when the bit is already clear -- when multiple interrupt levels are active and the APIC needs to update the ISR -- does the guest fall back to the MMIO EOI.

The paravirtual hypercall interface provides additional shortcuts for inter-processor interrupts:

Sending a TLB shootdown IPI to many vCPUs would otherwise require repeated APIC ICR programming and emulation. The hypercall encodes up to 128 destinations in two registers and handles the batch in one hypercall.

The ARM GIC (VGIC)

Arm's interrupt controller architecture is the Generic Interrupt Controller, or GIC. KVM's virtual GIC implementation is the VGIC. Interrupt IDs are organized in four ranges:

The kernel VGIC allows 64-1024 IRQs in steps of 32, configured via KVM_DEV_ARM_VGIC_GRP_NR_IRQS. Current Firecracker configures 128 total interrupt IDs: 32 SGIs and PPIs plus 96 SPIs available to its legacy-GSI allocator.

GICv2

GICv2 uses MMIO exclusively. The device type for KVM_CREATE_DEVICE is KVM_DEV_TYPE_ARM_VGIC_V2. Two MMIO regions must be placed via KVM_DEV_ARM_VGIC_GRP_ADDR:

The distributor holds global state; the CPU interface, one per vCPU, is the per-CPU window through which a running vCPU reads pending priority and signals EOI.

GICv3

GICv3 replaces the guest's per-CPU MMIO CPU interface with the ICC system-register interface, accessed with MRS and MSR. KVM's KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS device-attribute group exposes ICC and ICH state to userspace for save and restore. Device type: KVM_DEV_TYPE_ARM_VGIC_V3.

The memory layout changes substantially. Where GICv2 has one CPU interface region for all vCPUs, GICv3 introduces a redistributor -- a 128 KiB (KVM_VGIC_V3_REDIST_SIZE = 0x20000) per-vCPU MMIO region for private interrupt and LPI control state. LPI pending tables themselves reside in guest RAM. The distributor grows to 64 KiB (KVM_VGIC_V3_DIST_SIZE = 0x10000):

KVM associates each redistributor frame with a vCPU. Device-state attributes identify that vCPU by MPIDR, and the association depends on vCPU and redistributor creation order across save and restore.

The Interrupt Translation Service

LPIs are Locality-specific Peripheral Interrupts -- GICv3's mechanism for MSI delivery. A device writes an MSI to the ITS translation register; the ITS consults its guest-memory tables and turns the DeviceID/EventID pair into an LPI targeted at a redistributor. KVM exposes this via KVM_DEV_TYPE_ARM_VGIC_ITS with a 128 KiB MMIO region in Firecracker, placed at a 64 KiB-aligned address via KVM_VGIC_ITS_ADDR_TYPE.

The ITS Device Table maps each DeviceID to a per-device Interrupt Translation Table; entries in that table map EventIDs to physical LPIs. A Collection Table maps collection IDs to redistributors. Key control registers in the ITS MMIO space are GITS_CBASER (command queue base address), GITS_CWRITER and GITS_CREADR (command queue write and read pointers), and GITS_CTLR (enable).

Lifecycle Constraints

The VGIC imposes strict ordering on its initialization sequence. KVM_DEV_ARM_VGIC_CTRL_INIT must be called after all vCPUs are created. Before a snapshot, userspace uses KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES to flush LPI pending bits into the guest-memory pending tables; restore must then reproduce the guest memory, device state, and vCPU-to-redistributor associations coherently.

In the Firecracker source checked on July 10, 2026, the GICv3 regions occupy fixed offsets below the 1 GiB MMIO32_MEM_START: the distributor at MMIO32_MEM_START - 0x10000, the redistributors at dist_addr - (vcpu_count * 0x20000), and the ITS at redist_addr - 0x20000.

Why Clocks Are Hard Under Virtualization

Interrupt delivery can be made fast with the mechanisms above. Timekeeping is harder because the inaccuracy accumulates invisibly and manifests far from its cause.

There are three pieces of hardware vocabulary to keep separate. A clocksource is something the kernel can read to measure elapsed time, such as the TSC. A clock event device is something the kernel can program to interrupt it in the future, such as the LAPIC timer, HPET, PIT, or ARM architectural timer. A wall clock is civil time, tied to UTC and affected by NTP or administrator adjustments. Kernels need all three: monotonic elapsed time for scheduling and timeouts, timer interrupts to regain control, and wall time for filesystems, logs, and TLS.

The time-stamp counter -- read with RDTSC on x86 -- is a cheap way to measure elapsed time on a running CPU. RDTSC is not serializing, so software that needs ordering must use the documented fencing or RDTSCP sequence. That instruction-ordering issue is separate from cross-CPU synchronization: on systems without an architectural guarantee, TSC values on different CPUs can have offsets or drift. Linux therefore treats multi-socket and NUMA synchronization cautiously.

The TSC has also historically changed rate or stopped with CPU power states. Linux distinguishes X86_FEATURE_CONSTANT_TSC, whose rate does not change with P-states, from X86_FEATURE_NONSTOP_TSC, which does not stop in C-states. CPUID.80000007H:EDX[8] advertises the architectural invariant-TSC property. A hypervisor must expose a coherent set of these properties to the guest rather than letting the guest infer them from the physical host.

Migration to a host with a different TSC frequency requires both continuity and rate control. The hypervisor applies an offset so the visible counter does not jump and, when hardware supports it, a scaling ratio so it continues at the advertised guest frequency.

Legacy timekeeping via the PIT (Programmable Interval Timer, I/O ports 0x40-0x43, base frequency 1.193182 MHz) or the RTC (32.768 kHz crystal) relies on interrupt delivery rates that the hypervisor cannot always guarantee. When a host CPU is overloaded, timer interrupts arrive late; a guest that counts ticks can lose time while timer-driven work runs late.

Both VMX and SVM virtualize the TSC with an offset field -- TSC_OFFSET in the VMCS and the corresponding VMCB field on AMD -- so the guest reads a host-derived counter plus an offset. Both also support scaling. VMX multiplies the physical TSC by a 64-bit multiplier and shifts the product right by 48 bits before adding the offset; AMD SVM provides the TSC_RATIO MSR at 0xC0010104. KVM_SET_TSC_KHZ uses KVM_CAP_TSC_CONTROL for per-vCPU control or KVM_CAP_VM_TSC_CONTROL for the VM-wide default set before vCPU creation. KVM_GET_TSC_KHZ is gated by KVM_CAP_GET_TSC_KHZ for a vCPU or KVM_CAP_VM_TSC_CONTROL for a VM, and returns -EIO when the host TSC is unstable.

The IA32_TSC_ADJUST MSR (0x3B) records adjustments made through writes to the TSC or to IA32_TSC_ADJUST itself. Its reset value is 0. KVM virtualizes that guest-visible state separately from the VMCS TSC offset it uses to implement the virtual clock.

kvmclock: The Paravirtual Clock

The structural solution is to give the guest the conversion parameters rather than make it infer elapsed time from timer interrupts. kvmclock, using the shared pvclock structures, pairs a TSC sample with system time and tells the guest how to convert later TSC deltas to nanoseconds.

The guest detects KVM with CPUID leaf 0x40000000; the signature at EBX:ECX:EDX spells "KVMKVMKVM\0\0\0". Feature bits are at leaf 0x40000001 EAX:

The detection algorithm: check kvm_para_available(), then read cpuid_eax(0x40000001). If bit 3 is set, use the canonical MSRs MSR_KVM_SYSTEM_TIME_NEW (0x4b564d01) and MSR_KVM_WALL_CLOCK_NEW (0x4b564d00); if only bit 0 is set, use the deprecated pair 0x12 / 0x11.

The Wall Clock

The guest writes a 4-byte-aligned guest physical address to MSR_KVM_WALL_CLOCK_NEW (0x4b564d00). The hypervisor fills the structure at that address:

struct pvclock_wall_clock {
    u32 version;  /* seqlock */
    u32 sec;      /* wall-clock base for system_time == 0 */
    u32 nsec;
} __attribute__((__packed__));

This MSR is global -- not per-vCPU -- and records the wall-clock point corresponding to system time zero, usually guest boot. To compute current wall time, the guest adds that base to the elapsed system time obtained through the per-vCPU clock.

The System Time Clock

MSR_KVM_SYSTEM_TIME_NEW (0x4b564d01) is per-vCPU. The guest writes a 4-byte-aligned guest physical address with bit 0 as the enable bit. The hypervisor fills and updates the structure at that address; between updates, the guest extrapolates from the recorded TSC:

struct pvclock_vcpu_time_info {
    u32 version;           /* seqlock; odd = update in progress */
    u32 pad0;
    u64 tsc_timestamp;     /* host TSC at last update */
    u64 system_time;       /* host monotonic ns at last update */
    u32 tsc_to_system_mul; /* fixed-point multiplier */
    s8  tsc_shift;         /* shift before multiply: positive=left, negative=right */
    u8  flags;
    u8  pad[2];
} __attribute__((packed));  /* 32 bytes total */

The comment in arch/x86/include/asm/pvclock-abi.h is unambiguous: "these structs MUST NOT be changed" -- they are stable ABI shared between KVM and Xen guests.

Reading the Clock

The conversion from TSC ticks to nanoseconds uses the multiplier and shift:

delta = current_tsc - tsc_timestamp
if (tsc_shift >= 0): delta <<= tsc_shift
else:                delta >>= -tsc_shift
time_ns = ((delta * tsc_to_system_mul) >> 32) + system_time

The version field is a seqlock: read it before and after capturing the time fields; if either read is odd or the two values differ, the hypervisor updated the structure mid-read and the guest must retry. The seqlock protocol is what makes the update safe without a kernel lock in the guest read path.

Flags

Bit 0 is set when the host advertises KVM_FEATURE_CLOCKSOURCE_STABLE_BIT (bit 24 in CPUID.0x40000001). Without it, pvclock.c enforces global monotonicity by updating a last_value counter via atomic compare-and-swap on every read -- because without the host's guarantee, an unlucky migration could move the guest to a CPU with a slightly earlier TSC value, causing time to appear to go backward. With bit 0 set, the raw computed value is returned directly with no global serialization, enabling per-CPU vDSO reads.

kvmclock as a Linux Clocksource

kvmclock_init() in arch/x86/kernel/kvmclock.c registers kvm_clock with clocksource_register_hz(&kvm_clock, NSEC_PER_SEC). The default clocksource rating is 400, which wins over the HPET (rating 250) and the ACPI PM timer (rating 200). When the host exposes both X86_FEATURE_CONSTANT_TSC and X86_FEATURE_NONSTOP_TSC and !check_tsc_unstable(), the rating is reduced to 299 so that the native TSC clocksource -- which is cheaper, requiring no shared-memory read -- can win instead.

When flags bit 0 is set, kvmclock also calls kvm_sched_clock_init() to register the scheduler clock and exposes the fast path via vDSO, so that clock_gettime(CLOCK_MONOTONIC, ...) is serviced by a user-space shared-library read rather than a system call.

Per-vCPU pvclock_vcpu_time_info structures: the boot CPU uses a static array; hotplugged CPUs use dynamic allocation in kvmclock_setup_percpu(). TSC frequency is retrieved from these structures via kvm_get_tsc_khz().

Steal Time

The paravirtual clock tells the guest how much time has elapsed from the host's perspective. Steal time records involuntary runnable time during which the host did not schedule the vCPU. It does not include time when the guest vCPU was idle.

The guest writes a 64-byte-aligned guest physical address (stricter than the 4-byte alignment required by the clock MSRs) with bit 0 as the enable bit to MSR_KVM_STEAL_TIME (0x4b564d03). The structure must be zero-initialized before the MSR write. The hypervisor fills:

struct kvm_steal_time {
    __u64 steal;      /* ns vCPU was not scheduled (excludes idle time) */
    __u32 version;    /* seqlock; even/odd protocol */
    __u32 flags;      /* currently always 0 */
    __u8  preempted;  /* nonzero = vCPU currently descheduled */
    __u8  u8_pad[3];
    __u32 pad[11];
};

The steal field counts only involuntary non-run time -- host-scheduler preemption -- not idle time. A guest CPU consuming 100% of its allowed time shows zero steal; a vCPU that the host is not scheduling shows increasing steal. The preempted field is a hint the guest can use to avoid spinning on spinlocks when it knows the vCPU holding the lock has been descheduled.

VM-Level Clock Ioctls

Two VM-level ioctls expose the kvmclock value to the VMM for snapshot and restore purposes, gated by KVM_CAP_ADJUST_CLOCK:

struct kvm_clock_data {
    __u64 clock;      /* kvmclock nanosecond value */
    __u32 flags;
    __u32 pad0;
    __u64 realtime;   /* host CLOCK_REALTIME at snapshot (if KVM_CLOCK_REALTIME set) */
    __u64 host_tsc;   /* host TSC at snapshot (if KVM_CLOCK_HOST_TSC set) */
    __u32 pad[4];
};

KVM_GET_CLOCK reads the current kvmclock value; KVM_SET_CLOCK restores it. Flags in the structure:

KVM_KVMCLOCK_CTRL is a vCPU ioctl that causes KVM to set PVCLOCK_GUEST_STOPPED in that vCPU's shared pvclock structure. The guest clears the flag and touches its watchdogs, preventing a deliberate VMM pause from looking like a soft lockup. Userspace calls the ioctl after pausing the vCPU and before resuming it.

Clocks And Snapshots

Snapshotting turns guest timekeeping from a clock problem into a consistency problem. The VMM pauses vCPUs, records KVM clock state with KVM_GET_CLOCK, records the paravirtual clock MSRs that the guest uses, and later restores them with KVM_SET_CLOCK and vCPU register/MSR ioctls. On x86, KVM_KVMCLOCK_CTRL tells the guest that a vCPU was intentionally paused by host userspace so the Linux soft-lockup watchdog does not mistake the pause for a stuck CPU.

The hard policy question is what should happen to elapsed time. By default, current Firecracker restores kvm-clock without advancing it by the wall-clock interval since the snapshot. On x86_64 with host Linux 5.16 or newer, clock_realtime: true in the LoadSnapshot request opts into KVM_SET_CLOCK with KVM_CLOCK_REALTIME, adding that elapsed interval to the restored kvmclock value. The option can make time jump from the guest's perspective, so it is a workload policy rather than an automatic correction.

Firecracker also supports the VMClock device. On restore it changes the device's vm_generation_counter and disruption_marker, allowing guest userspace to detect that a discontinuity occurred; the device does not itself synchronize the guest clock. Chapter 16 returns to the release history and operational hazards. The rule here is that clocks and discontinuity indicators are guest-visible VM state, not ambient host facts.

Sources And Further Reading

• KVM API documentation (canonical): https://www.kernel.org/doc/html/latest/virt/kvm/api.html
• KVM x86 MSR documentation: https://www.kernel.org/doc/html/latest/virt/kvm/x86/msr.html
• KVM x86 hypercalls: https://www.kernel.org/doc/html/latest/virt/kvm/x86/hypercalls.html
• KVM x86 timekeeping: https://docs.kernel.org/virt/kvm/x86/timekeeping.html
• KVM x86 CPUID: https://docs.kernel.org/virt/kvm/x86/cpuid.html
• ARM VGICv2: https://docs.kernel.org/virt/kvm/devices/arm-vgic.html
• ARM VGICv3: https://docs.kernel.org/virt/kvm/devices/arm-vgic-v3.html
• ARM ITS: https://docs.kernel.org/virt/kvm/devices/arm-vgic-its.html
• include/uapi/linux/kvm.h: https://github.com/torvalds/linux/blob/master/include/uapi/linux/kvm.h
• arch/x86/include/uapi/asm/kvm.h: https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/kvm.h
• arch/x86/kvm/lapic.c: https://github.com/torvalds/linux/blob/master/arch/x86/kvm/lapic.c
• arch/x86/kvm/ioapic.h: https://github.com/torvalds/linux/blob/master/arch/x86/kvm/ioapic.h
• virt/kvm/eventfd.c: https://github.com/torvalds/linux/blob/master/virt/kvm/eventfd.c
• arch/x86/kernel/kvmclock.c: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/kvmclock.c
• arch/x86/kernel/pvclock.c: https://github.com/torvalds/linux/blob/master/arch/x86/kernel/pvclock.c
• arch/x86/include/asm/pvclock-abi.h: https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/pvclock-abi.h
• arch/x86/include/uapi/asm/kvm_para.h: https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/kvm_para.h
• Intel SDM Vol. 3C, VMX and posted-interrupt processing: https://www.intel.com/content/www/us/en/content-details/868148/intel-64-and-ia-32-architectures-software-developer-s-manual-volume-3c-system-programming-guide-part-3.html
• eventfd(2) man page: https://man7.org/linux/man-pages/man2/eventfd.2.html
• Original ioeventfd patch (LKML, August 2009): https://lkml.iu.edu/hypermail/linux/kernel/0908.2/03024.html
• LWN: KVM -- add support for irqfd (2009): https://lwn.net/Articles/332924/
• OASIS virtio v1.2 CS 01: https://docs.oasis-open.org/virtio/virtio/v1.2/cs01/virtio-v1.2-cs01.html
• Firecracker src/vmm/src/arch/x86_64/interrupts.rs: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/arch/x86_64/interrupts.rs
• Firecracker src/vmm/src/device_manager/mmio.rs: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/device_manager/mmio.rs
• Firecracker virtio-pci transport: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/devices/virtio/transport/pci/device.rs
• Firecracker src/vmm/src/devices/virtio/mod.rs: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/devices/virtio/mod.rs
• Firecracker src/vmm/src/vstate/vm.rs: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/vstate/vm.rs
• Firecracker src/vmm/src/arch/aarch64/gic/gicv3/mod.rs: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/arch/aarch64/gic/gicv3/mod.rs
• Firecracker CHANGELOG: https://github.com/firecracker-microvm/firecracker/blob/main/CHANGELOG.md
• Firecracker snapshot support documentation: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md
• rust-vmm kvm-ioctls crate: https://docs.rs/kvm-ioctls/latest/kvm_ioctls/
• Paired research note: Virtual Interrupts And Time