MicroVMs From The Bottom Up

Chapter 6: Guest Memory And Two-Dimensional Paging

A guest load begins with an address meaningful only inside the guest. How can the CPU reach the host page that backs it without letting the guest choose an arbitrary host address? The answer crosses four address spaces and two owners: the guest controls its virtual-to-physical mapping, while KVM and the host kernel control where those guest-physical pages reside.

Two-dimensional paging performs the two translations in hardware. The guest's page tables translate guest virtual to guest physical; a second, host-controlled tree translates guest physical to host physical. Intel calls the second tree EPT (Extended Page Tables). AMD calls it NPT (Nested Page Tables), also marketed as RVI (Rapid Virtualization Indexing). A cold TLB miss may walk both trees, while cached translations make the steady-state path much shorter. Before EPT and NPT, hypervisors maintained shadow page tables that collapsed the translations in software and paid to keep them synchronized.

The Memory Model

On a non-virtualized machine, DRAM is indexed by physical addresses. The CPU does not usually execute with those addresses directly once the kernel has turned paging on. Instead, each load, store, or instruction fetch starts with a virtual address. The MMU walks page tables rooted at CR3, checks permission bits, and produces a physical address. The TLB caches recent translations so the walk is not repeated on every instruction. If the page-table entry is missing or lacks permission, the CPU raises a page fault and the kernel decides whether to map a page, kill the task, or handle copy-on-write.

Physical address does not always mean RAM. The platform reserves parts of the physical address map for devices: a store to a memory-mapped UART or virtio register is still a CPU store, but the address decodes to a device register instead of a DRAM cell. Interrupt controllers can be device blocks in that same map: the x86 local APIC and I/O APIC are reached through fixed physical address windows. That is why a memory map is part of the machine's hardware contract. The OS owns that map and programs page tables so normal code can reach RAM and, when appropriate, device registers.

A guest kernel needs the same contract, but it cannot be allowed to program host physical memory. Its CR3 points to guest page tables that translate guest virtual addresses to guest physical addresses. Those GPAs are the physical addresses of the machine the guest thinks it owns. The host then needs a second translation, controlled by KVM, from guest physical addresses to the real host physical pages backing the VMM's memory allocation. EPT and NPT are that second MMU walk in hardware.

Four Address Spaces, Not Two

The KVM MMU documentation (Documentation/virt/kvm/x86/mmu.rst) defines four distinct address spaces that coexist in a virtualized x86-64 system. GPA is not HVA; confusing the two is the most common error in VMM memory code.

Symbol Name Controlled by
GVA Guest Virtual Address Guest OS -- CR3-rooted 4-level page tables inside the VM
GPA Guest Physical Address VMM -- KVM memory slots and the EPT/NPT structure
HVA Host Virtual Address VMM process mmap -- an ordinary pointer in the Firecracker address space
HPA Host Physical Address Host OS page tables -- where DRAM actually is

There are two related chains, and mixing them up causes bugs. Once mappings exist, the hardware fast path is:

GVA --(guest page tables, CR3)--> GPA
GPA --(EPT/NPT, host-controlled tables)--> HPA

KVM builds and repairs those EPT/NPT mappings through a setup and fault path that includes the VMM's host virtual address:

memslot GPA range --> HVA backing range
HVA --(host page tables)--> HPA
KVM installs GPA --> HPA entries into EPT/NPT

The guest believes its RAM starts at GPA zero; the VMM may map the backing at an HVA such as 0x7f3a80000000. The two are related only by the slot registration. Similarly, nothing guarantees that a GPA has the same numeric value as its HPA: the host kernel places the backing pages. EPT/NPT makes those distinctions invisible to guest code once KVM has built the mapping.

Memory Slots

KVM's model for guest memory is built around memory slots: named, numbered regions that declare "guest physical addresses from guest_phys_addr to guest_phys_addr + memory_size are backed by host virtual memory starting at userspace_addr." A slot is not the memory itself; it is a mapping declaration. The VMM supplies the backing memory by any means it chooses, and KVM uses the slot to build the EPT/NPT entries that make the translation fast.

KVM_SET_USER_MEMORY_REGION

KVM_SET_USER_MEMORY_REGION is _IOW(KVMIO, 0x46, struct kvm_userspace_memory_region), a VM ioctl issued on the VM file descriptor. It requires capability KVM_CAP_USER_MEMORY. The struct, from include/uapi/linux/kvm.h:

struct kvm_userspace_memory_region {
    __u32 slot;            /* bits 0-15: slot index; bits 16-31: address space ID */
    __u32 flags;
    __u64 guest_phys_addr; /* GPA base of this slot */
    __u64 memory_size;     /* bytes; 0 = delete this slot */
    __u64 userspace_addr;  /* HVA: host virtual address of backing memory */
};

The slot index in bits 0-15 is the identifier KVM uses to distinguish slots. Bits 16-31 carry the address space ID, used when KVM_CAP_MULTI_ADDRESS_SPACE is available -- irrelevant for most VMMs, which operate in address space zero. For an existing slot, the ioctl can move the mapping or change its flags, but it cannot resize it. Passing memory_size = 0 deletes the slot. Slots must not overlap in guest physical address space; the kernel rejects a conflict.

The legacy flags field has two bits that apply to KVM_SET_USER_MEMORY_REGION:

Flag Bit Meaning
KVM_MEM_LOG_DIRTY_PAGES 0x1 KVM maintains a dirty bitmap; retrieve with KVM_GET_DIRTY_LOG
KVM_MEM_READONLY 0x2 Guest writes produce KVM_EXIT_MMIO; requires KVM_CAP_READONLY_MEM

On ARM64, a write to a KVM_MEM_READONLY slot injects an abort into the guest rather than generating KVM_EXIT_MMIO -- a behavioral difference worth knowing if the code targets multiple architectures.

Slot Counts

As checked against Linux master on 2026-07-10, KVM_MEM_SLOTS_NUM is SHRT_MAX, or 32,767. x86 reserves three internal slots, leaving 32,764 user-visible slots in that source tree; arm64 leaves the default internal count at zero. These values replaced much smaller historical limits, including x86's former 509 user slots. They are still not an application constant. Query the running kernel with KVM_CHECK_EXTENSION(KVM_CAP_NR_MEMSLOTS) and treat its answer as authoritative.

Internally, the kernel stores memory slots in struct kvm_memory_slot (from include/linux/kvm_host.h), which holds the GPA base, page count, HVA backing address, dirty bitmap pointer, slot ID, and address-space ID. KVM indexes those slots for fast lookup by guest frame number and by slot ID. The data structure details matter for kernel work, but the VMM-facing rule is simpler: slot updates are whole mapping changes, and readers must see either the old mapping or the new one, never a half-updated slot.

The Extended Variant

A newer KVM_SET_USER_MEMORY_REGION2 (_IOW(KVMIO, 0x49, struct kvm_userspace_memory_region2)) extends the struct with guest_memfd_offset and guest_memfd fields. It requires capabilities KVM_CAP_GUEST_MEMFD and KVM_CAP_USER_MEMORY2. The extension exists for confidential computing -- Intel TDX and AMD SEV-SNP -- where guest memory must be isolated from the host even at the hypervisor level. Standard Firecracker deployments use the original ioctl.

The VMM mmap Pattern

The canonical sequence for allocating and registering guest RAM, following the LWN "Using the KVM API" article:

Host requirement: This code opens /dev/kvm and belongs on a bare-metal Linux host or a VM with nested virtualization. Keep it on an isolated machine prepared for KVM work. A registered HVA range must remain addressable while the VM uses its slot; removing its backing can turn the next guest access into an unresolved memory fault.

/* Step 1: allocate anonymous pages in the host process */
void *mem = mmap(NULL, size,
                 PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS,
                 -1, 0);

/* Step 2: declare the GPA->HVA mapping to KVM */
struct kvm_userspace_memory_region region = {
    .slot            = 0,
    .guest_phys_addr = 0x1000,
    .memory_size     = size,
    .userspace_addr  = (uint64_t)mem,
};
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

userspace_addr is an HVA -- a pointer in the VMM process's address space. The kernel has not touched the physical pages yet; the host kernel maps them lazily on first access. KVM records the slot's HVA range and uses it to build the EPT/NPT leaf entries that will resolve GPA to HPA when the guest faults in each page.

The backing can be anonymous memory (MAP_ANONYMOUS), file-backed memory (memfd or hugetlbfs), or device memory. The kernel recommends -- not requires -- that bits 20:0 of guest_phys_addr and userspace_addr be identical. The reason is alignment: both addresses must have the same offset within a 2 MiB boundary (2^21 bytes) for a host huge page to back a guest huge page without sub-page remapping. Violating this alignment is not an error, but it silently prevents huge-page EPT/NPT entries.

Firecracker's Slot Layout

Firecracker uses the vm-memory crate from the rust-vmm project, with two principal types:

pub type GuestRegionMmap = vm_memory::GuestRegionMmap<Option<AtomicBitmap>>; pub type GuestMemoryMmap = vm_memory::GuestRegionCollection<GuestRegionMmapExt>;

GuestRegionMmapExt wraps GuestRegionMmap and adds region_type (Dram or Hotpluggable), slot_from (starting KVM slot number), slot_size (uniform byte size per KVM slot), and plugged (a Mutex<BitVec> tracking which sub-slots are currently active). The AtomicBitmap type tracks dirty pages at page granularity using atomic operations, so multiple vCPU threads can mark pages concurrently without a lock. When dirty tracking is disabled, AtomicBitmap is None.

The slot registration is a From implementation in src/vmm/src/vstate/memory.rs:

impl From<&GuestMemorySlot<'_>> for kvm_userspace_memory_region {
    fn from(mem_slot: &GuestMemorySlot) -> Self {
        let flags = if mem_slot.slice.bitmap().is_some() {
            KVM_MEM_LOG_DIRTY_PAGES
        } else { 0 };
        kvm_userspace_memory_region {
            flags,
            slot: mem_slot.slot,
            guest_phys_addr: mem_slot.guest_addr.raw_value(),
            memory_size: mem_slot.slice.len() as u64,
            userspace_addr: mem_slot.slice.ptr_guard().as_ptr() as u64,
        }
    }
}

This From impl is the entire gap between Rust types and the raw ioctl struct. KVM_MEM_LOG_DIRTY_PAGES is set if and only if the AtomicBitmap is present -- a clean expression of the dirty-tracking flag as a type-level choice.

Firecracker's x86-64 GPA layout, checked on main on 2026-07-10, comes from src/vmm/src/arch/x86_64/layout.rs:

Region GPA Start Notes
Low RAM 0x0 Up to the 32-bit MMIO hole
EBDA / system data 0x9fc00 MPTable, ACPI tables
Kernel load 0x0010_0000 (1 MiB) HIMEM_START
32-bit MMIO gap 3 GiB-4 GiB Device BARs, LAPIC, IOAPIC
Mid RAM 4 GiB-256 GiB Second base-DRAM region
64-bit MMIO gap 256 GiB-512 GiB MMIO64_MEM_START, 256 GiB long
High RAM 512 GiB+ Third base-DRAM region for very large guests

The first MMIO hole spans 3 GiB to 4 GiB. A second, 256 GiB MMIO hole begins at 256 GiB. arch_memory_regions() can therefore split base DRAM into as many as three regions. Firecracker's hotpluggable-memory representation is separate and can divide a region into multiple KVM subslots.

Backing mode is determined at construction time in MmapRegion: anonymous() produces MAP_PRIVATE | MAP_ANONYMOUS memory (with optional hugepages), memfd_backed() produces MAP_SHARED memory via a memfd file descriptor, and snapshot_file() uses MAP_PRIVATE from a file for snapshot restore.

Two-Dimensional Paging: EPT And NPT

Shadow Paging: The Problem It Solved and Created

Before EPT and NPT existed, KVM used shadow page tables. A struct kvm_mmu_page held 512 shadow PTEs (SPTEs) that mapped GVA directly to HPA, effectively collapsing the two-level translation into one. That sounds efficient, but the maintenance cost was severe. KVM had to write-protect all guest page tables so it could intercept modifications; every guest CR3 load, every INVLPG, and every page-table write triggered a VM exit so KVM could rebuild or invalidate the corresponding shadow entries. The KVM MMU documentation notes that in EPT mode "neither invlpg nor CR3 loads and stores cause a vmexit in EPT mode, and kvm_set_cr3 is hardly ever called" -- describing, by contrast, how intrusive shadow paging was.

Shadow paging was also the source of a central MMU lock that serialized all vCPU threads on page-fault handling, a design that broke down catastrophically at scale. That lock is what motivated the TDP MMU rewrite discussed below.

The 24-Access Worst Case

A TLB miss under two-level nested paging on x86-64 requires up to 24 memory accesses in the worst case. The derivation: the guest has 4-level page tables (PML4 -> PDPT -> PD -> PT); each of those four guest-page-table entries is itself a GPA that must be resolved through the 4-level nested page table, costing 4 accesses per guest-table walk level plus 1 for the nested PML4 root. That gives 4 x 5 = 20 accesses for the guest walk, plus 4 more to translate the final GPA to HPA: 24 total. This is the cold-TLB worst case with no EPT or NPT TLB entries populated. In steady state the hardware TLBs cache the composed translations and most accesses cost nothing beyond a normal TLB hit.

Intel EPT

EPT was introduced in the Nehalem microarchitecture -- the first Intel Core i-series, around 2008. The "unrestricted guest" mode, which allows a guest to run in real mode without shadow paging, requires EPT and was added in the subsequent Westmere generation.

Enabling EPT. EPT is activated through VMCS Secondary Processor-Based VM-Execution Controls, encoding 0x401E (SECONDARY_VM_EXEC_CONTROL, confirmed in arch/x86/include/asm/vmx.h). Bit 1 of that field is "Enable EPT." Setting it to 1 activates hardware two-level paging for that VM.

EPT Pointer. Once EPT is enabled, the hardware needs to know where the root of the EPT paging structure lives. VMCS field EPT_POINTER (encoding 0x201A, confirmed in arch/x86/include/asm/vmx.h) is written via VMWRITE to supply that root. The EPTP bit layout:

Bits Meaning
2:0 EPT paging-structure memory type (0 = UC, 6 = WB; WB is normal)
5:3 Page-walk length minus 1 (3 = 4-level EPT, the current standard)
6 Enable accessed and dirty flags in EPT entries (requires CPU support; absent before Haswell)
11:7 Reserved, must be zero
51:12 Physical address of EPT PML4 table
63:52 Reserved

A 5-level EPT (PML5) was added for 57-bit guest physical addresses; bits 5:3 would be 4 for 5-level.

EPT leaf PTE fields. Each EPT entry is 8 bytes. The leaf PTE bits that matter most:

Bit Meaning
0 Read permission
1 Write permission
2 Execute permission (supervisor mode)
5:3 EPT memory type (6 = WB)
8 Accessed flag (set by hardware)
9 Dirty flag (set by hardware on write; leaf entries only)
51:12 Host physical page frame address

EPT violations and misconfigurations. An EPT violation exits when a guest access lacks sufficient EPT permission -- for example, a write to a read-only EPT entry. Exit reason EXIT_REASON_EPT_VIOLATION = 48 (from arch/x86/include/uapi/asm/vmx.h). An EPT misconfiguration exits when an EPT entry has an illegal format, such as a non-leaf entry with write permission set but read permission clear. Exit reason EXIT_REASON_EPT_MISCONFIG = 49. KVM uses EPT violations deliberately for MMIO interception: MMIO ranges are left unmapped in the EPT, so a guest access generates an EPT violation that KVM handles as KVM_EXIT_MMIO back to the VMM, without any explicit MMIO range registration in EPT.

flowchart TD
  A["Guest memory access (GPA)"]
  F{"Entry format valid?\n(e.g. write=1 but read=0 is illegal)"}
  G["EPT misconfiguration -> EXIT_REASON_EPT_MISCONFIG (49)"]
  B{"Entry present?\n(read | write | execute != 0)"}
  D{"Permission sufficient\nfor this access?"}
  E["EPT violation -> EXIT_REASON_EPT_VIOLATION (48)"]
  C["Hardware resolves GPA->HPA (no exit)"]

  A --> F
  F -- no --> G
  F -- yes --> B
  B -- no --> E
  B -- yes --> D
  D -- no --> E
  D -- yes --> C

EPTP switching. VM function 0 allows VMX non-root software to switch EPT roots without a full VM exit, by indexing into a hypervisor-controlled list of 512 8-byte EPTP entries via ECX. VMCS VM-function controls live at encoding 0x2018; the EPTP-list address lives at 0x2024. This is a niche optimization for workloads that need to quickly present different physical memory views to a guest.

AMD NPT

AMD introduced nested paging in 3rd-generation Opteron (codename Barcelona, 2007), one year before Intel's Nehalem. AMD's marketing name is Rapid Virtualization Indexing; the engineering name is NPT. Performance gains over shadow paging: VMware research measured up to 42%; Red Hat OLTP testing showed approximately 2x throughput improvement.

Enabling NPT. AMD's VM control block is the VMCB, a structure distinct from Intel's VMCS. Linux names bit 0 of the VMCB misc_ctl field SVM_MISC_ENABLE_NP; setting it activates nested paging for VMRUN.

nCR3. The nested page table root is held in nCR3 (Nested CR3), a 64-bit field at VMCB control area offset 0xB0 (confirmed in FreeBSD's sys/amd64/vmm/amd/vmcb.h as VMCB_OFF_NPT_BASE). It holds a host physical address -- the HPA of the top-level NPT paging structure. The guest's ordinary CR3 (gCR3) holds a GPA of the guest's own page-table root. Both pointers are active simultaneously; this is the fundamental asymmetry. The hardware consults gCR3 for GVA->GPA and nCR3 for GPA->HPA.

ASID. Each guest is assigned an Address Space Identifier at VMCB control area offset 0x58 (VMCB_OFF_ASID), so the hardware can tag TLB entries per-guest and avoid full TLB flushes on VM entry and exit.

Nested page faults. A nested page fault generates SVM exit code SVM_EXIT_NPF = 0x400 (from arch/x86/include/uapi/asm/svm.h). KVM module parameters kvm-amd.npt=0 and kvm-intel.ept=0 disable NPT and EPT respectively at module load time; the default for both is 1 (enabled for 64-bit and 32-bit PAE mode).

VMCB clean bits. Linux calls bit 4 of the clean-bits field VMCB_NPT. When set, it tells the processor that the nested-paging group, including nested_cr3, has not changed and may be reused without re-reading that group from memory.

The TDP MMU

KVM's TDP MMU (arch/x86/kvm/mmu/tdp_mmu.c) is an MMU implementation designed specifically for EPT/NPT. It avoids the per-page reverse mappings that shadow paging needs to find every SPTE associated with a guest page. Because TDP roots map GPA directly to HPA, page-fault work can proceed with much less shared bookkeeping and substantially more parallelism; the MMU lock still exists, but TDP faults do not serialize in the same way as the legacy write-side path.

When the TDP MMU was introduced as a 22-patch series in September 2020, Google measured an 89% reduction in demand-paging test duration on 416-vCPU VMs; previously 98% of time was spent waiting for the MMU lock. The work enabled live migration of 416-vCPU, 12 TiB VMs that had been impractical with the legacy path. The shadow MMU remains necessary when EPT/NPT is unavailable and for translation combinations that hardware cannot encode directly.

In TDP mode, the SPTE role has role.base.direct = true (direct GPA->HPA mapping), with role.base.cr0_wp and role.base.efer_nx unconditionally set to true -- unlike shadow paging, where they reflect actual guest CPU state. KVM supports 4 KiB (level-1 SPTE), 2 MiB (level-2), and 1 GiB (level-3) EPT/NPT entries. A large SPTE requires that the host supports the page size, that the guest PTE maps an equivalent range, that no write-protected pages exist in the range, and that the entire range falls within a single memory slot.

Dirty-Page Tracking

Two use cases drive dirty-page tracking: live migration (which must replay every write the guest makes after the first full copy) and snapshot diffing (which records only pages changed since the last snapshot). KVM offers a per-slot bitmap and a per-vCPU ring buffer. A VM selects the ring or the standalone bitmap as its primary interface; newer kernels can pair a dirty ring with a backup bitmap for writes produced outside a vCPU/ring context.

The Legacy Bitmap Interface

Setting KVM_MEM_LOG_DIRTY_PAGES (0x1) in kvm_userspace_memory_region.flags instructs the kernel to maintain a dirty bitmap for that slot -- one bit per 4 KiB page, bit 0 corresponding to the first page. The bitmap lives in kvm_memory_slot.dirty_bitmap. How KVM observes writes is an implementation detail: it can write-protect mappings and fault on the first write, or use hardware support such as Intel Page Modification Logging. Firecracker's hugepage documentation records the practical consequence for its supported hosts: dirty tracking makes KVM establish 4 KiB guest mappings, negating most of the benefit of hugepage-backed memory.

KVM_GET_DIRTY_LOG (_IOW(KVMIO, 0x42, struct kvm_dirty_log)) retrieves the bitmap for one slot:

struct kvm_dirty_log {
    __u32 slot;
    __u32 padding1;
    union {
        void __user *dirty_bitmap;
        __u64        padding2;
    };
};

By default, the kernel clears dirty bits atomically before the ioctl returns. KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 (capability value 168) defers that clearing to a subsequent KVM_CLEAR_DIRTY_LOG call. KVM_CLEAR_DIRTY_LOG (_IOWR(KVMIO, 0xc0, struct kvm_clear_dirty_log)) adds __u32 num_pages and __u64 first_page fields, enabling partial range clearing rather than whole-slot clearing -- useful for large slots where clearing the entire bitmap stalls the guest.

KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 supports two sub-flags: KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE (1 << 0) and KVM_DIRTY_LOG_INITIALLY_SET (1 << 1). When INITIALLY_SET is active, the bitmap starts all-ones, treating all pages as initially dirty. A VMM can then use KVM_CLEAR_DIRTY_LOG to mark pages clean and re-enable tracking in 64-page-granularity chunks instead of reprotecting an entire slot in one operation. KVM_DIRTY_LOG_INITIALLY_SET is incompatible with the dirty ring interface.

The Dirty Ring Interface

KVM_CAP_DIRTY_LOG_RING (capability value 192) enables the dirty ring. The ring is a per-vCPU mmap'd region, separate from kvm_run, containing struct kvm_dirty_gfn entries:

struct kvm_dirty_gfn {
    __u32 flags;  /* KVM_DIRTY_GFN_F_DIRTY = (1<<0), KVM_DIRTY_GFN_F_RESET = (1<<1) */
    __u32 slot;
    __u64 offset; /* page offset within slot */
};

The state machine is: flags = 0 means the entry is unused; flags = 1 (DIRTY set) means the kernel has recorded a dirty GFN; and flags = 1X (RESET set, with DIRTY ignored) means userspace has harvested it. Userspace reads ring entries without an ioctl, in sequence. It then issues KVM_RESET_DIRTY_RINGS (_IO(KVMIO, 0xc7)) before reading the harvested pages so KVM can re-enable dirty tracking for those GFNs.

If KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP is available, the ring can use per-slot bitmaps as a backup for dirtying that occurs without a vCPU/ring context. Userspace harvests that bitmap only after quiescing the remaining producers; it does not turn the two interfaces into interchangeable, concurrently consumed logs.

The ring has a genuine trade-off. In one reported test with an 800 MB/s random-write rate and a 24 GiB guest, dirty-ring migration took approximately 73 seconds, versus 55 seconds for the bitmap. That result is not a universal ranking; it shows that per-page ring harvesting and reset work can lose to bulk bitmap handling at a high dirty rate. The workload and migration algorithm decide which interface wins.

Firecracker's Dirty Tracking

Firecracker's MachineConfig (in src/vmm/src/vmm_config/machine_config.rs) has a track_dirty_pages: bool field, default false. When true, Firecracker sets KVM_MEM_LOG_DIRTY_PAGES on all memory slots, and each GuestRegionMmapExt receives a Some(AtomicBitmap) rather than None.

The snapshot flow in src/vmm/src/vmm_config/snapshot.rs uses store_dirty_bitmap() to read KVM's dirty log and merge it into the internal AtomicBitmap. dump_dirty() then iterates 64-bit words of the merged bitmap, seeking past clean regions using sparse-file semantics, and writes only dirty 4 KiB pages to the diff snapshot file. After a diff snapshot, Firecracker resets the dirty bitmap to baseline the next diff.

Without track_dirty_pages, Firecracker falls back to mincore(2) to identify resident pages. This mode requires swap to be disabled: a page swapped out appears as not-in-core and would be silently omitted from the snapshot. The trade-off is that mincore produces no write overhead at runtime, while track_dirty_pages introduces the write-protection overhead described above and forces 4 KiB granularity even when the host uses hugepages. Diff snapshots are currently in developer preview.

Userfaultfd for Snapshot Resume

When restoring a VM from a snapshot, the VMM must make guest memory available without eagerly reading the entire file. Firecracker supports two modes, controlled by LoadSnapshotParams.mem_backend.backend_type: File, where the host kernel services page faults through the page cache, and Uffd, where a dedicated userspace process services them through userfaultfd.

In the Uffd path, a separate userspace process receives the userfaultfd file descriptor over a Unix domain socket and responds to UFFD_EVENT_PAGEFAULT by issuing UFFDIO_COPY to populate individual pages on demand as the guest touches them. On Linux 5.10, the userfaultfd object is created via the userfaultfd(2) syscall; on Linux 6.1 and later it is created via /dev/userfaultfd. When the virtio-balloon deflates during a UFFD-backed resume, madvise(MADV_DONTNEED) triggers UFFD_EVENT_REMOVE, and the page handler must zero those pages rather than reloading from the snapshot file -- a subtle interaction between two separately designed subsystems that Firecracker's documentation explicitly warns about.

Memory Ballooning

Ballooning is the mechanism by which the host can reclaim memory from a running guest without stopping it. The guest OS voluntarily surrenders pages through a device driver, and the VMM releases the backing host memory. The protocol is virtio.

The Virtio Balloon Device

The virtio balloon device has device ID 5 (OASIS virtio 1.2 spec section 5.5.1). It uses up to five virtqueues: index 0 inflates the balloon, index 1 deflates it, index 2 carries statistics when VIRTIO_BALLOON_F_STATS_VQ is negotiated, index 3 carries free-page hints, and index 4 carries continuous free-page reports. The protocol is asymmetric: the host signals how many pages it wants by writing num_pages into virtio_balloon_config; the guest driver responds at its own pace. The host cannot force prompt cooperation.

The feature bits that define balloon behavior (from include/uapi/linux/virtio_balloon.h):

Bit Constant Meaning
0 VIRTIO_BALLOON_F_MUST_TELL_HOST Guest must notify host before reusing deflated pages
1 VIRTIO_BALLOON_F_STATS_VQ Enables stats virtqueue (index 2)
2 VIRTIO_BALLOON_F_DEFLATE_ON_OOM Guest deflates balloon instead of invoking OOM killer
3 VIRTIO_BALLOON_F_FREE_PAGE_HINT Guest reports free pages to host (index 3)
4 VIRTIO_BALLOON_F_PAGE_POISON Guest reports page-poison value via poison_val config field
5 VIRTIO_BALLOON_F_REPORTING Guest reports free pages via reporting queue for host to reclaim

The base config fields are __le32 num_pages (how many pages the host wants in the balloon) and __le32 actual (how many are currently held). Negotiated hinting and poisoning features add free_page_hint_cmd_id and poison_val. The stats queue exchanges packed, 10-byte struct virtio_balloon_stat tag-value pairs, with 16 defined tags as of Linux 6.12, including swap-in/out counts, major/minor faults, free and total memory, OOM kills, and direct and asynchronous reclaim statistics.

Firecracker's Balloon

Firecracker exposes the balloon through its REST API:

As checked on Firecracker main on 2026-07-10, the device supports traditional inflation and statistics, VIRTIO_BALLOON_F_DEFLATE_ON_OOM (bit 2), free-page hinting (bit 3, developer preview), and free-page reporting (bit 5).

When the guest inflates the balloon -- surrendering pages -- Firecracker discards the corresponding HVA range, normally with madvise(MADV_DONTNEED), so the host can reclaim its physical backing. Anonymous private pages are zero-filled if faulted in again. The Firecracker documentation stresses that the guest driver is untrusted: it controls the surrender rate, can report misleading statistics, and may stop cooperating. A platform must therefore be able to contain the Firecracker process at its full configured memory size; the balloon is not a hard security limit.

Oversubscription

Firecracker's design document states that microVMs can oversubscribe host CPU and memory; the degree is controlled by the operator. Guest RAM is mapped with MAP_NORESERVE, and host pages are generally populated on demand, so configured guest memory is not the same number as current resident memory. Allocation can still fail because of address-space limits, host policy, or later memory pressure; virtual reservation is not a promise that the host can satisfy every guest write.

Firecracker's production host setup guide (docs/prod-host-setup.md) recommends two settings that define the oversubscription envelope:

Disable insecure swap. The guide recommends an empty /proc/swaps, or an equivalently secure swap design, because guest memory written to host storage creates a data-remanence risk. Diff snapshots taken without KVM dirty tracking impose the stricter rule: swap must be disabled because mincore(2) does not report swapped-out pages as resident.

Disable KSM. echo 0 > /sys/kernel/mm/ksm/run. KSM (Kernel Same-page Merging) deduplicates pages with identical content across processes, saving physical RAM. The security cost is a timing side channel: by measuring how long certain memory operations take, a process can determine which pages are shared with another process -- leaking information about memory access patterns across VM boundaries. Disabling KSM removes this channel entirely.

Host-wide changes: Swap and KSM configuration requires administrative control and affects every workload on the machine. Apply the production-host policy during isolated host provisioning, not casually from a VMM startup script.

With swap and KSM disabled, virtio-balloon is the cooperative mechanism for returning guest-owned pages while a VM remains alive. Cgroup memory.limit_in_bytes (or the v2 equivalent memory.max) provides a hard ceiling on how much memory a Firecracker process can consume; exceeding it can invoke reclaim or the OOM policy rather than politely shrinking the guest. The operator's oversubscription ratio must leave room for actual guest working sets and host overhead, not merely compare configured slot sizes with installed RAM.

The next chapter turns from memory layout to virtual interrupts and time: how a guest receives events, completes device I/O, and keeps a clock when the hardware it sees is virtual.

Sources And Further Reading