Chapter 16: Snapshot And Restore

A Firecracker snapshot captures a paused machine, not merely a boot shortcut. The saved state includes guest memory, vCPU registers, in-kernel KVM state, and the emulated device state needed to continue execution. Restore rebuilds those objects in a new firecracker process and starts its vCPU threads in Paused state. The caller decides when they run.

That continuation property is both the value and the danger. A clone resumes with the kernel, application heaps, random-number generators, clocks, tokens, and protocol state present at the snapshot point. Restore avoids repeating initialization; it does not make duplicated state unique or current.

Creating A Snapshot

Snapshot creation is a runtime operation, but the microVM must already be paused:

PATCH /vm
{"state":"Paused"}

PUT /snapshot/create
{"snapshot_type":"Full","snapshot_path":"state.snap","mem_file_path":"memory.snap"}

These requests open KVM and snapshot files and therefore belong on an isolated Linux host with KVM access, or in a VM with nested virtualization. A successful create writes two artifacts. snapshot_path receives serialized VMM and KVM state. mem_file_path receives guest memory. Attached block images are external resources and are not copied into either file.

Pausing stops vCPU execution while the event loop remains alive. During device save, Firecracker calls each device's prepare_save. File-backed block devices drain outstanding operations and synchronize the backing file to the host filesystem. That does not package the file with the snapshot or guarantee that the host filesystem has committed it to durable media. The operator must keep the matching block images and coordinate any stronger durability requirement.

Snapshot creation also resets an active vsock transport. When the original VM or a restored VM resumes, the guest driver closes existing connections while listen sockets remain. Creating a snapshot is therefore not observationally invisible even though the guest CPUs are paused.

The state file is written and synchronized before the memory dump. A failure while writing memory can consequently leave an unusable partial artifact set; callers should publish the pair only after the API reports success.

The State File

The current writer serializes a header and MicrovmState with bitcode, then appends an eight-byte CRC-64:

The architecture-specific magic rejects an x86_64 state file on aarch64 and the reverse. The format version is independent of the Firecracker release version. The current loader requires the same major version and accepts a snapshot minor version no newer than its own; the patch component does not participate in that check. A 10,000,000-byte deserialization limit bounds the state file, not the guest-memory file.

The CRC detects accidental corruption. It is not authentication and says nothing about the block images or memory file. Snapshot artifacts can contain secrets and executable guest state, so an integrator must protect their confidentiality, integrity, ownership, and provenance outside Firecracker.

MicrovmState has five top-level parts:

pub struct MicrovmState { pub vm_info: VmInfo, pub kvm_state: KvmState, pub vm_state: VmState, pub vcpu_states: Vec<VcpuState>, pub device_states: DevicesState, }

VmInfo carries such configuration as memory size, SMT, boot-source paths, the static CPU-template choice, and huge-page policy. KvmState, VmState, and the per-vCPU vector preserve KVM capability modifiers, VM state, and CPU state. DevicesState now covers MMIO and optional PCI transports, ACPI VMGenID and VMClock, serial emulation, and the supported virtio devices. Vhost-user block is explicitly unsnapshottable. Vsock configuration and frontend state are saved, but live host connections are not continued.

Some process configuration is deliberately outside the state. Log and metrics destinations must be configured in the new process. MMDS configuration is saved with network-device state, but the MMDS data store is not. Host resources remain references: block paths must resolve, TAP devices must exist, and the vsock backend socket must be reachable when the new process restores devices.

Full And Differential Memory

snapshot_type defaults to Full. A full snapshot writes every plugged memory slot and seeks across unplugged virtio-mem slots, producing a file whose logical length describes the complete memory layout. The holes are sparse ranges; they are not ballooned pages. Creating the full image faults guest pages into the Firecracker process and can be much more expensive than loading that image later on demand.

Diff is a developer-preview format. It writes a full state file but only a layer of guest-memory pages considered dirty since boot or the preceding snapshot. A general diff layer is not a standalone memory image. Merge layers over the base memory file in creation order, then pair the resulting memory image with the state file produced alongside the last merged layer.

Firecracker has two ways to select pages for a diff. With track_dirty_pages: true, KVM_GET_DIRTY_LOG reports vCPU writes and the userspace AtomicBitmap covers writes performed through Firecracker's guest memory mappings. Firecracker conservatively marks activated virtqueue memory dirty after each snapshot because queue-object accesses are not marked during normal device operation.

Without KVM dirty tracking, Firecracker uses mincore(2) residency as an over-approximation. This works only with swap disabled: a dirty page moved to swap is not resident and can be omitted incorrectly. The fallback may also include clean resident pages. Dirty tracking is more precise but adds runtime cost and negates much of the benefit of huge pages.

flowchart LR
    A["Paused VM"] --> B{"Dirty tracking?"}
    B -->|"yes"| C["KVM dirty log"]
    B -->|"yes"| D["Userspace memory bitmap"]
    C --> E["Union selected pages"]
    D --> E
    B -->|"no"| F["mincore resident pages"]
    E --> G["Sparse diff layer"]
    F --> G
    G --> H["Merge over base in creation order"]
    H --> I["Resumable memory image"]

When a diff target already exists with the expected logical size, Firecracker writes selected pages into that file in place. This can merge directly into a base, but it also means the caller must not mistake a same-sized unrelated file for a disposable output. After a successful snapshot, Firecracker clears the dirty state used for the next interval and pre-marks virtqueue pages again.

Saving vCPU State

KVM does not expose a single save-vCPU ioctl. On x86_64, Firecracker reads KVM_GET_MP_STATE first because accepting pending APIC events can modify later state. It then reads general and special registers, XSAVE and XCR state, debug registers, LAPIC state, TSC frequency, CPUID, and selected MSRs. KVM_GET_VCPU_EVENTS comes last because earlier reads can affect pending exceptions, interrupts, NMIs, and SMIs.

Restore has its own dependency order. KVM_SET_CPUID2 precedes MP state. General registers precede vCPU events because KVM_SET_REGS clears pending exceptions. Special registers precede LAPIC state because they restore the APIC-base MSR, and LAPIC state precedes the saved MSRs because the TSC-deadline MSR depends on LAPIC timer mode. KVM_SET_VCPU_EVENTS is last. All other vCPUs must remain stopped while these event structures are restored.

Device state is saved before KVM state. An async block device can complete work and inject an interrupt while prepare_save drains it; saving KVM first would lose that interrupt from the restored machine.

Loading A Snapshot

PUT /snapshot/load is a pre-boot operation. Apart from logs and metrics, the new process must not already have a machine configuration. The request names the state file and a memory backend:

PUT /snapshot/load
{
  "snapshot_path": "state.snap",
  "mem_backend": {
    "backend_path": "memory.snap",
    "backend_type": "File"
  },
  "track_dirty_pages": false,
  "resume_vm": false
}

The older top-level mem_file_path spelling selects a file backend but is deprecated. track_dirty_pages is not stored in the snapshot; a caller that wants precise later diff snapshots must request it again during load.

Before building KVM objects, Firecracker validates the state file, applies network_overrides to TAP names and an optional vsock_override to the host socket, checks the memory-region description, and warns if the saved and host CPU vendor or manufacturer differs. There is no corresponding block-path override in the load request.

Restore then constructs KVM and vCPU fds, registers the restored memory regions, restores each vCPU, restores VM state, and reconstructs devices. ACPI device restore changes VMGenID and VMClock state only after KVM interrupt state has been restored, so their notifications are not overwritten. The vCPU host threads start paused. resume_vm: true resumes them before the load request completes; otherwise the caller later sends PATCH /vm with Resumed.

sequenceDiagram
    participant A as API caller
    participant F as Firecracker
    participant M as Memory backend
    participant K as KVM

    A->>F: PUT /snapshot/load
    F->>F: Validate state and apply overrides
    F->>M: Map file or register UFFD regions
    F->>K: Create VM and vCPUs
    F->>K: Register memory regions
    F->>K: Restore vCPU and VM state
    F->>F: Restore devices and start paused threads
    opt resume_vm is true
        F->>K: Resume vCPUs into KVM_RUN
    end
    F-->>A: 204 No Content

File-Backed Memory

The File backend maps each region from the memory image with MAP_PRIVATE. Reads fault pages through the host page cache. The first guest write creates a private anonymous copy, leaving the file unchanged. Multiple restored processes can therefore share clean page-cache pages while diverging only where each guest writes.

This is demand paging, not an eager copy. Startup can be quick because restore does not read all guest memory first; later page faults move part of the cost onto the guest's execution path. Working-set shape, storage latency, page-cache state, vCPU count, memory size, and devices all affect observed resume latency.

The mapped memory image must remain immutable for the life of the restored process. External modification can change data supplied by later faults and corrupt the guest. The file backend cannot restore hugetlbfs-backed snapshots; Firecracker requires UFFD for those.

Userfaultfd Memory

The Uffd backend gives fault servicing to an external process. Firecracker creates anonymous guest mappings, creates a userfaultfd, registers each region, and connects to the Unix socket named by backend_path. It sends the fd with SCM_RIGHTS and a JSON array of region mappings containing host address, size, backing offset, and page size. No continuing control protocol runs over that socket.

The handler reads UFFD_EVENT_PAGEFAULT and supplies data with operations such as UFFDIO_COPY. Because page faults can occur while Firecracker itself is restoring state, the handler must be listening and ready before the load request. Firecracker keeps its own UFFD fd open. If the handler disappears, future faults wait with no service and the guest can freeze indefinitely; there is no automatic fallback to the snapshot file.

Balloon discard produces UFFD_EVENT_REMOVE. A correct handler records those ranges as zero so that a later fault does not resurrect pre-discard data from the snapshot. The handler is part of the VM's availability and isolation boundary and must be monitored and confined accordingly.

Clone Uniqueness

Restoring one snapshot more than once duplicates everything already consumed or cached at the snapshot point. That includes kernel and userspace PRNG state, UUID sequences, /proc/sys/kernel/random/boot_id, credentials, TLS material, session tokens, database connection state, and application counters. Remapping a TAP device does not repair any of them.

Firecracker always exposes VMGenID. On restore it generates a new 16-byte ID, writes it into guest memory, and notifies the guest. Linux ACPI support arrived in 5.18; DeviceTree support arrived later and must be backported to the Linux 6.1 aarch64 kernels supported by Firecracker. A supported kernel mixes the new ID into its CSPRNG and reseeds. There is still a window before the guest handles the notification, and VMGenID cannot reset entropy cached inside an application library.

VMClock provides a separate userspace-visible generation counter. Firecracker increments it atomically before vCPUs resume. Guest software with the relevant VMClock kernel support can map or poll the device and rebuild userspace state when the counter changes. Upstream support is newer than Firecracker's current guest-kernel baseline, so deployments need an appropriate backport or newer kernel. Neither device can identify every duplicated token automatically; the application must define its resume boundary and reacquire per-instance state before serving work.

The safest snapshot point is after generic runtime initialization but before the guest obtains request-specific secrets or performs externally visible work. If the original VM and a clone both continue from the same state, a value meant to be used once can be used twice even if only one clone was created.

Clocks And Connections

By default, x86 kvmclock resumes from the saved value. Time spent while the snapshot was dormant is not added. clock_realtime: true asks KVM to advance kvmclock by elapsed host real time through KVM_CLOCK_REALTIME; it is x86_64 only, requires host support, and can expose a sudden jump to the guest. Guest wall-clock synchronization remains an operating-system responsibility.

Network connectivity is not guaranteed across resume. TAP overrides change the host endpoint, not IP addresses, routes, TCP sequence numbers, peer state, or credentials in guest memory. Vsock deliberately reports a transport reset and closes existing connections after resume. A clone protocol should treat external connections as invalid and reconnect explicitly.

Compatibility Boundaries

A matching snapshot format is necessary but not sufficient. Firecracker's documentation requires equivalent software and hardware configuration. Snapshots do not cross architectures, and Intel-to-AMD restore is unsupported even though current source only warns on a vendor mismatch before later restore steps fail or misbehave. CPU features exposed to the guest must remain invariant; a suitable CPU template can help only within its documented host set.

Host-kernel changes can alter the semantics of saved KVM state. Firecracker calls cross-kernel restore unstable and publishes only narrow tested cases. AArch64 snapshots also cannot cross GIC versions. Restore hosts must reproduce block paths and permissions and make required TAP and vsock endpoints available. cgroup v2 is strongly recommended because cgroup v1 is associated with high restore latency.

Treat state file, memory image, block images, CPU contract, host kernel, and external endpoints as one versioned artifact set. Valid CRC and compatible format numbers cannot make a mismatched set safe to resume.