Chapter 15: Boot And Configuration

Before InstanceStart, Firecracker has a control plane but no virtual machine. The API thread is listening, and the pre-boot controller has accumulated a VmResources configuration, but there is no KVM VM fd, registered guest memory, vCPU fd, or emulated device attached to a machine. PUT /actions with "action_type": "InstanceStart" turns that configuration into a running guest.

Firecracker's specification measures boot time from receipt of that request to the start of guest /sbin/init. Its 125 ms requirement applies under stated conditions, including a minimal kernel and root filesystem and no serial console. It is a useful design constraint, not a promise that every kernel, filesystem, host, and device configuration will boot in 125 ms.

The API Socket

The API uses HTTP/1.1 over a Unix domain socket, not TCP. Firecracker binds /run/firecracker.socket unless --api-sock selects another path. The localhost host in the OpenAPI document is an HTTP convention; it does not make the service listen on a network interface.

Unix permissions, the socket's containing directory, and the jailer's chroot determine who can reach the API. The firecracker process creates the socket. The jailer arranges the process environment and chroot, but it does not create an already-connected control channel and pass it across exec.

The pre-boot controller is stateful. Each successful PUT or PATCH changes the resource configuration held in memory. A later request can replace a boot source or a device with the same ID. InstanceStart consumes the resulting configuration; it is not a transaction that commits a batch of otherwise stateless calls.

Most device-identity operations are pre-boot only. Runtime operations include pause and resume, snapshot actions, metrics flushes, block and network rate updates, memory-management actions, and, when preview PCI support is enabled, device hotplug operations. Chapter 13 gives the complete controller split.

stateDiagram-v2 [*] --> NotStarted NotStarted --> Running: InstanceStart Running --> Paused: PATCH /vm Paused Paused --> Running: PATCH /vm Resumed Running --> Running: FlushMetrics or SendCtrlAltDel

SendCtrlAltDel injects keyboard input on x86_64. What the guest does with the sequence depends on its userspace and kernel configuration, so the request is not itself an Exited transition. Firecracker exits if the guest later performs the i8042 reset operation that Firecracker treats as shutdown.

GET / is valid before and after boot. Its InstanceInfo response contains app_name, id, state, and vmm_version. The state is one of "Not started", "Running", or "Paused".

The Boot Source

PUT /boot-source requires kernel_image_path and optionally accepts initrd_path and boot_args. Firecracker opens the kernel and initrd while processing the request, so the paths are resolved in the process's current mount namespace and chroot. Under the jailer, the files must be available inside that filesystem view.

On x86_64, current Firecracker loads an uncompressed ELF kernel. It prefers the PVH entry point when the image contains the appropriate note and otherwise uses the Linux 64-bit boot protocol. On aarch64 it loads the PE-formatted Linux Image. In neither case is there a BIOS, GRUB, or an in-guest bootloader stage. Chapter 10 traces both x86 handoffs.

If boot_args is absent, Firecracker starts with this command line:

reboot=k panic=1 nomodule 8250.nr_uarts=0 i8042.noaux i8042.nomux i8042.dumbkbd swiotlb=noforce

The builder then augments it. With the default MMIO transport it adds pci=off. A root virtio-block drive adds root=/dev/vda, or root=PARTUUID=<uuid> when partuuid is configured, plus ro or rw. A root virtio-pmem device similarly adds its /dev/pmemN root argument. An explicit boot_args string replaces the default string, but these transport and root arguments are still derived during device attachment.

An initrd is optional and does not change Firecracker's platform devices or transport requirements. It changes what early userspace can do before the kernel mounts its final root filesystem; it does not by itself remove a need for ACPI, virtio, or PCI support implied by the selected machine configuration.

A second PUT /boot-source before boot replaces the whole boot-source object. There is no boot-source PATCH.

Machine Configuration

Firecracker begins with one vCPU and 128 MiB of memory. GET /machine-config reports those defaults before any update. The PUT schema requires both vcpu_count and mem_size_mib; PATCH updates a subset of the existing values.

vcpu_count ranges from 1 through 32. With smt: true, it must be 1 or an even number, and SMT is supported only on x86_64. Each configured vCPU becomes a KVM vCPU and a host thread, but these are not the process's only threads: the API and VMM event loop also have threads, and optional facilities can add more.

mem_size_mib has no general 128 MiB minimum. That value is the default. Whether a smaller allocation can boot depends on the guest kernel, initrd, userspace, device layout, and architecture. Firecracker rejects zero-sized memory and applies configuration-specific constraints, such as requiring an even MiB count for 2 MiB huge pages.

track_dirty_pages enables the KVM dirty bitmap used for differential snapshots. Firecracker does not implement live migration. huge_pages: "2M" uses 2 MiB hugetlbfs pages for guest memory. The host must have enough pages in its pool: Firecracker maps with MAP_NORESERVE, so exhaustion can surface as SIGBUS on access. Current documentation also limits hugepage-backed snapshot restore to the userfaultfd backend and warns that 4 KiB dirty tracking removes the expected huge-page performance benefit. These options belong in a host capacity plan, not in a generic boot recipe.

CPU Templates

machine-config.cpu_template selects one of Firecracker's deprecated static templates. PUT /cpu-config supplies a custom template: CPUID and MSR modifiers on x86_64, or register and vCPU-feature modifiers on aarch64. Setting a custom template replaces the template stored in the machine configuration.

Templates create a controlled CPU feature surface for compatibility and snapshot portability. They are host- and architecture-sensitive, and the official documentation explicitly excludes them as a security boundary. The important lifecycle rule is simple: Firecracker resolves the template while creating vCPUs, so CPU identity cannot be changed after boot.

Block Devices

PUT /drives/{drive_id} creates or replaces a pre-boot block configuration. A virtio-block configuration names a backing file with path_on_host and sets is_read_only; a vhost-user block configuration names a Unix socket instead. Firecracker opens a file-backed device while processing the PUT, not at some later guest I/O.

is_root_device: true marks at most one block device as the root. Firecracker moves that device to the front of its block-device list, exposes it as /dev/vda, and derives the kernel root=, ro, or rw arguments. The caller does not need to duplicate those arguments in boot_args. A root virtio-pmem device is an alternative, so virtio-block is not Firecracker's only storage interface.

For file-backed virtio-block, io_engine is "Sync" by default or "Async" for the io_uring engine. Current Firecracker requires a host kernel newer than 5.10.51 for the async choice. cache_type: "Unsafe" does not advertise flush support. "Writeback" advertises VIRTIO_BLK_F_FLUSH and services guest flush requests with fsync; Firecracker does not advertise VIRTIO_BLK_F_CONFIG_WCE.

Read-only devices advertise VIRTIO_BLK_F_RO, allowing the guest driver to learn the restriction during feature negotiation. The host backing file is opened read-only as the enforcement mechanism.

After boot, PATCH /drives/{drive_id} can update the backing path and rate limiter. It can also notify the device after the backing file's size changed, causing a virtio configuration update. This operation is not atomic with host file changes and is documented for a cooperative guest that is not accessing the device during the update. It does not change is_root_device or is_read_only. Preview PCI support separately has runtime APIs for adding and removing eligible devices; that is a different operation from block PATCH.

Sharing a read-only base image and supplying per-VM writable state is a container integration concern, not a property of the block API. The firecracker-containerd design and the containerd book cover snapshotters and guest-side overlay assembly. Firecracker itself exposes the backing devices; it does not construct an overlay mount or run pivot_root in the guest.

Network Interfaces

PUT /network-interfaces/{iface_id} opens a pre-existing TAP interface named by host_dev_name. Firecracker neither creates the TAP nor attaches it to a bridge. That host networking work requires a bare-metal Linux host, or a VM with the required networking privileges, and must be completed before a jailed Firecracker process loses access to the relevant host paths and namespace.

guest_mac is optional. When it is present, Firecracker advertises VIRTIO_NET_F_MAC with that address. When it is absent, Firecracker does not invent a deterministic address; the guest driver chooses one. Configure the field explicitly when DHCP, policy, or identity depends on a stable MAC.

The optional mtu, from 68 through 65535, is advertised through VIRTIO_NET_F_MTU. If it is absent, that feature is not advertised and the guest uses its own default. An operator must account for every link in the host path, including bridges, tunnels, and overlays, rather than merely copy a physical NIC's MTU.

The RX and TX paths each accept bandwidth and operation token buckets. With refill_time measured in milliseconds, a bucket of size tokens has a steady rate of 1000 * size / refill_time tokens per second. Chapter 14 covers burst, overconsumption, and timer behavior. Runtime PATCH /network-interfaces/{iface_id} can replace the rate limiters, but it cannot change the TAP name, MAC, or MTU negotiated at boot.

Logs And Metrics

PUT /logger applies a pre-boot logger configuration. log_path can select a file or FIFO opened nonblocking; level, show_level, show_log_origin, and module control filtering and formatting. The current implementation accepts another logger update before boot, while the runtime controller rejects logger configuration. CLI logger flags configure the same global logger before API requests arrive.

PUT /metrics initializes a single metrics destination. metrics_path is required. emit_id can add the instance ID to each record, and properties can add operator-defined string fields. Initialization is one-shot. Firecracker flushes a JSON record every 60 seconds and on FlushMetrics.

Both destinations must already exist and be writable in Firecracker's filesystem view. Preparing files or FIFOs inside a jail is deployment work; do it on an isolated Linux host with the intended ownership and chroot layout. The writers do not block the VMM indefinitely: a failed nonblocking write is counted in logger or metrics counters. Metrics configuration is process telemetry, not guest state, and is not restored from a VM snapshot.

Instance Start

InstanceStart is a one-way lifecycle transition. Current Firecracker performs the important KVM work in this order:

sequenceDiagram
    participant A as API caller
    participant P as Pre-boot controller
    participant B as VM builder
    participant K as KVM

    A->>P: PUT /actions InstanceStart
    P->>B: build VmResources
    B->>K: KVM_CREATE_VM
    B->>K: KVM_CREATE_VCPU x N
    B->>K: KVM_SET_USER_MEMORY_REGION
    Note over B: Load kernel and initrd, then attach devices
    Note over B: Configure vCPU registers and KVM state
    Note over B: Start vCPU threads paused
    B->>K: Resume vCPUs into KVM_RUN
    B-->>A: 204 No Content

The builder allocates guest memory, creates the KVM VM, creates all vCPUs, and then registers base DRAM. It loads the kernel and optional initrd, attaches the configured devices through MMIO or optional PCI transport, configures the boot state, and moves each vCPU into a host thread in Paused state. The final resume lets those threads enter KVM_RUN. The boot-time measurement begins when the API receives InstanceStart, not at the later first guest instruction.

After boot, PATCH /vm can pause or resume the vCPUs. Pausing freezes guest CPU execution while the VMM event loop continues to process API and device events. It is not a complete process suspension and does not rewind the one-way InstanceStart transition.

Configuration Files

--config-file parses one JSON object into the same VmResources used by the pre-boot API and immediately calls the build-and-boot path. It is not a script that replays HTTP requests. --no-api requires a config file; without --no-api, Firecracker creates the API socket first, boots from the file, and keeps the socket for runtime operations.

Top-level keys use kebab case, such as boot-source, machine-config, network-interfaces, cpu-config, mmds-config, and memory-hotplug. Fields inside those objects retain their API names, such as kernel_image_path and vcpu_count.

The current VmmConfig requires a boot-source object and a drives array; the array may be empty. A root block drive is therefore not universally required: an initrd can provide userspace, and virtio-pmem can be the root device. Optional top-level resources include machine-config, cpu-config, logger, metrics, balloon, network-interfaces, vsock, entropy, pmem, mmds-config, and memory-hotplug. Transport selection, serial configuration, and some process controls remain CLI concerns rather than keys in this JSON structure.