MicroVMs From The Bottom Up

Chapter 14: Firecracker's Device Model

General-purpose VMMs expose broad device catalogs because their guests may need desktop, storage, networking, migration, and compatibility hardware. A guest can reach only the devices instantiated for its machine, but every emulator a deployment enables adds a parser and state machine to the trusted computing base. Firecracker starts from the narrower requirements of server and container workloads.

The result is a small catalog, not a frozen one. Current Firecracker has seven virtio device classes, two x86 legacy devices, an AArch64 RTC, and two always-present platform notification devices described through ACPI on x86-64 and the device tree on AArch64. Recent additions such as virtio-pmem, virtio-mem, optional PCI transport, and developer-preview hotplug show that each addition is evaluated rather than prohibited.

The Device Inventory

VirtioDeviceType in the current source names the seven paravirtual device classes. MMIO remains the default transport; --enable-pci puts all virtio devices behind virtio-pci instead. Firecracker still omits USB, GPU, audio, NVMe, SCSI, virtio-console, virtio-GPU, virtio-fs, virtio-input, and virtio-crypto.

Device Virtio ID Transport Notes
virtio-net 1 MMIO or PCI RX and TX queues; MMDS interception in the TX path
virtio-block 2 MMIO or PCI One queue; file-backed or developer-preview vhost-user backend
virtio-rng 4 MMIO or PCI One queue; randomness from aws-lc-rs
virtio-balloon 5 MMIO or PCI Inflate and deflate queues plus configured optional queues
virtio-vsock 19 MMIO or PCI Three queues; AF_UNIX backend on the host
virtio-mem 24 MMIO or PCI Runtime adjustment of requested hotplug memory
virtio-pmem 27 MMIO or PCI File-backed persistent-memory region and flush queue

The non-virtio set is also small. On x86-64, a 16550A-compatible serial device occupies COM1 at port 0x3f8 and GSI 4, and Firecracker's in-tree i8042 occupies ports 0x60 through 0x64 and GSI 1. The i8042 supplies the reset-command path that terminates the VMM process and the keyboard path used by SendCtrlAltDel; it is not a general PS/2 controller. On AArch64, the serial device and PL031 RTC are MMIO devices. VMGenID changes generation identity after restore, while VMClock reports clock discontinuities; both are always present. An optional boot timer is a measurement device rather than part of the guest workload interface.

The device set is therefore wider than the six-item list still quoted in FAQ.md, but it remains explicit and server-oriented. On x86-64, Firecracker uses ACPI for device discovery and platform notifications, not as a complete PC power-management model; for example, the guest has no ACPI poweroff device.

The Virtio Transports

Connecting a guest driver to a virtio device requires a transport for feature negotiation, queue addresses, notifications, and interrupts. Firecracker implements the modern MMIO transport and an optional PCI transport. The --enable-pci process flag selects PCI for all virtio devices; without it, all use MMIO.

The MMIO path assigns a fixed register window and legacy GSI to each device. On x86-64, Firecracker describes those resources both in DSDT AML and with the compatibility virtio_mmio.device= kernel parameter; AArch64 uses the flattened device tree. The PCI path adds an ECAM configuration region, BARs, common and notification capabilities, and MSI-X. It is also the prerequisite for developer-preview block, network, and pmem hotplug; the guest must rescan the PCI bus because Firecracker does not yet send a hotplug notification.

Each virtio-MMIO device occupies MMIO_LEN = 0x1000 bytes. On x86-64, the first slot is 0xc000_1000, immediately after the optional boot-timer slot at 0xc000_0000; later devices receive the next available 4 KiB range. MMIO devices are fixed at construction time. Runtime insertion and removal use PCI instead.

The Register Map

Every virtio-MMIO device exposes the same register layout at the base of its 4 KiB window, defined by virtio 1.2 section 4.2.2 and implemented in src/vmm/src/devices/virtio/transport/mmio.rs:

Offset Size R/W Field
0x000 4 RO MagicValue (0x7472_6976, ASCII "virt")
0x004 4 RO Version (0x2; not legacy 0x1)
0x008 4 RO DeviceID (virtio type integer)
0x00c 4 RO VendorID (0x0)
0x010 4 RO DeviceFeatures (page selected by 0x014)
0x014 4 WO DeviceFeaturesSel
0x020 4 WO DriverFeatures
0x024 4 WO DriverFeaturesSel
0x030 4 WO QueueSel
0x034 4 RO QueueNumMax (queue's max_size)
0x038 4 WO QueueNum
0x044 4 RW QueueReady
0x050 4 WO QueueNotify (NOTIFY_REG_OFFSET)
0x060 4 RO InterruptStatus (bit 0 = vring, bit 1 = config change)
0x064 4 WO InterruptACK
0x070 4 RW DeviceStatus
0x080 4 WO QueueDescLow
0x084 4 WO QueueDescHigh
0x090 4 WO QueueAvailLow
0x094 4 WO QueueAvailHigh
0x0a0 4 WO QueueUsedLow
0x0a4 4 WO QueueUsedHigh
0x0fc 4 RO ConfigGeneration
0x100-0xfff var RW Device-specific configuration space

The MagicValue at offset 0x000 reads 0x7472_6976 -- the ASCII bytes for "virt" stored little-endian. A guest driver that sees any other value knows immediately that the MMIO window does not contain a virtio device. The Version field reads 0x2, explicitly distinguishing this from the legacy virtio 0.9 transport (which reads 0x1); Firecracker does not support the legacy transport.

Initialization: The Status State Machine

Device initialization follows the virtio 1.2 status sequence, enforced for MMIO by MmioTransport::set_device_status(). The driver resets the device, adds ACKNOWLEDGE, adds DRIVER, negotiates features, adds FEATURES_OK, verifies that bit, configures queues, and finally adds DRIVER_OK. Firecracker accepts only those cumulative transitions and activates the backend on the last one.

Two additional bits signal abnormal states: DEVICE_NEEDS_RESET (bit 6) is set after an activation failure, and FAILED (bit 7) is accepted whenever the driver gives up. Clearing individual bits is invalid; reset is a write of zero. An invalid transition is logged and ignored.

flowchart LR A["Reset\n(write 0)"] --> B["ACKNOWLEDGE\n(bit 0)"] B --> C["DRIVER\n(bit 1)"] C --> D["Feature negotiation"] D --> E["FEATURES_OK\n(bit 3)"] E --> F["Re-read FEATURES_OK\n(confirm accepted)"] F --> G["Virtqueue setup"] G --> H["DRIVER_OK\n(bit 2)"]

Queue Notification and ioeventfd

The most frequent transport operation is the queue notification: the guest driver tells the backend that it placed buffers in the available ring. Without an acceleration path, KVM would return KVM_EXIT_MMIO to the vCPU thread for every notify write.

Firecracker avoids the userspace MMIO-exit path with KVM ioeventfd. For every queue, it registers that queue's eventfd at device_base + 0x050 with the queue index as the datamatch value. A matching guest write is consumed in KVM and signals the eventfd instead of returning KVM_EXIT_MMIO to the vCPU thread. The VMM thread wakes through epoll and processes the queue.

sequenceDiagram participant G as "Guest driver" participant K as "KVM (kernel)" participant E as "eventfd" participant V as "VMM thread" G->>K: write 0x050 (QueueNotify) Note over K: ioeventfd match -- no userspace MMIO exit K->>E: signal eventfd E->>V: epoll wakeup V->>V: process virtqueue V->>K: signal irqfd K->>G: inject virtio interrupt

The IRQ Limit

The MMIO transport still allocates one GSI per device. On x86-64, the legacy allocator spans GSI 5 through 23, but VMGenID and VMClock draw from that range too, so fewer than 19 virtio-MMIO devices fit in a complete machine. PCI devices use MSI-X routes from the much larger MSI GSI range and are constrained by PCI resources instead. These are transport limits, not the number of device classes Firecracker implements.

The Individual Devices

virtio-net

The network device has two queues: RX at index 0 and TX at index 1. There is no control virtqueue or multiqueue mode. Both queues have a maximum size of 256, and Firecracker bounds its frame buffer at 65,562 bytes.

The feature bits Firecracker always advertises include checksum offload on both sides (VIRTIO_NET_F_CSUM bit 0, VIRTIO_NET_F_GUEST_CSUM bit 1), TSO4 and TSO6 in both directions (bits 7, 8, 11, 12), UFO in both directions (bits 10, 14), receive buffer merging (VIRTIO_NET_F_MRG_RXBUF bit 15), VIRTIO_F_VERSION_1 (bit 32), and VIRTIO_RING_F_EVENT_IDX (bit 29). MAC address support (VIRTIO_NET_F_MAC bit 5) and MTU advertisement (VIRTIO_NET_F_MTU bit 3) are offered conditionally when the respective values are configured at VM creation time. Notably, VIRTIO_NET_F_STATUS (bit 16) and multiqueue support VIRTIO_NET_F_MQ (bit 22) are intentionally not advertised -- there is no live link status signalling and only one pair of queues.

The device configuration has a six-byte MAC field and a two-byte MTU field, with zeroed status and maximum-queue-pair positions retained to match virtio_net_config. RX and TX have separate rate limiters, so ingress and egress byte and operation budgets can be controlled independently.

virtio-block

The in-process, file-backed block device uses a single queue with a maximum of 256 descriptors and addresses storage in 512-byte sectors. It advertises VIRTIO_F_VERSION_1 and VIRTIO_RING_F_EVENT_IDX, adds VIRTIO_BLK_F_FLUSH for writeback caching, and adds VIRTIO_BLK_F_RO for a read-only drive. It does not advertise geometry, topology, size-max, or segment-max features. Its configuration space is one u64 capacity measured in 512-byte sectors.

The in-process backend implements VIRTIO_BLK_T_IN, VIRTIO_BLK_T_OUT, VIRTIO_BLK_T_FLUSH, and VIRTIO_BLK_T_GET_ID. Discard and write-zeroes requests are returned as unsupported.

The file-backed device supports synchronous calls or an asynchronous io_uring engine with 128 entries. Its 20-byte device identifier is derived from the backing file's st_dev, st_rdev, and st_ino. A separate developer-preview vhost-user backend connects to a Unix socket and delegates block processing to an external process; it does not accept Firecracker's file path, I/O engine, read-only, or rate-limiter fields, and current source rejects snapshotting that backend.

virtio-vsock

The vsock device provides a stream socket channel between the guest and a process on the host without requiring a network interface or IP routing. It has three queues: RX (index 0), TX (index 1), and Event (index 2), each sized 256 descriptors. Feature bits: VIRTIO_F_VERSION_1 (bit 32), VIRTIO_F_IN_ORDER (bit 35), and VIRTIO_RING_F_EVENT_IDX (bit 29). VIRTIO_VSOCK_F_SEQPACKET is not advertised.

The host CID is hardcoded as VSOCK_HOST_CID = 2, as the virtio-vsock specification reserves. The guest CID is user-configured at VM creation time.

The backend is an AF_UNIX socket on the host, not the vhost-vsock kernel module. This is deliberate. vhost would move the data path into the kernel, eliminating some context switches, but the kernel would then become directly reachable from the guest -- which is precisely the attack surface that the combination of KVM and seccomp filtering is designed to interpose. From Issue #650: "we don't want to use vhost since that would be another attack surface to directly expose the host kernel." The AF_UNIX backend stays in the firecracker userspace process, subject to the same seccomp filter as every other VMM operation.

Connection routing works as follows: connections that the host initiates arrive at the single UDS path configured at boot time; Firecracker forwards them into the guest on the appropriate port. Connections that the guest initiates cause Firecracker to open a UDS path derived from the configured path plus the target port number -- for example, if the UDS path is ./v.sock, a guest connection to port 52 causes Firecracker to connect to ./v.sock_52. The maximum per-packet buffer is MAX_PKT_BUF_SIZE = 64 * 1024 (64 KiB).

virtio-rng

The entropy device has one 256-entry queue and no device-specific configuration space. Its only advertised feature is VIRTIO_F_VERSION_1.

The entropy source is aws-lc-rs, AWS's Rust bindings to AWS-LC. Each request calls aws_lc_rs::rand::fill(). Firecracker caps one request at 64 KiB so overlapping or oversized descriptor chains cannot force an unbounded host allocation. The device also accepts byte and operation rate limits. With CONFIG_HW_RANDOM_VIRTIO, the guest exposes the source through /dev/hwrng and feeds it into the kernel's randomness subsystem.

virtio-balloon

The balloon always has inflate and deflate queues. Configuration may add a statistics queue, developer-preview free-page-hinting queue, and free-page-reporting queue, for a maximum of five. Each queue has a maximum size of 256. Its 12-byte configuration space reports the requested and actual page counts and carries Firecracker's free-page-hint command ID. Chapter 11 describes the page-ownership contract and why balloon input remains untrusted.

virtio-pmem and virtio-mem

The pmem device maps a file into a separate KVM memory slot at a 2 MiB-aligned guest physical range. Its configuration exposes the range's start and size. The one queue carries flush requests; Firecracker coalesces the backing-file msync work and can rate-limit those operations. A pmem device may be read-only or serve as the root device.

Virtio-mem reserves a hotpluggable region and uses one queue for plug, unplug, and state requests. Its configuration exposes the base address, region size, usable size, plugged size, requested size, and block size. Firecracker's API changes requested_size_mib at runtime; the guest driver then performs the corresponding block operations. The default guest block is 2 MiB, while the default host KVM-slot granularity is 128 MiB.

MMDS: Metadata Without a Second Network Interface

Cloud guests commonly need instance metadata. Firecracker's MMDS is not another virtio device: it adds no queue, IRQ, or transport window. Instead, the network backend diverts selected guest frames to an in-process metadata stack rather than writing them to the TAP fd. Chapter 17 owns the token model, Dumbo internals, and trust boundaries.

flowchart LR G["Guest\n(virtio-net driver)"] -->|"TX frame"| N["virtio-net\ndevice (VMM)"] N -->|"dst = MMDS addr?"| D{"Dumbo\ninspection"} D -->|"yes"| M["MmdsNetworkStack\n(HTTP/TCP/IPv4 in VMM)"] D -->|"no"| T["TAP fd\n(host network)"] M -->|"response frame"| N N -->|"RX frame"| G

MMDS adds guest-reachable protocol code to the existing network backend, but no new emulated hardware or process. It reuses the queue, interrupt, and frame-processing path already present for virtio-net.

Rate Limiters

Shared storage and network links need admission control beyond CPU and memory isolation. Firecracker's RateLimiter combines independent byte and operation token buckets. File-backed block, pmem, and entropy devices use one limiter each; network devices use separate RX and TX limiters. Serial output instead uses one byte TokenBucket and drops output that exceeds it.

The Token Bucket

The implementation lives in src/vmm/src/rate_limiter/mod.rs as TokenBucket:

pub struct TokenBucket {
    size: u64,                    // max capacity (tokens)
    initial_one_time_burst: u64,  // original burst budget, never replenished
    refill_time: u64,             // ms to refill from 0 to size
    one_time_burst: u64,          // remaining burst credit
    budget: u64,                  // current token budget
    last_update: Instant,
    processed_capacity: u64,      // size / gcd(size, refill_time_ns)
    processed_refill_time: u64,   // refill_time_ns / gcd(size, refill_time_ns)
}

TokenBucket::new() returns None when size or complete_refill_time_ms is zero, or when converting the refill interval to nanoseconds would overflow. An enabled bucket starts full (budget == size). Refill is computed on demand rather than on a tick:

refill_amount = (time_delta_ns * processed_capacity) / processed_refill_time

The fields processed_capacity and processed_refill_time are the GCD-reduced forms of size and refill_time_ns. This controls intermediate growth in the refill arithmetic. Firecracker advances last_update only by the time consumed to generate whole tokens, carrying the unused fractional-token time into the next call and minimizing lost credit.

The reduce() Call and Its Three Outcomes

When a device wants to process a request, it calls TokenBucket::reduce(n) where n is the token cost of the operation. The call returns one of three variants:

BucketReduction::Success -- the budget covered the request; tokens are deducted and processing proceeds immediately.

BucketReduction::Failure -- even after a passive replenish (update budget based on elapsed time since last_update), there are not enough tokens. The request is deferred.

BucketReduction::OverConsumption(f64) -- after consuming any one-time burst, the request remainder exceeds the bucket's total size. A strict "must have tokens" rule would block such a request forever. Firecracker allows it, subtracts the available regular budget, sets that budget to zero, and returns (request_remainder - available_budget) / bucket_size. The caller converts that ratio into a back-pressure delay.

The one_time_burst field is consumed first. If the burst allowance covers the entire request, the regular budget is not touched at all. This means a freshly-started microVM can absorb a startup I/O spike beyond the steady-state rate limit before the regular bucket begins draining.

The RateLimiter and Its Timer

pub struct RateLimiter {
    bandwidth: Option<TokenBucket>,   // byte tokens
    ops: Option<TokenBucket>,         // operation tokens
    timer_fd: TimerFd,
    timer_active: bool,
}

A single RateLimiter pairs two independent TokenBucket instances -- one for bandwidth (bytes), one for operation count -- with a TimerFd for deferred resumption. Both buckets must agree that a request can proceed; if either returns Failure, the timer is armed for REFILL_TIMER_DURATION = Duration::from_millis(100) (a compile-time constant, not configurable via API). On OverConsumption(ratio), the timer is armed for ratio * refill_time milliseconds instead. While timer_active == true, consume() returns false immediately, gating both buckets with a single timer regardless of which one triggered the throttle.

The TimerFd is created with the RateLimiter even when both buckets are disabled. update_buckets() can then enable limiting without needing a later timerfd_create(2), which may be unavailable under the thread's seccomp filter.

RateLimiter implements AsRawFd. The VMM's epoll loop watches the timer fd and calls event_handler() when it fires, which clears timer_active and allows the device's queue processing to resume.

API Schema and Runtime Updates

The TokenBucket object in Firecracker's OpenAPI spec (src/firecracker/swagger/firecracker.yaml) exposes three fields: size (int64, minimum 0 -- total token capacity), refill_time (int64, in milliseconds -- time to refill from empty to full), and one_time_burst (int64, optional). The effective steady-state rate in tokens per second is size / (refill_time / 1000).

A RateLimiter object wraps two optional TokenBucket instances under the keys bandwidth and ops. Current API configurations accept it on:

PUT /serial takes a single TokenBucket, not a two-bucket RateLimiter. It limits output bytes only and records dropped bytes rather than arming a timer and retrying them. The developer-preview vhost-user block backend delegates I/O and does not accept Firecracker's rate-limiter field.

To disable an existing rate limiter while the VM is running, send size: 0, refill_time: 0; TokenBucket::new() returns None and update_buckets() sets the field to None, removing the throttle entirely until a new non-zero configuration is applied.

The in-process virtio backends persist their limiter buckets with size, remaining one_time_burst, refill_time, current budget, and elapsed_ns since the last update. Restore reconstructs that budget and time base, but creates a fresh, inactive timerfd; an armed timer is not itself serialized. Serial's output bucket is reconstructed from restore-time VM configuration rather than this RateLimiterState.

Why the List Is Short

The threat model in docs/design.md treats vCPU threads as malicious as soon as they start:

"All vCPU threads are considered to be running malicious code as soon as they have been started; these malicious threads need to be contained."

Every configured emulator is guest-reachable through virtqueues, transport registers, or legacy I/O. Omitting a device removes its parsers, state transitions, snapshot state, and host interfaces from that path. Implemented devices still need strict descriptor validation and rate limiting; minimalism reduces the amount of code to defend but does not make the remaining code safe automatically.

The vsock backend illustrates one such boundary: Firecracker translates guest AF_VSOCK traffic to host Unix sockets instead of using the host's vhost-vsock data path. The vhost-user block option makes a different tradeoff by delegating a backend to another userspace process. Optional PCI similarly adds a transport when its performance and hotplug capabilities justify the extra machinery.

The current source therefore presents a governed catalog rather than a closed one. Pmem, memory hotplug, VMClock, PCI, and vhost-user block all arrived after Firecracker's original six-device description. What persists is the review criterion: each host interface and guest-reachable state machine must earn its place in the server-oriented machine model.

Sources and Further Reading