MicroVMs From The Bottom Up

Chapter 21: Host Networking For MicroVMs

A Firecracker guest owns a kernel, not a Linux network namespace in the host kernel. The common container bridge path cannot place one end of a veth pair inside that guest. Instead, virtio-net carries Ethernet frames across guest memory, Firecracker moves those frames through a TAP file descriptor, and the host kernel routes or bridges them from there.

Firecracker implements the guest-facing device and the TAP attachment. It does not allocate addresses, run DHCP, enable host forwarding, create a bridge, install NAT, or filter tenant traffic. Those are orchestrator and host-network responsibilities.

The TAP Attachment

Linux TUN/TAP connects a kernel network interface to a userspace file descriptor. TUN transfers layer-3 packets; TAP transfers layer-2 Ethernet frames. Firecracker uses TAP because virtio-net presents an Ethernet device to the guest.

For every configured network interface, Tap::open_named opens /dev/net/tun with O_RDWR | O_NONBLOCK | O_CLOEXEC and issues TUNSETIFF with these flags:

Flag Purpose
IFF_TAP Transfer Ethernet frames
IFF_NO_PI Omit the separate four-byte TUN/TAP protocol-information prefix
IFF_VNET_HDR Exchange virtio network offload metadata with each frame

host_dev_name is therefore an interface name such as tap0, despite the OpenAPI description calling it a path. It is resolved in the network namespace where the Firecracker process runs. With jailer --netns, the Jailer joins an existing namespace before Firecracker opens the TAP.

Creating a TAP or attaching to one the caller does not own requires CAP_NET_ADMIN. A persistent TAP can instead be created and assigned to the uid that will run Firecracker. The Jailer creates and owns the /dev/net/tun device node inside the jailed filesystem, but it does not create or configure the network interface in the selected network namespace.

Firecracker does not call TUNSETPERSIST. If TUNSETIFF creates a new nonpersistent TAP, closing the last fd removes it. The common Firecracker setup creates a persistent TAP before launch; that interface remains after the VMM closes its fd and must be managed by the host. TAP lifetime therefore depends on how the operator created it, not merely on the lifetime of the Tap Rust value.

The File Descriptor Contract

The single TAP fd is bidirectional and nonblocking:

Firecracker currently exposes one RX and one TX virtqueue, each with maximum size 256. It does not advertise VIRTIO_NET_F_MQ, and it opens one TAP queue fd per virtio-net device. The guest supplies RX buffers for device writes and TX buffers for device reads.

The TAP uses IFF_VNET_HDR. Firecracker sets the header size to sizeof(virtio_net_hdr_v1), currently 12 bytes, with TUNSETVNETHDRSZ. At activation it derives TUNSETOFFLOAD flags from negotiated virtio features for checksum and segmentation offload. This preserves the offload contract between the guest driver, the VMM, and the host TAP rather than forcing Firecracker to segment every packet itself.

Firecracker can advertise two optional configuration values to the guest:

The iface_id is only Firecracker's management identifier. It does not choose the interface name inside the guest, and API creation order does not guarantee guest enumeration order.

Packet Flow

The net device registers queue eventfds, the TAP fd, and one timerfd for each direction's rate limiter with the VMM event manager.

For guest TX:

  1. The guest adds a descriptor chain to the TX virtqueue and notifies the transport.
  2. KVM signals the queue eventfd registered by the MMIO or PCI transport.
  3. Firecracker validates the chain and frame length, then consumes one ops token and the frame length in byte tokens.
  4. An MMDS-eligible frame may be copied into the in-process MMDS stack. Otherwise writev sends the descriptor data through the TAP fd.
  5. Firecracker adds the chain to the used ring and signals the guest when notification rules require it.

For host-to-guest RX:

  1. The host kernel queues a frame on the TAP and makes its fd readable.
  2. Firecracker gathers guest RX descriptors and calls readv directly into those guest-memory buffers.
  3. It accounts the frame against the RX limiter.
  4. If tokens are available, it advances the used ring and may interrupt the guest. If not, it retains the completed frame as deferred and exposes it only after the limiter timer permits progress.
sequenceDiagram participant G as Guest virtio-net participant V as Firecracker net device participant T as TAP fd participant H as Host network stack G->>V: TX descriptors and queue notification V->>V: Validate and charge TX tokens V->>T: writev frame T->>H: Inject Ethernet frame H->>T: Queue Ethernet frame T->>V: Readable fd V->>G: readv into RX buffers V->>V: Charge RX tokens, defer completion if blocked V-->>G: Publish used entries and interrupt

This ordering matters when describing enforcement. TX limiting occurs before a normal frame enters the TAP. RX limiting occurs after the host kernel has already selected and queued the frame, and after Firecracker has read it into guest-provided memory; it delays guest-visible completion. The limiter does not remove the TAP fd from epoll. While blocked, handlers stop useful work, and the timer event resumes the deferred direction.

Host Topologies

The TAP is only an attachment point. Its host-side topology is independent of Firecracker.

Routed Or NATed

The Firecracker quick-start guide assigns a small subnet to each TAP, gives one address to the host side and another to the guest, enables host forwarding, and masquerades outbound traffic. A routed production network may advertise guest prefixes instead of applying NAT. Per-guest subnets avoid a shared layer-2 broadcast domain but require route, address, return-path, and firewall management.

Bridged

A Linux bridge can attach several TAP devices, or a TAP and a physical or virtual uplink, to one layer-2 domain. The bridge learns source MAC addresses and forwards frames without Firecracker's involvement. This makes broadcast, neighbor discovery, and guest-to-guest layer-2 traffic part of the deployment's security and scaling model. External connectivity may be bridged directly, routed, or NATed; a bridge does not imply one of those choices.

Namespaced

TAP devices may live in per-instance host network namespaces. The Jailer can join such a namespace with --netns. Namespaces allow identical TAP names and address plans across snapshot clones, but the platform still needs a host-side connection, often a veth pair, route, or bridge, between that namespace and the rest of the network. This is where host namespaces re-enter the design: they isolate the VMM's host network context, not the guest kernel itself.

All three setups touch host interfaces, routes, forwarding, and firewall state. They should be built in an isolated bare-metal Linux host or a nested- virtualization lab and integrated with the platform's existing network policy. The upstream network guide contains a disposable quick-start example; it is not a production policy template.

Guest Addressing

Firecracker does not assign an IP address inside the guest. A deployment can provide DHCP on the attached network, run an in-guest agent, bake static configuration into the image, or use the Linux kernel's ip= boot parameter. The last option configures an interface before init and is useful for root filesystems that omit a DHCP client and iproute2.

The Firecracker getting-started rootfs has its own convention in which the last four MAC octets encode the example IPv4 address. That is rootfs bootstrap behavior, not a virtio-net or Firecracker requirement.

MMDS can carry address, route, DNS, or other bootstrap data if guest software has been written to consume it. Firecracker does not automatically translate MMDS data into guest network configuration, and a guest needs enough initial network setup to reach the MMDS link-local address.

On snapshot restore, Firecracker normally reopens the saved TAP name. The network_overrides field can map a saved iface_id to a different host_dev_name, which is essential when clones use distinct TAPs. Guest IP and MAC state may still need orchestration so clones do not duplicate network identity.

Rate Limiting

Each interface has independent RX and TX RateLimiter objects. Each direction may contain two token buckets:

Bucket Token Meaning
bandwidth Bytes transferred by one frame
ops Frames or packet operations

A bucket's size is both its capacity and initial budget. refill_time is the number of milliseconds required to refill an empty bucket to size, so the sustained rate is size / refill_time, not necessarily size per second. one_time_burst is extra initial credit that does not replenish. A zero size or zero refill time disables that bucket.

For each frame, Firecracker consumes the ops token first and then byte tokens. If the byte check fails, it returns the ops token. Buckets replenish passively from elapsed monotonic time when used. When a check fails, a shared timerfd for that direction is armed for 100 ms so processing can retry. The constant is a blocked-retry interval, not the bucket's refill period.

A frame larger than the bucket capacity is allowed once as overconsumption and blocks subsequent work for a duration derived from the excess and the configured refill time. This prevents a bucket smaller than the maximum frame from making that frame permanently unsendable.

PUT /network-interfaces/{iface_id} sets pre-boot device configuration. PATCH /network-interfaces/{iface_id} can update or disable only the RX and TX rate-limit buckets after boot; it cannot change the TAP name, MAC, or MTU. Updated buckets start full. Rate limiting meters transfer at the virtio/TAP edge; it is not a substitute for cgroup CPU or memory limits, host qdiscs, firewall policy, or uplink capacity management.

MMDS is a special case. The TX path initially charges a candidate frame, then replenishes those tokens when MMDS consumes it. MMDS frames are therefore not accounted by the interface rate limiter.

Relationship To CNI

Common CNI bridge and point-to-point plugins create a veth pair because both ends are host-kernel network devices placed in different namespaces. Optional bandwidth plugins can install kernel qdiscs after interface creation. The plugin process exits; the host kernel retains the veth, routes, bridge membership, and traffic-control state.

Firecracker's TAP fd solves a different crossing: guest kernel to VMM to host kernel. An orchestrator may still use CNI machinery to configure the host side of that TAP or a surrounding namespace. CNI does not replace the virtio queues, the Firecracker event loop, or the TAP attachment.

The containerd book owns the detailed veth, namespace, bridge, and CNI lifecycle. For this book, the stable distinction is enough: veth joins two contexts inside one kernel; TAP exposes a host-kernel network device to the userspace VMM that serves another kernel.

MMDS And Host Filtering

On an MMDS-enabled net device, Firecracker examines candidate guest TX frames before sending them to TAP. Eligible ARP and IPv4 traffic may be consumed by the in-process MMDS stack, and responses are prioritized on the guest RX path. A host firewall cannot authorize access to those in-process responses because they do not traverse the TAP.

The inverse is equally important. MMDS is not a host-egress firewall. Frames on unselected interfaces go to TAP, and the net-device source documents a guest-descriptor TOCTOU that can change a destination after the MMDS decision. Operators must protect host services, including a host metadata address, with host network policy regardless of whether Firecracker MMDS is enabled. Chapter 17 covers the MMDS protocol and both forwarding cases in detail.

Sources And Further Reading