Chapter 22: MicroVMs As Containers

A container API does not require a container-shaped isolation boundary. It describes images, processes, filesystems, networks, and lifecycle operations. The runtime beneath that API may implement the sandbox with Linux namespaces, a virtual machine, or both.

That freedom produces three different meanings of microVM as container. firecracker-containerd places one or more OCI containers inside a Firecracker VM and exposes them through containerd's task API. Kata Containers maps a Kubernetes pod sandbox to a VM and each pod container to a process inside it. Flintlock does not run containers behind the task API at all: it provisions a microVM as a Kubernetes node and uses containerd to store the VM's OCI-packaged artifacts.

The distinction is the unit being isolated:

System	API object	VM mapping	Work inside the VM
firecracker-containerd	containerd task	one VM per `vm_id`	one or more OCI containers
Kata Containers	CRI pod sandbox	one VM per pod	the pod's container processes
Flintlock with CAPMVM	Cluster API machine	one VM per Kubernetes node	kubelet and a container runtime

The first two replace the runtime below a container interface. The third is an infrastructure provider. Treating all three as containerd shims hides the most important architectural choice.

The Runtime Boundary

Containerd separates task management from task implementation with the runtime v2 shim protocol. Before starting a task, containerd prepares its bundle and rootfs information, then invokes the configured runtime. The shim implements the containerd.task.v2.Task ttrpc service: calls such as Create, Start, Exec, Kill, Wait, and Delete cross this boundary, while the mechanism that creates the process remains below it.

A runtime name normally selects the executable. Containerd replaces dots with hyphens, keeps the final two name components, and prefixes containerd-shim-. Thus io.containerd.runc.v2 selects containerd-shim-runc-v2, while aws.firecracker selects containerd-shim-aws-firecracker. A runtime may also be configured with an explicit executable path.

The protocol does not require one shim per task. A shim may group several tasks behind one address, which is the opening both firecracker-containerd and Kata use: one host-side runtime owns a VM, then forwards container operations to an agent across the guest boundary.

flowchart LR subgraph host[Host] client[CRI or containerd client] ctr[containerd] shim[VM-aware shim] vmm[VMM] client --> ctr ctr -->|Task ttrpc| shim shim --> vmm end subgraph guest[Guest] agent[guest agent] workload[container processes] agent --> workload end shim -->|vsock control| agent

Containerd still owns images, snapshots, metadata, and the task API. The shim must translate their host-side representation into devices and messages a guest can consume. That translation, not merely starting a VMM, is the hard part.

firecracker-containerd

firecracker-containerd is a specialized containerd distribution with four parts: a control plugin compiled into the daemon, the containerd-shim-aws-firecracker host runtime, an agent in the guest, and a guest-rootfs builder. The control plugin adds VM lifecycle operations that the standard task service does not express. The runtime uses those operations for the VM and forwards container lifecycle requests to the guest agent.

One Shim Per VM

The OCI annotation aws.firecracker.vm.id assigns a container to a VM. The runtime's start path returns the address of an existing shim when another task uses the same ID. The result is one host shim and one Firecracker process per VM, with one guest agent serving all containers in that VM. If the annotation is absent, the runtime generates a UUID and treats the VM as a single-task VM.

This grouping decision is also a security decision. Containers sharing a vm_id share a guest kernel. The hardware boundary separates the group from the host, not one member of the group from another.

The host shim and guest agent speak ttrpc over Firecracker's virtio-vsock device. The control channel uses guest port 10789. Stdin, stdout, and stderr use ports allocated from 11000, three per task. Inside the VM, the agent invokes containerd-shim-runc-v1; the final workload is still an OCI container, but its namespaces, cgroups, and seccomp policy belong to the guest kernel.

Turning Snapshots Into Drives

A normal container runtime can mount a snapshot on the host and give the resulting path to runc. A Firecracker guest cannot traverse that host path. firecracker-containerd therefore uses block-backed snapshots. Its proof-of- concept snapshotter copies complete images, while its devmapper snapshotter uses thin-provisioned block devices. The guest mounts the resulting device as the container rootfs.

The integration was designed around Firecracker's static MMIO device model. It boots its guest with pci=off, reserves drive IDs with stub devices, and later uses PATCH /drives/{drive_id} to replace a stub's backing path before asking the agent to mount it. Current Firecracker has developer-preview PCI block hotplug, but that does not retroactively change this runtime's design: its kernel command line and drive handler still use the preallocated MMIO path.

The guest image uses /sbin/overlay-init as its init program. overlay-init assembles a read-only squashfs lower layer with a writable upper layer, then starts the systemd target that launches the agent. It is boot plumbing, not the agent itself.

One Network Per VM

Network setup also belongs to VM creation rather than to each container. The runtime invokes a CNI chain, creates a TAP device in the VM's network namespace, and uses the tc-redirect-tap plugin to redirect frames between the CNI-created veth and that TAP. Firecracker consumes the TAP fd as described in Chapter 21. Containers grouped in one VM consequently share the VM network configuration.

The project preserves containerd's runtime interface, but it is not a drop-in replacement for every containerd deployment. Its control plugin requires a specialized daemon, the documented quickstart disables CRI, and the project's roadmap still lists CRI conformance and Kubernetes compatibility as future work. The runtime v2 boundary is intact; the entire surrounding product surface is not automatically equivalent.

Kata Containers

Kata attaches the VM one level higher. Kubernetes asks its CRI implementation to create a pod sandbox, and a configured RuntimeClass selects Kata's containerd shim. Kata maps that sandbox to a VM. Later CreateContainer calls for the pod become processes in the same guest.

This mapping matches Kubernetes semantics: containers in one pod are already expected to share a network namespace and may share other namespaces and volumes. Kata substitutes a guest kernel for the host-kernel sandbox without adding a VM object to the Kubernetes API.

The Agent Owns Guest Containers

The host runtime boots the selected VMM and talks to kata-agent over a ttrpc channel. The agent's default control address is vsock://-1:1024, where -1 accepts any assigned guest CID. It receives the OCI specification in a CreateContainer request and creates the process with Kata's Rust rustjail implementation. There is no second runc binary in this path.

The guest agent is broader than a process launcher. Its RPC service also configures mounts, interfaces, routes, cgroups, devices, and sandbox state. Those operations must happen inside the guest because the host shim cannot use host namespace operations to configure a different kernel.

Storage And Network Translation

Kata chooses storage transport according to its VMM and snapshotter. With a file-backed snapshot, the runtime can start one virtiofsd per VM and export the container rootfs into the guest. With block-backed storage, it can attach a virtio-blk or virtio-scsi device instead. The devmapper path avoids exporting a host directory and lets the guest mount the snapshot's writable block device directly.

Networking starts with the namespace CNI prepared for the pod. The default Kata path creates a TAP device for the VM and installs traffic-control filters between the pod veth and the TAP. MACVTAP remains supported; the older bridge path is deprecated. From kubelet's perspective the pod has the CNI-assigned network. The packet path merely crosses virtio-net and a guest kernel before it reaches the workload.

Kata supports QEMU, Cloud Hypervisor, Firecracker, and the in-process Dragonball VMM. That range matters because storage, hotplug, device passthrough, and shared-filesystem support depend on the backend. Kata's Firecracker configuration remains a constrained path: it uses block-backed container storage, and the version pinned by Kata predates Firecracker's PCI hotplug preview. A statement about that backend is not a statement about every Kata deployment or every current Firecracker feature.

The repository contains both the established Go runtime and runtime-rs, its Rust shim implementation. Configuration selects the binary with runtime_path; Dragonball is embedded in runtime-rs, while other VMMs remain external processes. Release labels and defaults are moving targets, but the pod-to-VM mapping is the stable interface.

Flintlock

Flintlock uses container technology to provision VM artifacts, not to disguise a VM as a containerd task. It is a host service with a gRPC and HTTP API for creating, deleting, getting, and listing microVMs. A create request names the vCPU and memory allocation, kernel, optional initrd, root and additional volumes, network interfaces, metadata, and either the Firecracker or Cloud Hypervisor provider.

Containerd acts as Flintlock's artifact and state substrate. Kernel, initrd, and volume sources may be OCI image references. Flintlock pulls them through containerd, prepares snapshots or mounts, stores the desired and observed VM record, and reconciles that record into a running VMM. It does not register a runtime v2 shim and does not implement the Task service.

The original use case is a Kubernetes cluster whose nodes are microVMs on a bare-metal host. Cluster API Provider Microvm, or CAPMVM, supplies the MicrovmCluster, MicrovmMachine, and MicrovmMachineTemplate resources. Its machine controller translates a Cluster API machine into a Flintlock CreateMicroVM request and waits for the VM state to become CREATED. The guest then boots as a complete node, with its own kubelet and container runtime.

This produces another nested boundary:

bare-metal host
  flintlock
    microVM Kubernetes node
      kubelet and containerd
        pods and containers

The containers at the bottom are ordinary node workloads. Flintlock manages the machine above them. Its use of OCI images does not make the machine an OCI runtime task any more than storing a kernel in an object store makes the object store a hypervisor.

Where The Two Books Meet

The containerd book owns the container side of this boundary: runtime v2, ttrpc, snapshotters, OCI bundles, CNI, veth pairs, and traffic-control redirection. This book adds the translations forced by a second kernel: host paths become shared filesystems or virtual block devices, veth endpoints become TAP-backed virtio-net devices, and local runtime calls become agent RPCs over vsock.

The API can stay container-shaped while the isolation unit changes. To know which guarantee that API provides, ask one question first: does the VM contain a task, a pod, or an entire node?