Chapter 18: The Jailer

KVM contains guest execution, but a VMM exploit has already crossed that boundary. The resulting host process still needs a second containment layer. Firecracker's jailer prepares that layer while privileged, changes to an unprivileged uid and gid, and replaces itself with the firecracker binary.

The jailer does not issue KVM ioctls or build a VM. It creates the environment in which Firecracker later opens /dev/kvm: a reduced filesystem view, a new mount namespace, optional network and PID namespaces, optional cgroup membership, hard resource limits, a small set of device nodes, and cleaned process inheritance. Firecracker adds per-thread seccomp after exec.

These controls are not all automatic. Mount and filesystem isolation always run. Network namespace, PID namespace, and cgroup constraints depend on flags and external setup. Distinct host identities and trustworthy input paths are operator responsibilities.

Inputs And Jail Layout

Four command-line arguments are required:

The default chroot base is /srv/jailer. If the executable basename is firecracker and the ID is i1, the jail root is /srv/jailer/firecracker/i1/root. The copied executable appears as /firecracker inside that root, and the PID file as /firecracker.pid.

The jailer canonicalizes the executable and base paths and rejects traversal in cgroup file and parent names. It still treats its path inputs as trusted. The production guide requires the executable, chroot base, namespace path, and their parents to be protected from modification by unprivileged host users. An ID is expected to identify a fresh jail; the jailer does not empty a reused directory.

The executable is copied rather than hard-linked. The destination is opened with O_NOFOLLOW, a multiply linked destination is rejected, and ownership is changed to the target uid and gid. Besides avoiding link attacks, the separate inode prevents independent Firecracker processes from sharing executable page-cache pages through one file.

Sanitizing Inheritance

Before parsing arguments, sanitize_process closes every file descriptor from 3 through UINT_MAX with:

There is no older-kernel fallback in the current path. Failure aborts jailer startup. Standard input, output, and error remain until optional daemonization. The jailer then removes every environment variable.

This ordering prevents a launcher from accidentally carrying host directory fds, sockets, credentials, or loader settings across the jail boundary. Resources needed by Firecracker must be made visible deliberately inside the jail or through the namespaces and device nodes the jailer prepares.

The Setup Order

The source order in Env::run is the reliable reference; the prose in docs/jailer.md still differs on the namespace and final chroot details.

flowchart TD
    A["Close inherited fds and clear environment"] --> B["Validate arguments and create jail directory"]
    B --> C["Copy executable into jail"]
    C --> D["Join existing network namespace, if requested"]
    D --> E["Install RLIMIT_FSIZE and RLIMIT_NOFILE"]
    E --> F["Create and join configured cgroups"]
    F --> G["Open host /dev/null if daemonizing"]
    G --> H["Copy aarch64 cache and MIDR data"]
    H --> I["Unshare mount namespace and pivot_root"]
    I --> J["Create jailed directories and device nodes"]
    J --> K["Double-fork and redirect stdio, if daemonizing"]
    K --> L["Clone into PID namespace, if requested"]
    L --> M["Write PID file"]
    M --> N["Set uid/gid and exec Firecracker"]

The ordering is driven by path visibility and inheritance. The network namespace fd, cgroup filesystem, /dev/null, /proc/misc, and aarch64 sysfs data are host resources. They must be consumed before pivot_root removes the host filesystem from view. Cgroup membership and rlimits then flow through fork, clone, and exec to Firecracker and its later threads.

Network Namespace

--netns joins an existing network namespace; it does not create one or configure a TAP. The jailer opens the namespace path, calls setns(fd, CLONE_NEWNET), and closes the fd before changing its filesystem root. Firecracker inherits the resulting network namespace after privilege drop.

The orchestrator must create the namespace and its TAP and host networking in advance on an isolated Linux host. Those operations need host network privileges and do not belong in a casual shell recipe. Omitting --netns leaves Firecracker in the jailer's current network namespace.

Resource Limits

The jailer recognizes two --resource-limit name=value names:

It always installs the no-file limit, even when no resource-limit flags are given. For each configured resource, the soft and hard values are identical. Firecracker cannot raise the soft value back toward a larger inherited hard limit after the uid transition.

RLIMIT_FSIZE applies to files Firecracker writes, including snapshot, log, metrics, and other configured output paths inside its filesystem view. It is a process limit, not a storage quota for the whole jail. CPU, memory, and I/O budgets require cgroups or other host controls.

Cgroups

The jailer supports cgroup v1 and v2, with v1 still the CLI default. It creates no resource cgroup merely because the jailer was used. --cgroup properties create and configure leaves; a cgroup v2 --parent-cgroup can instead move the process directly into an existing cgroup when no properties are supplied.

For cgroup v1, the jailer discovers controller mounts in /proc/mounts and creates one leaf per referenced controller at:

<controller mount>/<parent cgroup>/<id>

The parent defaults to the executable basename. It writes property files, inherits empty cpuset values from ancestors when necessary, then attaches the single-threaded jailer through the leaf's tasks file. Firecracker threads created later inherit those memberships.

For cgroup v2, the jailer uses the unified hierarchy. For each requested controller it recursively writes +controller to applicable ancestor cgroup.subtree_control files, writes the leaf property, and attaches through cgroup.procs. The host hierarchy must already satisfy cgroup v2 delegation and no-internal-process rules; the jailer cannot make an arbitrarily populated parent valid.

The containerd book covers cgroup controllers and accounting in detail. The Firecracker-specific point is timing: the jailer joins the cgroup before guest threads and most VM resources exist. Kernel-created helpers such as the x86 kvm-pit thread can still require an external agent because they are not automatically placed in the Firecracker leaf.

Filesystem Isolation

The function named chroot in current source is a pivot_root sequence. It does not finish with the additional chroot(".") described by the older jailer document.

First the jailer calls unshare(CLONE_NEWNS). It recursively makes the mount tree MS_SLAVE, self-bind-mounts the jail directory so it is a mount point, changes into it, and creates old_root. It then performs:

pivot_root(".", "old_root")
chdir("/")
umount2("old_root", MNT_DETACH)
rmdir("old_root")

MS_SLAVE is the current source behavior; the documentation's worked example says MS_PRIVATE. The new mount namespace keeps the pivot and detach from changing the host's namespace. After detaching the old root, ordinary path resolution cannot traverse the host tree. Closing inherited fds before setup is important because an open directory fd could otherwise retain a reference outside the new root.

This sequence needs CAP_SYS_ADMIN and must only be exercised in an isolated Linux environment. The jailer normally runs as root to perform it. Firecracker does not retain those setup privileges after the final credential change.

The Filesystem Inside

After the pivot, the jailer creates /dev, /dev/net, and /run, sets the jail root and those directories to mode 0700, and makes the target uid/gid their owner. /run supports Firecracker's default API socket location.

It then creates owner-readable and owner-writable character-device nodes:

The jailer discovers the dynamic userfaultfd minor before the pivot. Omitting that node removes the device-file route to userfaultfd-backed restore; kernel and Firecracker support determine whether another creation path is available. On aarch64, the jailer also copies the host cache topology and MIDR_EL1 data that Firecracker needs when constructing the guest CPU description.

A device node is access to a kernel subsystem, not emulation inside the jail. The mount namespace hides the host /dev, while these new nodes expose only the specific device interfaces Firecracker is expected to open.

Daemon And PID Namespace Modes

With --daemonize, the jailer opens host /dev/null before the pivot, then uses a double fork after device setup. The first child calls setsid; the second fork prevents reacquiring a controlling terminal. The grandchild maps the saved /dev/null fd onto stdin, stdout, and stderr.

With --new-pid-ns, the final jailer process calls raw clone with CLONE_NEWPID. The child becomes PID 1 in the new namespace and immediately continues to credential setup and exec. The parent records the child's host PID and exits. PID 1 has special signal and orphan-reaping semantics, so this mode changes more than the number visible in /proc.

The jailer writes /firecracker.pid in both PID-namespace and ordinary modes. Orchestrators should use that file instead of assuming the PID that originally launched the jailer will remain the Firecracker PID, especially with daemonization.

Credential Handoff

The final command supplies the instance ID, startup timing values, and every argument after the jailer's -- separator. Rust's Unix CommandExt applies the requested gid and uid before execve, so the resulting Firecracker process runs under those numeric identities.

The jailer source does not call setgroups, capset, or PR_SET_NO_NEW_PRIVS in this handoff. Operators should launch it without unneeded supplementary groups and use a dedicated, unprivileged account. Changing from root to a non-root uid and executing the copied ordinary file causes normal Linux credential rules to remove root effective privilege, but the design should not rely on a reused account that owns unrelated host files.

Each Firecracker instance should use a unique uid and gid when practical. The production guide describes this as an extra layer, not an invariant enforced by the jailer. Unique identities limit cross-instance filesystem access if another containment layer fails; they do not make the shared kernel or host namespace private.

PR_SET_NO_NEW_PRIVS is set later by Firecracker's apply_filter on each thread immediately before the seccomp filter is installed. The jailer itself does not install seccomp. Chapter 19 follows that per-thread transition.

What The Jailer Does Not Contain

The jailer does not create host networking, filter guest egress, choose a safe CPU topology, disable KSM or swap, provision block images, or account every kernel helper thread. It also cannot choose workload-specific CPU, memory, I/O, and file-size budgets on its own.

Its role is narrower and concrete: remove accidental inherited authority, prepare an explicit filesystem and namespace view, join configured resource domains, and cross the uid/gid boundary before Firecracker parses guest-facing device traffic. The remaining host policy is the subject of Chapter 20 and the production host guide.

Sources And Further Reading

• Paired research note
• Jailer source
• Jailer process entry
• Mount-namespace and pivot sequence
• Cgroup implementation
• Resource-limit implementation
• Jailer documentation
• Production host setup
• Firecracker design
• Linux pivot_root(2)
• Linux cgroup v2