Chapter 10: Booting A Guest Kernel

A conventional VM can start in firmware such as SeaBIOS or OVMF and let a bootloader find the kernel. Firecracker instead receives the kernel image in its configuration, loads it into guest RAM, and constructs the CPU and memory state expected at the kernel's first instruction. On x86_64 it can make that handoff through either the Linux 64-bit boot protocol or the PVH direct boot ABI.

Both paths eventually reach start_kernel() in init/main.c, but they begin in different processor modes and use different data structures. The Linux path enters startup_64 with paging enabled and a struct boot_params pointer in RSI. The PVH path enters pvh_start_xen in 32-bit protected mode with an hvm_start_info pointer in EBX.

The Two Boot Paths

The Linux boot protocol tells a loader how to describe the command line, initrd, physical memory, and other platform data in struct boot_params. Its 64-bit entry contract additionally requires long mode, paging, identity mappings, segment descriptors, and RSI pointing at that structure.

PVH originated in Xen and uses a different contract. The loader starts the kernel in 32-bit protected mode with paging disabled and passes the smaller hvm_start_info structure in EBX. The kernel builds the state needed for long mode before joining the normal x86 startup path. A VMM may support either contract; Firecracker supports both.

Firecracker on x86_64 implements both. It inspects the loaded ELF kernel for a XEN_ELFNOTE_PHYS32_ENTRY note (type 18, name "Xen"); if the note is present, Firecracker uses PVH as the preferred path. If the note is absent, it falls back to the 64-bit Linux protocol. The choice is made from the loaded image at boot. The Linux path comes first here because it exposes the page tables and register state that Firecracker otherwise delegates to the PVH entry code.

vmlinux And bzImage

The image format determines how the VMM extracts both the entry point and the setup data.

vmlinux is the raw ELF produced by the kernel build. It is statically linked, 64-bit, uncompressed, and carries the full ELF header that any standard ELF loader can parse. The VMM loads its PT_LOAD segments directly into guest RAM and reads the ELF e_entry field to find startup_64, the first kernel instruction, in arch/x86/kernel/head_64.S. No decompression step runs, because there is nothing to decompress.

bzImage is a self-extracting boot image: the first (setup_sects + 1) x 512 bytes are a real-mode setup blob in 16-bit code, and the bytes after that are the protected-mode kernel compressed according to whatever CONFIG_KERNEL_* option was selected -- gzip, LZMA, or zstd. A bootloader that ingests a bzImage reads its setup_header from file offset 0x01F1 and selects one of the protocol entry points. The image's own decompressor expands the protected-mode kernel before the final kernel entry.

Firecracker requires an uncompressed ELF vmlinux on x86_64. It does not support bzImage. Build the guest kernel with make vmlinux. On aarch64, Firecracker instead requires the PE-format Image file produced by make Image.

The rust-vmm linux-loader crate, which Firecracker uses internally, supports three formats: raw ELF on x86_64 (for both the Linux protocol and PVH), bzImage on x86_64, and PE Image on aarch64/riscv64. For an ELF kernel, linux-loader returns the entry address and reports a PVH entry when the corresponding ELF note exists. Its separate bzImage loader extracts a setup_header. Firecracker uses the ELF loader and constructs either Linux boot parameters or PVH start information from its own machine layout.

The Setup Header and the Zero Page

The boot protocol's central data structure is struct boot_params, a 4096-byte, packed C struct that the Linux kernel defines in arch/x86/include/uapi/asm/bootparam.h. The kernel documentation calls the page it occupies the zero page, the traditional name for this zero-initialized parameter block. In the 64-bit protocol, the loader may place it anywhere that satisfies the addressing and identity-mapping rules, then communicates its guest-physical address in RSI at entry.

Embedded within boot_params at offset 0x01F1 is struct setup_header, a packed set of protocol fields. For a bzImage, the setup header is copied from the image file at the same byte offset. For an ELF vmlinux, the VMM fills in a minimal synthetic header, because the fields the kernel cares about at direct-boot time are only those the VMM itself controls -- loader type, command-line address, initrd address, and memory alignment.

The bzImage setup header contains two magic numbers written by arch/x86/boot/header.S:

boot_flag: .word 0xAA55
header:    .ascii "HdrS"        # 0x5372_6448 little-endian

boot_flag at image offset 0x01FE is 0xAA55, and header at 0x0202 is 0x5372_6448 (ASCII HdrS). The rust-vmm bzImage loader validates the latter. Firecracker's ELF loader validates the ELF image instead; its Linux boot configurator then writes both constants into the synthetic header passed to the guest. The protocol version field is a two-byte little-endian value at 0x0206, but Firecracker leaves it zero in this synthetic header.

For a loader consuming a bzImage, the version gates which fields it may read or write. Protocol 2.02 added cmd_line_ptr; 2.06 added cmdline_size; 2.09 added the setup_data linked list; and 2.12 added xloadflags, including XLF_KERNEL_64. Those compatibility rules explain the layout, but Firecracker does not copy a bzImage header on this path. It constructs the fields its supported ELF kernels consume.

Firecracker fills a small subset of the synthetic setup_header:

Firecracker sets type_of_loader to 0xFF, the catch-all value KERNEL_LOADER_OTHER defined by the boot protocol for VMMs and loaders without registered IDs.

Populating the Zero Page

The full boot_params struct is 4096 bytes. Most of it is zeroed. Beyond the setup_header section, two fields carry information the kernel cannot derive independently: the ACPI RSDP address and the physical memory map.

Firecracker places boot_params at guest-physical address 0x7000 (ZERO_PAGE_START). It fills acpi_rsdp_addr at boot_params[0x070] with the address 0x000E_0000, where it writes the RSDP. The memory map lives at boot_params[0x2D0] as an array of up to 128 boot_e820_entry structs -- each 20 bytes: a u64 start address, a u64 size, and a u32 type. The count of valid entries goes at boot_params[0x1E8] as a u8. Type 1 is E820_RAM (usable); type 2 is E820_RESERVED.

Firecracker's configure_64bit_boot() in src/vmm/src/arch/x86_64/mod.rs builds the table with four classes of entry:

1. [0x0000_0000, 0x9FC00) -> type 1: usable RAM below the Extended BIOS Data Area.
2. [0x9FC00, 0x9FC00 + 0x40400) -> type 2: reserved region covering the EBDA, the MP table, and the ACPI area.
3. [PCI_MMCONFIG_START, PCI_MMCONFIG_START + 256 MiB) -> type 2: the PCIe ECAM window.
4. [max(0x10_0000, region.start), region.end) -> type 1 per DRAM region: usable RAM from 1 MiB upward.

That table is what the kernel reads at startup_64 to construct its own memory model. It replaces the E820 query that a real BIOS would answer.

The complete boot_params assembly that Firecracker writes looks like this:

Together with the ACPI tables and device model, this page supplies the platform data that firmware and a bootloader would otherwise have passed to Linux.

Placing the Kernel, Initrd, and Command Line

The address constants for x86_64 in Firecracker are defined in src/vmm/src/arch/x86_64/layout.rs. A tour of the low guest-physical address space shows how tightly packed the boot data is:

The kernel load area begins at HIMEM_START = 0x0010_0000 (1 MiB). For ELF vmlinux, Firecracker's linux-loader maps the kernel's PT_LOAD segments at the guest physical addresses implied by the ELF image and the load base. The synthetic kernel_alignment field describes the alignment unit to Linux; it is not a claim that the ELF begins at a 16 MiB boundary.

The command line is a null-terminated C string at CMDLINE_START = 0x0002_0000 (128 KiB), with a maximum length of 2048 bytes including the null terminator. setup_header.cmd_line_ptr receives 0x0002_0000 as a 32-bit guest-physical address. Firecracker converts the command line to a C string and passes as_bytes_with_nul().len() as setup_header.cmdline_size, so this synthetic value includes the terminating null. The VMM enforces its own 2048-byte buffer limit before writing the string.

The initrd, if present, is placed at the top of the first DRAM region, page-aligned downward: align_down(lowmem_end - initrd_size, PAGE_SIZE). This keeps the initrd as high as possible to avoid colliding with the kernel's own data. setup_header.ramdisk_image receives the resulting 32-bit guest-physical address; setup_header.ramdisk_size receives the byte count. Firecracker's synthetic header leaves initrd_addr_max zero; its placement function and 32-bit ramdisk_image conversion keep this path in low memory rather than deriving a limit from a bzImage header.

KVM_TSS_ADDRESS = 0xFFFB_D000 is a special case. Before any vCPU can run, the VMM calls KVM_SET_TSS_ADDR with this guest-physical address while constructing the VM, before creating its vCPUs. KVM's x86 API requires a three-page region for this ioctl and documents the setting as Intel-specific. KVM also exposes KVM_SET_IDENTITY_MAP_ADDR for relocating its private identity-map page, but current Firecracker does not issue that ioctl and therefore does not choose a companion GPA.

Isolated KVM host required. Opening /dev/kvm requires read/write access under the host's device policy, and the ioctls here operate on a live VM fd. Exercise this path only on a disposable bare-metal Linux host or a VM with nested virtualization, not on a shared production host.

CPU State at the 64-bit Entry Point

The Linux 64-bit boot protocol specifies an entry state that is already in long mode with paging active. This is the fundamental difference from a traditional bootloader, which enters the kernel in 32-bit protected mode and leaves mode switching to the kernel's own startup code. The 64-bit direct-boot protocol hands off in the mode the kernel will run in, making the kernel's startup_64 path a simpler target.

Firecracker programs this state with three ioctls on the vCPU file descriptor: KVM_SET_REGS for general-purpose registers, KVM_SET_SREGS for segment registers and control registers, and KVM_SET_FPU for floating-point state. It applies them during vCPU configuration before the run loop starts.

General-Purpose Registers

RIP is the ELF e_entry field, which resolves to the startup_64 symbol in arch/x86/kernel/head_64.S. The value is fixed in that ELF image but can differ between kernel builds. Runtime KASLR does not rewrite the ELF header; the VMM reads e_entry rather than hardcoding an address.

RSI is the ABI contract: it must contain the 32-bit guest-physical address of struct boot_params. Current startup_64 immediately preserves it in R15 before making calls that could clobber RSI.

RFLAGS is 0x0000_0000_0000_0002. Bit 1 is always 1 by architectural definition; IF (bit 9) is 0, leaving interrupts disabled until the kernel enables them during start_kernel. All other GPRs are zero.

Control Registers and Segment State

Firecracker programs control registers and segment descriptors via KVM_SET_SREGS. Its LinuxBoot values are:

PE is bit 0 of CR0 (0x1), ET is bit 4 (0x10), and PG is bit 31 (0x8000_0000). All three must be set simultaneously -- turning on paging while not in protected mode is a fault. PAE is CR4 bit 5 (0x20); 64-bit paging requires it. LME and LMA together in EFER signal that the CPU is in long mode and that the current CS descriptor is 64-bit. KVM validates these relationships and will refuse to enter the guest if they are inconsistent.

The GDT lives at guest-physical address 0x500, with four 8-byte entries and a gdt.limit of 31. Firecracker writes four entries in src/vmm/src/arch/x86_64/gdt.rs:

sregs.cs uses selector 0x08 (index 1, 64-bit code). sregs.ds, es, fs, gs, and ss all use selector 0x10 (index 2, data). sregs.tr uses selector 0x18 (index 3, TSS). The IDT at 0x520 is a single null 8-byte entry with sregs.idt.limit = 7; the kernel will install its own interrupt handlers early in start_kernel.

The Linux boot protocol documentation names selectors __BOOT_CS (0x10) and __BOOT_DS (0x18) in its prescribed GDT layout. Firecracker instead loads selectors 0x08 and 0x10 together with equivalent code and data descriptor state through KVM_SET_SREGS. The important entry state is the active segment cache -- in particular a 64-bit code segment and flat data segments -- not reuse of Linux's GDT indices in guest RAM.

Boot-Time Page Tables

The boot page tables at PML4_START = 0x9000 create a minimal identity map covering guest-virtual [0, 1 GiB) using 2 MiB pages. Three levels suffice:

• PML4 at 0x9000: one entry pointing to the PDPTE -- 0xA003 (address 0xA000 with present and writable bits set).
• PDPTE at 0xA000: one entry pointing to the PDE array -- 0xB003 (address 0xB000 with present and writable bits set).
• PDE at 0xB000: 512 entries. Entry i is (i << 21) | 0x83: the 2 MiB page at physical address i x 2 MiB, with PS (bit 7), writable (bit 1), and present (bit 0) set.

This maps the half-open range [0, 1 GiB) with a 1:1 virtual-to-physical correspondence. The boot protocol requires the kernel's load range and the zero page both to be identity-mapped at entry, so that startup_64 can dereference RSI without a translation fault before it builds its own permanent page tables. Because 1 GiB of coverage includes 0x7000, 0x10_0000, 0x2_0000, and the initial stack at 0x8FF0, the three-level map is sufficient for boot.

Firecracker builds these tables in setup_page_tables() in src/vmm/src/arch/x86_64/regs.rs. startup_64 preserves the boot-parameter pointer, repairs the kernel's early page-table state for its actual load address, and switches to kernel-owned tables as early startup proceeds.

FPU State

KVM_SET_FPU initializes floating-point state to the architectural defaults: fcw = 0x037F (x87 control word with all exception masks set and double-extended precision, PC = 10b) and mxcsr = 0x1F80 (MXCSR with all SSE exception masks set and round-to-nearest). These are Firecracker's explicit initial values; the KVM API does not make this particular KVM_SET_FPU call part of the Linux boot protocol.

The Full Boot Sequence

With those pieces in place, the boot sequence from KVM_RUN to start_kernel is a straight line with no firmware detour:

sequenceDiagram
    participant VMM as VMM process
    participant KVM as KVM kernel module
    participant CPU as Guest vCPU
    participant K64 as "startup_64 (head_64.S)"
    participant SK as "start_kernel (main.c)"

    VMM->>KVM: KVM_SET_TSS_ADDR
    VMM->>KVM: KVM_CREATE_VCPU
    VMM->>VMM: load ELF segments at 0x10_0000
    VMM->>VMM: write boot_params at 0x7000
    VMM->>VMM: write cmdline at 0x2_0000
    VMM->>VMM: write initrd at top of DRAM
    VMM->>VMM: write GDT/IDT at 0x500/0x520
    VMM->>VMM: write page tables at 0x9000-0xB000
    VMM->>KVM: KVM_SET_REGS (RIP=e_entry, RSI=0x7000, ...)
    VMM->>KVM: KVM_SET_SREGS (CR0, CR3, CR4, EFER, GDT, ...)
    VMM->>KVM: KVM_SET_FPU
    VMM->>KVM: KVM_RUN
    KVM->>CPU: VMLAUNCH (Intel VMX) / VMRUN (AMD SVM)
    CPU->>K64: first instruction at RIP
    Note over K64: preserves RSI -> boot_params<br/>repairs early page tables
    K64->>K64: switches to kernel-owned page tables
    K64->>SK: x86_64_start_kernel() -> start_kernel()

No firmware, bootloader, or decompressor runs on this ELF LinuxBoot path. The first KVM_RUN enters the guest at the ELF entry point, after which Linux owns the transition from its early identity map to the normal kernel address space.

The PVH Path

PVH -- "Para-Virtualised Hardware," formally the x86/HVM direct boot ABI -- offers an alternative that removes the 64-bit paging requirement from the VMM. The kernel's PVH entry point accepts a 32-bit protected-mode state with paging disabled and transitions to long mode itself. That is a simpler entry contract for the VMM, but it means the kernel has more work to do before it can access 64-bit memory.

The ABI is signaled by an ELF PT_NOTE segment in the kernel image, with note name "Xen" (four bytes with null terminator) and note type XEN_ELFNOTE_PHYS32_ENTRY = 18. The note value is the 32-bit physical address of pvh_start_xen, implemented in arch/x86/platform/pvh/head.S; enlighten.c builds the Linux boot parameters after the assembly entry code reaches long mode. Firecracker's v1.12.0 changelog documents support for Linux kernels newer than 5.0 built with CONFIG_PVH=y.

The data structure at entry is hvm_start_info, placed at guest-physical address PVH_INFO_START = 0x6000 in Firecracker's layout:

#define XEN_HVM_START_MAGIC_VALUE 0x336ec578

struct hvm_start_info {
    uint32_t magic;          /* must == 0x336ec578          */
    uint32_t version;        /* 0 = v0, 1 = v1              */
    uint32_t flags;          /* SIF_xxx flags                */
    uint32_t nr_modules;     /* count of modules             */
    uint64_t modlist_paddr;  /* phys addr of hvm_modlist_entry[] */
    uint64_t cmdline_paddr;  /* phys addr of command line    */
    uint64_t rsdp_paddr;     /* phys addr of ACPI RSDP      */
    /* v1 additions: */
    uint64_t memmap_paddr;   /* phys addr of memory map      */
    uint32_t memmap_entries; /* entry count (0 = no map)     */
    uint32_t reserved;       /* must be zero                 */
};

The magic 0x336ec578 is the ASCII string "xEn3" with the high bit of 'E' set -- a Xen convention that predates the ABI's hypervisor-agnostic rebranding. The initrd rides in an hvm_modlist_entry (32 bytes: paddr u64, size u64, cmdline_paddr u64, reserved u64) pointed to by modlist_paddr. Firecracker writes version 1, a memory-map pointer and count, the command-line address, and an optional one-entry module list for the initrd. It currently leaves rsdp_paddr zero.

The CPU state the PVH ABI mandates differs from the Linux 64-bit protocol in two critical ways: there is no paging (CR0.PG = 0), and the pointer to the info struct goes in EBX rather than RSI. The full required state:

In Firecracker's PVH path, RBX = PVH_INFO_START = 0x6000. The GDT entries use 32-bit descriptors (0xC09B for code, 0xC093 for data, limit 0xFFFF_FFFF) instead of the 64-bit flags the Linux protocol uses. Its CR0 value is PE | ET, with PG clear, and CR4 is zero. pvh_start_xen copies the start information, enables PAE and long mode, activates its preconstructed page tables, and eventually jumps to startup_64 with a synthesized boot_params pointer in RSI.

PVH also gives Firecracker a direct-boot contract for non-Linux guests. Its PVH documentation covers FreeBSD as well as Linux; in both cases the loaded ELF must provide the expected entry note and implement the HVM start-info ABI.

Sources And Further Reading

• Linux x86 boot protocol specification (magic numbers, field offsets, 64-bit entry, setup_data, protocol version table): https://www.kernel.org/doc/html/latest/arch/x86/boot.html
• Zero-page offset table: https://docs.kernel.org/next/x86/zero-page.html
• struct boot_params, struct setup_header, boot_e820_entry, E820 type constants: https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/bootparam.h
• boot_flag: .word 0xAA55, header: .ascii "HdrS", version: .word 0x020f: https://github.com/torvalds/linux/blob/master/arch/x86/boot/header.S
• PVH/HVM direct boot ABI specification (CPU state at entry, CR0 requirement, EBX = hvm_start_info): https://xenbits.xen.org/docs/unstable/misc/pvh.html
• XEN_ELFNOTE_PHYS32_ENTRY = 18, ELF note name "Xen": https://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,elfnote.h.html
• hvm_start_info struct layout, magic 0x336ec578, hvm_modlist_entry: https://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,arch-x86,hvm,start_info.h.html
• Firecracker kernel format requirements (vmlinux ELF on x86_64, Image on aarch64): https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md
• Firecracker v1.12.0 PVH boot (PR #5048): https://github.com/firecracker-microvm/firecracker/blob/main/CHANGELOG.md
• rust-vmm linux-loader crate -- three formats (ELF, bzImage, PE); fields returned per format: https://github.com/rust-vmm/linux-loader/blob/main/README.md
• linux-loader design -- ELF entry point, setup_header extraction, init_size usage: https://github.com/rust-vmm/linux-loader/blob/main/DESIGN.md
• Linux PVH assembly entry and transition to startup_64: https://github.com/torvalds/linux/blob/master/arch/x86/platform/pvh/head.S
• Firecracker PVH implementation notes: https://github.com/firecracker-microvm/firecracker/blob/main/docs/pvh.md
• KVM_SET_TSS_ADDR and KVM_SET_IDENTITY_MAP_ADDR requirements: https://docs.kernel.org/virt/kvm/api.html
• Paired research note: Booting A Guest Kernel