Chapter 17: MMDS

A generic guest image still needs instance-specific data: identity, role, network settings, bootstrap material, or short-lived credentials. Firecracker's MicroVM Metadata Service, MMDS, supplies that data without a host TCP listener or a second service process. The host writes a JSON document through the Firecracker API socket. Guest software reads it through a small in-process Ethernet, ARP, IPv4, TCP, and HTTP stack.

The two paths have different trust assumptions. Access to the host API socket authorizes reads and writes to the store. Guest traffic is untrusted and gets a read-oriented HTTP surface, with an optional session-token requirement. MMDS V2 reduces a class of proxy and request-forgery attacks; it does not hide metadata from arbitrary code already executing inside the guest.

The Two Paths

The host path uses the same Unix-domain API as the rest of Firecracker:

PUT, PATCH, and GET /mmds are available before boot and remain available at runtime when MMDS was configured on a network device or restored from a snapshot. Configuration is pre-boot only because it changes which virtio-net devices own an MMDS network stack. PATCH follows RFC 7396: object members merge recursively and a null member removes the corresponding key. A patch requires an initialized store; a PUT initializes it, even when the replacement value is empty.

The store is one serde_json::Value behind an Arc<Mutex<Mmds>>. Its maximum serialized size defaults to the HTTP API request limit, currently 51,200 bytes, unless --mmds-size-limit selects another value. The limit is checked after a tentative merge, so a rejected patch leaves the old document unchanged.

The guest path lives in the virtio-net data path. Each selected network device has an MmdsNetworkStack, commonly called Dumbo. A guest TX frame addressed to the configured MMDS IPv4 address is diverted before the normal TAP write. Dumbo generates the response, and the net device gives pending MMDS frames priority when filling guest RX buffers.

Dumbo is not a socket listener in the host network namespace. It runs as part of net-device emulation in the VMM event loop. There is no MMDS sidecar and no dedicated MMDS thread.

Configuration

PUT /mmds/config accepts four fields:

The interface list must be nonempty, and every ID must already name a configured network device. Firecracker attaches a stack to listed devices and removes it from unlisted devices. A custom address must remain in the IPv4 link-local range. Guest routing must send that address through one of the selected interfaces; MMDS does not configure routes inside the guest.

V1 remains the schema default but is deprecated. A V1 configuration increments Firecracker's deprecated-API metric, logs a warning, and marks the HTTP response as deprecated. New integrations should select V2 explicitly.

imds_compat controls representation, not authorization. When false, an Accept: application/json request returns the selected JSON value, including arrays, numbers, booleans, and null. Plain-text output, which is also the default when no Accept header is supplied, follows EC2 IMDS-style traversal: an object becomes a newline-separated key listing, nested objects gain a / suffix, and a string loses its JSON quotes. Other JSON leaf types cannot be represented in that format and return 501 Not Implemented. With imds_compat: true, Dumbo uses the IMDS representation regardless of Accept.

Frame Interception

The selected stack recognizes two frame classes. An ARP request whose target protocol address equals the MMDS IP receives an answer using MMDS's synthetic MAC address 06:01:23:45:67:01. An IPv4 frame whose destination equals the MMDS IP is consumed by Dumbo. TCP can reach the HTTP handler; UDP and ICMP are absorbed without a response and increment the unusual-receive metric.

Everything else follows the ordinary TAP path. On a network interface without an MMDS stack, even traffic to the metadata address follows that path. The guest-visible route and selected interface therefore matter as much as the IP address.

Dumbo implements only the protocol needed for short metadata exchanges. It is a passive TCP server on port 80 with a small fixed request buffer and a bounded connection table. It does not implement a general host network stack: there is no active open, IP fragmentation, out-of-order TCP buffering, SACK, timestamp option, or conventional congestion control. Oversized or unsupported traffic is rejected or reset rather than handled like a full Linux TCP stack.

Guest HTTP methods are limited to GET and PUT. A GET resolves the URI as a JSON Pointer after collapsing repeated slashes. The only accepted guest PUT target is /latest/api/token; it mints a V2 session token and never invokes the host-side put_data or patch_data methods. Other methods return 405 with Allow: GET, PUT.

V1 And V2

V1 permits a metadata GET with no token. It also accepts a valid, invalid, or expired token so existing V1 clients can be measured before migration. Missing and invalid token counters still increment.

V2 requires a token on every metadata GET:

The TTL must be from 1 through 21,600 seconds. Firecracker accepts either its native X-metadata-token-ttl-seconds and X-metadata-token headers or the corresponding X-aws-ec2-... aliases. The token response echoes the TTL header variant used by the request. A missing, malformed, expired, or unauthentic token causes a V2 GET to return 401. A token request containing X-Forwarded-For is rejected with 400.

Any process in the guest that can reach MMDS can request its own token. V2 is designed to make a simple blind HTTP-proxy relay harder, not to distinguish one guest process from another. Metadata authorization remains a guest policy problem.

Token Construction

The token is a self-contained sealed expiration time, not a random identifier stored in a server-side session table. Its decoded representation contains a 12-byte nonce, an encrypted 8-byte expiration timestamp, and a 16-byte authentication tag. Standard base64 turns those 36 bytes into a 48-character response.

Each Mmds instance creates a random 256-bit AES-GCM key with AWS-LC. The additional authenticated data is microvmid=<instance_id>. Validation decodes the token, authenticates and decrypts it, then compares its expiration with the host monotonic clock. A token minted by another authority fails authentication; the instance ID adds another binding to the Firecracker identity.

Firecracker rejects encoded tokens longer than 70 characters before attempting decryption. It also rotates the key before encrypting more than u32::MAX tokens under one authority, invalidating tokens made with the old key. These limits bound parser and nonce-management behavior; they do not add a second authorization layer inside the guest.

The Host Network Boundary

MMDS interception is not a firewall. The net-device source states two reasons directly. First, an interface without an MMDS stack forwards metadata-addressed traffic to its TAP. Second, guest TX descriptors are shared memory: a malicious guest can race Firecracker's header check and rewrite the destination before the later TAP write. Firecracker deliberately avoids copying every outgoing packet merely to close that race.

Deployments must filter guest egress in the host network namespace so TAP traffic cannot reach host metadata or other restricted link-local services. The Firecracker production guide gives nftables and iptables-nft examples, but those privileged rules belong in isolated host provisioning, not in a guest or development-shell recipe. Broader tenant egress policy is still required; blocking only the default MMDS address does not make arbitrary guest traffic safe.

The API socket is the other boundary. MMDS does not authenticate host-side PUT, PATCH, or GET requests. Filesystem permissions, chroot layout, and the control-plane process must keep that socket away from untrusted callers. Do not place a secret in MMDS unless all code allowed to read metadata inside the guest is also allowed to receive that secret.

Snapshot Behavior

Snapshots persist enough configuration to recreate MMDS routing: each selected net device stores its MMDS network stack's MAC, IP, and TCP port, while device manager state stores the MMDS version and imds_compat setting. Active Dumbo TCP connections are not persisted.

The JSON data store is deliberately not persisted. Restore constructs a fresh store and a fresh token authority, binds it to the restored process's instance ID, and restores the selected network stacks around it. Metadata remains empty until the host sends a new PUT /mmds. Tokens minted before the snapshot fail under the new key.

This split prevents clone-specific metadata from silently propagating to every copy. It also creates a sequencing requirement: populate each clone's store before allowing guest work that depends on metadata. Restoring network configuration does not preserve responses, session tokens, or TCP connection state.

Operational Signals

MMDS metrics distinguish diverted frames, malformed or unusual packets, successful receives and sends, send errors, connection creation and cleanup, and missing or invalid tokens. The last two counters are especially useful during a V1-to-V2 migration because V1 records the condition without denying the request. Once V2 is enabled, the same counters accompany 401 responses.

Metrics do not prove that host filtering is correct. Traffic that bypasses a selected MMDS stack is ordinary net-device traffic and must be observed and controlled at the host networking layer.

Sources And Further Reading

• Paired research note
• MMDS design
• MMDS user guide
• MMDS data store
• MMDS request handling
• MMDS token authority
• MMDS network stack
• virtio-net MMDS interception
• MMDS configuration
• MMDS persistence
• Production host setup