[GH-ISSUE #390] feature(qemu-backend): add a QEMU sandbox backend for GPU, Windows and macOS sandboxes #99

New issue

Open

opened 2026-10-29 14:19:34 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 14:19:34 +00:00

Owner

Copy link

Originally created by @yingdi-shan on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/390

Affected component: Orchestrator / sandbox lifecycle (also: Firecracker runtime, Snapshot / template builder, Gateway / scheduler)

Problem statement

We need AgentENV to provide GPU sandboxes and Windows and macOS sandboxes in addition to today's Linux CPU sandboxes. AgentENV is built only on Firecracker, which rules out both:

• No device passthrough. Firecracker has no PCI or VFIO passthrough, so sandboxes cannot use GPUs (see #26, where the reply identified a QEMU backend as the necessary first step).
• Linux guests only. Firecracker boots a Linux kernel directly, has no UEFI/BIOS firmware, and offers only a minimal virtio-mmio device model. It cannot boot Windows or macOS.

The Firecracker dependency goes beyond the device model. Our capture path relies on our Firecracker fork's dirty-memory-range API, on a file-backed (MAP_PRIVATE) or uffd memory backend, and on MMDS for guest metadata. None of these has a direct QEMU equivalent. Several other parts of the stack also assume a Linux guest:

• guest configuration through kernel boot args (drive layout, DAMON region)
• the Linux envd guest agent
• volume mounting through Linux commands in the guest
• template builds that turn a Dockerfile/OCI image into a Linux rootfs

Use case

• GPU sandboxes: agent workloads that need a GPU in the sandbox (inference, CUDA builds, ML evaluation), scheduled on GPU nodes in the same cluster as CPU sandboxes.
• Windows sandboxes: agents that build, test or operate Windows software, or use a Windows desktop.
• macOS sandboxes: agents that build, test or operate macOS software, or use a macOS desktop.
• In every case, through the same E2B-compatible API, scheduler, volumes and OverlayBD storage layering as Linux sandboxes, not a second stack.

Current behavior and workarounds

• server.rs, the orchestrator and the template runner construct FirecrackerSandboxFactory / FirecrackerSandbox directly. No other VMM can be selected (#381 adds a factory alias for a mock backend).
• About 1.7k lines in src/sandbox/firecracker/ do not depend on Firecracker at all: OverlayBD snapshot sealing and restacking, dirty-memory-to-OverlayBD conversion, ProcessVmReader, SandboxResources, startup prefetch on the ublk device, and the MmdsMetadata type. A second backend cannot reuse them where they are.
• Workaround: run GPU, Windows and macOS sandboxes on separate stacks (Kata, plain VMs, or Mac hosting). That loses AgentENV's API, scheduling, snapshots, volumes and storage layering, and means operating several systems.

Desired behavior

• Linux under QEMU: Given a node configured with the QEMU backend, when a client creates a Linux sandbox through the existing API, then it boots under QEMU with the same rootfs (OverlayBD over ublk), networking (tap in netns), envd, volumes and custom-extension hooks as a Firecracker sandbox.
• GPU: Given a QEMU sandbox with a GPU assigned, when the guest runs nvidia-smi (or the equivalent), then the device is visible and usable.
• Windows: Given a Windows template, when a client creates a sandbox, then it boots under QEMU with UEFI and virtio drivers. Commands, files and process APIs work through the same API, backed by a Windows guest agent compatible with envd.
• macOS: Given a macOS template on a host where running macOS is licensed (see the open question below), when a client creates a sandbox, then it boots and the same API works through a macOS guest agent.
• Snapshots without passthrough: Given a QEMU sandbox (Linux or Windows) without passthrough devices, when it is paused/resumed or captured and launched, then this behaves as it does for Firecracker. The snapshot manifest records backend = "qemu", and a restore is routed only to nodes that can run that backend.
• Snapshots with passthrough: Given a sandbox with passthrough devices, when pause, capture or fork is requested, then the API rejects it with a clear error instead of producing a broken snapshot.
• Firecracker behavior, configuration and stored snapshots are unchanged.

Proposed approach

The work is split into phases, each of which can be shipped and reviewed on its own. Line counts are rough estimates based on the current Firecracker backend (about 6.2k lines of code plus 3.2k lines of tests). They cover the AgentENV host side only and exclude the guest agent ports in phase 4.

0. Remove Firecracker assumptions from shared code (refactor only, about 1–1.5k lines moved or changed)
   • Move the VMM-neutral code out of src/sandbox/firecracker/:
     • overlaybd_snapshot.rs restacking and memory-layer building → src/sandbox/ublk/
     • process_vm_reader.rs
     • resources.rs → src/sandbox/
     • the local startup-prefetch half of startup_pack.rs
     • the MmdsMetadata type
     • drive and boot-argument helpers
   • Make convert_dirty_memory_to_overlaybd take a neutral dirty-range type instead of firecracker_client::models::DirtyMemoryRanges.
   • Split FirecrackerCommonConfig into a shared sandbox config and the Firecracker-only fields. Use serde(flatten) and add tests that persisted paused states and snapshot configs from before the split still load.
   • Replace the FirecrackerCaptureArtifacts downcast in publish_captured with a recording hook the backend supplies.
   • Rename the FirecrackerManifest and firecracker_version Rust identifiers, keeping their stored names.
   • Select the factory from configuration instead of hard-wiring FirecrackerSandboxFactory (coordinate with #381).
1. Linux and GPU cold start (about 3k lines of code + 1–1.5k of tests)
   • QMP client, QEMU microvm/q35 command-line builder, process lifecycle and logs
   • virtio-blk on ublk devices; rate limits as throttle groups
   • tap networking
   • guest metadata delivery to replace MMDS: a host-side responder in the sandbox netns, or fw_cfg plus an envd change
   • [qemu] config and dependency/setup packaging
   • VFIO device assignment from node config
2. Pause/resume and snapshots (about 3–4k lines of code + 1.5k of tests)
   • VM state through QMP migration to a file
   • memory restore from the ublk memory device
   • rootfs and attached-drive sealing through the shared phase-0 code
3. Parity (about 2–3k lines of code + 1–2k of tests)
   • fork, incremental dirty-page capture, startup-pack recording, volume slot rebinding and freeze, warm process pool
4. Windows and macOS guests (to be estimated after a guest-agent spike)
   • UEFI firmware (OVMF); virtio drivers baked into Windows templates
   • guest metadata and network setup without kernel boot args (DHCP in the sandbox netns, or the guest agent)
   • Windows and macOS ports of the guest agent behind the envd API
   • volume mounting through the agent instead of Linux commands
   • templates built from disk images instead of Dockerfile/OCI rootfs builds
   • graphical console access (VNC/SPICE) if desktop/computer-use agents need it

Open question: memory model (phase 2). Upstream QEMU does not expose dirty RAM ranges outside migration. Its options each lose something Firecracker gives us today:

• x-ignore-shared requires share=on, which breaks copy-on-write for forks.
• mapped-ram (QEMU ≥ 9.0) restores eagerly.
• postcopy/userfaultfd adds complexity.

Incremental capture probably needs a small QEMU patch (about 0.5–1.5k lines of C) that we package the same way as our Firecracker fork. We should prototype QEMU memory restore from a ublk device before committing to the phase 2 design.

Open question: where macOS can run. Apple's macOS license allows macOS to run in a virtual machine only on Apple-branded hardware. AgentENV hosts are Linux with KVM, ublk and network namespaces. The options are:

• (a) Linux on Apple hardware, running QEMU/KVM. This is practical only on Intel Macs.
• (b) A separate backend on macOS hosts using Apple's Virtualization.framework. That is not QEMU, and the host has no ublk or netns, so storage and networking would need their own design.

This needs a decision, and legal review, before macOS work starts. It may become its own issue.

Compatibility and operational impact

API/config changes:

• New [qemu] section and a per-node backend selection. Firecracker stays the default.
• Templates need a guest OS attribute (Linux/Windows/macOS) so scheduling and the guest agent match the template.
• No E2B API change for Linux cold start. Snapshot operations on passthrough sandboxes return an explicit error.

Snapshot or storage format changes:

• Snapshots are VMM-specific: a QEMU capture can be restored only by QEMU. SandboxSnapshotManifest.backend already exists, so "qemu" is added to KNOWN_BACKENDS.
• Layer and repository formats are unchanged. Existing records default to "firecracker".

New host/runtime requirements:

• QEMU binary (possibly patched) and OVMF firmware shipped through config/deps_manifest.toml.
• For GPUs: IOMMU enabled, VFIO-bound devices, and runtime-user access to /dev/vfio.
• Windows guests: operators are responsible for Windows licensing and activation in their templates.
• macOS: Apple hardware (see the open question above).
• The QEMU backend is KVM-only; PVM is not supported.

Scheduler:

• Nodes advertise their backends, devices and supported guest OSes.
• Placement for create, restore and fork must match backend, guest OS and the required devices.

Upgrade and rollback considerations:

• Phase 0 must read existing paused states and snapshot configs unchanged.
• Rolling back to a release without QEMU leaves QEMU snapshots unrestorable; those releases should reject them explicitly rather than misinterpret them.

Alternatives considered

• Cloud Hypervisor: Rust, an API closer to Firecracker's, VFIO support, snapshot/restore and Windows guests. It does not support macOS guests, and its device ecosystem is narrower. QEMU is the one VMM that covers GPU, Windows and macOS. Cloud Hypervisor could still be evaluated for the Linux/GPU path in the phase 2 prototype.
• Kata Containers for GPU workloads: a separate stack. It doesn't integrate with AgentENV snapshots, volumes or scheduling, and doesn't cover desktop OSes.
• Extending Firecracker with PCI/VFIO and firmware boot: a large divergence from upstream that we would have to maintain alone.

Non-goals

• Replacing Firecracker or changing its defaults.
• Converting snapshots between VMMs.
• Snapshotting, pausing or forking sandboxes with passthrough devices (VFIO device state is generally not migratable).
• PVM support for QEMU.
• Running macOS guests on non-Apple hardware.
• A generic Vmm trait abstraction before a QEMU prototype shows where the shared boundary really is.

Contribution

I can implement it

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @yingdi-shan on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/390 **Affected component**: Orchestrator / sandbox lifecycle (also: Firecracker runtime, Snapshot / template builder, Gateway / scheduler) ### Problem statement We need AgentENV to provide GPU sandboxes and Windows and macOS sandboxes in addition to today's Linux CPU sandboxes. AgentENV is built only on Firecracker, which rules out both: - **No device passthrough.** Firecracker has no PCI or VFIO passthrough, so sandboxes cannot use GPUs (see #26, where the reply identified a QEMU backend as the necessary first step). - **Linux guests only.** Firecracker boots a Linux kernel directly, has no UEFI/BIOS firmware, and offers only a minimal virtio-mmio device model. It cannot boot Windows or macOS. The Firecracker dependency goes beyond the device model. Our capture path relies on our Firecracker fork's dirty-memory-range API, on a file-backed (MAP_PRIVATE) or uffd memory backend, and on MMDS for guest metadata. None of these has a direct QEMU equivalent. Several other parts of the stack also assume a Linux guest: - guest configuration through kernel boot args (drive layout, DAMON region) - the Linux envd guest agent - volume mounting through Linux commands in the guest - template builds that turn a Dockerfile/OCI image into a Linux rootfs ### Use case - **GPU sandboxes:** agent workloads that need a GPU in the sandbox (inference, CUDA builds, ML evaluation), scheduled on GPU nodes in the same cluster as CPU sandboxes. - **Windows sandboxes:** agents that build, test or operate Windows software, or use a Windows desktop. - **macOS sandboxes:** agents that build, test or operate macOS software, or use a macOS desktop. - In every case, through the same E2B-compatible API, scheduler, volumes and OverlayBD storage layering as Linux sandboxes, not a second stack. ### Current behavior and workarounds - `server.rs`, the orchestrator and the template runner construct `FirecrackerSandboxFactory` / `FirecrackerSandbox` directly. No other VMM can be selected (#381 adds a factory alias for a mock backend). - About 1.7k lines in `src/sandbox/firecracker/` do not depend on Firecracker at all: OverlayBD snapshot sealing and restacking, dirty-memory-to-OverlayBD conversion, `ProcessVmReader`, `SandboxResources`, startup prefetch on the ublk device, and the `MmdsMetadata` type. A second backend cannot reuse them where they are. - Workaround: run GPU, Windows and macOS sandboxes on separate stacks (Kata, plain VMs, or Mac hosting). That loses AgentENV's API, scheduling, snapshots, volumes and storage layering, and means operating several systems. ### Desired behavior - **Linux under QEMU:** Given a node configured with the QEMU backend, when a client creates a Linux sandbox through the existing API, then it boots under QEMU with the same rootfs (OverlayBD over ublk), networking (tap in netns), envd, volumes and custom-extension hooks as a Firecracker sandbox. - **GPU:** Given a QEMU sandbox with a GPU assigned, when the guest runs `nvidia-smi` (or the equivalent), then the device is visible and usable. - **Windows:** Given a Windows template, when a client creates a sandbox, then it boots under QEMU with UEFI and virtio drivers. Commands, files and process APIs work through the same API, backed by a Windows guest agent compatible with envd. - **macOS:** Given a macOS template on a host where running macOS is licensed (see the open question below), when a client creates a sandbox, then it boots and the same API works through a macOS guest agent. - **Snapshots without passthrough:** Given a QEMU sandbox (Linux or Windows) without passthrough devices, when it is paused/resumed or captured and launched, then this behaves as it does for Firecracker. The snapshot manifest records `backend = "qemu"`, and a restore is routed only to nodes that can run that backend. - **Snapshots with passthrough:** Given a sandbox with passthrough devices, when pause, capture or fork is requested, then the API rejects it with a clear error instead of producing a broken snapshot. - Firecracker behavior, configuration and stored snapshots are unchanged. ### Proposed approach The work is split into phases, each of which can be shipped and reviewed on its own. Line counts are rough estimates based on the current Firecracker backend (about 6.2k lines of code plus 3.2k lines of tests). They cover the AgentENV host side only and exclude the guest agent ports in phase 4. 0. **Remove Firecracker assumptions from shared code (refactor only, about 1–1.5k lines moved or changed)** - Move the VMM-neutral code out of `src/sandbox/firecracker/`: - `overlaybd_snapshot.rs` restacking and memory-layer building → `src/sandbox/ublk/` - `process_vm_reader.rs` - `resources.rs` → `src/sandbox/` - the local startup-prefetch half of `startup_pack.rs` - the `MmdsMetadata` type - drive and boot-argument helpers - Make `convert_dirty_memory_to_overlaybd` take a neutral dirty-range type instead of `firecracker_client::models::DirtyMemoryRanges`. - Split `FirecrackerCommonConfig` into a shared sandbox config and the Firecracker-only fields. Use `serde(flatten)` and add tests that persisted paused states and snapshot configs from before the split still load. - Replace the `FirecrackerCaptureArtifacts` downcast in `publish_captured` with a recording hook the backend supplies. - Rename the `FirecrackerManifest` and `firecracker_version` Rust identifiers, keeping their stored names. - Select the factory from configuration instead of hard-wiring `FirecrackerSandboxFactory` (coordinate with #381). 1. **Linux and GPU cold start (about 3k lines of code + 1–1.5k of tests)** - QMP client, QEMU `microvm`/`q35` command-line builder, process lifecycle and logs - virtio-blk on ublk devices; rate limits as throttle groups - tap networking - guest metadata delivery to replace MMDS: a host-side responder in the sandbox netns, or fw_cfg plus an envd change - `[qemu]` config and dependency/setup packaging - VFIO device assignment from node config 2. **Pause/resume and snapshots (about 3–4k lines of code + 1.5k of tests)** - VM state through QMP migration to a file - memory restore from the ublk memory device - rootfs and attached-drive sealing through the shared phase-0 code 3. **Parity (about 2–3k lines of code + 1–2k of tests)** - fork, incremental dirty-page capture, startup-pack recording, volume slot rebinding and freeze, warm process pool 4. **Windows and macOS guests (to be estimated after a guest-agent spike)** - UEFI firmware (OVMF); virtio drivers baked into Windows templates - guest metadata and network setup without kernel boot args (DHCP in the sandbox netns, or the guest agent) - Windows and macOS ports of the guest agent behind the envd API - volume mounting through the agent instead of Linux commands - templates built from disk images instead of Dockerfile/OCI rootfs builds - graphical console access (VNC/SPICE) if desktop/computer-use agents need it **Open question: memory model (phase 2).** Upstream QEMU does not expose dirty RAM ranges outside migration. Its options each lose something Firecracker gives us today: - `x-ignore-shared` requires `share=on`, which breaks copy-on-write for forks. - `mapped-ram` (QEMU ≥ 9.0) restores eagerly. - postcopy/userfaultfd adds complexity. Incremental capture probably needs a small QEMU patch (about 0.5–1.5k lines of C) that we package the same way as our Firecracker fork. We should prototype QEMU memory restore from a ublk device before committing to the phase 2 design. **Open question: where macOS can run.** Apple's macOS license allows macOS to run in a virtual machine only on Apple-branded hardware. AgentENV hosts are Linux with KVM, ublk and network namespaces. The options are: - (a) Linux on Apple hardware, running QEMU/KVM. This is practical only on Intel Macs. - (b) A separate backend on macOS hosts using Apple's Virtualization.framework. That is not QEMU, and the host has no ublk or netns, so storage and networking would need their own design. This needs a decision, and legal review, before macOS work starts. It may become its own issue. ### Compatibility and operational impact **API/config changes:** - New `[qemu]` section and a per-node backend selection. Firecracker stays the default. - Templates need a guest OS attribute (Linux/Windows/macOS) so scheduling and the guest agent match the template. - No E2B API change for Linux cold start. Snapshot operations on passthrough sandboxes return an explicit error. **Snapshot or storage format changes:** - Snapshots are VMM-specific: a QEMU capture can be restored only by QEMU. `SandboxSnapshotManifest.backend` already exists, so `"qemu"` is added to `KNOWN_BACKENDS`. - Layer and repository formats are unchanged. Existing records default to `"firecracker"`. **New host/runtime requirements:** - QEMU binary (possibly patched) and OVMF firmware shipped through `config/deps_manifest.toml`. - For GPUs: IOMMU enabled, VFIO-bound devices, and runtime-user access to `/dev/vfio`. - Windows guests: operators are responsible for Windows licensing and activation in their templates. - macOS: Apple hardware (see the open question above). - The QEMU backend is KVM-only; PVM is not supported. **Scheduler:** - Nodes advertise their backends, devices and supported guest OSes. - Placement for create, restore and fork must match `backend`, guest OS and the required devices. **Upgrade and rollback considerations:** - Phase 0 must read existing paused states and snapshot configs unchanged. - Rolling back to a release without QEMU leaves QEMU snapshots unrestorable; those releases should reject them explicitly rather than misinterpret them. ### Alternatives considered - **Cloud Hypervisor:** Rust, an API closer to Firecracker's, VFIO support, snapshot/restore and Windows guests. It does not support macOS guests, and its device ecosystem is narrower. QEMU is the one VMM that covers GPU, Windows and macOS. Cloud Hypervisor could still be evaluated for the Linux/GPU path in the phase 2 prototype. - **Kata Containers for GPU workloads:** a separate stack. It doesn't integrate with AgentENV snapshots, volumes or scheduling, and doesn't cover desktop OSes. - **Extending Firecracker with PCI/VFIO and firmware boot:** a large divergence from upstream that we would have to maintain alone. ### Non-goals - Replacing Firecracker or changing its defaults. - Converting snapshots between VMMs. - Snapshotting, pausing or forking sandboxes with passthrough devices (VFIO device state is generally not migratable). - PVM support for QEMU. - Running macOS guests on non-Apple hardware. - A generic `Vmm` trait abstraction before a QEMU prototype shows where the shared boundary really is. ### Contribution I can implement it ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:34 +00:00

bsakaguchi commented 2026-10-29 14:19:34 +00:00

Author

Owner

Copy link

@dushulin commented on GitHub (Oct 28, 2026):

@yingdi-shan hi, This is a capability that we very much need. Has there been any discussion on the technical solution?

<!-- gh-comment-id:6460297456 --> @dushulin commented on GitHub (Oct 28, 2026): @yingdi-shan hi, This is a capability that we very much need. Has there been any discussion on the technical solution?

bsakaguchi commented 2026-10-29 14:19:35 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Oct 28, 2026):

We are currently working on v0.3.0, which introduces a database to track relationships among layers, enabling online garbage collection of snapshot data. This work will follow once v0.3.0 is complete.

<!-- gh-comment-id:6474148086 --> @yingdi-shan commented on GitHub (Oct 28, 2026): We are currently working on v0.3.0, which introduces a database to track relationships among layers, enabling online garbage collection of snapshot data. This work will follow once v0.3.0 is complete.

bsakaguchi referenced this issue 2026-10-29 14:20:42 +00:00

[PR #100] [CLOSED] fix(orchestrator): restore pause state after allocation failure #178

bsakaguchi referenced this issue 2026-10-29 14:20:45 +00:00

[PR #108] [MERGED] Fix sandbox lifecycle bugs and improve network policy handling #183

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#99

No description provided.