[GH-ISSUE #381] (feature: mock-backend) Add a compile-time mock sandbox backend for KVM-less integration testing #97

New issue

Open

opened 2026-10-29 14:19:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:34 +00:00

Owner

Originally created by @tm-worker on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/381

Affected component: Orchestrator / sandbox lifecycle
Version: v0.2.3 (commit 4968423)

Problem statement

Every sandbox lifecycle operation requires /dev/kvm, so tests/integration.rs — the fc::*, process::* and ublk suites — cannot run on KVM-less CI runners (verified: 21 device-dependent failures on a KVM-less linux/arm64 container). The same limitation blocks any tooling that wants to exercise the agent's REST surface (lifecycle, snapshots, node metrics, data-plane proxy) outside a KVM host.

It also holds back end-to-end integration testing of the full stack: an E2E test that boots the agent together with the scheduler-plane / gateway components and drives a real sandbox lifecycle is impossible without provisioning a KVM host first — developers who just want to work on or verify the control plane have to set up virtualization before they can run anything.

Use case

CI pipelines (every push / PR): run the sandbox lifecycle, snapshot and node-metrics suites on ordinary container runners — no KVM-capable self-hosted fleet required.
Developer workstations (daily): anyone hacking on the orchestrator, snapshot manager, or the API/proxy face wants to exercise the real REST surface locally without provisioning a VM host.
Control-plane integration testing (per release): boot agent + gateway / scheduler components together and drive a real sandbox lifecycle end to end on any machine.

Current behavior and workarounds

Today every backend operation requires /dev/kvm and the host setup checks enforce it at startup. Workarounds and why they fall short:

Provision a KVM host / self-hosted runners: real hardware and fleet maintenance; still blocks laptops and borrowed CI containers.
Skip the device-dependent tests in CI: leaves the sandbox lifecycle unexercised on most runners — a coverage gap, not a workaround.
Hand-write an external fake for the REST API: duplicates the surface and drifts from the real orchestrator / snapshot / proxy code paths — it tests the fake, not the agent.

The in-process MockSandboxBackend / MockBackendFactory already cover exactly what's missing: they implement the full SandboxBackendFactory trait including build_from_snapshot and build_from_paused_state, and host_interaction_ip() is configurable so the agent's real /proxy forwarding code can dial a caller-provided target. The only blocker is #[cfg(test)] gating — the release build cannot reach them, and server.rs hard-wires FirecrackerSandboxFactory.

Desired behavior

Feature off ⇒ production builds byte-identical.
Feature on ⇒ full lifecycle + snapshot capture/publish + node metrics exercised end-to-end on the real REST/orchestrator/snapshot/proxy paths, no /dev/kvm.
E2E: the agent (mock backend) together with the gateway / scheduler-plane services boots and drives the sandbox lifecycle on kind or any KVM-less machine — a deployment manifest for the mock image ships with the PR.
tests/integration.rs runnable on KVM-less CI runners for everything except the device-dependent tests (unchanged).

Proposed approach

A small, inert-by-default test-support patch (13 files, ~+477/-18, verified on linux/arm64):

mock-backend cargo feature. With the feature off, production builds are byte-identical (cargo check verified both ways).
Feature-gate sandbox/mock (stays #[cfg(test)]-compatible) plus two existing test-support constructors that the mock path needs (SandboxSnapshotManifest::for_test, SnapshotPublishMetadata::mock).
DefaultBackendFactory type alias for Orchestrator's default F: flips to the mock under the feature; every hardcoded Arc<Orchestrator> reference (ObservabilityService, ApiImpl) follows via one switch.
server-mock bin: the production server.rs wiring copied verbatim with four diffs — mock factory (data-plane IP from AENV_MOCK_HOST_IP, so /proxy dials a caller-provided target through the agent's REAL forwarding code), VM-capability/env checks skipped, template seeding via the real publish API (AENV_MOCK_TEMPLATE_IDS; an empty repository 400s before the factory ever runs), shutdown blocks adjusted.
Mock capture fidelity: snapshot() / capture_to_dir write a real artifact set (the same writer the snapshot-manager tests seed with) instead of a dangling for_test manifest — the capture → publish → load_runnable loop stays on real code.
Dockerfile.agentenv-mock producing an agentenv-runtime-mock image (structurally the production image; the mock needs no KVM/ublk).

Observed behavior (evidence, KVM-less linux/arm64 container)

create → 201 (seeded template → real load_runnable → mock backend → real envd access token), status/pause → 204 (idempotent), connect → 201/200, delete → 204
snapshot → 201 through the REAL capture → publish chain; listed via GET /snapshots
GET /nodes returns live NodeMetrics (allocated CPU/memory counts the mock sandboxes)
Regression evidence: the library's unit-test suite — 962 tests covering orchestrator lifecycle, the snapshot manager, the API face and sandbox-backend contracts — passes identically with the feature enabled and disabled. Under the feature, the API-level tests run against the mock backend through the swapped default factory, i.e. the existing REST contract tests gain KVM-less coverage for free. (tests/integration.rs device-dependent failures are pre-existing and unrelated.)

Compatibility and operational impact

API/config changes: two feature-gated pub items, one type alias, one new bin (server-mock) and one cargo feature (mock-backend) — all inert with the feature off; no production API changes.
Snapshot or storage format changes: none in production. The mock backend's capture path writes test-support artifacts (the same writer the snapshot-manager tests seed); the production capture → publish → load_runnable chain is untouched.
New host/runtime requirements: none added — mock mode performs no KVM/ublk/network setup.
Upgrade and rollback: disable the feature; no production code path changes.

Alternatives considered

KVM-capable self-hosted CI runners: solves CI only, at real hardware and maintenance cost, and still blocks local development and ad-hoc tooling.
Software emulation (qemu/TCG) instead of KVM: not viable — Firecracker hard-requires /dev/kvm; there is no TCG path to fall back to.
An external hand-written fake server replicating the REST API: this is the status quo pain — it drifts from the real orchestrator/snapshot/proxy code and validates the fake rather than the agent. The proposal reuses the real stack with only the VM backend swapped.
Keep tests skipped on KVM-less runners: permanent coverage gap for every contributor without a VM host.

Non-goals

Not a production sandbox backend; the mock mode is for tests and local/integration environments only.
No KVM emulation or VM-interpreter work.
No changes to production lifecycle, snapshot/storage formats, or host setup requirements.
Not a replacement for the real integration tests on KVM hosts — those keep running where /dev/kvm exists.

Contribution

I can implement it — a local branch with the full change is ready and verified (see Observed behavior); happy to open focused PRs (feature gate + bin, and the capture-fidelity fix, as separate commits/PRs if preferred) once the approach is aligned.

Originally created by @tm-worker on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/381 **Affected component**: Orchestrator / sandbox lifecycle **Version**: v0.2.3 (commit 4968423) ### Problem statement Every sandbox lifecycle operation requires `/dev/kvm`, so `tests/integration.rs` — the `fc::*`, `process::*` and `ublk` suites — cannot run on KVM-less CI runners (verified: 21 device-dependent failures on a KVM-less linux/arm64 container). The same limitation blocks any tooling that wants to exercise the agent's REST surface (lifecycle, snapshots, node metrics, data-plane proxy) outside a KVM host. It also holds back **end-to-end integration testing of the full stack**: an E2E test that boots the agent together with the scheduler-plane / gateway components and drives a real sandbox lifecycle is impossible without provisioning a KVM host first — developers who just want to work on or verify the control plane have to set up virtualization before they can run anything. ### Use case - **CI pipelines (every push / PR)**: run the sandbox lifecycle, snapshot and node-metrics suites on ordinary container runners — no KVM-capable self-hosted fleet required. - **Developer workstations (daily)**: anyone hacking on the orchestrator, snapshot manager, or the API/proxy face wants to exercise the real REST surface locally without provisioning a VM host. - **Control-plane integration testing (per release)**: boot agent + gateway / scheduler components together and drive a real sandbox lifecycle end to end on any machine. ### Current behavior and workarounds Today every backend operation requires `/dev/kvm` and the host setup checks enforce it at startup. Workarounds and why they fall short: - **Provision a KVM host / self-hosted runners**: real hardware and fleet maintenance; still blocks laptops and borrowed CI containers. - **Skip the device-dependent tests in CI**: leaves the sandbox lifecycle unexercised on most runners — a coverage gap, not a workaround. - **Hand-write an external fake for the REST API**: duplicates the surface and drifts from the real orchestrator / snapshot / proxy code paths — it tests the fake, not the agent. The in-process `MockSandboxBackend` / `MockBackendFactory` already cover exactly what's missing: they implement the full `SandboxBackendFactory` trait including `build_from_snapshot` and `build_from_paused_state`, and `host_interaction_ip()` is configurable so the agent's **real** `/proxy` forwarding code can dial a caller-provided target. The only blocker is `#[cfg(test)]` gating — the release build cannot reach them, and `server.rs` hard-wires `FirecrackerSandboxFactory`. ### Desired behavior - Feature off ⇒ production builds byte-identical. - Feature on ⇒ full lifecycle + snapshot capture/publish + node metrics exercised end-to-end on the real REST/orchestrator/snapshot/proxy paths, no `/dev/kvm`. - E2E: the agent (mock backend) together with the gateway / scheduler-plane services boots and drives the sandbox lifecycle on kind or any KVM-less machine — a deployment manifest for the mock image ships with the PR. - `tests/integration.rs` runnable on KVM-less CI runners for everything except the device-dependent tests (unchanged). ### Proposed approach A small, inert-by-default test-support patch (13 files, ~+477/-18, verified on linux/arm64): 1. `mock-backend` cargo feature. With the feature off, production builds are byte-identical (`cargo check` verified both ways). 2. Feature-gate `sandbox/mock` (stays `#[cfg(test)]`-compatible) plus two existing test-support constructors that the mock path needs (`SandboxSnapshotManifest::for_test`, `SnapshotPublishMetadata::mock`). 3. `DefaultBackendFactory` type alias for `Orchestrator`'s default `F`: flips to the mock under the feature; every hardcoded `Arc<Orchestrator>` reference (ObservabilityService, ApiImpl) follows via one switch. 4. `server-mock` bin: the production `server.rs` wiring copied verbatim with four diffs — mock factory (data-plane IP from `AENV_MOCK_HOST_IP`, so `/proxy` dials a caller-provided target through the agent's REAL forwarding code), VM-capability/env checks skipped, template seeding via the real publish API (`AENV_MOCK_TEMPLATE_IDS`; an empty repository 400s before the factory ever runs), shutdown blocks adjusted. 5. Mock capture fidelity: `snapshot()` / `capture_to_dir` write a real artifact set (the same writer the snapshot-manager tests seed with) instead of a dangling `for_test` manifest — the capture → publish → load_runnable loop stays on real code. 6. `Dockerfile.agentenv-mock` producing an `agentenv-runtime-mock` image (structurally the production image; the mock needs no KVM/ublk). ### Observed behavior (evidence, KVM-less linux/arm64 container) - create → 201 (seeded template → real `load_runnable` → mock backend → real envd access token), status/pause → 204 (idempotent), connect → 201/200, delete → 204 - snapshot → 201 through the REAL capture → publish chain; listed via `GET /snapshots` - `GET /nodes` returns live `NodeMetrics` (allocated CPU/memory counts the mock sandboxes) - Regression evidence: the library's unit-test suite — **962 tests covering orchestrator lifecycle, the snapshot manager, the API face and sandbox-backend contracts** — passes identically with the feature **enabled and disabled**. Under the feature, the API-level tests run against the mock backend through the swapped default factory, i.e. the existing REST contract tests gain KVM-less coverage for free. (`tests/integration.rs` device-dependent failures are pre-existing and unrelated.) ### Compatibility and operational impact - **API/config changes**: two feature-gated `pub` items, one type alias, one new bin (`server-mock`) and one cargo feature (`mock-backend`) — all inert with the feature off; no production API changes. - **Snapshot or storage format changes**: none in production. The mock backend's capture path writes test-support artifacts (the same writer the snapshot-manager tests seed); the production capture → publish → load_runnable chain is untouched. - **New host/runtime requirements**: none added — mock mode performs no KVM/ublk/network setup. - **Upgrade and rollback**: disable the feature; no production code path changes. ### Alternatives considered - **KVM-capable self-hosted CI runners**: solves CI only, at real hardware and maintenance cost, and still blocks local development and ad-hoc tooling. - **Software emulation (qemu/TCG) instead of KVM**: not viable — Firecracker hard-requires `/dev/kvm`; there is no TCG path to fall back to. - **An external hand-written fake server replicating the REST API**: this is the status quo pain — it drifts from the real orchestrator/snapshot/proxy code and validates the fake rather than the agent. The proposal reuses the real stack with only the VM backend swapped. - **Keep tests skipped on KVM-less runners**: permanent coverage gap for every contributor without a VM host. ### Non-goals - Not a production sandbox backend; the mock mode is for tests and local/integration environments only. - No KVM emulation or VM-interpreter work. - No changes to production lifecycle, snapshot/storage formats, or host setup requirements. - Not a replacement for the real integration tests on KVM hosts — those keep running where `/dev/kvm` exists. ### Contribution **I can implement it** — a local branch with the full change is ready and verified (see Observed behavior); happy to open focused PRs (feature gate + bin, and the capture-fidelity fix, as separate commits/PRs if preferred) once the approach is aligned.

bsakaguchi referenced this issue

2026-10-29 14:20:40 +00:00

[PR #98] fix(overlaybd): validate index bounds before allocation #177