kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #101] [RFC] Exploring an end-to-end ARM64 support path for AgentENV: findings from a validated DGX Spark deployment #35

New issue

Closed

opened 2026-10-29 14:18:38 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:18:38 +00:00

Owner

Originally created by @leether on GitHub (Jul 31, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/101

Affected component

Cross-cutting / other

RFC intent and possible closure

This is a request for design feedback, not a request to declare ARM64 supported
or to merge one large implementation. A useful outcome could be any of the
following:

ARM64 server/runtime support is currently out of scope; the issue can close
with that position recorded.
The project is open to an initial support tier; the issue can close after
that tier and its compatibility boundary are recorded and focused follow-up
issues or PRs are identified.
The project needs more evidence before choosing a direction; the issue can
close or be deferred with the missing validation and revisit conditions
recorded.

Problem statement

This RFC asks whether ARM64 server/runtime support is within AgentENV's scope
and, if so, what initial level of support the project might intend to provide
and test.

A server that compiles on aarch64 is an important milestone, but it does not
by itself prove that the selected Firecracker build, guest kernel, tools drive,
ublk/OverlayBD path, seccomp policy, and snapshot lifecycle work together.

There are two adjacent open contributions as of 2026-07-31:

#86 proposes ARM64 release
artifacts, installer selection, and multi-architecture container publishing.
#94 proposes an ARM64
source-build path, architecture-aware Firecracker and guest-kernel downloads,
ARM tools-drive construction, and openEuler recognition.

This issue is not a replacement for either PR. It first asks whether the
project would like to pursue ARM64 server/runtime support. If the answer is
yes, this case study may help distinguish a build or installation milestone
from an end-to-end supported runtime.

We independently assembled and operated one complete AgentENV stack on an
NVIDIA DGX Spark. This work did not use or validate #86 or #94. It demonstrates
that one full ARM64 combination is feasible, but it does not prove that every
ARM64 host, kernel, or distribution is compatible.

Item	What it covers	What remains outside its stated validation
#60	General request for ARM support	The intended support boundary and acceptance criteria
#63	Template build failure caused by using an incompatible upstream Firecracker binary	A complete compatible dependency set and snapshot regression
#86	Prebuilt ARM server release, installer selection, multi-arch container	End-to-end runtime behavior is outside its stated scope
#94	ARM source build, architecture-aware dependencies, tools drive, openEuler	Prebuilt ARM server and multi-arch publishing are excluded; native ARM unit and integration tests were not run
This issue	One independent native ARM runtime case and a possible staged path	It does not request adoption of one downstream implementation

One dependency observation may be relevant when reviewing those paths. At the
current #94 head (3134f27), its manifest selects
firecracker-1.15.1-patch-v1-aarch64 and vmlinux-6.1.175-aarch64 from the
aenv-deps
release. That release tag resolves to source commit
90288c3,
whose serial restore path predates
#5764 and does
not save and restore the full UART device state.

This is a source/provenance comparison, not a claim that I executed #94's
published binary or reproduced a failure through #94. It is evidence that an
ARM snapshot baseline should be selected and tested explicitly instead of
being inferred from an asset name.

Use case

The concrete case is a single-node AgentENV server on a native ARM64 host. If
the project later chooses to pursue an end-to-end ARM runtime tier, a supported
profile would need to:

install or build every host and guest runtime artifact for the same
architecture;
resolve an OCI image through the declared OverlayBD/ublk storage tier;
build a template and restore sandboxes from its snapshot;
sustain repeated envd RPCs and console output after restore;
pause and resume a sandbox;
recover envd after a crash or SIGKILL; and
reject or deliberately migrate an incompatible snapshot before starting
Firecracker.

The independently validated profile was:

Layer	Validated value
Host hardware	NVIDIA DGX Spark, bare metal, 20 ARM CPUs (`Cortex-X925` / `Cortex-A725`), 4 KiB pages
Host OS	Ubuntu 24.04, Linux `6.17.0-1021-nvidia`, `aarch64`
Host capabilities	`/dev/kvm` present; `CONFIG_BLK_DEV_UBLK=m`; real `uvm-ublk-daemon` and OverlayBD data path
AgentENV assembly	`0.1.0-arm64.11`
AgentENV validated production source	`d79fd2d`
Public reference branch	`leether/arm64-assembly-line`
Firecracker base	`kvcache-ai/firecracker-next` `v1.17.0-next.1` / `fee2681`
Firecracker exact build	`b792de1`, `aarch64-unknown-linux-gnu`
Firecracker runtime ID	`fc-v1.17.0-next.1+agentenv.arm64.1.b792de1`
Firecracker release	`agentenv-v1.17.0-next.1-arm64.1`
Guest kernel	Ubuntu/NVIDIA `6.17.0-1021-nvidia` ARM64 kernel image
Tools drive	downstream `0.1.0-arm64.3`
`envd`	`0.6.13` / `bfdbb24`

This is one validated combination, not a proposal that AgentENV adopt these
exact versions.

Current behavior and workarounds

End-to-end runtime evidence

The public reference branch contains a native ARM regression that deliberately:

keeps UART TX active while taking a snapshot;
checks that envd stderr is not /dev/console;
writes 16 KiB directly to /dev/console after restore, so log redirection
cannot hide a broken virtual UART;
completes 100 consecutive envd RPCs after restore; and
kills envd and verifies supervisor recovery without recreating the sandbox.

The automated gate is documented in
arm64-assembly.md
and implemented in
uart_snapshot.rs.

Separate production-promotion observations on this single DGX host recorded
three snapshot-based starts at 90 ms, 79 ms, and 71 ms; 300/300 repeated execs;
two 16 KiB console-write checks; pause/resume; and envd supervisor recovery.
These are operator-recorded observations, not additional iterations encoded in
the public automated test.

Preliminary single-host performance context

End-to-end CLI pilot

We also ran a small warm-cache pilot on the same DGX host, using the same 2
vCPU / 1024 MiB sandbox size. Snapshot starts used the default alpine
template; cold starts used localhost:5000/alpine:3.20 from the local registry.
The modes were interleaved, and no vLLM requests were running during the
measurement.

End-to-end CLI measurement	Snapshot (n=5)	Cold start (n=5)
Successful create, first exec, and delete	5/5	5/5
`start -d` p50	105.7 ms	541.7 ms
`start -d` range	101.0–287.5 ms	531.8–714.5 ms
Start to first successful exec p50	179.9 ms	613.0 ms

Within this one host and measurement boundary, snapshot start p50 was about
5.1x faster than cold start, and time to first successful exec was about 3.4x
faster. Both modes had a slower first iteration, so this sample is too small to
claim p95 or p99. A publishable benchmark should first run five warm-up
iterations per mode and then measure at least 100 iterations.

This pilot reports only two end-to-end measurements: start -d, and start to
first successful exec. Those values include CLI, API, orchestration, and
readiness overhead and are not used below to judge the README latency targets.

Repository snapshot benchmark

AgentENV's current
README
states that snapshot-backed environments boot or resume in under 50 ms, pause
in under 100 ms, and create incremental snapshots in under 100 ms even under
heavy disk modification. The same repository now contains a direct backend
snapshot_benchmark.rs
with explicit operation boundaries. The README does not say that this file
produced the published numbers, so this is a reproduction of the repository's
closest code-defined benchmark, not a claim about the maintainers' original
benchmark environment.

We copied that benchmark file unchanged at commit a33198e and verified the
source SHA256 (abc15b944e54639774868a43a4ebf46c52b8fe62309878de4f98418565d6aa6b)
before building it against the ARM reference branch. The timed operations,
1-vCPU setting, base 128 MiB guest, and heavy-data preparation were unchanged.
Firecracker, the guest kernel, tools drive, and envd were the same artifacts as
the validated production assembly. The benchmark used an isolated AgentENV
home, runtime directory, ublk daemon, image cache, network CIDRs, and
ubuntu:24.04 rootfs. Other production services remained running.

We then ran the repository's full Criterion mode unchanged: 3 seconds of
warm-up, a 20-second target measurement window, and 10 samples per benchmark.
Criterion extended several windows when necessary. Its output reports
[lower bound, point estimate, upper bound]:

Code-defined backend measurement	README target	Criterion result	Observation
Hot `snapshot_resume`	<50 ms	[78.530, 82.124, 85.371] ms	Target not reproduced
`snapshot_creation` (`sandbox.pause()`)	<100 ms	[47.602, 52.598, 58.493] ms	Under target
`snapshot_creation_1gdisk` (1 GiB direct write, `sync`, drop caches)	<100 ms under heavy disk modification	[47.376, 51.445, 55.950] ms	Under target
Cold `snapshot_resume`	No separate README threshold	[103.61, 107.66, 111.47] ms	Additional context
`snapshot_creation_1gmem`	README claim is about disk modification	[272.64, 302.17, 333.07] ms	Additional context
50-way `concurrent_resume`	No separate README threshold	[317.68, 324.41, 331.88] ms	Additional context

To check repeatability outside Criterion's aggregation, we launched three new
benchmark processes and retained all 30 bounded samples for both cold and hot
resume. Each round produced exactly one roughly 30-second stall: cold sample 6
and hot sample 7. Across all samples, cold resume had median 106.2 ms and hot
resume median 80.9 ms, but each had a 3/30 stall rate. The 27 sub-second cold
samples were 91.70–118.71 ms (mean 103.51 ms); the 27 sub-second hot samples
were 61.55–85.06 ms (mean 76.95 ms). The retained stalls were
30.91–31.04 seconds cold and 30.52–30.54 seconds hot. Their duration is close
to the benchmark's explicit 30-second socket timeout, but the current logs do
not establish which wait consumed it; that is correlation, not a root-cause
claim. Criterion's summarized interval did not expose this periodic stall, so
both the central estimate and the bounded raw-sample behavior are reported.

The benchmark times FirecrackerSandbox::resume_from_snapshot_config() and
sandbox.pause() directly. It excludes CLI/API/orchestrator latency and does
not separately time user-facing persistent snapshot publication or object
storage. The file also has no separate snapshot-boot case; hot and cold resume
are the closest code-defined paths. Within that exact backend boundary, the
base and heavy-disk capture paths met the two numerical under-100-ms thresholds,
while hot resume did not meet the under-50-ms threshold. This is code-path
evidence, not a full reproduction of every README lifecycle boundary. It is
also not an ARM-versus-x86 comparison; that requires the same commit, artifacts,
runner, and background-load policy on a comparable x86 host.

Compatibility lessons from the bring-up

AgentENV needs its patched Firecracker API. A same-numbered upstream
binary may run but reject AgentENV's downstream drive fields such as
direct, as #63 demonstrates.
Snapshot behavior has a minimum baseline. Our earlier
v1.15.1-patch-v1-lineage assembly reproduced a deterministic UART TX stall
after snapshot restore. The behavior matches the bug fixed by Firecracker
#5764 and disappeared after moving to a v1.17.0-next.1 lineage containing
that fix. We did not perform a single-commit bisect, so this should be read
as a strong compatibility finding, not proof that only one commit differed.
A guest workaround can mask a device bug. Keeping envd logs away from
/dev/console prevents the original HTTP 504 symptom, but it does not prove
that UART TX survived restore. The regression therefore exercises the
console device directly.
The build target and seccomp policy are coupled. The validated
Firecracker is an ARM GNU build. Reusing the ARM musl policy unchanged did
not cover libc-specific signal and process behavior; an empty fallback would
silently weaken isolation. The release therefore carries a reviewed,
default-deny ARM GNU policy and fails if it is missing.
The guest kernel is part of the product. Our selected distribution
kernel has CONFIG_IP_PNP disabled, so the tools init applies the final
AgentENV-injected ip= argument in userspace. The 6.1 ARM reference config
in kvcache-ai/firecracker enables CONFIG_IP_PNP. Either approach can be
made to work, but the required capability or fallback needs to be explicit.
Snapshot identity is larger than a version string. A downstream runtime
ID is useful provenance, but it is not by itself a complete compatibility
key. ARM restore compatibility can also depend on CPU features, GIC version,
host/KVM behavior, Firecracker snapshot format, guest kernel, and guest tools.
Restoring multiple clones also duplicates guest userspace state. Firecracker's
VMGenID helps the kernel RNG only when the ARM guest has DeviceTree VMGenID
support (upstream Linux added it in 6.10; a 6.1 guest needs a backport), and
it does not generically rotate userspace tokens, machine identity, or cached
random state.

Desired behavior

If the project chooses to pursue ARM64 server/runtime support, a possible
outcome would be a documented set of support tiers with observable acceptance
criteria:

Build tier: the server and tools drive compile for ARM64, and selected
artifacts are architecture-correct and checksummed.
Install tier: a declared host profile can install or provision the
complete dependency set with preflight checks and a reproducible BOM.
Runtime tier: native ARM KVM tests cover template creation,
OverlayBD/ublk attachment, active-UART snapshot restore, repeated envd
execution, pause/resume, and supervisor recovery.
Release tier: the project publishes supported profiles, exact dependency
provenance, security policy, upgrade/rebuild rules, rollback guidance, and
downloadable test evidence.

Disabling ublk may be a valid reduced profile, as #94 proposes for openEuler
kernels without CONFIG_BLK_DEV_UBLK, but the documentation should identify
which snapshot, on-demand-loading, and OverlayBD guarantees are then
unavailable. A build-only profile and an end-to-end runtime profile should not
share the same unqualified support label.

Proposed approach

Would maintainers be open to discussing whether ARM64 server/runtime support
belongs in AgentENV's roadmap? If so, the following questions may help define a
staged path:

Project scope: Is ARM64 server/runtime support something the project
would like to pursue? If so, should the initial goal be build-only, source
installation, prebuilt releases, or an end-to-end supported runtime? Which
tier would #86 or #94 satisfy if merged?
Reference profiles and preflight: Which host distributions, kernel/KVM
versions, CPU/GIC combinations, page sizes, cgroup mode, and ublk capability
should be supported or checked before host setup?
Versioned dependency contract: What are the canonical source and release
coordinates for Firecracker, guest kernel, tools/envd, ublk daemon, and
OverlayBD? Who would own downstream patches, upstream synchronization,
provenance, and end-of-life?
Guest lifecycle contract: Which kernel options should be required, when
should a userspace ip= fallback be supported, and how should ARM snapshot
clones handle VMGenID, entropy, boot_id, machine identity, and
application-level unique state before user code runs?
Snapshot compatibility: Which hardware and software fields should form
the compatibility key? How should placement avoid incompatible hosts, and
should legacy or mismatched snapshots fail closed, warn, or use an explicitly
tested compatibility map?
Build and isolation targets: Should the canonical ARM64 Firecracker
target be GNU, musl, or both? Should production startup fail closed when a
matching default-deny seccomp policy is absent? What checksum, signing, and
SBOM model should release artifacts use?
Native validation: If the project pursues a runtime or release tier,
where could KVM + ublk ARM tests run, and which results should block release?
Suggested coverage includes clean install, real OverlayBD/ublk I/O, template
build, active-UART restore, repeated envd RPCs, pause/resume,
clone-uniqueness checks, snapshot mismatch, missing seccomp policy, and a
clearly defined latency benchmark.

If maintainers consider ARM64 support in scope and indicate an initial tier,
the public reference can be split into focused contributions. We can provide
the tested case, help validate #86/#94 on DGX Spark, and implement or review
agreed pieces; this issue does not ask for one large downstream merge.

Compatibility and operational impact

API/config changes:

Explicit support tiers may require capability reporting and preflight output.
A snapshot compatibility record may require new metadata fields beyond one
Firecracker runtime ID.

Snapshot or storage format changes:

Legacy snapshots need an explicit reject, migrate, or compatibility-map
policy.
Clone-uniqueness handling may require a post-restore guest lifecycle hook.

New host/runtime requirements:

Native ARM64 KVM for a runtime tier.
ublk/OverlayBD only for tiers that promise those storage features.
A guest kernel and seccomp policy matching the declared profile.

Upgrade and rollback considerations:

Changing Firecracker snapshot format, guest kernel, or tools drive may require
template rebuilds.
Canary promotion and rollback are useful reference operational practices,
but need not be ARM-specific acceptance criteria for an initial source-build
tier.

Alternatives considered

Comment only on #94. That would mix an independent runtime case with
review of one source-build implementation and could be misread as validation
of that PR.
Comment only on #86. The independent assembly did not use #86, and #86's
release scope does not define runtime compatibility.
Report only the UART incident. Firecracker already fixed the device bug;
the remaining AgentENV question is dependency selection and regression
ownership.
Publish only the downstream recipe. One working BOM does not define
portable support tiers, snapshot placement, clone uniqueness, or lifecycle
policy.

Non-goals

This issue does not claim that #86 or #94 has been validated on DGX Spark.
It does not assume that AgentENV has already committed to supporting ARM64.
It does not claim that one DGX Spark validates all ARM CPUs, GIC versions,
kernels, page sizes, or distributions.
It does not present the five-run CLI pilot, one Criterion run, or 30 bounded
resume samples as a cross-architecture or production tail-latency benchmark.
It does not assume that the repository benchmark is the undisclosed original
methodology behind the README numbers; it reports the exact code boundary
and commit used for comparison.
It does not require AgentENV to adopt the downstream kernel, exact version
set, release scripts, or runtime-ID implementation unchanged.
It does not reopen the Firecracker UART bug fixed in #5764.
It does not make canary promotion and rollback prerequisites for accepting an
ARM source-build contribution.

Contribution

I can help test or review it.

Pre-submission checklist

I searched existing issues and discussions. #60, #63, #86, and #94 are
adjacent work, but none currently defines the possible end-to-end ARM64
runtime support path discussed here.
I described a concrete problem and use case, not only a proposed
technology.
I understand that acceptance of the problem does not imply approval of a
particular implementation.

Originally created by @leether on GitHub (Jul 31, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/101 ### Affected component Cross-cutting / other ### RFC intent and possible closure This is a request for design feedback, not a request to declare ARM64 supported or to merge one large implementation. A useful outcome could be any of the following: 1. ARM64 server/runtime support is currently out of scope; the issue can close with that position recorded. 2. The project is open to an initial support tier; the issue can close after that tier and its compatibility boundary are recorded and focused follow-up issues or PRs are identified. 3. The project needs more evidence before choosing a direction; the issue can close or be deferred with the missing validation and revisit conditions recorded. ### Problem statement This RFC asks whether ARM64 server/runtime support is within AgentENV's scope and, if so, what initial level of support the project might intend to provide and test. A server that compiles on `aarch64` is an important milestone, but it does not by itself prove that the selected Firecracker build, guest kernel, tools drive, ublk/OverlayBD path, seccomp policy, and snapshot lifecycle work together. There are two adjacent open contributions as of 2026-07-31: - [#86](https://github.com/kvcache-ai/AgentENV/pull/86) proposes ARM64 release artifacts, installer selection, and multi-architecture container publishing. - [#94](https://github.com/kvcache-ai/AgentENV/pull/94) proposes an ARM64 source-build path, architecture-aware Firecracker and guest-kernel downloads, ARM tools-drive construction, and openEuler recognition. This issue is not a replacement for either PR. It first asks whether the project would like to pursue ARM64 server/runtime support. If the answer is yes, this case study may help distinguish a build or installation milestone from an end-to-end supported runtime. We independently assembled and operated one complete AgentENV stack on an NVIDIA DGX Spark. This work did not use or validate #86 or #94. It demonstrates that one full ARM64 combination is feasible, but it does not prove that every ARM64 host, kernel, or distribution is compatible. ### Related work and scope | Item | What it covers | What remains outside its stated validation | | --- | --- | --- | | [#60](https://github.com/kvcache-ai/AgentENV/issues/60) | General request for ARM support | The intended support boundary and acceptance criteria | | [#63](https://github.com/kvcache-ai/AgentENV/issues/63) | Template build failure caused by using an incompatible upstream Firecracker binary | A complete compatible dependency set and snapshot regression | | [#86](https://github.com/kvcache-ai/AgentENV/pull/86) | Prebuilt ARM server release, installer selection, multi-arch container | End-to-end runtime behavior is outside its stated scope | | [#94](https://github.com/kvcache-ai/AgentENV/pull/94) | ARM source build, architecture-aware dependencies, tools drive, openEuler | Prebuilt ARM server and multi-arch publishing are excluded; native ARM unit and integration tests were not run | | This issue | One independent native ARM runtime case and a possible staged path | It does not request adoption of one downstream implementation | One dependency observation may be relevant when reviewing those paths. At the current #94 head (`3134f27`), its manifest selects `firecracker-1.15.1-patch-v1-aarch64` and `vmlinux-6.1.175-aarch64` from the [`aenv-deps`](https://github.com/kvcache-ai/firecracker/releases/tag/aenv-deps) release. That release tag resolves to source commit [`90288c3`](https://github.com/kvcache-ai/firecracker/commit/90288c397d6d3fa576f9d367d997bdeebf835f10), whose serial restore path predates [#5764](https://github.com/firecracker-microvm/firecracker/pull/5764) and does not save and restore the full UART device state. This is a source/provenance comparison, not a claim that I executed #94's published binary or reproduced a failure through #94. It is evidence that an ARM snapshot baseline should be selected and tested explicitly instead of being inferred from an asset name. ### Use case The concrete case is a single-node AgentENV server on a native ARM64 host. If the project later chooses to pursue an end-to-end ARM runtime tier, a supported profile would need to: 1. install or build every host and guest runtime artifact for the same architecture; 2. resolve an OCI image through the declared OverlayBD/ublk storage tier; 3. build a template and restore sandboxes from its snapshot; 4. sustain repeated `envd` RPCs and console output after restore; 5. pause and resume a sandbox; 6. recover `envd` after a crash or `SIGKILL`; and 7. reject or deliberately migrate an incompatible snapshot before starting Firecracker. The independently validated profile was: | Layer | Validated value | | --- | --- | | Host hardware | NVIDIA DGX Spark, bare metal, 20 ARM CPUs (`Cortex-X925` / `Cortex-A725`), 4 KiB pages | | Host OS | Ubuntu 24.04, Linux `6.17.0-1021-nvidia`, `aarch64` | | Host capabilities | `/dev/kvm` present; `CONFIG_BLK_DEV_UBLK=m`; real `uvm-ublk-daemon` and OverlayBD data path | | AgentENV assembly | `0.1.0-arm64.11` | | AgentENV validated production source | [`d79fd2d`](https://github.com/leether/AgentENV/commit/d79fd2d9e591e941eef5e51553fc154aed4446d8) | | Public reference branch | [`leether/arm64-assembly-line`](https://github.com/leether/AgentENV/tree/leether/arm64-assembly-line) | | Firecracker base | `kvcache-ai/firecracker-next` `v1.17.0-next.1` / `fee2681` | | Firecracker exact build | [`b792de1`](https://github.com/leether/firecracker-next/commit/b792de1c4618092b0d712c59e5d2a195c97ff5bd), `aarch64-unknown-linux-gnu` | | Firecracker runtime ID | `fc-v1.17.0-next.1+agentenv.arm64.1.b792de1` | | Firecracker release | [`agentenv-v1.17.0-next.1-arm64.1`](https://github.com/leether/firecracker-next/releases/tag/agentenv-v1.17.0-next.1-arm64.1) | | Guest kernel | Ubuntu/NVIDIA `6.17.0-1021-nvidia` ARM64 kernel image | | Tools drive | downstream `0.1.0-arm64.3` | | `envd` | `0.6.13` / `bfdbb24` | This is one validated combination, not a proposal that AgentENV adopt these exact versions. ### Current behavior and workarounds #### End-to-end runtime evidence The public reference branch contains a native ARM regression that deliberately: - keeps UART TX active while taking a snapshot; - checks that `envd` stderr is not `/dev/console`; - writes 16 KiB directly to `/dev/console` after restore, so log redirection cannot hide a broken virtual UART; - completes 100 consecutive `envd` RPCs after restore; and - kills `envd` and verifies supervisor recovery without recreating the sandbox. The automated gate is documented in [`arm64-assembly.md`](https://github.com/leether/AgentENV/blob/leether/arm64-assembly-line/docs/src/internals/arm64-assembly.md) and implemented in [`uart_snapshot.rs`](https://github.com/leether/AgentENV/blob/leether/arm64-assembly-line/tests/integration/uart_snapshot.rs). Separate production-promotion observations on this single DGX host recorded three snapshot-based starts at 90 ms, 79 ms, and 71 ms; 300/300 repeated execs; two 16 KiB console-write checks; pause/resume; and `envd` supervisor recovery. These are operator-recorded observations, not additional iterations encoded in the public automated test. #### Preliminary single-host performance context ##### End-to-end CLI pilot We also ran a small warm-cache pilot on the same DGX host, using the same 2 vCPU / 1024 MiB sandbox size. Snapshot starts used the default `alpine` template; cold starts used `localhost:5000/alpine:3.20` from the local registry. The modes were interleaved, and no vLLM requests were running during the measurement. | End-to-end CLI measurement | Snapshot (n=5) | Cold start (n=5) | | --- | ---: | ---: | | Successful create, first exec, and delete | 5/5 | 5/5 | | `start -d` p50 | 105.7 ms | 541.7 ms | | `start -d` range | 101.0–287.5 ms | 531.8–714.5 ms | | Start to first successful exec p50 | 179.9 ms | 613.0 ms | Within this one host and measurement boundary, snapshot start p50 was about 5.1x faster than cold start, and time to first successful exec was about 3.4x faster. Both modes had a slower first iteration, so this sample is too small to claim p95 or p99. A publishable benchmark should first run five warm-up iterations per mode and then measure at least 100 iterations. This pilot reports only two end-to-end measurements: `start -d`, and start to first successful `exec`. Those values include CLI, API, orchestration, and readiness overhead and are not used below to judge the README latency targets. ##### Repository snapshot benchmark AgentENV's current [`README`](https://github.com/kvcache-ai/AgentENV/blob/a33198e6718e22e8b8e3a3a784ca12d97f63370e/README.md#L19-L21) states that snapshot-backed environments boot or resume in under 50 ms, pause in under 100 ms, and create incremental snapshots in under 100 ms even under heavy disk modification. The same repository now contains a direct backend [`snapshot_benchmark.rs`](https://github.com/kvcache-ai/AgentENV/blob/a33198e6718e22e8b8e3a3a784ca12d97f63370e/crates/benchmarks/benches/snapshot_benchmark.rs) with explicit operation boundaries. The README does not say that this file produced the published numbers, so this is a reproduction of the repository's closest code-defined benchmark, not a claim about the maintainers' original benchmark environment. We copied that benchmark file unchanged at commit `a33198e` and verified the source SHA256 (`abc15b944e54639774868a43a4ebf46c52b8fe62309878de4f98418565d6aa6b`) before building it against the ARM reference branch. The timed operations, 1-vCPU setting, base 128 MiB guest, and heavy-data preparation were unchanged. Firecracker, the guest kernel, tools drive, and envd were the same artifacts as the validated production assembly. The benchmark used an isolated AgentENV home, runtime directory, ublk daemon, image cache, network CIDRs, and `ubuntu:24.04` rootfs. Other production services remained running. We then ran the repository's full Criterion mode unchanged: 3 seconds of warm-up, a 20-second target measurement window, and 10 samples per benchmark. Criterion extended several windows when necessary. Its output reports `[lower bound, point estimate, upper bound]`: | Code-defined backend measurement | README target | Criterion result | Observation | | --- | ---: | ---: | --- | | Hot `snapshot_resume` | <50 ms | [78.530, **82.124**, 85.371] ms | Target not reproduced | | `snapshot_creation` (`sandbox.pause()`) | <100 ms | [47.602, **52.598**, 58.493] ms | Under target | | `snapshot_creation_1gdisk` (1 GiB direct write, `sync`, drop caches) | <100 ms under heavy disk modification | [47.376, **51.445**, 55.950] ms | Under target | | Cold `snapshot_resume` | No separate README threshold | [103.61, **107.66**, 111.47] ms | Additional context | | `snapshot_creation_1gmem` | README claim is about disk modification | [272.64, **302.17**, 333.07] ms | Additional context | | 50-way `concurrent_resume` | No separate README threshold | [317.68, **324.41**, 331.88] ms | Additional context | To check repeatability outside Criterion's aggregation, we launched three new benchmark processes and retained all 30 bounded samples for both cold and hot resume. Each round produced exactly one roughly 30-second stall: cold sample 6 and hot sample 7. Across all samples, cold resume had median 106.2 ms and hot resume median 80.9 ms, but each had a 3/30 stall rate. The 27 sub-second cold samples were 91.70–118.71 ms (mean 103.51 ms); the 27 sub-second hot samples were 61.55–85.06 ms (mean 76.95 ms). The retained stalls were 30.91–31.04 seconds cold and 30.52–30.54 seconds hot. Their duration is close to the benchmark's explicit 30-second socket timeout, but the current logs do not establish which wait consumed it; that is correlation, not a root-cause claim. Criterion's summarized interval did not expose this periodic stall, so both the central estimate and the bounded raw-sample behavior are reported. The benchmark times `FirecrackerSandbox::resume_from_snapshot_config()` and `sandbox.pause()` directly. It excludes CLI/API/orchestrator latency and does not separately time user-facing persistent snapshot publication or object storage. The file also has no separate snapshot-boot case; hot and cold resume are the closest code-defined paths. Within that exact backend boundary, the base and heavy-disk capture paths met the two numerical under-100-ms thresholds, while hot resume did not meet the under-50-ms threshold. This is code-path evidence, not a full reproduction of every README lifecycle boundary. It is also not an ARM-versus-x86 comparison; that requires the same commit, artifacts, runner, and background-load policy on a comparable x86 host. #### Compatibility lessons from the bring-up 1. **AgentENV needs its patched Firecracker API.** A same-numbered upstream binary may run but reject AgentENV's downstream drive fields such as `direct`, as #63 demonstrates. 2. **Snapshot behavior has a minimum baseline.** Our earlier `v1.15.1-patch-v1`-lineage assembly reproduced a deterministic UART TX stall after snapshot restore. The behavior matches the bug fixed by Firecracker #5764 and disappeared after moving to a `v1.17.0-next.1` lineage containing that fix. We did not perform a single-commit bisect, so this should be read as a strong compatibility finding, not proof that only one commit differed. 3. **A guest workaround can mask a device bug.** Keeping `envd` logs away from `/dev/console` prevents the original HTTP 504 symptom, but it does not prove that UART TX survived restore. The regression therefore exercises the console device directly. 4. **The build target and seccomp policy are coupled.** The validated Firecracker is an ARM GNU build. Reusing the ARM musl policy unchanged did not cover libc-specific signal and process behavior; an empty fallback would silently weaken isolation. The release therefore carries a reviewed, default-deny ARM GNU policy and fails if it is missing. 5. **The guest kernel is part of the product.** Our selected distribution kernel has `CONFIG_IP_PNP` disabled, so the tools init applies the final AgentENV-injected `ip=` argument in userspace. The 6.1 ARM reference config in `kvcache-ai/firecracker` enables `CONFIG_IP_PNP`. Either approach can be made to work, but the required capability or fallback needs to be explicit. 6. **Snapshot identity is larger than a version string.** A downstream runtime ID is useful provenance, but it is not by itself a complete compatibility key. ARM restore compatibility can also depend on CPU features, GIC version, host/KVM behavior, Firecracker snapshot format, guest kernel, and guest tools. Restoring multiple clones also duplicates guest userspace state. Firecracker's VMGenID helps the kernel RNG only when the ARM guest has DeviceTree VMGenID support (upstream Linux added it in 6.10; a 6.1 guest needs a backport), and it does not generically rotate userspace tokens, machine identity, or cached random state. ### Desired behavior If the project chooses to pursue ARM64 server/runtime support, a possible outcome would be a documented set of support tiers with observable acceptance criteria: - **Build tier:** the server and tools drive compile for ARM64, and selected artifacts are architecture-correct and checksummed. - **Install tier:** a declared host profile can install or provision the complete dependency set with preflight checks and a reproducible BOM. - **Runtime tier:** native ARM KVM tests cover template creation, OverlayBD/ublk attachment, active-UART snapshot restore, repeated `envd` execution, pause/resume, and supervisor recovery. - **Release tier:** the project publishes supported profiles, exact dependency provenance, security policy, upgrade/rebuild rules, rollback guidance, and downloadable test evidence. Disabling ublk may be a valid reduced profile, as #94 proposes for openEuler kernels without `CONFIG_BLK_DEV_UBLK`, but the documentation should identify which snapshot, on-demand-loading, and OverlayBD guarantees are then unavailable. A build-only profile and an end-to-end runtime profile should not share the same unqualified support label. ### Proposed approach Would maintainers be open to discussing whether ARM64 server/runtime support belongs in AgentENV's roadmap? If so, the following questions may help define a staged path: 1. **Project scope:** Is ARM64 server/runtime support something the project would like to pursue? If so, should the initial goal be build-only, source installation, prebuilt releases, or an end-to-end supported runtime? Which tier would #86 or #94 satisfy if merged? 2. **Reference profiles and preflight:** Which host distributions, kernel/KVM versions, CPU/GIC combinations, page sizes, cgroup mode, and ublk capability should be supported or checked before host setup? 3. **Versioned dependency contract:** What are the canonical source and release coordinates for Firecracker, guest kernel, tools/`envd`, ublk daemon, and OverlayBD? Who would own downstream patches, upstream synchronization, provenance, and end-of-life? 4. **Guest lifecycle contract:** Which kernel options should be required, when should a userspace `ip=` fallback be supported, and how should ARM snapshot clones handle VMGenID, entropy, `boot_id`, machine identity, and application-level unique state before user code runs? 5. **Snapshot compatibility:** Which hardware and software fields should form the compatibility key? How should placement avoid incompatible hosts, and should legacy or mismatched snapshots fail closed, warn, or use an explicitly tested compatibility map? 6. **Build and isolation targets:** Should the canonical ARM64 Firecracker target be GNU, musl, or both? Should production startup fail closed when a matching default-deny seccomp policy is absent? What checksum, signing, and SBOM model should release artifacts use? 7. **Native validation:** If the project pursues a runtime or release tier, where could KVM + ublk ARM tests run, and which results should block release? Suggested coverage includes clean install, real OverlayBD/ublk I/O, template build, active-UART restore, repeated `envd` RPCs, pause/resume, clone-uniqueness checks, snapshot mismatch, missing seccomp policy, and a clearly defined latency benchmark. If maintainers consider ARM64 support in scope and indicate an initial tier, the public reference can be split into focused contributions. We can provide the tested case, help validate #86/#94 on DGX Spark, and implement or review agreed pieces; this issue does not ask for one large downstream merge. ### Compatibility and operational impact API/config changes: - Explicit support tiers may require capability reporting and preflight output. - A snapshot compatibility record may require new metadata fields beyond one Firecracker runtime ID. Snapshot or storage format changes: - Legacy snapshots need an explicit reject, migrate, or compatibility-map policy. - Clone-uniqueness handling may require a post-restore guest lifecycle hook. New host/runtime requirements: - Native ARM64 KVM for a runtime tier. - ublk/OverlayBD only for tiers that promise those storage features. - A guest kernel and seccomp policy matching the declared profile. Upgrade and rollback considerations: - Changing Firecracker snapshot format, guest kernel, or tools drive may require template rebuilds. - Canary promotion and rollback are useful reference operational practices, but need not be ARM-specific acceptance criteria for an initial source-build tier. ### Alternatives considered 1. **Comment only on #94.** That would mix an independent runtime case with review of one source-build implementation and could be misread as validation of that PR. 2. **Comment only on #86.** The independent assembly did not use #86, and #86's release scope does not define runtime compatibility. 3. **Report only the UART incident.** Firecracker already fixed the device bug; the remaining AgentENV question is dependency selection and regression ownership. 4. **Publish only the downstream recipe.** One working BOM does not define portable support tiers, snapshot placement, clone uniqueness, or lifecycle policy. ### Non-goals - This issue does not claim that #86 or #94 has been validated on DGX Spark. - It does not assume that AgentENV has already committed to supporting ARM64. - It does not claim that one DGX Spark validates all ARM CPUs, GIC versions, kernels, page sizes, or distributions. - It does not present the five-run CLI pilot, one Criterion run, or 30 bounded resume samples as a cross-architecture or production tail-latency benchmark. - It does not assume that the repository benchmark is the undisclosed original methodology behind the README numbers; it reports the exact code boundary and commit used for comparison. - It does not require AgentENV to adopt the downstream kernel, exact version set, release scripts, or runtime-ID implementation unchanged. - It does not reopen the Firecracker UART bug fixed in #5764. - It does not make canary promotion and rollback prerequisites for accepting an ARM source-build contribution. ### Contribution I can help test or review it. ### Pre-submission checklist - [x] I searched existing issues and discussions. #60, #63, #86, and #94 are adjacent work, but none currently defines the possible end-to-end ARM64 runtime support path discussed here. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi closed this issue

2026-10-29 14:18:39 +00:00

bsakaguchi commented

2026-10-29 14:18:39 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Aug 1, 2026):

Thank you, this analysis and the arm64 benchmark data are very valuable!

@yingdi-shan commented on GitHub (Aug 1, 2026): Thank you, this analysis and the arm64 benchmark data are very valuable!

bsakaguchi commented

2026-10-29 14:18:39 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Aug 8, 2026):

Fixed in PR #86.

@yingdi-shan commented on GitHub (Aug 8, 2026): Fixed in PR #86.

bsakaguchi referenced this issue

2026-10-29 14:20:02 +00:00

[PR #35] [CLOSED] feat: integrate openclaw/crabbox as AgentENV sandbox client #132