[PR #298] [MERGED] feat(snapshot): accelerate cold resume with startup first-touch manifests #327

New issue

Closed

opened 2026-10-29 14:22:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/298
Author: @huajq
Created: 9/16/2026
Status: ✅ Merged
Merged: 9/18/2026
Merged by: @yingdi-shan

Base: main ← Head: trace-manifest

📝 Commits (2)

79ce937 feat(snapshot): record startup first-touch and upload manifests decoupled
0bf3cb6 feat(snapshot): prefetch startup manifests concurrently at resume

📊 Changes

52 files changed (+5218 additions, -113 deletions)

View changed files

📝 config/default.toml (+17 -0)
📝 crates/e2e-tests/tests/snapshot_oss_e2e_test.rs (+5 -0)
📝 src/api/impls/image_build/tests.rs (+1 -1)
📝 src/bin/server.rs (+6 -0)
📝 src/cfg.rs (+40 -0)
📝 src/sandbox/backend.rs (+9 -2)
📝 src/sandbox/firecracker/config.rs (+15 -0)
📝 src/sandbox/firecracker/mod.rs (+3 -1)
📝 src/sandbox/firecracker/sandbox.rs (+178 -33)
➕ src/sandbox/firecracker/startup_pack.rs (+327 -0)
📝 src/sandbox/manifest.rs (+21 -0)
📝 src/sandbox/mock.rs (+8 -2)
📝 src/sandbox/mod.rs (+4 -3)
📝 src/sandbox/ublk/device.rs (+105 -1)
📝 src/sandbox/ublk/mod.rs (+1 -1)
📝 src/snapshot/manager.rs (+25 -6)
📝 src/snapshot/mock.rs (+1 -0)
📝 src/snapshot/mod.rs (+6 -0)
📝 src/snapshot/repository/backends/oss/client.rs (+14 -0)
📝 src/snapshot/repository/backends/oss/repository.rs (+268 -5)

...and 32 more files

📄 Description

What

Two-commit change that cuts snapshot cold-resume time by prefetching the guest's early memory working set:

Capture/upload: at snapshot publish (OSS backend, feature enabled), a throwaway recording VM re-resumes the snapshot on a dedicated memory device while a daemon-side recorder observes first-touch pages in a bounded window (min 200ms / quiet 300ms / max 2s). A detached continuation then expands the trace into a compact manifest — address-merged ranges sorted by each range's earliest first-touch rank — uploads it as memory-startup.pack, and attaches a sha256-bound descriptor to the committed record. Publish never waits on any of this (measured ~16s → ~6s).
Consume: at resume, a prefetch task downloads the manifest (magic probe, then full body gated on the committed sha256), translates ranges to final-object 256KiB cache blocks with the shared planner (tar unwrap / zfile jump table / merged LSMT index), merges adjacent blocks into capped 16MiB runs, and refills them concurrently through the same batched source-read path and shared full-file cache that background downloads use. Runs are delivered sorted by earliest first-touch rank. Resume never blocks on prefetch completion; pages not yet prefetched fault on-demand as usual.

Why

Cold resume of large-memory snapshots is dominated by on-demand remote reads: the guest stalls through hundreds of MiB of 256KiB faults before envd can answer its health probe. Recording once where the hot pages are and pulling them with few large ordered requests removes that serialization without changing any image formats or the read path.

N/A — design iterated and A/B-validated offline (trace/pack variants evaluated and rejected in earlier rounds).

Scope and non-goals

Included: recording infra (ublk recorder + daemon RPC + recording VM orchestration), manifest codec, decoupled publish continuation, resolver/daemon prefetch path, graceful-shutdown drain semantics.

Excluded: first-program/workload-phase coverage (recording window covers boot+envd only), rootfs/attached-drive manifests, POSIX backend manifests, P2P transport of manifests, order-preserving per-block pull (A/B-tested and rejected: 2.4× more requests, slower envd-ready).

Design and behavior changes

Manifest format: AENVMF01 header + optional exact-order page prefix + rank-sorted (offset,len) ranges; single version; unknown versions are skipped (plain on-demand resume). 51,338 pages → 784 ranges = 12.6KB.
Delivery: merged runs preserve large requests (~500 GETs); rank ordering makes boot-critical blocks land first. A/B data showed ordering adds no measurable benefit at current scale but costs nothing (same runs/requests) and is the semantically correct priority.
Failure handling: recording/build/upload/attach failures publish normally without a descriptor; prefetch failure/cancel/timeout (default 30s) never fails the resume; loader election dedups prefetch vs guest faults in the shared cache; shutdown stops new continuations and drains in-flight ones (bounded).
Concurrency: prefetch shares the download scheduler's admission gate, block slots and cache entries with on-demand and background refills; same-manifest concurrent resumes dedup by descriptor sha256.

Compatibility and operations

Public API or generated protocol: no public API change; internal daemon RPC gains PrefetchStartupPack (best-effort, always answers Ok).
Configuration or defaults: new [snapshot.memory_startup_pack] section (enabled, consume_enabled, record window/budget, consume timeout) — all default off, no behavior change unless enabled.
Snapshot manifest, artifact layout, or storage format: new optional artifact memory-startup.pack and optional memory_startup record field; records without it resume exactly as before.
Upgrade and rollback: mixed-version clusters degrade to plain on-demand resume on both sides.
Host requirements, permissions, ports, or dependencies: none new.

Validation

make fmt (via cargo fmt --all -- --check)
make clippy — ran cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets (clean) instead of the full target
make test-unit — ran focused suites instead (see below)
Relevant Rust integration tests — covered by cluster E2E below instead (KVM/ublk host)
make -C services test — N/A (services/ untouched)
Generated clients/server regenerated — N/A (no generated code touched)
Documentation updated — feature is off by default; docs to follow with enablement
Benchmarks or performance comparison completed

Commands and results:

cargo test -p overlaybd startup            # 8 passed (manifest codec, prefetch task, run ordering)
cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets   # clean
cargo fmt --all -- --check                 # clean

Cluster A/B (AMD EPYC node, production snapshot chain 01a04285…, OSS backend, cold cache per run; median of 3):

                     create total   envd wait    on-demand @envd_ready   prefetch(manifest)
baseline (no manifest)   16.2s        11.6s           ~694MB total              —
with manifest             3.4s         1.9s            ~46MB              ~413MB / ~500 GETs
                                                      (final ~120MB)     (100% done @envd_ready)

(envd-ready ~6× faster; create ~4.8× faster; on-demand traffic at envd_ready ~93% lower. Manifest cost: 2.4s recording window + 12.6KB object, fully off the publish path.)

Risks and reviewer notes

Prefetch adds a ~400MB/2s network burst per cold resume when enabled (bounded by the shared admission gate; A/B showed no regression for co-tenant traffic).
Recording boots one extra throwaway VM per publish (bounded window, network-isolated, cleaned up on all paths).
Record descriptor attach is a read-modify-write without CAS (documented; same model as alias binding).
Suggested review order: storage/overlaybd/src/startup_manifest.rs (format), storage/overlaybd/src/backend/cache/startup_pack_task.rs (consume path), then the capture commit.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/298 **Author:** [@huajq](https://github.com/huajq) **Created:** 9/16/2026 **Status:** ✅ Merged **Merged:** 9/18/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `trace-manifest` --- ### 📝 Commits (2) - [`79ce937`](https://github.com/kvcache-ai/AgentENV/commit/79ce9371e218282a335dd9b7e96f1ab526d140e8) feat(snapshot): record startup first-touch and upload manifests decoupled - [`0bf3cb6`](https://github.com/kvcache-ai/AgentENV/commit/0bf3cb63248dfd44dfe1005ac6a6dbf0c6380b7c) feat(snapshot): prefetch startup manifests concurrently at resume ### 📊 Changes **52 files changed** (+5218 additions, -113 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+17 -0) 📝 `crates/e2e-tests/tests/snapshot_oss_e2e_test.rs` (+5 -0) 📝 `src/api/impls/image_build/tests.rs` (+1 -1) 📝 `src/bin/server.rs` (+6 -0) 📝 `src/cfg.rs` (+40 -0) 📝 `src/sandbox/backend.rs` (+9 -2) 📝 `src/sandbox/firecracker/config.rs` (+15 -0) 📝 `src/sandbox/firecracker/mod.rs` (+3 -1) 📝 `src/sandbox/firecracker/sandbox.rs` (+178 -33) ➕ `src/sandbox/firecracker/startup_pack.rs` (+327 -0) 📝 `src/sandbox/manifest.rs` (+21 -0) 📝 `src/sandbox/mock.rs` (+8 -2) 📝 `src/sandbox/mod.rs` (+4 -3) 📝 `src/sandbox/ublk/device.rs` (+105 -1) 📝 `src/sandbox/ublk/mod.rs` (+1 -1) 📝 `src/snapshot/manager.rs` (+25 -6) 📝 `src/snapshot/mock.rs` (+1 -0) 📝 `src/snapshot/mod.rs` (+6 -0) 📝 `src/snapshot/repository/backends/oss/client.rs` (+14 -0) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+268 -5) _...and 32 more files_ </details> ### 📄 Description ## What Two-commit change that cuts snapshot cold-resume time by prefetching the guest's early memory working set: 1. **Capture/upload**: at snapshot publish (OSS backend, feature enabled), a throwaway recording VM re-resumes the snapshot on a dedicated memory device while a daemon-side recorder observes first-touch pages in a bounded window (min 200ms / quiet 300ms / max 2s). A detached continuation then expands the trace into a compact manifest — address-merged ranges sorted by each range's earliest first-touch rank — uploads it as `memory-startup.pack`, and attaches a sha256-bound descriptor to the committed record. Publish never waits on any of this (measured ~16s → ~6s). 2. **Consume**: at resume, a prefetch task downloads the manifest (magic probe, then full body gated on the committed sha256), translates ranges to final-object 256KiB cache blocks with the shared planner (tar unwrap / zfile jump table / merged LSMT index), merges adjacent blocks into capped 16MiB runs, and refills them concurrently through the same batched source-read path and shared full-file cache that background downloads use. Runs are delivered sorted by earliest first-touch rank. Resume never blocks on prefetch completion; pages not yet prefetched fault on-demand as usual. ## Why Cold resume of large-memory snapshots is dominated by on-demand remote reads: the guest stalls through hundreds of MiB of 256KiB faults before envd can answer its health probe. Recording once where the hot pages are and pulling them with few large ordered requests removes that serialization without changing any image formats or the read path. ## Related issue N/A — design iterated and A/B-validated offline (trace/pack variants evaluated and rejected in earlier rounds). ## Scope and non-goals Included: recording infra (ublk recorder + daemon RPC + recording VM orchestration), manifest codec, decoupled publish continuation, resolver/daemon prefetch path, graceful-shutdown drain semantics. Excluded: first-program/workload-phase coverage (recording window covers boot+envd only), rootfs/attached-drive manifests, POSIX backend manifests, P2P transport of manifests, order-preserving per-block pull (A/B-tested and rejected: 2.4× more requests, slower envd-ready). ## Design and behavior changes - **Manifest format**: `AENVMF01` header + optional exact-order page prefix + rank-sorted `(offset,len)` ranges; single version; unknown versions are skipped (plain on-demand resume). 51,338 pages → 784 ranges = **12.6KB**. - **Delivery**: merged runs preserve large requests (~500 GETs); rank ordering makes boot-critical blocks land first. A/B data showed ordering adds no measurable benefit at current scale but costs nothing (same runs/requests) and is the semantically correct priority. - **Failure handling**: recording/build/upload/attach failures publish normally without a descriptor; prefetch failure/cancel/timeout (default 30s) never fails the resume; loader election dedups prefetch vs guest faults in the shared cache; shutdown stops new continuations and drains in-flight ones (bounded). - **Concurrency**: prefetch shares the download scheduler's admission gate, block slots and cache entries with on-demand and background refills; same-manifest concurrent resumes dedup by descriptor sha256. ## Compatibility and operations - Public API or generated protocol: no public API change; internal daemon RPC gains `PrefetchStartupPack` (best-effort, always answers `Ok`). - Configuration or defaults: new `[snapshot.memory_startup_pack]` section (`enabled`, `consume_enabled`, record window/budget, consume timeout) — **all default off**, no behavior change unless enabled. - Snapshot manifest, artifact layout, or storage format: new optional artifact `memory-startup.pack` and optional `memory_startup` record field; records without it resume exactly as before. - Upgrade and rollback: mixed-version clusters degrade to plain on-demand resume on both sides. - Host requirements, permissions, ports, or dependencies: none new. ## Validation - [x] `make fmt` (via `cargo fmt --all -- --check`) - [ ] `make clippy` — ran `cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets` (clean) instead of the full target - [ ] `make test-unit` — ran focused suites instead (see below) - [ ] Relevant Rust integration tests — covered by cluster E2E below instead (KVM/ublk host) - [ ] `make -C services test` — N/A (`services/` untouched) - [ ] Generated clients/server regenerated — N/A (no generated code touched) - [ ] Documentation updated — feature is off by default; docs to follow with enablement - [x] Benchmarks or performance comparison completed Commands and results: ```text cargo test -p overlaybd startup # 8 passed (manifest codec, prefetch task, run ordering) cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets # clean cargo fmt --all -- --check # clean ``` Cluster A/B (AMD EPYC node, production snapshot chain 01a04285…, OSS backend, cold cache per run; median of 3): ```text create total envd wait on-demand @envd_ready prefetch(manifest) baseline (no manifest) 16.2s 11.6s ~694MB total — with manifest 3.4s 1.9s ~46MB ~413MB / ~500 GETs (final ~120MB) (100% done @envd_ready) ``` (envd-ready ~6× faster; create ~4.8× faster; on-demand traffic at envd_ready ~93% lower. Manifest cost: 2.4s recording window + 12.6KB object, fully off the publish path.) ## Risks and reviewer notes - Prefetch adds a ~400MB/2s network burst per cold resume when enabled (bounded by the shared admission gate; A/B showed no regression for co-tenant traffic). - Recording boots one extra throwaway VM per publish (bounded window, network-isolated, cleaned up on all paths). - Record descriptor attach is a read-modify-write without CAS (documented; same model as alias binding). - Suggested review order: `storage/overlaybd/src/startup_manifest.rs` (format), `storage/overlaybd/src/backend/cache/startup_pack_task.rs` (consume path), then the capture commit. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:14 +00:00