[PR #267] [CLOSED] feat(snapshot): prefetch envd memory pages at resume for faster envd-ready #305

New issue

Closed

opened 2026-10-29 14:22:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:00 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/267
Author: @huajq
Created: 9/9/2026
Status: ❌ Closed

Base: main ← Head: feat/envd-prefetch-wip2

📝 Commits (2)

06fda16 feat(snapshot): prefetch envd memory pages at resume for faster envd-ready
8ac10b2 fix(snapshot): address AI review findings on envd prefetch

📊 Changes

26 files changed (+1147 additions, -9 deletions)

View changed files

📝 .gitignore (+1 -0)
📝 CLAUDE.md (+7 -0)
📝 docs/src/concepts/snapshots.md (+19 -0)
📝 src/sandbox/firecracker/config.rs (+24 -0)
📝 src/sandbox/firecracker/manifest.rs (+7 -0)
📝 src/sandbox/firecracker/mod.rs (+1 -0)
➕ src/sandbox/firecracker/pagedump.rs (+93 -0)
📝 src/sandbox/firecracker/sandbox.rs (+190 -0)
📝 src/sandbox/ublk/device.rs (+20 -0)
📝 src/snapshot/mod.rs (+7 -0)
➕ src/snapshot/prefetch.rs (+150 -0)
📝 src/snapshot/repository/backends/oss/client.rs (+2 -0)
📝 src/snapshot/repository/backends/oss/repository.rs (+56 -1)
📝 src/snapshot/repository/backends/oss/resolver.rs (+23 -1)
📝 src/snapshot/repository/backends/posixfs/artifacts.rs (+77 -2)
📝 src/snapshot/repository/backends/posixfs/runtime.rs (+43 -3)
📝 storage/ublk-daemon/Cargo.toml (+1 -0)
📝 storage/ublk-daemon/src/client.rs (+27 -0)
📝 storage/ublk-daemon/src/protocol.rs (+11 -0)
📝 storage/ublk-daemon/src/server.rs (+192 -0)

...and 6 more files

📄 Description

What

Record the guest init daemon's (envd) resident guest-physical address ranges at snapshot capture and bulk-prefetch those pages into the node's remote-block cache before the VM resumes, so envd's early startup no longer waits on one-at-a-time remote reads.

Why

Resuming a snapshot stalls envd-ready on a serial demand-page-fault chain (~256 KiB × ~21 ms OSS RTT per read, bounded by vCPU count). Measured on a cold OSS-backed snapshot: envd-ready improves from 5.47 s to 2.13 s (-61%) with prefetch enabled.

Closes #

Scope and non-goals

Included: GPA-range export via a new aenv-pagedump tools-drive binary; memory-prefetch.json artifact publish/resolve on OSS and posix_fs; a Prefetch RPC on the ublk daemon; resume-time trigger; tests and docs.
Excluded (non-goals): KVM pre-fault / KVM_PRE_FAULT_MEMORY (requires kernel ≥ 6.11, not available on the fleet); page-fault trace recording/replay; changes to snapshot layering or capture content; any modification of the guest kernel.

Design and behavior changes

capture: tools-drive aenv-pagedump reads /proc/<envd-pid>/maps+pagemap
         → GPA ranges → artifacts/{id}/memory-prefetch.json (best-effort)
resume:  resolver downloads manifest (404 tolerated) → start_resume triggers
         ublk-daemon Prefetch RPC (fire-and-forget) → 6 workers × 4 MiB chunks
         via ImageFile::read_at → remote-block cache warmed before guest runs

Strictly best-effort at every hop: missing binary, missing/invalid/oversized manifest, or RPC failure all fall back silently to the existing on-demand path; resume never fails because of prefetch.
Manifest stores GPA ranges only, so chained snapshots (delta layers) stay valid.
Export runs inside a bounded spawn_blocking + dedicated runtime with a 10 s timeout so a wedged envd can never stall capture.
Older tools drives (no aenv-pagedump) simply produce no manifest — behavior unchanged from today.

Compatibility and operations

Public API or generated protocol: additive Prefetch variant on the internal ublk-daemon Unix-socket protocol; old client → new daemon fails safe (warn only); new client → old daemon unaffected. No external API change.
Configuration or defaults: N/A — feature is always on, no new config keys.
Snapshot manifest, artifact layout, or storage format: adds one optional fixed artifact artifacts/{id}/memory-prefetch.json; FirecrackerSnapshotManifest gains a #[serde(skip)] runtime-only field; both are backward compatible.
Upgrade and rollback: safe — snapshots with or without the manifest resume identically; rolling back simply stops producing/consuming manifests.
Host requirements, permissions, ports, or dependencies: tools drive must be rebuilt to include aenv-pagedump (new tools-image/pagedump crate + Dockerfile stage); older tools drives degrade gracefully.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes) — N/A
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

cargo test -p agentenv --lib         → 838 passed, 0 failed
cargo test -p uvm-ublk-daemon        → 48 + 37 passed, 0 failed
cargo fmt --all -- --check           → clean
cargo clippy --all-targets -D warnings → clean
E2E on OSS validation bucket (cold resume):
  without prefetch: envd-ready 5.47 s | with prefetch: 2.13 s (-61%)
  export path verified in guest: "exported via aenv-pagedump binary"
  capture export overhead: ~0.68 s (shell pipeline) → ~0.1 s (tools-drive binary)

Skipped checks and reasons: make -C services test — no services/ changes. Generated-code regeneration — N/A (hand-written protocol variant, no codegen involved).

Risks and reviewer notes

Capture makes bounded (10 s timeout) guest RPCs before pause — a new dependency on envd liveness at capture time; mitigated by the timeout and silent fallback.
Prefetch is bounded by MAX_PREFETCH_RANGES (4096) / MAX_PREFETCH_BYTES (256 MiB); over-cap manifests are disabled, never truncated.
GPA ranges equal memory-image offsets only for single-region guests (mem ≤ 3328 MiB); documented as best-effort for larger guests.
Key files: src/sandbox/firecracker/pagedump.rs, src/snapshot/prefetch.rs, storage/ublk-daemon/src/server.rs (handle_prefetch), tools-image/pagedump/src/main.rs.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/267 **Author:** [@huajq](https://github.com/huajq) **Created:** 9/9/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/envd-prefetch-wip2` --- ### 📝 Commits (2) - [`06fda16`](https://github.com/kvcache-ai/AgentENV/commit/06fda161fd33f99864fbfe3d16e2795f977d7d25) feat(snapshot): prefetch envd memory pages at resume for faster envd-ready - [`8ac10b2`](https://github.com/kvcache-ai/AgentENV/commit/8ac10b25bb49a1b55da28870e3f1ded9cdccd066) fix(snapshot): address AI review findings on envd prefetch ### 📊 Changes **26 files changed** (+1147 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+1 -0) 📝 `CLAUDE.md` (+7 -0) 📝 `docs/src/concepts/snapshots.md` (+19 -0) 📝 `src/sandbox/firecracker/config.rs` (+24 -0) 📝 `src/sandbox/firecracker/manifest.rs` (+7 -0) 📝 `src/sandbox/firecracker/mod.rs` (+1 -0) ➕ `src/sandbox/firecracker/pagedump.rs` (+93 -0) 📝 `src/sandbox/firecracker/sandbox.rs` (+190 -0) 📝 `src/sandbox/ublk/device.rs` (+20 -0) 📝 `src/snapshot/mod.rs` (+7 -0) ➕ `src/snapshot/prefetch.rs` (+150 -0) 📝 `src/snapshot/repository/backends/oss/client.rs` (+2 -0) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+56 -1) 📝 `src/snapshot/repository/backends/oss/resolver.rs` (+23 -1) 📝 `src/snapshot/repository/backends/posixfs/artifacts.rs` (+77 -2) 📝 `src/snapshot/repository/backends/posixfs/runtime.rs` (+43 -3) 📝 `storage/ublk-daemon/Cargo.toml` (+1 -0) 📝 `storage/ublk-daemon/src/client.rs` (+27 -0) 📝 `storage/ublk-daemon/src/protocol.rs` (+11 -0) 📝 `storage/ublk-daemon/src/server.rs` (+192 -0) _...and 6 more files_ </details> ### 📄 Description ## What Record the guest init daemon's (envd) resident guest-physical address ranges at snapshot capture and bulk-prefetch those pages into the node's remote-block cache before the VM resumes, so envd's early startup no longer waits on one-at-a-time remote reads. ## Why Resuming a snapshot stalls envd-ready on a serial demand-page-fault chain (~256 KiB × ~21 ms OSS RTT per read, bounded by vCPU count). Measured on a cold OSS-backed snapshot: envd-ready improves from **5.47 s to 2.13 s (-61%)** with prefetch enabled. ## Related issue Closes # ## Scope and non-goals - Included: GPA-range export via a new `aenv-pagedump` tools-drive binary; `memory-prefetch.json` artifact publish/resolve on OSS and posix_fs; a `Prefetch` RPC on the ublk daemon; resume-time trigger; tests and docs. - Excluded (non-goals): KVM pre-fault / `KVM_PRE_FAULT_MEMORY` (requires kernel ≥ 6.11, not available on the fleet); page-fault trace recording/replay; changes to snapshot layering or capture content; any modification of the guest kernel. ## Design and behavior changes ``` capture: tools-drive aenv-pagedump reads /proc/<envd-pid>/maps+pagemap → GPA ranges → artifacts/{id}/memory-prefetch.json (best-effort) resume: resolver downloads manifest (404 tolerated) → start_resume triggers ublk-daemon Prefetch RPC (fire-and-forget) → 6 workers × 4 MiB chunks via ImageFile::read_at → remote-block cache warmed before guest runs ``` - Strictly best-effort at every hop: missing binary, missing/invalid/oversized manifest, or RPC failure all fall back silently to the existing on-demand path; resume never fails because of prefetch. - Manifest stores GPA ranges only, so chained snapshots (delta layers) stay valid. - Export runs inside a bounded `spawn_blocking` + dedicated runtime with a 10 s timeout so a wedged envd can never stall capture. - Older tools drives (no `aenv-pagedump`) simply produce no manifest — behavior unchanged from today. ## Compatibility and operations - Public API or generated protocol: additive `Prefetch` variant on the internal ublk-daemon Unix-socket protocol; old client → new daemon fails safe (warn only); new client → old daemon unaffected. No external API change. - Configuration or defaults: N/A — feature is always on, no new config keys. - Snapshot manifest, artifact layout, or storage format: adds one optional fixed artifact `artifacts/{id}/memory-prefetch.json`; `FirecrackerSnapshotManifest` gains a `#[serde(skip)]` runtime-only field; both are backward compatible. - Upgrade and rollback: safe — snapshots with or without the manifest resume identically; rolling back simply stops producing/consuming manifests. - Host requirements, permissions, ports, or dependencies: tools drive must be rebuilt to include `aenv-pagedump` (new `tools-image/pagedump` crate + Dockerfile stage); older tools drives degrade gracefully. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [x] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) — N/A - [x] Documentation updated - [x] Benchmarks or performance comparison completed Commands and results: ```text cargo test -p agentenv --lib → 838 passed, 0 failed cargo test -p uvm-ublk-daemon → 48 + 37 passed, 0 failed cargo fmt --all -- --check → clean cargo clippy --all-targets -D warnings → clean E2E on OSS validation bucket (cold resume): without prefetch: envd-ready 5.47 s | with prefetch: 2.13 s (-61%) export path verified in guest: "exported via aenv-pagedump binary" capture export overhead: ~0.68 s (shell pipeline) → ~0.1 s (tools-drive binary) ``` Skipped checks and reasons: `make -C services test` — no `services/` changes. Generated-code regeneration — N/A (hand-written protocol variant, no codegen involved). ## Risks and reviewer notes - Capture makes bounded (10 s timeout) guest RPCs before pause — a new dependency on envd liveness at capture time; mitigated by the timeout and silent fallback. - Prefetch is bounded by MAX_PREFETCH_RANGES (4096) / MAX_PREFETCH_BYTES (256 MiB); over-cap manifests are disabled, never truncated. - GPA ranges equal memory-image offsets only for single-region guests (mem ≤ 3328 MiB); documented as best-effort for larger guests. - Key files: `src/sandbox/firecracker/pagedump.rs`, `src/snapshot/prefetch.rs`, `storage/ublk-daemon/src/server.rs` (`handle_prefetch`), `tools-image/pagedump/src/main.rs`. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:00 +00:00