[PR #219] feat: add mincore snapshot pre-fault #261

New issue

Open

opened 2026-10-29 14:21:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/219
Author: @Hygge-Gezelligheid
Created: 8/26/2026
Status: 🔄 Open

Base: main ← Head: agent/mincore-prefault-mvp

📝 Commits (10+)

ccb2f79 feat: add mincore snapshot prefault
3e9c872 chore: use firecracker v1.15.1-patch-v2
352706d fix: keep mincore profiling enabled with swap
4e6bf83 feat: record mincore profiling baseline
3d46c00 docs: describe mincore working-set prefault
14e95b4 fix: preserve prefault restore failures
74fcad9 fix: profile only newly resident mincore pages
5adf169 bench: profile mincore restore stages
26544dd bench: add controlled prefault restore comparison
335f671 feat: validate complete pre-fault restore

📊 Changes

98 files changed (+5241 additions, -148 deletions)

View changed files

📝 Cargo.lock (+2 -1)
📝 Cargo.toml (+1 -1)
📝 config/default.toml (+15 -0)
📝 config/deps_manifest.toml (+2 -2)
📝 crates/benchmarks/Cargo.toml (+1 -0)
📝 crates/benchmarks/benches/snapshot_benchmark.rs (+1178 -19)
📝 crates/linux-cap/src/lib.rs (+10 -3)
📝 docs/src/SUMMARY.md (+1 -0)
➕ docs/src/concepts/working-set-prefault.md (+57 -0)
📝 docs/src/configuration/reference.md (+21 -0)
➕ docs/src/internals/agentenv-prefault-experiment-report.md (+73 -0)
➕ docs/src/internals/agentenv-prefault-fixed512-raw-20260830.log (+270 -0)
➕ docs/src/internals/agentenv-prefault-product-raw-20260830.log (+178 -0)
➕ docs/src/internals/investigate-concurrent-kvm-prefault-backing-scaling.md (+20 -0)
📝 src/cfg.rs (+36 -0)
📝 src/sandbox/firecracker/config.rs (+6 -0)
📝 src/sandbox/firecracker/instance.rs (+280 -1)
📝 src/sandbox/firecracker/manifest.rs (+393 -1)
➕ src/sandbox/firecracker/mincore_tracking.rs (+357 -0)
📝 src/sandbox/firecracker/mod.rs (+12 -2)

...and 78 more files

📄 Description

Scope

This draft implements the public AgentENV mincore snapshot pre-fault path, including correctness repairs required for the production restore path and reproducible benchmark coverage.

Base: main@58732505ba415ac691f2eee2fe7b1af39ba7058e
Firecracker implementation dependency: kvcache-ai/firecracker#21
Shipping dependency: a Firecracker binary published from that PR (the currently available v1.15.1-patch-v2 artifact does not return the completion stats this revision requires).

What changes

Persist and carry working-set metadata through committed snapshots, factory/override restores, paused restores, and forks; profile through the envd-ready boundary rather than envd-initialized.
Validate and normalize GPA ranges, apply platform/capability gates before guest-memory API use, and issue pre-fault before resume.
Require Firecracker completion statistics for a successful benchmark sample: requested bytes must equal completed bytes and remaining bytes must be zero.
Regenerate the Firecracker client schema/models, including pre-fault completion statistics and corrected response definitions.
Add a benchmark-only explicit --max-prefault-bytes override and a deterministic fixed-512-MiB mechanism benchmark. Product defaults remain unchanged at 256 MiB.
Add production-path tests, including committed snapshot/factory startup coverage rather than private-field-only setup.

Validation

Passed on Madsys:

cargo test -p agentenv --lib prefault_stats — 3 passed
cargo test -p agentenv --lib manifest — 33 passed
cargo test -p firecracker_client
cargo check -p agentenv-benchmarks --bench snapshot
cargo bench -p agentenv-benchmarks --bench snapshot --no-run
cargo fmt --check and git diff --check
make test-unit PROFILE=debug — AgentENV 806 passed / 4 ignored; linux-cap 3 passed

Firecracker-specific unit, integration, and API-route validation is recorded in firecracker#21.

Performance evidence

The included raw logs and report distinguish an isolated fixed working-set mechanism test from the product path. All fixed-512-MiB samples verified requested == completed == 536,870,912 and remaining == 0.

Product resume → envd-ready

Resource-cold (no anchor sandbox; not a claim of physically cold host cache): 20 ABBA samples per arm; mean 142.24 ms without pre-fault vs 123.96 ms with pre-fault, a 12.8% / 18.27 ms improvement.
Hot shared resource: 85.22 ms without vs 85.65 ms with pre-fault; effectively neutral (+0.5%).

Fixed 512 MiB multi-vCPU mechanism test

All configurations pre-fault the same deterministic 512 MiB GPA range; only snapshot vCPU/worker count changes. Each row is 10 samples of complete pre-fault wall time, not guest scan or end-to-end resume time.

vCPU / workers	Mean	Median	Relative to 1 worker
1	257.329 ms	257.458 ms	1.00×
2	292.257 ms	294.133 ms	0.88× (slower)
4	234.023 ms	231.807 ms	1.10×
8	195.458 ms	193.150 ms	1.32×

Per-worker monotonic timelines show the 2-worker ioctls overlap by about 99.8–100%, so this is not userspace dispatch/join serialization. In a focused 1→2 diagnostic, wall time rose 231.2 → 276.7 ms (+19%) while completed uBlk read bytes changed only 578.4 → 589.0 MiB (+1.8%); successful reads rose 4,580 → 6,012 (+31%) and mean request size fell about 126 → 98 KiB.

The supported conclusion is therefore limited: workers execute concurrently and completion accounting is correct, but this host does not show near-linear multi-vCPU pre-fault scaling. The 2-worker regression is accompanied by backing-read request fragmentation, placing the remaining question below userspace worker dispatch in the KVM / host page-fault / backing-uBlk path. This PR deliberately makes no multi-vCPU performance claim.

The product command measured here is snapshot load → envd-ready; it is not a webpage/application first-request benchmark.

Review / merge boundary

Do not merge or release this AgentENV PR until the Firecracker dependency PR is accepted and a matching immutable binary asset is published. The project CI jobs for this head have passed; a separate code-review workflow is still in progress.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/219 **Author:** [@Hygge-Gezelligheid](https://github.com/Hygge-Gezelligheid) **Created:** 8/26/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `agent/mincore-prefault-mvp` --- ### 📝 Commits (10+) - [`ccb2f79`](https://github.com/kvcache-ai/AgentENV/commit/ccb2f796b04257c090d791f9d51124ba5051110b) feat: add mincore snapshot prefault - [`3e9c872`](https://github.com/kvcache-ai/AgentENV/commit/3e9c872a7612c08faf5bc1f861a166c4e2f9dd07) chore: use firecracker v1.15.1-patch-v2 - [`352706d`](https://github.com/kvcache-ai/AgentENV/commit/352706d3204a0a2050c674b8cdd98823b58d56c9) fix: keep mincore profiling enabled with swap - [`4e6bf83`](https://github.com/kvcache-ai/AgentENV/commit/4e6bf8327dd66d8f1af37e3a39550a4fa582d46f) feat: record mincore profiling baseline - [`3d46c00`](https://github.com/kvcache-ai/AgentENV/commit/3d46c001f9b8f2eb6e65a29a8a0a9f55bca602f4) docs: describe mincore working-set prefault - [`14e95b4`](https://github.com/kvcache-ai/AgentENV/commit/14e95b4a391f529e0a526ce905619f0823006253) fix: preserve prefault restore failures - [`74fcad9`](https://github.com/kvcache-ai/AgentENV/commit/74fcad933d0a9b8e3b2384b61649be14e0598190) fix: profile only newly resident mincore pages - [`5adf169`](https://github.com/kvcache-ai/AgentENV/commit/5adf1698ec1a7abec1f3d93bb9510ec15045e288) bench: profile mincore restore stages - [`26544dd`](https://github.com/kvcache-ai/AgentENV/commit/26544dd5833633cad653c9dbc948923eefff156e) bench: add controlled prefault restore comparison - [`335f671`](https://github.com/kvcache-ai/AgentENV/commit/335f671d38c6bd9f73648ea5b8f0b6f3fd9a6773) feat: validate complete pre-fault restore ### 📊 Changes **98 files changed** (+5241 additions, -148 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+2 -1) 📝 `Cargo.toml` (+1 -1) 📝 `config/default.toml` (+15 -0) 📝 `config/deps_manifest.toml` (+2 -2) 📝 `crates/benchmarks/Cargo.toml` (+1 -0) 📝 `crates/benchmarks/benches/snapshot_benchmark.rs` (+1178 -19) 📝 `crates/linux-cap/src/lib.rs` (+10 -3) 📝 `docs/src/SUMMARY.md` (+1 -0) ➕ `docs/src/concepts/working-set-prefault.md` (+57 -0) 📝 `docs/src/configuration/reference.md` (+21 -0) ➕ `docs/src/internals/agentenv-prefault-experiment-report.md` (+73 -0) ➕ `docs/src/internals/agentenv-prefault-fixed512-raw-20260830.log` (+270 -0) ➕ `docs/src/internals/agentenv-prefault-product-raw-20260830.log` (+178 -0) ➕ `docs/src/internals/investigate-concurrent-kvm-prefault-backing-scaling.md` (+20 -0) 📝 `src/cfg.rs` (+36 -0) 📝 `src/sandbox/firecracker/config.rs` (+6 -0) 📝 `src/sandbox/firecracker/instance.rs` (+280 -1) 📝 `src/sandbox/firecracker/manifest.rs` (+393 -1) ➕ `src/sandbox/firecracker/mincore_tracking.rs` (+357 -0) 📝 `src/sandbox/firecracker/mod.rs` (+12 -2) _...and 78 more files_ </details> ### 📄 Description ## Scope This draft implements the public AgentENV mincore snapshot pre-fault path, including correctness repairs required for the production restore path and reproducible benchmark coverage. - Base: `main@58732505ba415ac691f2eee2fe7b1af39ba7058e` - Firecracker implementation dependency: [kvcache-ai/firecracker#21](https://github.com/kvcache-ai/firecracker/pull/21) - Shipping dependency: a Firecracker binary published from that PR (the currently available `v1.15.1-patch-v2` artifact does not return the completion stats this revision requires). ## What changes - Persist and carry working-set metadata through committed snapshots, factory/override restores, paused restores, and forks; profile through the `envd-ready` boundary rather than `envd-initialized`. - Validate and normalize GPA ranges, apply platform/capability gates before guest-memory API use, and issue pre-fault before resume. - Require Firecracker completion statistics for a successful benchmark sample: requested bytes must equal completed bytes and remaining bytes must be zero. - Regenerate the Firecracker client schema/models, including pre-fault completion statistics and corrected response definitions. - Add a benchmark-only explicit `--max-prefault-bytes` override and a deterministic fixed-512-MiB mechanism benchmark. Product defaults remain unchanged at 256 MiB. - Add production-path tests, including committed snapshot/factory startup coverage rather than private-field-only setup. ## Validation Passed on Madsys: - `cargo test -p agentenv --lib prefault_stats` — 3 passed - `cargo test -p agentenv --lib manifest` — 33 passed - `cargo test -p firecracker_client` - `cargo check -p agentenv-benchmarks --bench snapshot` - `cargo bench -p agentenv-benchmarks --bench snapshot --no-run` - `cargo fmt --check` and `git diff --check` - `make test-unit PROFILE=debug` — AgentENV 806 passed / 4 ignored; linux-cap 3 passed Firecracker-specific unit, integration, and API-route validation is recorded in [firecracker#21](https://github.com/kvcache-ai/firecracker/pull/21). ## Performance evidence The included raw logs and report distinguish an isolated fixed working-set mechanism test from the product path. All fixed-512-MiB samples verified `requested == completed == 536,870,912` and `remaining == 0`. ### Product resume → envd-ready - **Resource-cold** (no anchor sandbox; not a claim of physically cold host cache): 20 ABBA samples per arm; mean 142.24 ms without pre-fault vs 123.96 ms with pre-fault, a 12.8% / 18.27 ms improvement. - **Hot shared resource**: 85.22 ms without vs 85.65 ms with pre-fault; effectively neutral (+0.5%). ### Fixed 512 MiB multi-vCPU mechanism test All configurations pre-fault the same deterministic 512 MiB GPA range; only snapshot vCPU/worker count changes. Each row is 10 samples of complete pre-fault wall time, not guest scan or end-to-end resume time. | vCPU / workers | Mean | Median | Relative to 1 worker | | ---: | ---: | ---: | ---: | | 1 | 257.329 ms | 257.458 ms | 1.00× | | 2 | 292.257 ms | 294.133 ms | 0.88× (slower) | | 4 | 234.023 ms | 231.807 ms | 1.10× | | 8 | 195.458 ms | 193.150 ms | 1.32× | Per-worker monotonic timelines show the 2-worker ioctls overlap by about 99.8–100%, so this is not userspace dispatch/join serialization. In a focused 1→2 diagnostic, wall time rose 231.2 → 276.7 ms (+19%) while completed uBlk read bytes changed only 578.4 → 589.0 MiB (+1.8%); successful reads rose 4,580 → 6,012 (+31%) and mean request size fell about 126 → 98 KiB. The supported conclusion is therefore limited: workers execute concurrently and completion accounting is correct, but this host does not show near-linear multi-vCPU pre-fault scaling. The 2-worker regression is accompanied by backing-read request fragmentation, placing the remaining question below userspace worker dispatch in the KVM / host page-fault / backing-uBlk path. This PR deliberately makes no multi-vCPU performance claim. The product command measured here is `snapshot load → envd-ready`; it is not a webpage/application first-request benchmark. ## Review / merge boundary Do not merge or release this AgentENV PR until the Firecracker dependency PR is accepted and a matching immutable binary asset is published. The project CI jobs for this head have passed; a separate code-review workflow is still in progress. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 14:21:34 +00:00

bsakaguchi referenced this issue

2026-10-29 14:21:58 +00:00

[PR #261] fix(sandbox): shard the persistent serial log directory #298

bsakaguchi referenced this issue

2026-10-29 14:21:58 +00:00

[PR #265] [CLOSED] fix(sandbox): clean up empty serial logs and expire inactive logs #301

bsakaguchi referenced this issue

2026-10-29 14:22:03 +00:00