[PR #314] [MERGED] feat(snapshot): add startup manifest prefetch for POSIX snapshots #338

New issue

Closed

opened 2026-10-29 14:22:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/314
Author: @Hygge-Gezelligheid
Created: 9/22/2026
Status: ✅ Merged
Merged: 9/29/2026
Merged by: @yingdi-shan

Base: main ← Head: agent/posix-startup-pack

📝 Commits (3)

81f27cb feat(snapshot): record and resolve POSIX startup manifests
cfa561a feat(snapshot): prefetch POSIX startup through shared memory devices
9ce53f8 test(snapshot): validate and document POSIX startup prefetch

📊 Changes

17 files changed (+2433 additions, -200 deletions)

View changed files

📝 config/default.toml (+3 -3)
📝 docs/src/configuration/reference.md (+38 -0)
➕ scripts/verify-posix-startup-pack-smoke.py (+108 -0)
📝 src/sandbox/firecracker/sandbox.rs (+77 -22)
📝 src/sandbox/firecracker/startup_pack.rs (+1119 -13)
📝 src/sandbox/manifest.rs (+2 -2)
📝 src/sandbox/ublk/device.rs (+10 -2)
📝 src/snapshot/manager.rs (+7 -6)
📝 src/snapshot/mod.rs (+2 -2)
📝 src/snapshot/repository/backends/oss/repository.rs (+99 -71)
📝 src/snapshot/repository/backends/posixfs/backend.rs (+259 -4)
📝 src/snapshot/repository/backends/posixfs/catalog.rs (+232 -6)
📝 src/snapshot/repository/backends/posixfs/runtime.rs (+33 -1)
📝 src/snapshot/startup_pack.rs (+303 -65)
📝 src/snapshot/types/snapshot.rs (+3 -3)
📝 tests/integration.rs (+3 -0)
➕ tests/integration/posix_startup_pack.rs (+135 -0)

📄 Description

Overview

Extend the startup memory manifest flow from PR #298 to POSIX snapshots. The capture and publication stages asynchronously save the startup manifest. During restore, the logical memory ranges in the manifest are prefetched through the same memory ublk device that Firecracker actually uses.

Recording and consumption are disabled by default, and the OSS prefetch path remains unchanged.

Implementation

Atomically save the manifest and attach a SHA-256 descriptor under the snapshot record lock. Retain the capture lease during background recording; coordinate publication with deletion to prevent background tasks from recreating deleted snapshot records.
Resolve POSIX manifests as LocalPath, validate their size, hash, and memory layout, then perform buffered device reads concurrently with restore.
Reuse the existing trace and manifest formats. The local consumer reads logical memory ranges directly; the OSS consumer continues to use the existing physical-range planner and cache executor.
Share in-flight prefetch by actual device generation and manifest digest. Each shared task uses one reader and a 1 MiB buffer, with at most four readers reading concurrently within the process.
Use cancellation and completion notifications that retain their state. One consumer leaving does not cancel tasks for other consumers. During shutdown, first request that the VM stop, then wait for reads to exit; retain the device reference until reads finish.
Continue normal on-demand reads if the manifest is missing or corrupt, or prefetch fails. Older snapshots remain compatible.
Conservatively skip io_engine = 2, for which device prefetch has not yet been validated; leave the existing I/O engine settings unchanged.

No separate disk cache, whole-layer prefetch policy, or public protocol changes are introduced. Private performance test code is not included in production changes.

Configuration

Use the existing [snapshot.memory_startup_pack] configuration:

enabled: controls recording and publication.
consume_enabled: controls prefetch during restore.

Both default to false. Consumption can be disabled independently for snapshots that already contain a manifest.

Validation

Formatting checks and strict Clippy checks across all targets.
28 startup prefetch tests, 23 POSIX repository tests, and 15 OSS repository tests.
Coverage includes manifest validation, publication/deletion races, shared-task cancellation, queued-read cancellation, completion notifications, device lifetime during blocked reads, and direct-I/O skip.
Hardware integration tests cover POSIX capture → asynchronous manifest attachment → normal resolution → restore → stop.
Release-mode bare-metal performance tests using the final simplified consumer.

Performance

Experiment 1: Memory-device lifecycle; lower-file cache uncontrolled (historical)

On Madsys-4-2, five randomized blocks of serial comparisons were completed using the same native 128 MiB Ubuntu-family snapshot, release test program and daemon, and buffered-read configuration ioEngine=0.

Mode	Five times to ready (ms)	Mean
Cold, prefetch disabled	123.309、137.345、108.428、135.402、113.886	123.674 ms
Cold, prefetch enabled	70.689、65.997、68.000、78.939、79.556	72.636 ms
Hot control	89.818、96.125、76.852、75.908、87.060	85.153 ms

With prefetch enabled, the mean time to ready decreased by 51.038 ms (41.3%), with improvement in all five paired blocks.

The fixed-input test follows the standard cold/hot device lifecycle and calls the normal snapshot restore path. Timing starts before the restore call and ends at envd-ready, including device acquisition, manifest validation, and prefetch work before ready. Repository resolution and baseline preparation are excluded from this interval. Cold releases the previous memory device reference; Hot retains a warm instance. No global cache clearing or lower-file cache eviction is performed, so the lower-file page cache may already be warm.

All five prefetch runs overlapped with VM restore and completed before ready. From the observation point after device acquisition to the read-drain observation point after ready, mean memory-device reads increased from 40.841 MiB to 56.505 MiB, an increase of approximately 38.4%. These are device-level counters, not physical disk reads or exact guest page-fault counts.

Experiment 2: Controlled memory lower-file page cache (current head)

This extension measures how the host Linux page cache of the snapshot's memory lower file changes restore latency and the benefit of startup prefetch. It tests commit 8a16c6f7d80d92fbb774ca0ac597918f5aafc09d on Madsys-4-2, using the same fixed one-vCPU, 128 MiB Ubuntu-family POSIX snapshot and buffered I/O configuration ioEngine=0.

Each of eight blocks contains four serial restores in a frozen randomized order: file-warm/prefetch-off, file-warm/prefetch-on, file-cold/prefetch-off, and file-cold/prefetch-on. All 32 formal restores and all block admission/exit checks passed; four pilot restores are excluded, and neither reserve block was needed.

Memory lower-file page cache	Prefetch off: mean / median	Prefetch on: mean / median	Reduction in mean ready latency
File-warm: all pages resident	118.849 / 116.199 ms	83.369 / 82.891 ms	29.85%
File-cold: no pages resident	290.141 / 302.017 ms	126.960 / 127.002 ms	56.24%

Cache preparation and verification: All four conditions first perform an untimed, prefetch-off baseline restore, then stop it and wait for acknowledgement that its memory-device reference has been released. All four timed restores therefore use the device-cold lifecycle; “file-warm” does not mean a warm VM or the hot control from Experiment 1. After baseline release, both treatments fully read and SHA-256-check a new, independently copied memory lower file. File-warm retains those cached pages. File-cold applies one POSIX_FADV_DONTNEED operation only to that private file. Immediately before timing, mincore must show 14,472/14,472 resident pages for file-warm or 0/14,472 for file-cold; every formal trial met its threshold. The file is 59,277,312 bytes, with 4,096-byte host pages. The final residency probe is outside the timer, with a recorded probe-to-timer gap of at most 1 microsecond. No file-content read or diagnostic I/O occurs in that gap. No global cache clearing or changes to other users' files, services, or devices are performed.

Timing scope: Timing starts immediately before the normal restore constructor and start(), and ends when start() returns at envd-ready. Memory-device acquisition, manifest validation, prefetch, VM restore, and guest readiness are included. Repository resolution, the baseline restore/release, file hashing, cache preparation, and the final pre-timer residency probe are excluded. Each trial has its own process and private runtime; both baseline and timed memory-device releases are acknowledged, and final checks show no remaining experiment processes or Unix sockets.

Paired differences are calculated within each block. The intervals below are descriptive 95% Student-t intervals over eight paired differences (7 degrees of freedom):

Paired comparison	Mean difference	95% interval
File-warm: prefetch on minus off	-35.480 ms	[-44.337, -26.622] ms
File-cold: prefetch on minus off	-163.181 ms	[-185.740, -140.622] ms
Prefetch off: file-cold minus file-warm	+171.292 ms	[+150.333, +192.251] ms
Prefetch on: file-cold minus file-warm	+43.590 ms	[+35.241, +51.939] ms

Prefetch is faster in all eight pairs under each file-cache condition. These measurements support a substantial benefit for this fixed snapshot when its memory lower-file pages are initially absent from the host page cache, and show that the file-cache state materially affects measured restore latency.

Scope of “file-cold”: Only this memory lower file's Linux page-cache residency is controlled. Rootfs file pages, filesystem metadata, process state, and NVMe controller/SSD caches may remain warm. This measures snapshot restore to envd-ready, rather than Ubuntu boot from scratch or the entire user-request-to-ready path. Logical ublk counters are not physical disk traffic or guest page-fault counts; post-ready residency and counters have their own observation timestamps. Experiments 1 and 2 use different measurement builds and cache treatments, and their samples are reported separately.

Limitations

Prefetch competes with on-demand reads for CPU, cache, and device bandwidth. Cancellation and timeouts can prevent subsequent reads but cannot interrupt reads already blocked in the kernel. Such reads may delay final device release; the VM is requested to stop before waiting.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/314 **Author:** [@Hygge-Gezelligheid](https://github.com/Hygge-Gezelligheid) **Created:** 9/22/2026 **Status:** ✅ Merged **Merged:** 9/29/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `agent/posix-startup-pack` --- ### 📝 Commits (3) - [`81f27cb`](https://github.com/kvcache-ai/AgentENV/commit/81f27cb5946c0fc1042cebadbf9b0ee7879161e5) feat(snapshot): record and resolve POSIX startup manifests - [`cfa561a`](https://github.com/kvcache-ai/AgentENV/commit/cfa561aa13e21b47ca4be09dc4f53d6598f00a4d) feat(snapshot): prefetch POSIX startup through shared memory devices - [`9ce53f8`](https://github.com/kvcache-ai/AgentENV/commit/9ce53f83268a9c33aa6dff1cb28f2ba7d0310fc7) test(snapshot): validate and document POSIX startup prefetch ### 📊 Changes **17 files changed** (+2433 additions, -200 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+3 -3) 📝 `docs/src/configuration/reference.md` (+38 -0) ➕ `scripts/verify-posix-startup-pack-smoke.py` (+108 -0) 📝 `src/sandbox/firecracker/sandbox.rs` (+77 -22) 📝 `src/sandbox/firecracker/startup_pack.rs` (+1119 -13) 📝 `src/sandbox/manifest.rs` (+2 -2) 📝 `src/sandbox/ublk/device.rs` (+10 -2) 📝 `src/snapshot/manager.rs` (+7 -6) 📝 `src/snapshot/mod.rs` (+2 -2) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+99 -71) 📝 `src/snapshot/repository/backends/posixfs/backend.rs` (+259 -4) 📝 `src/snapshot/repository/backends/posixfs/catalog.rs` (+232 -6) 📝 `src/snapshot/repository/backends/posixfs/runtime.rs` (+33 -1) 📝 `src/snapshot/startup_pack.rs` (+303 -65) 📝 `src/snapshot/types/snapshot.rs` (+3 -3) 📝 `tests/integration.rs` (+3 -0) ➕ `tests/integration/posix_startup_pack.rs` (+135 -0) </details> ### 📄 Description ## Overview Extend the startup memory manifest flow from PR #298 to POSIX snapshots. The capture and publication stages asynchronously save the startup manifest. During restore, the logical memory ranges in the manifest are prefetched through **the same memory ublk device that Firecracker actually uses**. Recording and consumption are disabled by default, and the OSS prefetch path remains unchanged. ## Implementation - Atomically save the manifest and attach a SHA-256 descriptor under the snapshot record lock. Retain the capture lease during background recording; coordinate publication with deletion to prevent background tasks from recreating deleted snapshot records. - Resolve POSIX manifests as `LocalPath`, validate their size, hash, and memory layout, then perform buffered device reads concurrently with restore. - Reuse the existing trace and manifest formats. The local consumer reads logical memory ranges directly; the OSS consumer continues to use the existing physical-range planner and cache executor. - Share in-flight prefetch by actual device generation and manifest digest. Each shared task uses one reader and a 1 MiB buffer, with at most four readers reading concurrently within the process. - Use cancellation and completion notifications that retain their state. One consumer leaving does not cancel tasks for other consumers. During shutdown, first request that the VM stop, then wait for reads to exit; retain the device reference until reads finish. - Continue normal on-demand reads if the manifest is missing or corrupt, or prefetch fails. Older snapshots remain compatible. - Conservatively skip `io_engine = 2`, for which device prefetch has not yet been validated; leave the existing I/O engine settings unchanged. No separate disk cache, whole-layer prefetch policy, or public protocol changes are introduced. Private performance test code is not included in production changes. ## Configuration Use the existing `[snapshot.memory_startup_pack]` configuration: - `enabled`: controls recording and publication. - `consume_enabled`: controls prefetch during restore. Both default to `false`. Consumption can be disabled independently for snapshots that already contain a manifest. ## Validation - Formatting checks and strict Clippy checks across all targets. - 28 startup prefetch tests, 23 POSIX repository tests, and 15 OSS repository tests. - Coverage includes manifest validation, publication/deletion races, shared-task cancellation, queued-read cancellation, completion notifications, device lifetime during blocked reads, and direct-I/O skip. - Hardware integration tests cover POSIX capture → asynchronous manifest attachment → normal resolution → restore → stop. - Release-mode bare-metal performance tests using the final simplified consumer. ## Performance ### Experiment 1: Memory-device lifecycle; lower-file cache uncontrolled (historical) On Madsys-4-2, five randomized blocks of serial comparisons were completed using the same native 128 MiB Ubuntu-family snapshot, release test program and daemon, and buffered-read configuration `ioEngine=0`. | Mode | Five times to ready (ms) | Mean | | --- | --- | ---: | | Cold, prefetch disabled | 123.309、137.345、108.428、135.402、113.886 | 123.674 ms | | Cold, prefetch enabled | 70.689、65.997、68.000、78.939、79.556 | 72.636 ms | | Hot control | 89.818、96.125、76.852、75.908、87.060 | 85.153 ms | With prefetch enabled, the mean time to ready decreased by **51.038 ms (41.3%)**, with improvement in all five paired blocks. The fixed-input test follows the standard cold/hot device lifecycle and calls the normal snapshot restore path. Timing starts before the restore call and ends at envd-ready, including device acquisition, manifest validation, and prefetch work before ready. Repository resolution and baseline preparation are excluded from this interval. Cold releases the previous memory device reference; Hot retains a warm instance. No global cache clearing or lower-file cache eviction is performed, so the lower-file page cache may already be warm. All five prefetch runs overlapped with VM restore and completed before ready. From the observation point after device acquisition to the read-drain observation point after ready, mean memory-device reads increased from **40.841 MiB to 56.505 MiB**, an increase of approximately **38.4%**. These are device-level counters, not physical disk reads or exact guest page-fault counts. ### Experiment 2: Controlled memory lower-file page cache (current head) This extension measures how the **host Linux page cache of the snapshot's memory lower file** changes restore latency and the benefit of startup prefetch. It tests commit `8a16c6f7d80d92fbb774ca0ac597918f5aafc09d` on Madsys-4-2, using the same fixed one-vCPU, 128 MiB Ubuntu-family POSIX snapshot and buffered I/O configuration `ioEngine=0`. Each of eight blocks contains four serial restores in a frozen randomized order: file-warm/prefetch-off, file-warm/prefetch-on, file-cold/prefetch-off, and file-cold/prefetch-on. All **32 formal restores** and all block admission/exit checks passed; four pilot restores are excluded, and neither reserve block was needed. | Memory lower-file page cache | Prefetch off: mean / median | Prefetch on: mean / median | Reduction in mean ready latency | | --- | ---: | ---: | ---: | | File-warm: all pages resident | 118.849 / 116.199 ms | 83.369 / 82.891 ms | **29.85%** | | File-cold: no pages resident | 290.141 / 302.017 ms | 126.960 / 127.002 ms | **56.24%** | **Cache preparation and verification:** All four conditions first perform an untimed, prefetch-off baseline restore, then stop it and wait for acknowledgement that its memory-device reference has been released. All four timed restores therefore use the device-cold lifecycle; “file-warm” does not mean a warm VM or the hot control from Experiment 1. After baseline release, both treatments fully read and SHA-256-check a new, independently copied memory lower file. File-warm retains those cached pages. File-cold applies one `POSIX_FADV_DONTNEED` operation only to that private file. Immediately before timing, `mincore` must show **14,472/14,472 resident pages for file-warm or 0/14,472 for file-cold**; every formal trial met its threshold. The file is 59,277,312 bytes, with 4,096-byte host pages. The final residency probe is outside the timer, with a recorded probe-to-timer gap of at most 1 microsecond. No file-content read or diagnostic I/O occurs in that gap. No global cache clearing or changes to other users' files, services, or devices are performed. **Timing scope:** Timing starts immediately before the normal restore constructor and `start()`, and ends when `start()` returns at envd-ready. Memory-device acquisition, manifest validation, prefetch, VM restore, and guest readiness are included. Repository resolution, the baseline restore/release, file hashing, cache preparation, and the final pre-timer residency probe are excluded. Each trial has its own process and private runtime; both baseline and timed memory-device releases are acknowledged, and final checks show no remaining experiment processes or Unix sockets. Paired differences are calculated within each block. The intervals below are descriptive 95% Student-t intervals over eight paired differences (7 degrees of freedom): | Paired comparison | Mean difference | 95% interval | | --- | ---: | ---: | | File-warm: prefetch on minus off | -35.480 ms | [-44.337, -26.622] ms | | File-cold: prefetch on minus off | -163.181 ms | [-185.740, -140.622] ms | | Prefetch off: file-cold minus file-warm | +171.292 ms | [+150.333, +192.251] ms | | Prefetch on: file-cold minus file-warm | +43.590 ms | [+35.241, +51.939] ms | Prefetch is faster in all eight pairs under each file-cache condition. These measurements support a substantial benefit for this fixed snapshot when its memory lower-file pages are initially absent from the host page cache, and show that the file-cache state materially affects measured restore latency. **Scope of “file-cold”:** Only this memory lower file's Linux page-cache residency is controlled. Rootfs file pages, filesystem metadata, process state, and NVMe controller/SSD caches may remain warm. This measures snapshot restore to envd-ready, rather than Ubuntu boot from scratch or the entire user-request-to-ready path. Logical ublk counters are not physical disk traffic or guest page-fault counts; post-ready residency and counters have their own observation timestamps. Experiments 1 and 2 use different measurement builds and cache treatments, and their samples are reported separately. ## Limitations Prefetch competes with on-demand reads for CPU, cache, and device bandwidth. Cancellation and timeouts can prevent subsequent reads but cannot interrupt reads already blocked in the kernel. Such reads may delay final device release; the VM is requested to stop before waiting. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:23 +00:00