[PR #424] [CLOSED] feat(snapshot): bound and reclaim managed snapshot storage #426

New issue

Closed

opened 2026-10-29 14:23:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:23:17 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/424
Author: @guozy18
Created: 10/22/2026
Status: ❌ Closed

Base: main ← Head: sync/managed-snapshot-quota-reclamation

📝 Commits (2)

526131d feat(sandbox): add per-sandbox max managed-snapshot/ folder size
ab889be fix(snapshot): reclaim managed generations safely

📊 Changes

10 files changed (+764 additions, -112 deletions)

View changed files

📝 config/default.toml (+7 -0)
📝 docs/src/configuration/env-vars.md (+1 -0)
📝 docs/src/configuration/reference.md (+1 -0)
📝 src/cfg.rs (+39 -0)
📝 src/sandbox/firecracker/overlaybd_snapshot.rs (+106 -34)
📝 src/sandbox/firecracker/sandbox.rs (+265 -46)
📝 storage/ublk-daemon/src/client.rs (+123 -30)
📝 tests/integration/fc.rs (+75 -0)
📝 tests/integration/snapshot.rs (+1 -0)
📝 tests/integration/snapshot_attached_drive.rs (+146 -2)

📄 Description

What

Reclaim unused managed snapshot generations independently, and add an optional capacity limit that rejects new managed captures once the configured usage threshold is reached.

Why

Before: snapshot and pause artifacts already live in separate <sandbox_id>/<uuid>/ directories, but their lifetime is tied to one shared PersistentSnapshotRootGuard for the sandbox root. A long-lived sandbox or fork tree can therefore keep obsolete generations on disk even after their individual snapshots are no longer needed.

After: each generation has its own guard. Captured snapshots, paused states and the launch memory baseline retain only the generation they need. A generation is deleted when its last reference drops; running VMs keep independent references to the disk layers they still use.

Source changes: 49db0db and 30324f0.

Scope and non-goals

Managed snapshot ownership, capacity checks and capture failure handling. Temporal pause is outside this PR.

Design and behavior changes

Reclamation is safe because:

After restack, managed lower layers referenced by the live image.json are retained in a sibling runtime-layers/ directory (overlaybd/runtime-layers/ for the root filesystem). The implementation prefers hard links and falls back to sparse copies when linking fails. Removing the source generation therefore does not invalidate those layer paths in the live device's configuration.
Snapshot staging adopts runtime-owned lower layers, including runtime-layers/, into the new generation's inherited-layers/, or compacts them into managed-base.commit. Each snapshot's artifact configuration remains self-contained.

Memory captures keep the launch baseline plus one cumulative delta, rather than chaining every intermediate capture. Firecracker dirty bits accumulate since launch, so the launch baseline remains pinned for as long as it is needed.

During a full snapshot/pause capture, one mutation flag is shared across rootfs and attached-drive restacks. Once any restack changes live writable state, a later failure in that capture is terminal. At the daemon RPC layer, a failed send or lost response is terminal once sending has begun, because the restack outcome is unknown. Failure before sending is not marked terminal by the RPC client; the enclosing capture still applies its mutation tracking. The separate volume-only snapshot path retains per-volume failure classification.

The capacity check runs before creating a managed generation:

Configure [firecracker].managed_snapshot_max_total_bytes or AENV_FIRECRACKER_MANAGED_SNAPSHOT_MAX_TOTAL_BYTES; the default is unlimited.
Measure allocated file blocks under the resolved managed snapshot root, counting each device/inode once and skipping symlinks. Fork descendants can share the inherited root and its limit.
Reject a new managed snapshot, fork capture or managed pause when current usage is at or above the limit. The VM has not been paused at this point, so rejection leaves it running without a compensating resume.

Compatibility and operations

The managed directory shape remains <managed-snapshot-base>/<sandbox_id>/<uuid>/. The change is the ownership and reclamation unit: one guard per generation instead of one guard for the whole sandbox root. Runtime layer retention is handled separately in a runtime-layers/ directory next to each live image configuration.

The limit is a pre-capture admission check, not a reservation or a strict write-time disk quota. A single capture or concurrent captures can exceed it. Referenced generations are never automatically evicted, and caller-managed persistent pause directories retain their existing behavior. Leaving the limit unset preserves the previous admission behavior.

Public API schemas and published snapshot formats are unchanged.

Validation

Passed on Linux, with real KVM integration checks:

make fmt and git diff --check.
cargo clippy --locked --workspace --all-targets --all-features -- -D warnings.
cargo test --locked -p agentenv --lib: 1,027 passed, 4 ignored.
cargo test --locked -p uvm-ublk-daemon --lib: 52 passed.
cargo test --locked -p uvm-ublk-daemon --test ublk_daemon_test: 37 passed.
16 KVM integration checks: capture/resume (7), attached-drive and volume ownership/failures (5), snapshot publication/lifecycle (3), and zero-quota rejection with the guest still running (1).

The four ignored library tests require privileged network/capability fixtures. Full HTTP/OSS E2E and performance benchmarks were not run as part of this validation.

Risks and reviewer notes

Review generation ownership in sandbox.rs, runtime lower-layer retention and staging in overlaybd_snapshot.rs, and handling of unknown restack outcomes in the daemon client. These paths determine when artifacts can be deleted and when a failed capture can safely resume the VM.

Checklist

One coherent storage lifecycle change, with regression coverage.
Configuration and environment-variable documentation updated.
Generated API files unchanged.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/424 **Author:** [@guozy18](https://github.com/guozy18) **Created:** 10/22/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `sync/managed-snapshot-quota-reclamation` --- ### 📝 Commits (2) - [`526131d`](https://github.com/kvcache-ai/AgentENV/commit/526131d875a988068f138b1ca83b0a34026ade47) feat(sandbox): add per-sandbox max managed-snapshot/ folder size - [`ab889be`](https://github.com/kvcache-ai/AgentENV/commit/ab889be6a5708bc87a90e683999a74b1c65ecdf0) fix(snapshot): reclaim managed generations safely ### 📊 Changes **10 files changed** (+764 additions, -112 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+7 -0) 📝 `docs/src/configuration/env-vars.md` (+1 -0) 📝 `docs/src/configuration/reference.md` (+1 -0) 📝 `src/cfg.rs` (+39 -0) 📝 `src/sandbox/firecracker/overlaybd_snapshot.rs` (+106 -34) 📝 `src/sandbox/firecracker/sandbox.rs` (+265 -46) 📝 `storage/ublk-daemon/src/client.rs` (+123 -30) 📝 `tests/integration/fc.rs` (+75 -0) 📝 `tests/integration/snapshot.rs` (+1 -0) 📝 `tests/integration/snapshot_attached_drive.rs` (+146 -2) </details> ### 📄 Description ## What Reclaim unused managed snapshot generations independently, and add an optional capacity limit that rejects new managed captures once the configured usage threshold is reached. ## Why **Before:** snapshot and pause artifacts already live in separate `<sandbox_id>/<uuid>/` directories, but their lifetime is tied to one shared `PersistentSnapshotRootGuard` for the sandbox root. A long-lived sandbox or fork tree can therefore keep obsolete generations on disk even after their individual snapshots are no longer needed. **After:** each generation has its own guard. Captured snapshots, paused states and the launch memory baseline retain only the generation they need. A generation is deleted when its last reference drops; running VMs keep independent references to the disk layers they still use. ## Related issue Source changes: `49db0db` and `30324f0`. ## Scope and non-goals Managed snapshot ownership, capacity checks and capture failure handling. Temporal pause is outside this PR. ## Design and behavior changes Reclamation is safe because: 1. After restack, managed lower layers referenced by the live `image.json` are retained in a sibling `runtime-layers/` directory (`overlaybd/runtime-layers/` for the root filesystem). The implementation prefers hard links and falls back to sparse copies when linking fails. Removing the source generation therefore does not invalidate those layer paths in the live device's configuration. 2. Snapshot staging adopts runtime-owned lower layers, including `runtime-layers/`, into the new generation's `inherited-layers/`, or compacts them into `managed-base.commit`. Each snapshot's artifact configuration remains self-contained. Memory captures keep the launch baseline plus one cumulative delta, rather than chaining every intermediate capture. Firecracker dirty bits accumulate since launch, so the launch baseline remains pinned for as long as it is needed. During a full snapshot/pause capture, one mutation flag is shared across rootfs and attached-drive restacks. Once any restack changes live writable state, a later failure in that capture is terminal. At the daemon RPC layer, a failed send or lost response is terminal once sending has begun, because the restack outcome is unknown. Failure before sending is not marked terminal by the RPC client; the enclosing capture still applies its mutation tracking. The separate volume-only snapshot path retains per-volume failure classification. The capacity check runs before creating a managed generation: - Configure `[firecracker].managed_snapshot_max_total_bytes` or `AENV_FIRECRACKER_MANAGED_SNAPSHOT_MAX_TOTAL_BYTES`; the default is unlimited. - Measure allocated file blocks under the resolved managed snapshot root, counting each device/inode once and skipping symlinks. Fork descendants can share the inherited root and its limit. - Reject a new managed snapshot, fork capture or managed pause when current usage is at or above the limit. The VM has not been paused at this point, so rejection leaves it running without a compensating resume. ## Compatibility and operations The managed directory shape remains `<managed-snapshot-base>/<sandbox_id>/<uuid>/`. The change is the ownership and reclamation unit: one guard per generation instead of one guard for the whole sandbox root. Runtime layer retention is handled separately in a `runtime-layers/` directory next to each live image configuration. The limit is a pre-capture admission check, not a reservation or a strict write-time disk quota. A single capture or concurrent captures can exceed it. Referenced generations are never automatically evicted, and caller-managed persistent pause directories retain their existing behavior. Leaving the limit unset preserves the previous admission behavior. Public API schemas and published snapshot formats are unchanged. ## Validation Passed on Linux, with real KVM integration checks: - `make fmt` and `git diff --check`. - `cargo clippy --locked --workspace --all-targets --all-features -- -D warnings`. - `cargo test --locked -p agentenv --lib`: 1,027 passed, 4 ignored. - `cargo test --locked -p uvm-ublk-daemon --lib`: 52 passed. - `cargo test --locked -p uvm-ublk-daemon --test ublk_daemon_test`: 37 passed. - 16 KVM integration checks: capture/resume (7), attached-drive and volume ownership/failures (5), snapshot publication/lifecycle (3), and zero-quota rejection with the guest still running (1). The four ignored library tests require privileged network/capability fixtures. Full HTTP/OSS E2E and performance benchmarks were not run as part of this validation. ## Risks and reviewer notes Review generation ownership in `sandbox.rs`, runtime lower-layer retention and staging in `overlaybd_snapshot.rs`, and handling of unknown restack outcomes in the daemon client. These paths determine when artifacts can be deleted and when a failed capture can safely resume the VM. ## Checklist - [x] One coherent storage lifecycle change, with regression coverage. - [x] Configuration and environment-variable documentation updated. - [x] Generated API files unchanged. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:23:17 +00:00