[PR #256] [MERGED] feat(snapshot): compress layers at publish time when uploading to OSS/ACR #297

New issue

Closed

opened 2026-10-29 14:21:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/256
Author: @huajq
Created: 9/7/2026
Status: ✅ Merged
Merged: 9/8/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/snapshot-publish-compression

📝 Commits (1)

d03b5a6 feat(snapshot): compress layers at publish time when uploading to OSS/ACR

📊 Changes

25 files changed (+1316 additions, -482 deletions)

View changed files

📝 config/default.toml (+14 -24)
📝 crates/e2e-tests/tests/snapshot_oss_e2e_test.rs (+243 -15)
📝 docs/src/configuration/reference.md (+22 -16)
📝 docs/src/internals/p2p-design.md (+2 -0)
📝 src/cfg.rs (+19 -25)
📝 src/image/local_layer.rs (+4 -2)
📝 src/sandbox/firecracker/overlaybd_snapshot.rs (+8 -124)
📝 src/sandbox/firecracker/sandbox.rs (+4 -28)
📝 src/sandbox/mod.rs (+1 -1)
📝 src/sandbox/ublk/overlaybd.rs (+26 -41)
📝 src/snapshot/image_export/service.rs (+2 -0)
📝 src/snapshot/manager.rs (+25 -2)
📝 src/snapshot/p2p.rs (+106 -12)
📝 src/snapshot/repository/backends/common/acr/client.rs (+1 -1)
📝 src/snapshot/repository/backends/common/acr/publisher.rs (+316 -25)
📝 src/snapshot/repository/backends/common/mod.rs (+1 -0)
➕ src/snapshot/repository/backends/common/recontainerize.rs (+379 -0)
📝 src/snapshot/repository/backends/mod.rs (+1 -0)
📝 src/snapshot/repository/backends/oss/mod.rs (+22 -2)
📝 src/snapshot/repository/backends/oss/repository.rs (+97 -33)

...and 5 more files

📄 Description

What

Redesign snapshot layer compression: local layers always stay raw, and compression happens once when layers are uploaded to OSS/ACR. A single new switch, [snapshot.publish_compression] (enabled / algorithm (lz4|zstd) / workers, on by default), replaces the legacy capture-time knobs — [memory_snapshot].compression_* and [template_build] are removed from the configuration schema.

Why

Benchmarks show capture-time compression is a local pessimization: the guest working set is unchanged by compression (~62 MiB in the test setup), and while layer files are page-cache-warm the saved bytes never materialize — yet decompression CPU is paid on every fault (cold resume: raw 63.8 ms vs lz4 +18% vs zstd +106%; hot path identical at ~15 ms). The bytes compression actually saves only matter on the network path: OSS/ACR upload and cross-node resume.

Closes # (no tracking issue; motivation is the measurement analysis above)

Scope and non-goals

Included: publish-time compression in the OSS managed-layer and ACR source-registry upload paths (memory layers, rootfs read-write deltas, attached-drive deltas uniformly); removal of capture-time compression and the template-build compression override pipeline; P2P digest-key consistency guard; config, docs, and tests.
Non-goals: no Repository trait changes; POSIX backend unchanged (local stays raw); zero read-path changes (ZFile is already transparent via switch-file magic detection); full P2P alignment for compressed digests (left as a TODO).

Design and behavior changes

Capture path: memory / rootfs / attached-drive layers are always sealed raw (daemon close_seal unchanged).
Publish path: new shared helper prepare_layer_upload (common/recontainerize.rs) probes is_zfile, recontainerizes raw local layers with compact_layers into a temporary ZFile, then hashes and uploads the compressed bytes. ManagedLayer.digest/size, OSS object keys, and ACR blob annotations all reference the actually-uploaded bytes. Remote lowers are re-referenced untouched; already-ZFile inputs are passed through (idempotent).
Resolution/resume: unchanged — consuming nodes auto-detect the format by content magic and need no configuration or flag.
Failure handling: compression/hash failures abort the publish through the existing rollback path (delete_prefix); no partial uploads; temp files are cleaned up by guard.
Known trade-off: ZFile layers record uuid = None, so P2P uuid-keyed acceleration does not apply to them (the daemon virtual-size probe falls back to a full ImageFile open, one-time).

Compatibility and operations

Public API or generated protocol: unchanged.
Configuration or defaults: new [snapshot.publish_compression] section (enabled by default, lz4) — OSS/ACR uploads start compressing on upgrade; set enabled = false to keep raw uploads. Legacy keys removed from the schema; unknown TOML keys are ignored by the config parser, so stale configs keep starting.
Snapshot manifest, artifact layout, or storage format: when enabled, manifest digests/sizes reference compressed bytes; readers auto-detect by content magic, and existing repositories (raw or previously compressed layers) remain readable.
Upgrade and rollback: rollback-safe — disabling the switch resumes raw uploads; no data migration needed. Note the default is on, so existing OSS deployments will upload compressed layers after upgrade (readers auto-detect; downgrade-safe).
Host requirements, permissions, ports, or dependencies: none added.

Validation

make fmt (cargo fmt --all -- --check)
make clippy (cargo clippy -p agentenv -p agentenv-e2e-tests --all-targets -- -D warnings)
make test-unit (cargo test -p agentenv --lib: 817/818; the single failure is a pre-existing flaky api::proxy test that passes in isolation, unrelated)
Relevant Rust integration tests (fc::memory_snapshot_format_matches_config_and_resumes, fc::backend_pause_state_round_trips_through_encoded_artifacts)
Documentation updated (AGENTS/CLAUDE.md, reference.md, default.toml, p2p-design.md)
Benchmarks or performance comparison completed

Commands and results:

cargo fmt --all -- --check                                    # pass
cargo clippy -p agentenv -p agentenv-e2e-tests --all-targets -- -D warnings  # pass
cargo test -p agentenv --lib                                  # 817 passed, 1 flaky (unrelated)

# OSS e2e (minio), incl. the new compression case
cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test -- --include-ignored --test-threads=1
# 5 passed (incl. snapshot_oss_publish_compresses_raw_layers_when_enabled)

# KVM integration (real Firecracker pause/resume)
cargo test -p agentenv --test integration --release -- fc::memory_snapshot_format_matches_config_and_resumes fc::backend_pause_state_round_trips_through_encoded_artifacts
# 2 passed

# Real OSS end-to-end (bucket agentenv-oss-validation):
# publish (compression on): memory layer 53MB -> 34.7MB; fresh-home cross-node
# resume succeeded with guest data intact; cached layer verified as ZFile magic;
# guest runtime upper stays raw.

Performance comparison (real OSS, release build; template-build scenario, memory layer 87.8 → 31.9 MiB, 2.75×):

Metric	raw	publish_compression (lz4)
Build publish phase (capture + compress + upload)	2.28 s	1.02 s (2.2× faster)
Memory-layer upload time	1.60 s	0.49 s
Cold start from template (fresh node, OSS caches cleared, mean of 3)	3.01 s (3.42/3.17/2.44)	2.05 s (2.02/2.07/2.06), −32%
Bytes downloaded during cold start	~64 MiB	~30 MiB

Evidence for removing capture-time compression: cold resume raw 63.8 ms vs lz4 75.1 ms (+18%) vs zstd 131.4 ms (+106%); hot path identical at ~15 ms; guest working set (~62 MiB) is codec-independent.

Skipped checks and reasons: make -C services test (services/ untouched); no generated-code changes.

Risks and reviewer notes

Core invariant: manifest digest/size must equal the actually-uploaded (compressed) bytes — guarded by unit tests in recontainerize.rs and the minio e2e. Please focus review on common/recontainerize.rs (new), the three upload branches in oss/repository.rs, and acr/publisher.rs::upload_local_delta.
P2P: digest-keyed publication is skipped when the committed record names different bytes than the local descriptor (conservative; see the TODO in p2p.rs about propagating compressed paths to restore acceleration).
ZFile layers get uuid = None: P2P uuid-keyed acceleration is unavailable for them (pre-existing graceful degradation, not a regression).
Benchmarks ran on a shared test host, so absolute numbers carry some noise; the compressed runs show noticeably tighter variance (2.02–2.07 s vs 2.44–3.42 s) because less of the critical path is network-exposed.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/256 **Author:** [@huajq](https://github.com/huajq) **Created:** 9/7/2026 **Status:** ✅ Merged **Merged:** 9/8/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/snapshot-publish-compression` --- ### 📝 Commits (1) - [`d03b5a6`](https://github.com/kvcache-ai/AgentENV/commit/d03b5a6cfdd3aa3b9128f0404019b9a821dad56a) feat(snapshot): compress layers at publish time when uploading to OSS/ACR ### 📊 Changes **25 files changed** (+1316 additions, -482 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+14 -24) 📝 `crates/e2e-tests/tests/snapshot_oss_e2e_test.rs` (+243 -15) 📝 `docs/src/configuration/reference.md` (+22 -16) 📝 `docs/src/internals/p2p-design.md` (+2 -0) 📝 `src/cfg.rs` (+19 -25) 📝 `src/image/local_layer.rs` (+4 -2) 📝 `src/sandbox/firecracker/overlaybd_snapshot.rs` (+8 -124) 📝 `src/sandbox/firecracker/sandbox.rs` (+4 -28) 📝 `src/sandbox/mod.rs` (+1 -1) 📝 `src/sandbox/ublk/overlaybd.rs` (+26 -41) 📝 `src/snapshot/image_export/service.rs` (+2 -0) 📝 `src/snapshot/manager.rs` (+25 -2) 📝 `src/snapshot/p2p.rs` (+106 -12) 📝 `src/snapshot/repository/backends/common/acr/client.rs` (+1 -1) 📝 `src/snapshot/repository/backends/common/acr/publisher.rs` (+316 -25) 📝 `src/snapshot/repository/backends/common/mod.rs` (+1 -0) ➕ `src/snapshot/repository/backends/common/recontainerize.rs` (+379 -0) 📝 `src/snapshot/repository/backends/mod.rs` (+1 -0) 📝 `src/snapshot/repository/backends/oss/mod.rs` (+22 -2) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+97 -33) _...and 5 more files_ </details> ### 📄 Description ## What Redesign snapshot layer compression: local layers always stay raw, and compression happens once when layers are uploaded to OSS/ACR. A single new switch, `[snapshot.publish_compression]` (`enabled` / `algorithm` (`lz4`|`zstd`) / `workers`, on by default), replaces the legacy capture-time knobs — `[memory_snapshot].compression_*` and `[template_build]` are removed from the configuration schema. ## Why Benchmarks show capture-time compression is a local pessimization: the guest working set is unchanged by compression (~62 MiB in the test setup), and while layer files are page-cache-warm the saved bytes never materialize — yet decompression CPU is paid on every fault (cold resume: raw 63.8 ms vs lz4 +18% vs zstd +106%; hot path identical at ~15 ms). The bytes compression actually saves only matter on the network path: OSS/ACR upload and cross-node resume. ## Related issue Closes # (no tracking issue; motivation is the measurement analysis above) ## Scope and non-goals - Included: publish-time compression in the OSS managed-layer and ACR source-registry upload paths (memory layers, rootfs read-write deltas, attached-drive deltas uniformly); removal of capture-time compression and the template-build compression override pipeline; P2P digest-key consistency guard; config, docs, and tests. - Non-goals: no `Repository` trait changes; POSIX backend unchanged (local stays raw); zero read-path changes (ZFile is already transparent via switch-file magic detection); full P2P alignment for compressed digests (left as a TODO). ## Design and behavior changes - Capture path: memory / rootfs / attached-drive layers are always sealed raw (daemon `close_seal` unchanged). - Publish path: new shared helper `prepare_layer_upload` (`common/recontainerize.rs`) probes `is_zfile`, recontainerizes raw local layers with `compact_layers` into a temporary ZFile, then hashes and uploads the compressed bytes. `ManagedLayer.digest/size`, OSS object keys, and ACR blob annotations all reference the actually-uploaded bytes. Remote lowers are re-referenced untouched; already-ZFile inputs are passed through (idempotent). - Resolution/resume: unchanged — consuming nodes auto-detect the format by content magic and need no configuration or flag. - Failure handling: compression/hash failures abort the publish through the existing rollback path (`delete_prefix`); no partial uploads; temp files are cleaned up by guard. - Known trade-off: ZFile layers record `uuid = None`, so P2P uuid-keyed acceleration does not apply to them (the daemon virtual-size probe falls back to a full `ImageFile` open, one-time). ## Compatibility and operations - Public API or generated protocol: unchanged. - Configuration or defaults: new `[snapshot.publish_compression]` section (**enabled by default**, `lz4`) — OSS/ACR uploads start compressing on upgrade; set `enabled = false` to keep raw uploads. Legacy keys removed from the schema; unknown TOML keys are ignored by the config parser, so stale configs keep starting. - Snapshot manifest, artifact layout, or storage format: when enabled, manifest digests/sizes reference compressed bytes; readers auto-detect by content magic, and existing repositories (raw or previously compressed layers) remain readable. - Upgrade and rollback: rollback-safe — disabling the switch resumes raw uploads; no data migration needed. Note the default is **on**, so existing OSS deployments will upload compressed layers after upgrade (readers auto-detect; downgrade-safe). - Host requirements, permissions, ports, or dependencies: none added. ## Validation - [x] `make fmt` (`cargo fmt --all -- --check`) - [x] `make clippy` (`cargo clippy -p agentenv -p agentenv-e2e-tests --all-targets -- -D warnings`) - [x] `make test-unit` (`cargo test -p agentenv --lib`: 817/818; the single failure is a pre-existing flaky `api::proxy` test that passes in isolation, unrelated) - [x] Relevant Rust integration tests (`fc::memory_snapshot_format_matches_config_and_resumes`, `fc::backend_pause_state_round_trips_through_encoded_artifacts`) - [x] Documentation updated (AGENTS/CLAUDE.md, `reference.md`, `default.toml`, `p2p-design.md`) - [x] Benchmarks or performance comparison completed Commands and results: ```text cargo fmt --all -- --check # pass cargo clippy -p agentenv -p agentenv-e2e-tests --all-targets -- -D warnings # pass cargo test -p agentenv --lib # 817 passed, 1 flaky (unrelated) # OSS e2e (minio), incl. the new compression case cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test -- --include-ignored --test-threads=1 # 5 passed (incl. snapshot_oss_publish_compresses_raw_layers_when_enabled) # KVM integration (real Firecracker pause/resume) cargo test -p agentenv --test integration --release -- fc::memory_snapshot_format_matches_config_and_resumes fc::backend_pause_state_round_trips_through_encoded_artifacts # 2 passed # Real OSS end-to-end (bucket agentenv-oss-validation): # publish (compression on): memory layer 53MB -> 34.7MB; fresh-home cross-node # resume succeeded with guest data intact; cached layer verified as ZFile magic; # guest runtime upper stays raw. ``` Performance comparison (real OSS, release build; template-build scenario, memory layer 87.8 → 31.9 MiB, 2.75×): | Metric | raw | publish_compression (lz4) | |---|---|---| | Build publish phase (capture + compress + upload) | 2.28 s | **1.02 s (2.2× faster)** | | Memory-layer upload time | 1.60 s | 0.49 s | | Cold start from template (fresh node, OSS caches cleared, mean of 3) | 3.01 s (3.42/3.17/2.44) | **2.05 s (2.02/2.07/2.06), −32%** | | Bytes downloaded during cold start | ~64 MiB | ~30 MiB | Evidence for removing capture-time compression: cold resume raw 63.8 ms vs lz4 75.1 ms (+18%) vs zstd 131.4 ms (+106%); hot path identical at ~15 ms; guest working set (~62 MiB) is codec-independent. Skipped checks and reasons: `make -C services test` (`services/` untouched); no generated-code changes. ## Risks and reviewer notes - Core invariant: manifest digest/size must equal the actually-uploaded (compressed) bytes — guarded by unit tests in `recontainerize.rs` and the minio e2e. Please focus review on `common/recontainerize.rs` (new), the three upload branches in `oss/repository.rs`, and `acr/publisher.rs::upload_local_delta`. - P2P: digest-keyed publication is skipped when the committed record names different bytes than the local descriptor (conservative; see the TODO in `p2p.rs` about propagating compressed paths to restore acceleration). - ZFile layers get `uuid = None`: P2P uuid-keyed acceleration is unavailable for them (pre-existing graceful degradation, not a regression). - Benchmarks ran on a shared test host, so absolute numbers carry some noise; the compressed runs show noticeably tighter variance (2.02–2.07 s vs 2.44–3.42 s) because less of the critical path is network-exposed. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:55 +00:00