[PR #249] [MERGED] fix: reuse published layers when cloning volumes #288

New issue

Closed

opened 2026-10-29 14:21:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/249
Author: @yingdi-shan
Created: 9/6/2026
Status: ✅ Merged
Merged: 9/8/2026
Merged by: @yingdi-shan

Base: main ← Head: fix/volume-layer-reuse

📝 Commits (1)

946e0c4 fix: reuse published layers when cloning volumes

📊 Changes

1 file changed (+312 additions, -19 deletions)

View changed files

📝 src/volume.rs (+312 -19)

📄 Description

What

Reuse committed repository layer references when cloning an unreserved, published volume. Cloning now creates a volume record and materializes its image config on mount, preserving each inherited layer's digest and size. Reserved source volumes continue cloning their local files so unpublished writes are retained.

Why

A template build investigated in #247 spent 33.556 seconds deleting its builder and another 33.375 seconds publishing the next cache seed. Both operations hashed the same inherited 7.43 GB BuildKit cache layer. The local cache config lacked descriptors even though its volume record already contained the published layer references; reproducing the Rust hash loop against that layer took 33.98 seconds.

Reusing published references eliminates seed republication and supplies descriptor-bearing configs to subsequent writable clones, so unchanged inherited layers bypass full-file hashing during publication. Existing cache generations benefit without rewriting their old local configs.

Diagnosed while testing #247. This PR is based directly on main and does not depend on or modify #247.

Scope and non-goals

The change is confined to volume cloning and its regression tests. It applies to both POSIX and OSS repositories through the existing repository interfaces. New local data still follows normal publication and integrity checks.

Design and behavior changes

After validating source status, reservation, and size, an unreserved source with committed layers supplies those references directly to the child.
The existing materializer resolves managed and external layers when the child is mounted. Local cache files from the source are unnecessary, and deleting the parent does not invalidate the child's committed backing.
Reserved sources and sources without committed backing retain the existing local clone path.

Compatibility and operations

Public API or generated protocol: unchanged.
Configuration or defaults: unchanged.
Snapshot manifest, artifact layout, or storage format: unchanged; uses existing volume layer references and lazy materialization.
Upgrade and rollback: no migration. Existing published volumes use the new path immediately.
Host requirements, permissions, ports, or dependencies: unchanged.

Validation

make fmt
make clippy (workspace, all targets and features, warnings denied).
make test-unit: 812 ordinary AgentENV tests, 4 capability tests, 3 Linux capability tests, 8 ublk tests, 46 daemon tests, and both shell verification scripts passed.
cargo test -p agentenv --lib volume::tests: 5 passed.
git merge-tree --write-tree HEAD fork/feat/template-buildkit: clean merge with #247 at c53bc78.

Regression coverage includes stale and absent local configs, cold managers, managed and external layers, parent deletion, successive read-only/writable generations, unpublished writes from reserved sources, and source state/size validation.

Live BuildKit/VM integration and a post-fix wall-clock benchmark have not been run. BuildKit remains in the separate, unmerged #247. Services and generated code are untouched.

Risks and reviewer notes

The optimization requires an unreserved source because a reserved volume can contain local changes newer than its committed backing. Tests exercise that distinction. Content addressing and repository ownership remain authoritative for published layers.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests.
Logs and examples contain no credentials, tokens, or private registry information.
Generated code is untouched.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/249 **Author:** [@yingdi-shan](https://github.com/yingdi-shan) **Created:** 9/6/2026 **Status:** ✅ Merged **Merged:** 9/8/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `fix/volume-layer-reuse` --- ### 📝 Commits (1) - [`946e0c4`](https://github.com/kvcache-ai/AgentENV/commit/946e0c419ee544841459b9a3bcfe8767f7ffd734) fix: reuse published layers when cloning volumes ### 📊 Changes **1 file changed** (+312 additions, -19 deletions) <details> <summary>View changed files</summary> 📝 `src/volume.rs` (+312 -19) </details> ### 📄 Description ## What Reuse committed repository layer references when cloning an unreserved, published volume. Cloning now creates a volume record and materializes its image config on mount, preserving each inherited layer's digest and size. Reserved source volumes continue cloning their local files so unpublished writes are retained. ## Why A template build investigated in #247 spent 33.556 seconds deleting its builder and another 33.375 seconds publishing the next cache seed. Both operations hashed the same inherited 7.43 GB BuildKit cache layer. The local cache config lacked descriptors even though its volume record already contained the published layer references; reproducing the Rust hash loop against that layer took 33.98 seconds. Reusing published references eliminates seed republication and supplies descriptor-bearing configs to subsequent writable clones, so unchanged inherited layers bypass full-file hashing during publication. Existing cache generations benefit without rewriting their old local configs. ## Related issue Diagnosed while testing #247. This PR is based directly on `main` and does not depend on or modify #247. ## Scope and non-goals The change is confined to volume cloning and its regression tests. It applies to both POSIX and OSS repositories through the existing repository interfaces. New local data still follows normal publication and integrity checks. ## Design and behavior changes - After validating source status, reservation, and size, an unreserved source with committed layers supplies those references directly to the child. - The existing materializer resolves managed and external layers when the child is mounted. Local cache files from the source are unnecessary, and deleting the parent does not invalidate the child's committed backing. - Reserved sources and sources without committed backing retain the existing local clone path. ## Compatibility and operations - Public API or generated protocol: unchanged. - Configuration or defaults: unchanged. - Snapshot manifest, artifact layout, or storage format: unchanged; uses existing volume layer references and lazy materialization. - Upgrade and rollback: no migration. Existing published volumes use the new path immediately. - Host requirements, permissions, ports, or dependencies: unchanged. ## Validation - [x] `make fmt` - [x] `make clippy` (workspace, all targets and features, warnings denied). - [x] `make test-unit`: 812 ordinary AgentENV tests, 4 capability tests, 3 Linux capability tests, 8 ublk tests, 46 daemon tests, and both shell verification scripts passed. - [x] `cargo test -p agentenv --lib volume::tests`: 5 passed. - [x] `git merge-tree --write-tree HEAD fork/feat/template-buildkit`: clean merge with #247 at `c53bc78`. Regression coverage includes stale and absent local configs, cold managers, managed and external layers, parent deletion, successive read-only/writable generations, unpublished writes from reserved sources, and source state/size validation. Live BuildKit/VM integration and a post-fix wall-clock benchmark have not been run. BuildKit remains in the separate, unmerged #247. Services and generated code are untouched. ## Risks and reviewer notes The optimization requires an unreserved source because a reserved volume can contain local changes newer than its committed backing. Tests exercise that distinction. Content addressing and repository ownership remain authoritative for published layers. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] Generated code is untouched. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:51 +00:00