[PR #289] [MERGED] Enhance tools with versioned releases and lifecycle test coverage #319

New issue

Closed

opened 2026-10-29 14:22:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/289
Author: @guozy18
Created: 9/14/2026
Status: ✅ Merged
Merged: 9/16/2026
Merged by: @yingdi-shan

Base: main ← Head: main

📝 Commits (3)

733ed8e feat(tools): resolve immutable releases for launch and restore
37edd3d test(tools): cover sharing and version retention in lifecycle tests
1e8dfd3 feat(tools): support reproducible OCI tools releases

📊 Changes

22 files changed (+845 additions, -207 deletions)

View changed files

📝 .github/actions/sandbox-ci-setup/action.yml (+5 -1)
📝 .github/workflows/release.yml (+3 -0)
📝 deploy/docker/Dockerfile.agentenv (+7 -10)
📝 docs/src/configuration/reference.md (+17 -5)
📝 docs/src/internals/sandbox-testing.md (+1 -1)
📝 src/image/oci_image.rs (+3 -2)
📝 src/image/resolver.rs (+49 -0)
📝 src/sandbox/firecracker/config.rs (+1 -16)
📝 src/sandbox/firecracker/sandbox.rs (+53 -14)
📝 src/sandbox/ublk/device.rs (+120 -43)
📝 src/sandbox/ublk/mod.rs (+1 -1)
📝 src/setup/deps.rs (+210 -20)
📝 src/setup/mod.rs (+6 -2)
📝 src/setup/overlaybd.rs (+101 -0)
📝 storage/ublk-daemon/src/client.rs (+1 -1)
📝 storage/ublk-daemon/src/protocol.rs (+4 -3)
📝 storage/ublk-daemon/src/server.rs (+9 -7)
📝 storage/ublk-daemon/tests/ublk_daemon_test.rs (+10 -5)
📝 tests/integration/fc.rs (+179 -0)
📝 tools-image/Dockerfile (+12 -40)

...and 2 more files

📄 Description

What

Publish tools as ordinary OCI rootfs images and load immutable tools releases through shared read-only ublk devices. Snapshots continue to store only tools_drive_version.

For example, after the default changes from 0.1.0 to 0.1.1, new sandboxes use 0.1.1. A cold node restoring a snapshot created with 0.1.0 fetches that recorded release instead of requiring an operator to preinstall its tools drive.

Why

Tools updates should follow normal runtime deployments while retaining old snapshot behavior. Publishers should not have to produce OverlayBD-native images, and snapshot storage should not contain duplicate tools layers.

Implements the tools release and snapshot restore design discussed for this change; no linked issue.

Scope and non-goals

Includes tools image publication, dependency preparation, shared device lifecycle, and launch/restore resolution. Running and paused sandboxes retain their original tools version. This PR does not publish a tools release or change the bundled default version.

Design and behavior changes

flowchart LR
    V[Configured or snapshot tools version] --> R[Resolve immutable tools release]
    R --> O[OCI rootfs: pinned local conversion]
    R --> N[Native OverlayBD: lazy reads]
    O --> U[Shared read-only ublk device]
    N --> U
    R --> E[Legacy wrapper: download tools.ext4]

The tools Dockerfile exports the rootfs directly and adds io.agentenv.tools-drive.format=oci-rootfs-v1. v1 fixes the OverlayBD converter at v1.0.18-aenv.1, layer identity, and sparse 64 GiB geometry so old releases can be reconstructed after runtime upgrades. The virtual capacity is not a 64 GiB download or allocation.
Converted layers are retained in the versioned tools dependency directory, independently of image-cache eviction. Relative paths keep dependency bundles relocatable; --setup-only prepares the default tools release during packaging.
Sandboxes using one release share a read-only ublk device. One idle tools device is retained for reuse. An open block-device fd prevents last-close cache invalidation, while normal memory-pressure reclamation remains possible.
Remote tools layers automatically download in the background after guest readiness, independently of user-disk and memory policies. Converter installation is serialized across concurrent release resolutions.
Tools and BuildKit image imports reuse the OCI conversion pipeline with separate resolver settings. Legacy ext4 wrappers and explicit local ext4 imports remain supported.

Compatibility and operations

Public API: unchanged. The internal ublk acquire request can omit capacity so the daemon obtains it from the image; deploy server and daemon together.
Configuration/defaults: no new user-facing option. config/default.toml is unchanged, and the bundled tools release remains 0.1.0.
Snapshots: retain only the existing version field. Tools do not become managed snapshot layers and are not uploaded to snapshot storage. Nodes must resolve retained versions through the configured tools URL template.
Release/upgrade: publish and validate a new tools image, then update [tools].version in config/deps_manifest.toml for the next runtime build. During the first mixed-runtime rollout, keep deployments on the old tools release until every node supports oci-rootfs-v1, then switch the default.
Rollback: older runtimes cannot restore snapshots created with OCI rootfs tools. Keep published tools versions immutable and available while snapshots reference them.
Host dependencies: reuse existing regctl, OverlayBD, ublk, and Firecracker dependencies. A separate pinned converter is installed only when the runtime's installed converter cannot be reused.

Validation

Validated after rebasing onto main at 715a4f6, in an isolated Linux environment with a non-root runtime user.

cargo fmt --all -- --check
cargo clippy --offline --locked --workspace --all-targets --all-features -- -D warnings
cargo test --offline --locked -p agentenv --lib -- --test-threads=1: 886 passed, 4 ignored; includes BuildKit, image resolution, tools installation, and snapshot tests.
cargo test --offline --locked -p uvm-ublk-daemon --test ublk_daemon_test acquire_overlaybd: 3 passed.
Actual VM integration: default lifecycle and snapshot round-trip; ordinary OCI tools conversion with concurrent shared-device launches; template creation and derived-snapshot launch.

The OCI tools integration test is part of the existing tests/integration/fc.rs suite and uses a local image fixture. It checks a real block device, sharing, read-only access, continued use after a peer stops, and version/device retention across pause/resume. The converter regression reproduces the concurrent-install failure without the lock and verifies one download with the lock.

Four existing ignored library tests remain skipped. No new performance comparison or full BuildKit CLI E2E run is claimed. Go services and generated API schemas are unchanged by this PR.

Risks and reviewer notes

Review tools resolver isolation, converter installation/reproducibility, and shared-device release ordering. Restoring an old release depends on retaining its source image; there is intentionally no snapshot-storage fallback for tools.

Checklist

One coherent tools-distribution change with no unrelated refactoring.
New behavior has focused regression and repository integration coverage.
Examples and validation details contain no credentials or private registry information.
No generated code was manually changed by this PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/289 **Author:** [@guozy18](https://github.com/guozy18) **Created:** 9/14/2026 **Status:** ✅ Merged **Merged:** 9/16/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (3) - [`733ed8e`](https://github.com/kvcache-ai/AgentENV/commit/733ed8eef08358b32ab5caa08115e1c535cc75a3) feat(tools): resolve immutable releases for launch and restore - [`37edd3d`](https://github.com/kvcache-ai/AgentENV/commit/37edd3d275d4cc5e07fc18053a68b6503b7d3858) test(tools): cover sharing and version retention in lifecycle tests - [`1e8dfd3`](https://github.com/kvcache-ai/AgentENV/commit/1e8dfd34e956941460eadab0bc78dd6b54bd6dd9) feat(tools): support reproducible OCI tools releases ### 📊 Changes **22 files changed** (+845 additions, -207 deletions) <details> <summary>View changed files</summary> 📝 `.github/actions/sandbox-ci-setup/action.yml` (+5 -1) 📝 `.github/workflows/release.yml` (+3 -0) 📝 `deploy/docker/Dockerfile.agentenv` (+7 -10) 📝 `docs/src/configuration/reference.md` (+17 -5) 📝 `docs/src/internals/sandbox-testing.md` (+1 -1) 📝 `src/image/oci_image.rs` (+3 -2) 📝 `src/image/resolver.rs` (+49 -0) 📝 `src/sandbox/firecracker/config.rs` (+1 -16) 📝 `src/sandbox/firecracker/sandbox.rs` (+53 -14) 📝 `src/sandbox/ublk/device.rs` (+120 -43) 📝 `src/sandbox/ublk/mod.rs` (+1 -1) 📝 `src/setup/deps.rs` (+210 -20) 📝 `src/setup/mod.rs` (+6 -2) 📝 `src/setup/overlaybd.rs` (+101 -0) 📝 `storage/ublk-daemon/src/client.rs` (+1 -1) 📝 `storage/ublk-daemon/src/protocol.rs` (+4 -3) 📝 `storage/ublk-daemon/src/server.rs` (+9 -7) 📝 `storage/ublk-daemon/tests/ublk_daemon_test.rs` (+10 -5) 📝 `tests/integration/fc.rs` (+179 -0) 📝 `tools-image/Dockerfile` (+12 -40) _...and 2 more files_ </details> ### 📄 Description ## What Publish tools as ordinary OCI rootfs images and load immutable tools releases through shared read-only ublk devices. Snapshots continue to store only `tools_drive_version`. For example, after the default changes from `0.1.0` to `0.1.1`, new sandboxes use `0.1.1`. A cold node restoring a snapshot created with `0.1.0` fetches that recorded release instead of requiring an operator to preinstall its tools drive. ## Why Tools updates should follow normal runtime deployments while retaining old snapshot behavior. Publishers should not have to produce OverlayBD-native images, and snapshot storage should not contain duplicate tools layers. ## Related issue Implements the tools release and snapshot restore design discussed for this change; no linked issue. ## Scope and non-goals Includes tools image publication, dependency preparation, shared device lifecycle, and launch/restore resolution. Running and paused sandboxes retain their original tools version. This PR does not publish a tools release or change the bundled default version. ## Design and behavior changes ```mermaid flowchart LR V[Configured or snapshot tools version] --> R[Resolve immutable tools release] R --> O[OCI rootfs: pinned local conversion] R --> N[Native OverlayBD: lazy reads] O --> U[Shared read-only ublk device] N --> U R --> E[Legacy wrapper: download tools.ext4] ``` - The tools Dockerfile exports the rootfs directly and adds `io.agentenv.tools-drive.format=oci-rootfs-v1`. v1 fixes the OverlayBD converter at `v1.0.18-aenv.1`, layer identity, and sparse 64 GiB geometry so old releases can be reconstructed after runtime upgrades. The virtual capacity is not a 64 GiB download or allocation. - Converted layers are retained in the versioned tools dependency directory, independently of image-cache eviction. Relative paths keep dependency bundles relocatable; `--setup-only` prepares the default tools release during packaging. - Sandboxes using one release share a read-only ublk device. One idle tools device is retained for reuse. An open block-device fd prevents last-close cache invalidation, while normal memory-pressure reclamation remains possible. - Remote tools layers automatically download in the background after guest readiness, independently of user-disk and memory policies. Converter installation is serialized across concurrent release resolutions. - Tools and BuildKit image imports reuse the OCI conversion pipeline with separate resolver settings. Legacy ext4 wrappers and explicit local ext4 imports remain supported. ## Compatibility and operations - **Public API:** unchanged. The internal ublk acquire request can omit capacity so the daemon obtains it from the image; deploy server and daemon together. - **Configuration/defaults:** no new user-facing option. `config/default.toml` is unchanged, and the bundled tools release remains `0.1.0`. - **Snapshots:** retain only the existing version field. Tools do not become managed snapshot layers and are not uploaded to snapshot storage. Nodes must resolve retained versions through the configured tools URL template. - **Release/upgrade:** publish and validate a new tools image, then update `[tools].version` in `config/deps_manifest.toml` for the next runtime build. During the first mixed-runtime rollout, keep deployments on the old tools release until every node supports `oci-rootfs-v1`, then switch the default. - **Rollback:** older runtimes cannot restore snapshots created with OCI rootfs tools. Keep published tools versions immutable and available while snapshots reference them. - **Host dependencies:** reuse existing regctl, OverlayBD, ublk, and Firecracker dependencies. A separate pinned converter is installed only when the runtime's installed converter cannot be reused. ## Validation Validated after rebasing onto `main` at `715a4f6`, in an isolated Linux environment with a non-root runtime user. - [x] `cargo fmt --all -- --check` - [x] `cargo clippy --offline --locked --workspace --all-targets --all-features -- -D warnings` - [x] `cargo test --offline --locked -p agentenv --lib -- --test-threads=1`: 886 passed, 4 ignored; includes BuildKit, image resolution, tools installation, and snapshot tests. - [x] `cargo test --offline --locked -p uvm-ublk-daemon --test ublk_daemon_test acquire_overlaybd`: 3 passed. - [ ] Actual VM integration: default lifecycle and snapshot round-trip; ordinary OCI tools conversion with concurrent shared-device launches; template creation and derived-snapshot launch. The OCI tools integration test is part of the existing `tests/integration/fc.rs` suite and uses a local image fixture. It checks a real block device, sharing, read-only access, continued use after a peer stops, and version/device retention across pause/resume. The converter regression reproduces the concurrent-install failure without the lock and verifies one download with the lock. Four existing ignored library tests remain skipped. No new performance comparison or full BuildKit CLI E2E run is claimed. Go services and generated API schemas are unchanged by this PR. ## Risks and reviewer notes Review tools resolver isolation, converter installation/reproducibility, and shared-device release ordering. Restoring an old release depends on retaining its source image; there is intentionally no snapshot-storage fallback for tools. ## Checklist - [x] One coherent tools-distribution change with no unrelated refactoring. - [x] New behavior has focused regression and repository integration coverage. - [x] Examples and validation details contain no credentials or private registry information. - [x] No generated code was manually changed by this PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:10 +00:00