[PR #347] [MERGED] feat(image): build and publish reusable images with the CLI #367

New issue

Closed

opened 2026-10-29 14:22:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-29 14:22:40 +00:00

Owner

Copy link

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/347
Author: @Primary33
Created: 10/9/2026
Status: ✅ Merged
Merged: 10/16/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/local-image-builds

---

📝 Commits (1)

• 2c295eb feat: build and publish reusable images with the CLI

📊 Changes

28 files changed (+2169 additions, -206 deletions)

View changed files

📝 crates/aenv/src/client/templates.rs (+10 -0)
📝 crates/aenv/src/commands/build.rs (+116 -32)
📝 crates/aenv/src/commands/pull.rs (+1 -0)
📝 crates/aenv/src/main.rs (+1 -1)
📝 docs/src/concepts/sandboxes/starting.md (+56 -1)
📝 scripts/buildkit/test.py (+33 -0)
📝 src/api/generated/src/models.rs (+37 -1)
📝 src/api/impls/image_build.rs (+223 -85)
📝 src/api/impls/image_build/cleanup.rs (+9 -3)
➕ src/api/impls/image_build/record.rs (+206 -0)
📝 src/api/impls/image_build/tests.rs (+388 -5)
📝 src/api/impls/image_build/worker.rs (+5 -5)
📝 src/api/impls/template.rs (+84 -12)
📝 src/api/openapi.yml (+21 -4)
📝 src/api/proxy.rs (+3 -2)
📝 src/bin/server.rs (+2 -2)
📝 src/digest.rs (+10 -0)
📝 src/image/buildkit.rs (+1 -6)
📝 src/image/cache/graph.rs (+69 -37)
📝 src/image/cache/service.rs (+78 -2)

...and 8 more files

📄 Description

What

Build and publish reusable OverlayBD images through the shared repository with aenv build --image --name NAME[:TAG] [--tag TAG]... CONTEXT, and reference them as aenv/name:tag or by digest.

Image builds reuse the template build API instead of adding build endpoints: POST /v3/templates with "imageOnly": true creates the build, and the existing builder, status, logs, and delete endpoints drive it. Image tags belong to their image name, like E2B template tags, and a build points its tags at the image it publishes.

Why

Users need a complete image-building flow whose results survive local cache eviction and resolve on another repository-backed node. A digest alone forces every consumer to copy a new reference after each rebuild, so tags give a published image a mutable name while running workloads keep the immutable digest (review feedback in #378).

Reusing the template build protocol keeps one builder lifecycle, one gateway routing model, and one status/logs surface for both kinds of build.

Related issue

Design proposal: #378.

Builds on #386 (merged), which moved the layer reference types and the volume backing config out of snapshot. The tag management API (#387) builds on this change.

Image-build foundation for the four-part replacement of #325. Independent of Compose runtime (#349); the image catalog (#359) and Compose builds (#348) build on this change and will be rebased onto it.

Scope and non-goals

Standalone image CLI, imageOnly builds on the template build API, image publication and resolution, cache ownership, build-time tags, and aenv/name:tag resolution. Assigning, listing, and deleting tags after a build is #387; listing image names and inspecting or deleting images by digest is #359. Protected tags and reclamation of unreferenced layers are out of scope.

Design and behavior changes

Builds:

• POST /v3/templates with imageOnly: true requires name (name or name:tag); tags adds more tags, and no tag means latest. cpuCount/memoryMB are rejected. The response carries names and tags.
• The build record is a repository document, images/builds/{buildID}.json, not a template record, so template listings and aliases are unaffected and GET /templates/{id} returns 404 for it. Status moves waiting → building → ready/error through the same compare-and-swap pattern as template records, which also excludes a start on another node.
• PUT /templates/{id}/builds/{id}/builder starts the worker and rejects startCmd/readyCmd for imageOnly builds. The BuildKit tunnel, DELETE …/builder, …/status, and …/logs work unchanged; a ready status reports imageDigest. BuildKit builds for the worker's own platform, which is the server's, as for template builds.
• After the solve, the image is imported, the worker is released, then the image is published, its tags are pointed at it, and the record is marked ready, in the same position where a template build publishes its snapshot. Failures are recorded before resources are released. Build finalization is the same as for template builds. The scheduler binding ends with the build; there is no node-local result state.
• DELETE /templates/{id} removes an imageOnly build's record and logs and keeps the published image and its tags.

Images:

• Stdout of aenv build --image contains only sha256:<64 hex digits>, the digest of the canonical OverlayBD description; diagnostics, including the assigned tags, go to stderr.
• Immutable layers and images/descriptions/{digest}.json are committed before ready; local paths are materialized during resolution. Only commit-store-owned files are tracked as local hard commits.
• aenv/name[:tag] is accepted wherever a sha256: reference is. The resolver reads the tag and continues with its digest, so node caches, sandboxes, and snapshots record only the digest and a moved tag never changes an existing workload.

Tags:

• A build points the tag in --name NAME:TAG and every --tag TAG at its image, so a tag always points to an image built under its name.
• Tags are stored at images/tags/{name}/{tag}.json as {imageDigest, createdAtUnixMs}; #387 reads and manages the same documents.

Layering and repository:

• No Repository or backend methods are added. With the moves in #386, ImageResolver::publish_image takes the layer-storing step as a closure (the API passes snapshot::store::store_volume_layers), so image does not depend on snapshot and module_cycles stays acyclic. The server passes the repository when it builds the resolver (ImageResolver::with_repository).
• DocumentKey::Image holds a digest that only DocumentKey::image can construct, so an image key cannot escape the repository. Image documents share the images/ root, and no collection prefix contains another.

Gateway: no changes. imageOnly builds use the template build routes.

Compatibility and operations

• Public API or generated protocol: additive AgentENV extensions to E2B schemas (imageOnly on TemplateBuildRequestV3, imageDigest on TemplateBuildInfo). Short references starting with aenv/ now name image tags instead of resolving through the search registries; fully qualified references such as docker.io/aenv/... are unchanged. A bare sha256: image now means a published image description.
• Configuration or defaults: none.
• Snapshot manifest, artifact layout, or storage format: new repository prefixes images/descriptions/, images/builds/, and images/tags/{name}/.
• Upgrade and rollback: upgrade the CLI and server together. Older servers cannot run imageOnly builds or resolve published digests or tags.
• Host requirements, permissions, ports, or dependencies: builders and consumers must use the same repository; multi-node POSIX deployments require a shared filesystem.

Validation

Rebased onto b086553 (main with #386). The checks below ran on 2238155, whose tree is identical to 2c295eb; the ones marked "via #387" ran on #387 at 06ec1d5, which contains this change. Run on a Linux test host as a non-root user with isolated AENV_TEST_STATE_DIR and AENV_TEST_DEPS_PATH:

• make fmt
• make clippy
• make test-unit
• Relevant Rust integration tests
• make -C services test (required when services/ changes)
• Generated clients/server regenerated with the documented make target
• Documentation updated
• Benchmarks or performance comparison completed

Commands and results:

make fmt                        passed
make clippy                     passed
make test-unit PROFILE=debug    passed: agentenv 1001 passed, 4 ignored; aenv 115 passed;
                                capability, envd, and ublk suites passed. Includes module_cycles,
                                the imageOnly create/builder/cleanup/status/delete tests, image
                                publication and tag resolution tests, and repository layout tests
openapi-generator 7.22.0        regenerating src/api/generated (rust-axum, as cargo adev codegen
                                server does) reproduces the committed code, apart from one rustfmt
                                line break in the Node model that regenerating main produces too
scripts/buildkit/test.py        10 passed against a live server with a POSIX repository, which
                                then holds image documents only under images/{descriptions,tags,builds}/
test_native_image_build (OSS)   passed against a server with an S3-compatible repository
cross-node check (OSS)          node A builds; node B, with its own home and cache and the same
                                bucket, reports the build ready with its imageDigest and cold
                                starts by digest and by aenv/name:v1, reading layers from the bucket
cargo test -p agentenv --test integration --test orchestrator_integration
                                via #387: 23 passed, 1 ignored
cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test -- --ignored
                                via #387: 9 passed
scripts/buildkit/test.py (OSS)  via #387: 10 passed

Skipped checks and reasons: no services/ changes. The integration test config sets [pool.block] enabled = true, because with a config file the ublk daemon reads pool settings only from that file and a partial file disables the pool; this is already the case on main. The cross-node check ran as two isolated servers on one host, and no arm64 host was available. Benchmarks: not applicable.

Risks and reviewer notes

Review the imageOnly branches of the template build endpoints (src/api/impls/template.rs BuildRecord, src/api/impls/image_build.rs, src/api/impls/image_build/record.rs), publication and resolution (src/image/resolver.rs, src/image/mod.rs, src/image/catalog.rs), and cache ownership (src/image/cache/).

Tag writes are last-writer-wins, and a tag is resolved to a digest once per launch. Deleting an imageOnly build checks for an active build only on the node that serves the request, like template deletion. Failed publication may leave immutable layers for reuse.

Changes since the last review:

• Dropped the hard-coded --opt platform=linux/amd64.
• Reverted the unrelated finish_image_build change (the failure-path wait and the log flush order).
• Moved tags and builds to images/tags/ and images/builds/, and descriptions to images/descriptions/, so every image document shares one root.
• Removed the gateway special case; the resolver gets the repository at construction (the hand-written Debug stays because dyn Repository is not Debug); allocate_image_build uses build_record and BuildRecord; removed the CLI's parse_image_tag; ImageTag is {name, tag}; image code shares one now_unix_ms.
• Split the moves into #386 and the tag API into #387. The shared document update helper is left for a follow-up refactor that also covers volume/catalog.rs and snapshot/catalog.rs.

Checklist

• The PR contains one coherent change and no unrelated formatting or refactoring.
• New behavior is covered by tests, or I explained why testing is impractical.
• Logs and examples contain no credentials, tokens, or private registry information.
• I did not manually edit generated code without updating its source and regenerating it.

---

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/347 **Author:** [@Primary33](https://github.com/Primary33) **Created:** 10/9/2026 **Status:** ✅ Merged **Merged:** 10/16/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/local-image-builds` --- ### 📝 Commits (1) - [`2c295eb`](https://github.com/kvcache-ai/AgentENV/commit/2c295eba295f12399632eb713bdd10912936c269) feat: build and publish reusable images with the CLI ### 📊 Changes **28 files changed** (+2169 additions, -206 deletions) <details> <summary>View changed files</summary> 📝 `crates/aenv/src/client/templates.rs` (+10 -0) 📝 `crates/aenv/src/commands/build.rs` (+116 -32) 📝 `crates/aenv/src/commands/pull.rs` (+1 -0) 📝 `crates/aenv/src/main.rs` (+1 -1) 📝 `docs/src/concepts/sandboxes/starting.md` (+56 -1) 📝 `scripts/buildkit/test.py` (+33 -0) 📝 `src/api/generated/src/models.rs` (+37 -1) 📝 `src/api/impls/image_build.rs` (+223 -85) 📝 `src/api/impls/image_build/cleanup.rs` (+9 -3) ➕ `src/api/impls/image_build/record.rs` (+206 -0) 📝 `src/api/impls/image_build/tests.rs` (+388 -5) 📝 `src/api/impls/image_build/worker.rs` (+5 -5) 📝 `src/api/impls/template.rs` (+84 -12) 📝 `src/api/openapi.yml` (+21 -4) 📝 `src/api/proxy.rs` (+3 -2) 📝 `src/bin/server.rs` (+2 -2) 📝 `src/digest.rs` (+10 -0) 📝 `src/image/buildkit.rs` (+1 -6) 📝 `src/image/cache/graph.rs` (+69 -37) 📝 `src/image/cache/service.rs` (+78 -2) _...and 8 more files_ </details> ### 📄 Description ## What Build and publish reusable OverlayBD images through the shared repository with `aenv build --image --name NAME[:TAG] [--tag TAG]... CONTEXT`, and reference them as `aenv/name:tag` or by digest. Image builds reuse the template build API instead of adding build endpoints: `POST /v3/templates` with `"imageOnly": true` creates the build, and the existing builder, status, logs, and delete endpoints drive it. Image tags belong to their image name, like E2B template tags, and a build points its tags at the image it publishes. ## Why Users need a complete image-building flow whose results survive local cache eviction and resolve on another repository-backed node. A digest alone forces every consumer to copy a new reference after each rebuild, so tags give a published image a mutable name while running workloads keep the immutable digest (review feedback in #378). Reusing the template build protocol keeps one builder lifecycle, one gateway routing model, and one status/logs surface for both kinds of build. ## Related issue Design proposal: #378. Builds on #386 (merged), which moved the layer reference types and the volume backing config out of `snapshot`. The tag management API (#387) builds on this change. Image-build foundation for the four-part replacement of #325. Independent of Compose runtime (#349); the image catalog (#359) and Compose builds (#348) build on this change and will be rebased onto it. ## Scope and non-goals Standalone image CLI, imageOnly builds on the template build API, image publication and resolution, cache ownership, build-time tags, and `aenv/name:tag` resolution. Assigning, listing, and deleting tags after a build is #387; listing image names and inspecting or deleting images by digest is #359. Protected tags and reclamation of unreferenced layers are out of scope. ## Design and behavior changes Builds: - `POST /v3/templates` with `imageOnly: true` requires `name` (`name` or `name:tag`); `tags` adds more tags, and no tag means `latest`. `cpuCount`/`memoryMB` are rejected. The response carries `names` and `tags`. - The build record is a repository document, `images/builds/{buildID}.json`, not a template record, so template listings and aliases are unaffected and `GET /templates/{id}` returns 404 for it. Status moves waiting → building → ready/error through the same compare-and-swap pattern as template records, which also excludes a start on another node. - `PUT /templates/{id}/builds/{id}/builder` starts the worker and rejects `startCmd`/`readyCmd` for imageOnly builds. The BuildKit tunnel, `DELETE …/builder`, `…/status`, and `…/logs` work unchanged; a ready status reports `imageDigest`. BuildKit builds for the worker's own platform, which is the server's, as for template builds. - After the solve, the image is imported, the worker is released, then the image is published, its tags are pointed at it, and the record is marked ready, in the same position where a template build publishes its snapshot. Failures are recorded before resources are released. Build finalization is the same as for template builds. The scheduler binding ends with the build; there is no node-local result state. - `DELETE /templates/{id}` removes an imageOnly build's record and logs and keeps the published image and its tags. Images: - Stdout of `aenv build --image` contains only `sha256:<64 hex digits>`, the digest of the canonical OverlayBD description; diagnostics, including the assigned tags, go to stderr. - Immutable layers and `images/descriptions/{digest}.json` are committed before `ready`; local paths are materialized during resolution. Only commit-store-owned files are tracked as local hard commits. - `aenv/name[:tag]` is accepted wherever a `sha256:` reference is. The resolver reads the tag and continues with its digest, so node caches, sandboxes, and snapshots record only the digest and a moved tag never changes an existing workload. Tags: - A build points the tag in `--name NAME:TAG` and every `--tag TAG` at its image, so a tag always points to an image built under its name. - Tags are stored at `images/tags/{name}/{tag}.json` as `{imageDigest, createdAtUnixMs}`; #387 reads and manages the same documents. Layering and repository: - No `Repository` or backend methods are added. With the moves in #386, `ImageResolver::publish_image` takes the layer-storing step as a closure (the API passes `snapshot::store::store_volume_layers`), so `image` does not depend on `snapshot` and `module_cycles` stays acyclic. The server passes the repository when it builds the resolver (`ImageResolver::with_repository`). - `DocumentKey::Image` holds a digest that only `DocumentKey::image` can construct, so an image key cannot escape the repository. Image documents share the `images/` root, and no collection prefix contains another. Gateway: no changes. imageOnly builds use the template build routes. ## Compatibility and operations - Public API or generated protocol: additive AgentENV extensions to E2B schemas (`imageOnly` on `TemplateBuildRequestV3`, `imageDigest` on `TemplateBuildInfo`). Short references starting with `aenv/` now name image tags instead of resolving through the search registries; fully qualified references such as `docker.io/aenv/...` are unchanged. A bare `sha256:` image now means a published image description. - Configuration or defaults: none. - Snapshot manifest, artifact layout, or storage format: new repository prefixes `images/descriptions/`, `images/builds/`, and `images/tags/{name}/`. - Upgrade and rollback: upgrade the CLI and server together. Older servers cannot run imageOnly builds or resolve published digests or tags. - Host requirements, permissions, ports, or dependencies: builders and consumers must use the same repository; multi-node POSIX deployments require a shared filesystem. ## Validation Rebased onto `b086553` (main with #386). The checks below ran on `2238155`, whose tree is identical to `2c295eb`; the ones marked "via #387" ran on #387 at `06ec1d5`, which contains this change. Run on a Linux test host as a non-root user with isolated `AENV_TEST_STATE_DIR` and `AENV_TEST_DEPS_PATH`: - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [x] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [x] Generated clients/server regenerated with the documented `make` target - [x] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results: ```text make fmt passed make clippy passed make test-unit PROFILE=debug passed: agentenv 1001 passed, 4 ignored; aenv 115 passed; capability, envd, and ublk suites passed. Includes module_cycles, the imageOnly create/builder/cleanup/status/delete tests, image publication and tag resolution tests, and repository layout tests openapi-generator 7.22.0 regenerating src/api/generated (rust-axum, as cargo adev codegen server does) reproduces the committed code, apart from one rustfmt line break in the Node model that regenerating main produces too scripts/buildkit/test.py 10 passed against a live server with a POSIX repository, which then holds image documents only under images/{descriptions,tags,builds}/ test_native_image_build (OSS) passed against a server with an S3-compatible repository cross-node check (OSS) node A builds; node B, with its own home and cache and the same bucket, reports the build ready with its imageDigest and cold starts by digest and by aenv/name:v1, reading layers from the bucket cargo test -p agentenv --test integration --test orchestrator_integration via #387: 23 passed, 1 ignored cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test -- --ignored via #387: 9 passed scripts/buildkit/test.py (OSS) via #387: 10 passed ``` Skipped checks and reasons: no `services/` changes. The integration test config sets `[pool.block] enabled = true`, because with a config file the ublk daemon reads pool settings only from that file and a partial file disables the pool; this is already the case on main. The cross-node check ran as two isolated servers on one host, and no arm64 host was available. Benchmarks: not applicable. ## Risks and reviewer notes Review the imageOnly branches of the template build endpoints (`src/api/impls/template.rs` `BuildRecord`, `src/api/impls/image_build.rs`, `src/api/impls/image_build/record.rs`), publication and resolution (`src/image/resolver.rs`, `src/image/mod.rs`, `src/image/catalog.rs`), and cache ownership (`src/image/cache/`). Tag writes are last-writer-wins, and a tag is resolved to a digest once per launch. Deleting an imageOnly build checks for an active build only on the node that serves the request, like template deletion. Failed publication may leave immutable layers for reuse. Changes since the last review: - Dropped the hard-coded `--opt platform=linux/amd64`. - Reverted the unrelated `finish_image_build` change (the failure-path wait and the log flush order). - Moved tags and builds to `images/tags/` and `images/builds/`, and descriptions to `images/descriptions/`, so every image document shares one root. - Removed the gateway special case; the resolver gets the repository at construction (the hand-written `Debug` stays because `dyn Repository` is not `Debug`); `allocate_image_build` uses `build_record` and `BuildRecord`; removed the CLI's `parse_image_tag`; `ImageTag` is `{name, tag}`; image code shares one `now_unix_ms`. - Split the moves into #386 and the tag API into #387. The shared document update helper is left for a follow-up refactor that also covers `volume/catalog.rs` and `snapshot/catalog.rs`. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- _{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

bsakaguchi 2026-10-29 14:22:40 +00:00

• closed this issue
• added the
  pull-request
  label

bsakaguchi referenced this issue 2026-10-29 14:22:48 +00:00

[PR #367] [MERGED] refactor(repository): store build logs and cache state as shared documents #379

bsakaguchi referenced this issue 2026-10-29 14:22:48 +00:00

[PR #368] [MERGED] refactor: move the durable repository out of the snapshot module #381

bsakaguchi referenced this issue 2026-10-29 14:22:49 +00:00

[PR #369] [MERGED] refactor(volume): implement the volume catalog once on repository documents #382

bsakaguchi referenced this issue 2026-10-29 14:22:51 +00:00

[PR #373] [MERGED] test(config): fail when the reference or bundled config drifts from cfg.rs #383

bsakaguchi referenced this issue 2026-10-29 14:22:51 +00:00

[PR #370] [MERGED] refactor(snapshot): move catalog, publish and resolve logic out of repository backends #384

bsakaguchi referenced this issue 2026-10-29 14:22:51 +00:00

[PR #374] [MERGED] refactor(sandbox): reach guest envd through one backend method #386

bsakaguchi referenced this issue 2026-10-29 14:22:52 +00:00

[PR #375] [MERGED] refactor(gateway): route from one table and share sandbox list cursor vectors #387

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#367

No description provided.