[PR #254] [MERGED] feat: build templates with managed BuildKit workers #295

New issue

Closed

opened 2026-10-29 14:21:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/254
Author: @yingdi-shan
Created: 9/7/2026
Status: ✅ Merged
Merged: 9/11/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/buildkit-integration

📝 Commits (6)

3b8c195 feat: build templates with managed BuildKit workers
077b995 fix(buildkit): simplify workers and complete build validation
e0d2ffa feat(release): bundle BuildKit client with aenv CLI
0b514f3 test(buildkit): wait for idle node metrics before baseline
2cecd36 fix(buildkit): preserve template API and publish images on server
51c966a fix(buildkit): bound build and tunnel concurrency

📊 Changes

76 files changed (+6953 additions, -731 deletions)

View changed files

📝 .github/workflows/integration-tests.yml (+19 -12)
📝 .github/workflows/release.yml (+25 -9)
📝 Cargo.lock (+7 -10)
📝 Makefile (+12 -1)
📝 build.rs (+10 -0)
➕ config/buildkit-version (+1 -0)
📝 config/default.toml (+11 -0)
📝 crates/aenv/Cargo.toml (+2 -2)
➕ crates/aenv/src/client/buildkit.rs (+199 -0)
📝 crates/aenv/src/client/mod.rs (+6 -0)
📝 crates/aenv/src/commands/build.rs (+388 -512)
📝 crates/aenv/src/main.rs (+2 -2)
📝 crates/aenv/src/progress.rs (+60 -0)
📝 deploy/docker-compose.yml (+6 -0)
📝 docs/src/concepts/templates.md (+173 -21)
📝 docs/src/configuration/reference.md (+17 -0)
📝 docs/src/deployment/docker-compose.md (+9 -0)
📝 docs/src/getting-started/aenv-cli.md (+49 -8)
➕ scripts/buildkit/README.md (+61 -0)
➕ scripts/buildkit/fixture/.dockerignore (+1 -0)

...and 56 more files

📄 Description

What

Build Dockerfile contexts with BuildKit in managed microVMs and automatically publish the resulting runnable template on the server. The CLI streams context and BuildKit traffic; successful completion needs no client image-submission request.

Why

Support standard Dockerfile semantics, multi-stage builds, instruction-cache reuse, and persistent cache mounts while preserving the existing template API and remote storage format.

Closes #147. Split from #247 alongside #252 (volume deletion) and #253 (OCI UID/GID startup).

Scope and non-goals

BuildKit image import, worker/cache lifecycle, node and gateway APIs, CLI packaging, configuration, tests, and documentation. CLI argument compatibility is outside this change. General volume deletion and OCI account resolution remain in the companion PRs; stage selection, SSH forwarding, remote/stdin contexts, and cross-platform emulation remain outside this flow.

Design and behavior changes

Existing POST /v3/templates allocates template/build IDs with its unchanged request and response.
PUT /templates/{templateID}/builds/{buildID}/builder claims the waiting build and prepares its worker. It returns the unique imageName to use with the BuildKit image exporter. Already-started builds return 409.
Poll existing build status for building, then use authenticated WebSocket GET on the builder resource for native BuildKit traffic.
The server watches BuildKit history for successful completion of this build's image exporter, takes its result descriptor, verifies OCI content, and publishes the template. History replay tolerates a dropped observation stream and ignores earlier cached builds. Existing connections can drain before worker shutdown.
Existing status reports final ready or error. DELETE on the builder cancels before publication; publication already in progress finishes independently of the client.

The existing declarative build endpoint keeps its behavior. There is no competing build-creation collection or digest-submission endpoint. Gateway allocation records the build's node binding; subsequent builder and status requests follow it.

Prepared builder snapshots are reused by configuration. Every build receives a separate writable cache child. Cache-head updates and retirements are atomic, cleanup retries retain ownership, and failed/cancelled builds retain the previous seed. Build deadlines and recovery release abandoned workers; public sandbox operations hide builders. Stop BuildKit and checkpoint its cache before stopping the worker. Preserve image startup, health checks, and configuration; package a private aenv-buildctl beside the CLI.

Compatibility and operations

Public API or generated protocol: all 33 pre-existing operations retain their requests and responses, and all 55 pre-existing schemas are unchanged. Only the builder resource is added. Regenerated server bindings are included.
Configuration or defaults: BuildKit v0.33.0, 16 vCPUs, 32768 MiB memory, and a 262144 MiB cache. Smaller hosts can override builder resources.
Snapshot manifest, artifact layout, or storage format: existing catalog keys, artifact paths, and layers need no migration. Builder records and the cache head use the separate template-build namespace. Startup shell is optional; old records retain Bash behavior and round-trip without adding it.
Upgrade and rollback: upgrade the gateway and build nodes before using BuildKit. CLI build arguments use a context directory. Older servers ignore/drop the new shell field when rewriting records, so deriving templates from BuildKit images on an older server can fail or change shell behavior. Existing declarative APIs remain available.
Host requirements, permissions, ports, or dependencies: existing sandbox prerequisites plus bundled buildctl; Linux/macOS CLI uses a private Unix socket.

Validation

make fmt
make clippy
make test-unit
Relevant live BuildKit integration tests (results below)
make -C services test fmt-check vet
make agentenv-server
Documentation updated
Benchmarks or performance comparison

Commands and results:

make fmt clippy test-unit AENV_TEST_STATE_DIR=<isolated path> AENV_TEST_DEPS_PATH=<test dependencies>: passed
  877 AgentENV unit tests, 4 privileged AgentENV tests, 55 CLI tests,
  3 linux-cap tests, 56 ublk/daemon tests, capability runner and installer checks
make -C services test fmt-check vet: passed
make agentenv-server: regenerated the OpenAPI server
make build-server build-aenv: passed
Structural OpenAPI comparison against the merge base: 33 existing operations and 55 schemas unchanged
AENV_BIN=<built CLI> BUILDCTL_BIN=<v0.33.0 buildctl> python3 scripts/buildkit/test.py <eight core cases>: passed
  client exit after solve; context/instruction/cache mounts; concurrent builds;
  failure/cancellation/isolation/recovery; deadlines; startup overrides;
  external Dockerfile/secrets; terminal progress

Focused coverage includes real-repository 404/409 allocation, preserving a template named builds, legacy snapshot serialization, stale history rejection, exporter descriptor selection, failed builds, history reconnection, and final Solve-response delivery during publication.

Live tests use an isolated node with a 2-vCPU, 2048-MiB builder and an 8192-MiB cache. The client-exit regression kills the CLI immediately after a successful BuildKit solve and verifies that the server publishes a runnable template without any image-submission request. Successful cases verify worker and volume cleanup, leaving one ready cache seed.

Risks and reviewer notes

Concurrent builders multiply resource use. Cache sharing follows the repository's API-key trust boundary, and publication time depends on storage throughput. Review the completion watcher in src/image/buildkit/history.rs, builder lifecycle in src/api/impls/image_build/, and gateway allocation/routing.

Merge #252 and #253 first for optimized deletion and numeric image users. The current live run covers the standalone BuildKit branch; numeric-user and combined-branch validation must be repeated with #253. Full Rust integration suites, a live multi-node gateway deployment, and throughput benchmarks were not run for this revision.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior has focused regression coverage.
Logs and examples contain no credentials, tokens, or private registry information.
Schema changes include regenerated bindings.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/254 **Author:** [@yingdi-shan](https://github.com/yingdi-shan) **Created:** 9/7/2026 **Status:** ✅ Merged **Merged:** 9/11/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/buildkit-integration` --- ### 📝 Commits (6) - [`3b8c195`](https://github.com/kvcache-ai/AgentENV/commit/3b8c1959230c2edcae9b8d380644493c073fc315) feat: build templates with managed BuildKit workers - [`077b995`](https://github.com/kvcache-ai/AgentENV/commit/077b995af0d3fad3ca0fad00367a1955892bf63e) fix(buildkit): simplify workers and complete build validation - [`e0d2ffa`](https://github.com/kvcache-ai/AgentENV/commit/e0d2ffa520290c04484d6084bcbe632c152d2ac8) feat(release): bundle BuildKit client with aenv CLI - [`0b514f3`](https://github.com/kvcache-ai/AgentENV/commit/0b514f3472e09ceb8da3689a0ad21b0a18118777) test(buildkit): wait for idle node metrics before baseline - [`2cecd36`](https://github.com/kvcache-ai/AgentENV/commit/2cecd36e11bf8eea1795a7e641fc4fc331769527) fix(buildkit): preserve template API and publish images on server - [`51c966a`](https://github.com/kvcache-ai/AgentENV/commit/51c966a19a5efbae2b2a2500a8400ae433610bbb) fix(buildkit): bound build and tunnel concurrency ### 📊 Changes **76 files changed** (+6953 additions, -731 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/integration-tests.yml` (+19 -12) 📝 `.github/workflows/release.yml` (+25 -9) 📝 `Cargo.lock` (+7 -10) 📝 `Makefile` (+12 -1) 📝 `build.rs` (+10 -0) ➕ `config/buildkit-version` (+1 -0) 📝 `config/default.toml` (+11 -0) 📝 `crates/aenv/Cargo.toml` (+2 -2) ➕ `crates/aenv/src/client/buildkit.rs` (+199 -0) 📝 `crates/aenv/src/client/mod.rs` (+6 -0) 📝 `crates/aenv/src/commands/build.rs` (+388 -512) 📝 `crates/aenv/src/main.rs` (+2 -2) 📝 `crates/aenv/src/progress.rs` (+60 -0) 📝 `deploy/docker-compose.yml` (+6 -0) 📝 `docs/src/concepts/templates.md` (+173 -21) 📝 `docs/src/configuration/reference.md` (+17 -0) 📝 `docs/src/deployment/docker-compose.md` (+9 -0) 📝 `docs/src/getting-started/aenv-cli.md` (+49 -8) ➕ `scripts/buildkit/README.md` (+61 -0) ➕ `scripts/buildkit/fixture/.dockerignore` (+1 -0) _...and 56 more files_ </details> ### 📄 Description ## What Build Dockerfile contexts with BuildKit in managed microVMs and automatically publish the resulting runnable template on the server. The CLI streams context and BuildKit traffic; successful completion needs no client image-submission request. ## Why Support standard Dockerfile semantics, multi-stage builds, instruction-cache reuse, and persistent cache mounts while preserving the existing template API and remote storage format. ## Related issue Closes #147. Split from #247 alongside #252 (volume deletion) and #253 (OCI UID/GID startup). ## Scope and non-goals BuildKit image import, worker/cache lifecycle, node and gateway APIs, CLI packaging, configuration, tests, and documentation. CLI argument compatibility is outside this change. General volume deletion and OCI account resolution remain in the companion PRs; stage selection, SSH forwarding, remote/stdin contexts, and cross-platform emulation remain outside this flow. ## Design and behavior changes 1. Existing `POST /v3/templates` allocates template/build IDs with its unchanged request and response. 2. `PUT /templates/{templateID}/builds/{buildID}/builder` claims the waiting build and prepares its worker. It returns the unique `imageName` to use with the BuildKit image exporter. Already-started builds return `409`. 3. Poll existing build status for `building`, then use authenticated WebSocket `GET` on the builder resource for native BuildKit traffic. 4. The server watches BuildKit history for successful completion of this build's image exporter, takes its result descriptor, verifies OCI content, and publishes the template. History replay tolerates a dropped observation stream and ignores earlier cached builds. Existing connections can drain before worker shutdown. 5. Existing status reports final `ready` or `error`. `DELETE` on the builder cancels before publication; publication already in progress finishes independently of the client. The existing declarative build endpoint keeps its behavior. There is no competing build-creation collection or digest-submission endpoint. Gateway allocation records the build's node binding; subsequent builder and status requests follow it. Prepared builder snapshots are reused by configuration. Every build receives a separate writable cache child. Cache-head updates and retirements are atomic, cleanup retries retain ownership, and failed/cancelled builds retain the previous seed. Build deadlines and recovery release abandoned workers; public sandbox operations hide builders. Stop BuildKit and checkpoint its cache before stopping the worker. Preserve image startup, health checks, and configuration; package a private `aenv-buildctl` beside the CLI. ## Compatibility and operations - Public API or generated protocol: all 33 pre-existing operations retain their requests and responses, and all 55 pre-existing schemas are unchanged. Only the builder resource is added. Regenerated server bindings are included. - Configuration or defaults: BuildKit v0.33.0, 16 vCPUs, 32768 MiB memory, and a 262144 MiB cache. Smaller hosts can override builder resources. - Snapshot manifest, artifact layout, or storage format: existing catalog keys, artifact paths, and layers need no migration. Builder records and the cache head use the separate `template-build` namespace. Startup `shell` is optional; old records retain Bash behavior and round-trip without adding it. - Upgrade and rollback: upgrade the gateway and build nodes before using BuildKit. CLI build arguments use a context directory. Older servers ignore/drop the new shell field when rewriting records, so deriving templates from BuildKit images on an older server can fail or change shell behavior. Existing declarative APIs remain available. - Host requirements, permissions, ports, or dependencies: existing sandbox prerequisites plus bundled buildctl; Linux/macOS CLI uses a private Unix socket. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [x] Relevant live BuildKit integration tests (results below) - [x] `make -C services test fmt-check vet` - [x] `make agentenv-server` - [x] Documentation updated - [ ] Benchmarks or performance comparison Commands and results: ```text make fmt clippy test-unit AENV_TEST_STATE_DIR=<isolated path> AENV_TEST_DEPS_PATH=<test dependencies>: passed 877 AgentENV unit tests, 4 privileged AgentENV tests, 55 CLI tests, 3 linux-cap tests, 56 ublk/daemon tests, capability runner and installer checks make -C services test fmt-check vet: passed make agentenv-server: regenerated the OpenAPI server make build-server build-aenv: passed Structural OpenAPI comparison against the merge base: 33 existing operations and 55 schemas unchanged AENV_BIN=<built CLI> BUILDCTL_BIN=<v0.33.0 buildctl> python3 scripts/buildkit/test.py <eight core cases>: passed client exit after solve; context/instruction/cache mounts; concurrent builds; failure/cancellation/isolation/recovery; deadlines; startup overrides; external Dockerfile/secrets; terminal progress ``` Focused coverage includes real-repository `404`/`409` allocation, preserving a template named `builds`, legacy snapshot serialization, stale history rejection, exporter descriptor selection, failed builds, history reconnection, and final Solve-response delivery during publication. Live tests use an isolated node with a 2-vCPU, 2048-MiB builder and an 8192-MiB cache. The client-exit regression kills the CLI immediately after a successful BuildKit solve and verifies that the server publishes a runnable template without any image-submission request. Successful cases verify worker and volume cleanup, leaving one ready cache seed. ## Risks and reviewer notes Concurrent builders multiply resource use. Cache sharing follows the repository's API-key trust boundary, and publication time depends on storage throughput. Review the completion watcher in `src/image/buildkit/history.rs`, builder lifecycle in `src/api/impls/image_build/`, and gateway allocation/routing. Merge #252 and #253 first for optimized deletion and numeric image users. The current live run covers the standalone BuildKit branch; numeric-user and combined-branch validation must be repeated with #253. Full Rust integration suites, a live multi-node gateway deployment, and throughput benchmarks were not run for this revision. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior has focused regression coverage. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] Schema changes include regenerated bindings. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:55 +00:00