[PR #28] [CLOSED] feat(template): E2B build-context uploads, COPY/ADD steps, and alias rebuild (full PR — see stacked PRs) #127

New issue

Closed

opened 2026-10-29 14:19:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:59 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/28
Author: @JoyboyBrian
Created: 7/28/2026
Status: ❌ Closed

Base: main ← Head: feat/e2b-template-build-context

📝 Commits (7)

93dd9c9 fix(api): return the JSON error envelope for unmatched routes
92a4bb8 feat(template): E2B build-context uploads, COPY/ADD steps, and alias rebuild
2052736 docs: unwrap hard-wrapped lines in the E2B template build section
a9227cb fix(template): harden COPY step semantics and memory use
fb24859 fix(template): make build-context upload URLs single-use
b632852 fix(template): address review findings and create WORKDIR directories
6f4b6a5 fix(template): address round-2 review findings

📊 Changes

34 files changed (+4849 additions, -129 deletions)

View changed files

📝 config/default.toml (+24 -0)
📝 docs/src/integration/e2b.md (+28 -0)
➕ src/api/build_files.rs (+287 -0)
📝 src/api/generated/src/apis/templates.rs (+29 -0)
📝 src/api/generated/src/models.rs (+158 -0)
📝 src/api/generated/src/server/mod.rs (+172 -0)
📝 src/api/impls/mod.rs (+4 -0)
📝 src/api/impls/template.rs (+95 -0)
📝 src/api/impls/template_helpers.rs (+185 -13)
📝 src/api/mod.rs (+1 -0)
📝 src/api/openapi.yml (+48 -0)
📝 src/api/proxy.rs (+30 -1)
📝 src/api/server.rs (+3 -2)
📝 src/cfg.rs (+223 -0)
📝 src/sandbox/backend.rs (+14 -0)
📝 src/sandbox/envd.rs (+35 -0)
📝 src/sandbox/process.rs (+12 -0)
📝 src/snapshot/manager.rs (+8 -0)
➕ src/snapshot/repository/backends/oss/build_files.rs (+169 -0)
📝 src/snapshot/repository/backends/oss/mod.rs (+1 -0)

...and 14 more files

📄 Description

Summary

The stock E2B SDK's template builder could not build Dockerfiles containing COPY, and could not rebuild an existing alias. This PR closes both gaps so AsyncTemplate.build works end to end against AgentENV:

Build-context uploads. The SDK resolves every COPY through GET /templates/{templateID}/files/{hash}; AgentENV had no such route, so the SDK died on a 404 with an empty body. This PR implements the endpoint (201 + {present, url}, matching the E2B spec) plus a streaming PUT endpoint the returned URL points at. Archives are stored in the snapshot repository (posix_fs and oss backends), so in multi-node deployments any node can issue the link, accept the upload, and read it back at build time — no gateway or routing changes needed. The SDK PUTs with a bare HTTP client (no auth headers), so the URL carries a bearer grant persisted in the shared repository — the same shape as an object-store presigned URL. Grants are single-use, and stored archives are immutable: a hash addresses its content, so keeping the first archive costs nothing and means no build can observe its context change while it runs. Spent grants and expired archives are pruned opportunistically. A new optional [template_build] config section (all defaults provided) bounds the upload size, the decompressed archive size, and the URL TTL, and can override the public base URL.
COPY/ADD execution. The uploaded archive is rewritten host-side to final absolute guest paths per Docker COPY semantics — single file, directory contents-into-dest, globs, WORKDIR-relative destinations (including a relative WORKDIR, which now resolves against the current one), root:root ownership, --chown/--chmod — then streamed into the build sandbox via envd and extracted with a single tar -xpf. --chown is resolved to numeric ids inside the build sandbox, the way Docker resolves names against the image's own /etc/passwd, and written into the rewritten tar headers, so a copy can only ever change the files it creates. The rewrite runs in two passes (index paths, then stream each entry's bytes) so an archive is never held in memory, and rejects archives past the configured decompressed budget. Its mapping logic is a pure function with unit tests; ../absolute entry escapes are rejected.
Alias rebuilds. Requesting a build under an existing alias no longer fails with 400 … cannot rebind. The alias keeps resolving to the previous template while the new build runs, and moves atomically when the new build commits (E2B rebuild semantics). A failed rebuild leaves the old binding untouched; the previous template stays addressable by id. Implemented in both backends, with the alias bind reordered to be the final fallible operation of publish.
Error envelope. Unmatched control-plane routes now return the spec's {"code":404,"message":"…"} instead of an empty body, so future gaps surface as a readable 404 rather than a client-side JSON parse error (separate first commit).

Behavior notes

A failed rebuild keeps the requested name on the errored record so listings show what failed; alias resolution is owned by the alias binding and keeps pointing at the previous, working template.
COPY app.py /opt where /opt is an existing directory fails the build with a clear error; write directory destinations with a trailing slash (/opt/), the style E2B's own docs use. Documented in docs/src/integration/e2b.md.
COPY --chown applies only to the files the copy creates, and an unknown user fails the build rather than silently landing as root.
COPY requires tar in the base image (RUN already requires /bin/bash); documented.
Verifying that an upload matches its filesHash was considered and rejected: the SDK derives that hash from the COPY arguments plus per-file metadata and content in its own traversal order, not from the archive bytes, so recomputing it server-side would pin the server to SDK internals. Single-use grants and immutable archives close the same window without that coupling.
envd currently answers the files-upload endpoint with a 2xx text/plain body while the generated client expects JSON; the upload path tolerates the decode error after a completed upload (commented inline — the root fix belongs in the envd http-client spec).

Testing

make fmt, cargo clippy --workspace --all-targets --all-features -- -D warnings, and the agentenv/envd/linux-cap unit suites. New unit tests cover the Docker COPY plan semantics (gzip, globs, ownership, path escapes, the decompressed budget), the build-file stores, single-use upload grants and their pruning, archive immutability, and alias rebind behavior in create/publish.
End-to-end on a two-node docker-compose deployment (gateway + scheduler + two runtime nodes) with E2B Python SDK 2.35.0: from_image, from_dockerfile without COPY, from_dockerfile with COPY, alias rebuild, cold sandboxes, and the full exec/files runtime suite all pass. Separately verified in a booted sandbox that COPY --chown leaves pre-existing files under the destination untouched, that a relative WORKDIR resolves against the previous one, that an upload URL is rejected on replay, and that a stored archive keeps its original bytes across a repeat upload on both nodes.
Generated server code regenerated from the spec change via make agentenv-server.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/28 **Author:** [@JoyboyBrian](https://github.com/JoyboyBrian) **Created:** 7/28/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/e2b-template-build-context` --- ### 📝 Commits (7) - [`93dd9c9`](https://github.com/kvcache-ai/AgentENV/commit/93dd9c9581fc19c132e200da5d96495c6631f539) fix(api): return the JSON error envelope for unmatched routes - [`92a4bb8`](https://github.com/kvcache-ai/AgentENV/commit/92a4bb8574282903905b98b61fb03f3261d7196e) feat(template): E2B build-context uploads, COPY/ADD steps, and alias rebuild - [`2052736`](https://github.com/kvcache-ai/AgentENV/commit/205273606a2d01ecf3db067e158211cbd038e00a) docs: unwrap hard-wrapped lines in the E2B template build section - [`a9227cb`](https://github.com/kvcache-ai/AgentENV/commit/a9227cb555410d5013e9258e74223592bf5513b7) fix(template): harden COPY step semantics and memory use - [`fb24859`](https://github.com/kvcache-ai/AgentENV/commit/fb248599fe0a8880b5d636b1b8ab06f956b67fe6) fix(template): make build-context upload URLs single-use - [`b632852`](https://github.com/kvcache-ai/AgentENV/commit/b6328520bfb49a6bf891293a8d9d76d66b18ad8f) fix(template): address review findings and create WORKDIR directories - [`6f4b6a5`](https://github.com/kvcache-ai/AgentENV/commit/6f4b6a5b0d51855fe90aaa645849db92099e4963) fix(template): address round-2 review findings ### 📊 Changes **34 files changed** (+4849 additions, -129 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+24 -0) 📝 `docs/src/integration/e2b.md` (+28 -0) ➕ `src/api/build_files.rs` (+287 -0) 📝 `src/api/generated/src/apis/templates.rs` (+29 -0) 📝 `src/api/generated/src/models.rs` (+158 -0) 📝 `src/api/generated/src/server/mod.rs` (+172 -0) 📝 `src/api/impls/mod.rs` (+4 -0) 📝 `src/api/impls/template.rs` (+95 -0) 📝 `src/api/impls/template_helpers.rs` (+185 -13) 📝 `src/api/mod.rs` (+1 -0) 📝 `src/api/openapi.yml` (+48 -0) 📝 `src/api/proxy.rs` (+30 -1) 📝 `src/api/server.rs` (+3 -2) 📝 `src/cfg.rs` (+223 -0) 📝 `src/sandbox/backend.rs` (+14 -0) 📝 `src/sandbox/envd.rs` (+35 -0) 📝 `src/sandbox/process.rs` (+12 -0) 📝 `src/snapshot/manager.rs` (+8 -0) ➕ `src/snapshot/repository/backends/oss/build_files.rs` (+169 -0) 📝 `src/snapshot/repository/backends/oss/mod.rs` (+1 -0) _...and 14 more files_ </details> ### 📄 Description ## Summary The stock E2B SDK's template builder could not build Dockerfiles containing `COPY`, and could not rebuild an existing alias. This PR closes both gaps so `AsyncTemplate.build` works end to end against AgentENV: - **Build-context uploads.** The SDK resolves every `COPY` through `GET /templates/{templateID}/files/{hash}`; AgentENV had no such route, so the SDK died on a 404 with an empty body. This PR implements the endpoint (`201` + `{present, url}`, matching the E2B spec) plus a streaming `PUT` endpoint the returned URL points at. Archives are stored in the snapshot repository (`posix_fs` and `oss` backends), so in multi-node deployments any node can issue the link, accept the upload, and read it back at build time — no gateway or routing changes needed. The SDK `PUT`s with a bare HTTP client (no auth headers), so the URL carries a bearer grant persisted in the shared repository — the same shape as an object-store presigned URL. Grants are single-use, and stored archives are immutable: a hash addresses its content, so keeping the first archive costs nothing and means no build can observe its context change while it runs. Spent grants and expired archives are pruned opportunistically. A new optional `[template_build]` config section (all defaults provided) bounds the upload size, the decompressed archive size, and the URL TTL, and can override the public base URL. - **COPY/ADD execution.** The uploaded archive is rewritten host-side to final absolute guest paths per Docker `COPY` semantics — single file, directory contents-into-dest, globs, `WORKDIR`-relative destinations (including a relative `WORKDIR`, which now resolves against the current one), root:root ownership, `--chown`/`--chmod` — then streamed into the build sandbox via envd and extracted with a single `tar -xpf`. `--chown` is resolved to numeric ids inside the build sandbox, the way Docker resolves names against the image's own `/etc/passwd`, and written into the rewritten tar headers, so a copy can only ever change the files it creates. The rewrite runs in two passes (index paths, then stream each entry's bytes) so an archive is never held in memory, and rejects archives past the configured decompressed budget. Its mapping logic is a pure function with unit tests; `..`/absolute entry escapes are rejected. - **Alias rebuilds.** Requesting a build under an existing alias no longer fails with `400 … cannot rebind`. The alias keeps resolving to the previous template while the new build runs, and moves atomically when the new build commits (E2B rebuild semantics). A failed rebuild leaves the old binding untouched; the previous template stays addressable by id. Implemented in both backends, with the alias bind reordered to be the final fallible operation of publish. - **Error envelope.** Unmatched control-plane routes now return the spec's `{"code":404,"message":"…"}` instead of an empty body, so future gaps surface as a readable 404 rather than a client-side JSON parse error (separate first commit). ## Behavior notes - A failed rebuild keeps the requested name on the errored record so listings show what failed; alias *resolution* is owned by the alias binding and keeps pointing at the previous, working template. - `COPY app.py /opt` where `/opt` is an existing directory fails the build with a clear error; write directory destinations with a trailing slash (`/opt/`), the style E2B's own docs use. Documented in `docs/src/integration/e2b.md`. - `COPY --chown` applies only to the files the copy creates, and an unknown user fails the build rather than silently landing as root. - `COPY` requires `tar` in the base image (`RUN` already requires `/bin/bash`); documented. - Verifying that an upload matches its `filesHash` was considered and rejected: the SDK derives that hash from the COPY arguments plus per-file metadata and content in its own traversal order, not from the archive bytes, so recomputing it server-side would pin the server to SDK internals. Single-use grants and immutable archives close the same window without that coupling. - envd currently answers the files-upload endpoint with a 2xx `text/plain` body while the generated client expects JSON; the upload path tolerates the decode error after a completed upload (commented inline — the root fix belongs in the envd http-client spec). ## Testing - `make fmt`, `cargo clippy --workspace --all-targets --all-features -- -D warnings`, and the `agentenv`/`envd`/`linux-cap` unit suites. New unit tests cover the Docker COPY plan semantics (gzip, globs, ownership, path escapes, the decompressed budget), the build-file stores, single-use upload grants and their pruning, archive immutability, and alias rebind behavior in create/publish. - End-to-end on a two-node docker-compose deployment (gateway + scheduler + two runtime nodes) with E2B Python SDK 2.35.0: `from_image`, `from_dockerfile` without COPY, **`from_dockerfile` with COPY**, **alias rebuild**, cold sandboxes, and the full exec/files runtime suite all pass. Separately verified in a booted sandbox that `COPY --chown` leaves pre-existing files under the destination untouched, that a relative `WORKDIR` resolves against the previous one, that an upload URL is rejected on replay, and that a stored archive keeps its original bytes across a repeat upload on both nodes. - Generated server code regenerated from the spec change via `make agentenv-server`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:19:59 +00:00