kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #147] Design: native single-stage Dockerfile builds orchestrated by `aenv` #49

New issue

Closed

opened 2026-10-29 14:18:49 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:18:49 +00:00

Owner

Originally created by @JoyboyBrian on GitHub (Aug 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/147

This issue supersedes the build-context transport portion of #141 following maintainer review. aenv orchestrates an ordinary sandbox as a build VM; build context is streamed into that VM through envd and is never persisted by AgentENV host components or the snapshot repository. This milestone establishes the build-session and context-transport path; it introduces no execution caching and no stock E2B SDK COPY compatibility.

Goal

aenv build builds a supported single-stage Dockerfile from a local context into a named AgentENV snapshot/template.

Instruction handling (v1)

Executed: one non-scratch FROM; shell-form RUN; ENV; WORKDIR; USER; COPY from the local build context; local-file/directory ADD with COPY behavior.
Mapped to startup metadata (existing behavior, unchanged): ENTRYPOINT/CMD — ENTRYPOINT takes precedence and replaces CMD; exec-form arrays are shell-quoted into one command; the ready command remains unset. Full Docker runtime/PID 1 compatibility is not claimed.
Warned and ignored: EXPOSE, VOLUME, LABEL; STOPSIGNAL (a deliberate change from today's silent skip).
Rejected with actionable errors: ARG — deliberately replacing aenv build's current mapping of ARG to a persisted ENV, which is not Docker build-arg semantics; exec-form RUN; SHELL; FROM scratch; a second FROM and COPY --from; remote URL sources and automatic archive extraction for ADD; COPY --chown/--chmod; heredocs; special files and unsupported link types; any other instruction.

The local context honors .dockerignore; regular files and directories only. COPY/ADD destinations without a trailing / are resolved with a guest Filesystem/Stat before transfer — an existing directory gets Docker's copy-into behavior; forms still ambiguous after the stat are rejected before transfer. Extraction runs as root and content is root-owned, matching Docker's --chown-less default. USER affects subsequent RUN instructions via envd's per-request execution user; the CLI's streaming process-start path gains the username plumbing its unary path already has. The server-side template path is unchanged and continues to execute RUN as the boot default user (a documented divergence). The base image must provide /bin/bash and tar.

Build workflow

aenv parses and validates the supported subset up front — exactly one FROM enforced, \ line continuations and multi-key ENV handled — and rejects unsupported syntax before any VM is created.
aenv creates an ordinary cold sandbox from the FROM image with an explicit build-session TTL and runs a periodic keepalive for the lifetime of the build, including while a step or transfer is in progress.
Instructions execute sequentially: RUN through envd with the effective environment, working directory, and user; COPY/ADD packaged client-side, streamed through envd, and interpreted/extracted inside the VM; metadata instructions update the effective client-side context.
After all instructions succeed, aenv captures the sandbox as a named snapshot, supplying the final command context and startup metadata.
The build sandbox is deleted after success or failure; abandoned sessions fall back to the sandbox timeout and eviction behavior. A retry creates a new build sandbox and re-executes the build; no build cache is used in this milestone.

Server-side changes (two, both small)

Capture-time metadata. SandboxSnapshotRequest today carries only name, so a client-orchestrated build would lose its final ENV/WORKDIR/USER and startup settings. Optional final-context and startup fields are added to the snapshot capture request: when present, the supplied context replaces the sandbox's stored context wholesale (no field-level merging) and is applied only while publishing; the startup command's embedded context is derived from the same final context. No mutable set-context endpoint or runtime metadata state is introduced.
Gateway streaming classification. Multipart envd uploads are not classified as streaming today and inherit the deadline for ordinary proxied requests; the classifier will be extended. This also fixes large aenv upload transfers through the gateway — closing an existing gap rather than adding surface.

No context store, external coordinator, or snapshot repository format change is introduced.

Security and failure semantics

AgentENV host components do not persist build-context bytes.
Context packaging happens on the client; archive interpretation and extraction happen inside the build VM.
Client-side selection enforces context boundaries and .dockerignore.
Unsupported paths, source types, and Dockerfile syntax fail before transfer where possible.
Failure to execute, transfer, capture, or publish fails the build and removes the build VM.
Publishing an existing alias continues to fail with the existing alias-conflict behavior.

Acceptance criteria

A native aenv build E2E builds and launches a snapshot from a single-stage Dockerfile containing RUN, ENV, WORKDIR, USER, COPY, and local ADD — including RUN executing as the effective user and a stat-resolved single-file copy onto an existing directory.
Copied content and the final environment, workdir, user, and startup behavior survive snapshot capture and launch.
Unsupported multi-stage and extended ADD/COPY forms fail with actionable errors.
The build works through the gateway without staging context on the AgentENV host filesystem. With the ordinary proxy timeout configured shorter than the transfer duration, the multipart envd upload still completes through the streaming path.
Existing server-side template builds remain unchanged.

Reuse from #73/#74

The previous stack is not carried forward as-is. The planning/destination logic and test cases from #73 and the envd transfer and guest-side handling from #74 may be adapted for this design, with their open review findings addressed and revalidated in the native implementation before reuse; #74's in-guest ownership lookup remains the basis for the deferred --chown follow-up. The client-side planning and orchestration logic stays in aenv; a shared crate can be extracted later if a second consumer appears.

Non-goals

Stock E2B SDK COPY compatibility
Host-side build-context staging
Multi-stage builds
Per-step execution caching and OverlayBD checkpoints
Cross-build context deduplication
Full Docker/BuildKit compatibility

If this scope and the capture-time metadata approach look right, I will wait for acknowledgement before starting implementation from the latest main.

Originally created by @JoyboyBrian on GitHub (Aug 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/147 This issue supersedes the build-context transport portion of #141 following maintainer review. `aenv` orchestrates an ordinary sandbox as a build VM; build context is streamed into that VM through envd and is never persisted by AgentENV host components or the snapshot repository. This milestone establishes the build-session and context-transport path; it introduces no execution caching and no stock E2B SDK `COPY` compatibility. ## Goal `aenv build` builds a supported single-stage Dockerfile from a local context into a named AgentENV snapshot/template. ## Instruction handling (v1) - **Executed:** one non-`scratch` `FROM`; shell-form `RUN`; `ENV`; `WORKDIR`; `USER`; `COPY` from the local build context; local-file/directory `ADD` with `COPY` behavior. - **Mapped to startup metadata (existing behavior, unchanged):** `ENTRYPOINT`/`CMD` — `ENTRYPOINT` takes precedence and replaces `CMD`; exec-form arrays are shell-quoted into one command; the ready command remains unset. Full Docker runtime/PID 1 compatibility is not claimed. - **Warned and ignored:** `EXPOSE`, `VOLUME`, `LABEL`; `STOPSIGNAL` (a deliberate change from today's silent skip). - **Rejected with actionable errors:** `ARG` — deliberately replacing `aenv build`'s current mapping of `ARG` to a persisted `ENV`, which is not Docker build-arg semantics; exec-form `RUN`; `SHELL`; `FROM scratch`; a second `FROM` and `COPY --from`; remote URL sources and automatic archive extraction for `ADD`; `COPY --chown`/`--chmod`; heredocs; special files and unsupported link types; any other instruction. The local context honors `.dockerignore`; regular files and directories only. `COPY`/`ADD` destinations without a trailing `/` are resolved with a guest `Filesystem/Stat` before transfer — an existing directory gets Docker's copy-into behavior; forms still ambiguous after the stat are rejected before transfer. Extraction runs as root and content is root-owned, matching Docker's `--chown`-less default. `USER` affects subsequent `RUN` instructions via envd's per-request execution user; the CLI's streaming process-start path gains the username plumbing its unary path already has. The server-side template path is unchanged and continues to execute `RUN` as the boot default user (a documented divergence). The base image must provide `/bin/bash` and `tar`. ## Build workflow 1. `aenv` parses and validates the supported subset up front — exactly one `FROM` enforced, `\` line continuations and multi-key `ENV` handled — and rejects unsupported syntax before any VM is created. 2. `aenv` creates an ordinary cold sandbox from the `FROM` image with an explicit build-session TTL and runs a periodic keepalive for the lifetime of the build, including while a step or transfer is in progress. 3. Instructions execute sequentially: `RUN` through envd with the effective environment, working directory, and user; `COPY`/`ADD` packaged client-side, streamed through envd, and interpreted/extracted inside the VM; metadata instructions update the effective client-side context. 4. After all instructions succeed, `aenv` captures the sandbox as a named snapshot, supplying the final command context and startup metadata. 5. The build sandbox is deleted after success or failure; abandoned sessions fall back to the sandbox timeout and eviction behavior. A retry creates a new build sandbox and re-executes the build; no build cache is used in this milestone. ## Server-side changes (two, both small) 1. **Capture-time metadata.** `SandboxSnapshotRequest` today carries only `name`, so a client-orchestrated build would lose its final `ENV`/`WORKDIR`/`USER` and startup settings. Optional final-context and startup fields are added to the snapshot capture request: when present, the supplied context replaces the sandbox's stored context wholesale (no field-level merging) and is applied only while publishing; the startup command's embedded context is derived from the same final context. No mutable set-context endpoint or runtime metadata state is introduced. 2. **Gateway streaming classification.** Multipart envd uploads are not classified as streaming today and inherit the deadline for ordinary proxied requests; the classifier will be extended. This also fixes large `aenv upload` transfers through the gateway — closing an existing gap rather than adding surface. No context store, external coordinator, or snapshot repository format change is introduced. ## Security and failure semantics - AgentENV host components do not persist build-context bytes. - Context packaging happens on the client; archive interpretation and extraction happen inside the build VM. - Client-side selection enforces context boundaries and `.dockerignore`. - Unsupported paths, source types, and Dockerfile syntax fail before transfer where possible. - Failure to execute, transfer, capture, or publish fails the build and removes the build VM. - Publishing an existing alias continues to fail with the existing alias-conflict behavior. ## Acceptance criteria - A native `aenv build` E2E builds and launches a snapshot from a single-stage Dockerfile containing `RUN`, `ENV`, `WORKDIR`, `USER`, `COPY`, and local `ADD` — including `RUN` executing as the effective user and a stat-resolved single-file copy onto an existing directory. - Copied content and the final environment, workdir, user, and startup behavior survive snapshot capture and launch. - Unsupported multi-stage and extended `ADD`/`COPY` forms fail with actionable errors. - The build works through the gateway without staging context on the AgentENV host filesystem. With the ordinary proxy timeout configured shorter than the transfer duration, the multipart envd upload still completes through the streaming path. - Existing server-side template builds remain unchanged. ## Reuse from #73/#74 The previous stack is not carried forward as-is. The planning/destination logic and test cases from #73 and the envd transfer and guest-side handling from #74 may be adapted for this design, with their open review findings addressed and revalidated in the native implementation before reuse; #74's in-guest ownership lookup remains the basis for the deferred `--chown` follow-up. The client-side planning and orchestration logic stays in `aenv`; a shared crate can be extracted later if a second consumer appears. ## Non-goals - Stock E2B SDK `COPY` compatibility - Host-side build-context staging - Multi-stage builds - Per-step execution caching and OverlayBD checkpoints - Cross-build context deduplication - Full Docker/BuildKit compatibility If this scope and the capture-time metadata approach look right, I will wait for acknowledgement before starting implementation from the latest `main`.

bsakaguchi closed this issue

2026-10-29 14:18:50 +00:00

bsakaguchi commented

2026-10-29 14:18:50 +00:00

Author

Owner

@JoyboyBrian commented on GitHub (Aug 9, 2026):

Hi @yingdi-shan, I’ve posted the focused design distilled from #141 and closed the superseded stack. Before starting implementation, could you confirm that the v1 scope and the capture-time metadata approach here match your intent? If so, I’ll begin from the latest main.

@JoyboyBrian commented on GitHub (Aug 9, 2026): Hi @yingdi-shan, I’ve posted the focused design distilled from #141 and closed the superseded stack. Before starting implementation, could you confirm that the v1 scope and the capture-time metadata approach here match your intent? If so, I’ll begin from the latest `main`.

bsakaguchi commented

2026-10-29 14:18:51 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Aug 10, 2026):

Looks good to me.

PR #143 has added support for exec-form and heredocs.
I suggest pausing after each RUN/COPY command to form a new overlaybd layer (this can be implemented when we add cache support).

@yingdi-shan commented on GitHub (Aug 10, 2026): Looks good to me. 1. PR #143 has added support for exec-form and heredocs. 2. I suggest pausing after each RUN/COPY command to form a new overlaybd layer (this can be implemented when we add cache support).

bsakaguchi commented

2026-10-29 14:18:51 +00:00

Author

Owner

@JoyboyBrian commented on GitHub (Aug 10, 2026):

Thanks. I’ll preserve the exec-form and RUN heredoc support added by #143 in the v1 native workflow, and keep per-step pause/layer creation deferred to the cache milestone. I’ll update the issue text accordingly and start implementation from the latest main.

@JoyboyBrian commented on GitHub (Aug 10, 2026): Thanks. I’ll preserve the exec-form and RUN heredoc support added by #143 in the v1 native workflow, and keep per-step pause/layer creation deferred to the cache milestone. I’ll update the issue text accordingly and start implementation from the latest `main`.

bsakaguchi referenced this issue

2026-10-29 14:20:07 +00:00

[PR #49] [CLOSED] feat(ublk): add weight-based I/O scheduler for fair sandbox bandwidth sharing #140