[GH-ISSUE #141] RFC: per-step OverlayBD layer caching and build-context transport for E2B template builds #51

New issue

Closed

opened 2026-10-29 14:18:50 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented 2026-10-29 14:18:50 +00:00

Owner

Copy link

Originally created by @JoyboyBrian on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/141

Status (2026-08-09): superseded. Maintainer review chose a different direction — a native single-stage Dockerfile workflow orchestrated by aenv, streaming context into the build VM via envd with no host-side persistence. The shared durable context store, the coordinator, and stock E2B SDK COPY compatibility are out of scope; per-step caching and multi-stage support are separate follow-ups. Focused design: #147. The original proposal below is kept for reference.

TL;DR: Adopt the reviewer's per-step cache, while retaining a slim shared context-staging path required by stock-SDK upload timing and cross-node routing; validate it through Phase 0/0.5 before enabling per-step reuse.

• Status: open for discussion
• Related: #28 (original full PR and concept discussion), #71 #72 #73 #74 (active COPY/ADD stack), #70 (parked alias rebuild)
• Prompted by review feedback on #71 proposing (a) uploading build context directly into the build sandbox via envd, and (b) deduplicating builds by snapshotting after each COPY/RUN step so each step yields a cacheable, reusable OverlayBD layer, found via a tree of cache keys with prefix matching.

Affected component

Snapshot / template builder (cross-cutting: API server / proxy, gateway / scheduler in later phases).

RFC intent and possible closure

The #71 review proposes an execution model — cache every mutating build step as an OverlayBD layer and resume the longest completed prefix — that this RFC recommends adopting, subject to the evidence and constraints gathered here. One stock-SDK constraint remains to be placed: context uploads complete before the build-start request, and with AgentENV's current independent request routing those bytes need a staging home until the consuming step runs. The RFC therefore proposes retaining a smaller context store as transport and a present oracle — not as the build cache — and asks for decisions on:

1. the per-step cache model (key structure, context bindings, publication semantics) in §Proposed approach;
2. the metadata coordinator for multi-node OSS deployments (§Metadata coordination) — required from Phase 0, because upload bindings and build-state transitions need linearizable writes that the OSS backend cannot express today;
3. the attempt state machine, promotion outbox, and the reconciler actors that make crashes converge (§Failure contract, §Cache publication);
4. the form of context staging (shared store vs the alternatives in §Alternatives considered);
5. the phasing (including a final-key-only Phase 0.5) and default-off gating of the cache;
6. how the current stack (#71–#74) should be reshaped to fit.

Useful closure outcomes: agreement on the target model with the stack revised to match; agreement on a different staging design from §Alternatives; or a recorded decision that execution caching is deferred and the stack lands as a compatibility-only milestone.

Problem statement

The stock E2B SDK's Template.build() has one hard correctness requirement AgentENV does not meet, and one capability the protocol anticipates that AgentENV lacks:

1. Context transport (correctness). The SDK resolves every COPY/ADD through GET /templates/{templateID}/files/{hash} and PUTs a tar archive to the returned URL with a bare HTTP client, finishing all uploads before it sends the build-start request (Template.build). Without a place to accept those uploads and make the bytes available to the build node, COPY builds cannot work at all.
2. Execution reuse (performance and cache semantics). Today's template builder re-executes every step of every build. The E2B protocol exposes per-step cache controls — TemplateStep.force, TemplateBuildStartV2.force, NodeDetail.cachedBuilds exist in AgentENV's src/api/openapi.yml — which AgentENV parses and ignores; implementing a step cache gives those fields their intended function (the schema exposes cache control, though it does not by itself mandate any particular cache granularity).

The open PR stack (#71–#74) implements transport via a repository-backed archive store, host-side COPY planning, and envd-streamed execution — but no execution caching. The #71 review asks whether the durable store is necessary and proposes per-step layer caching as the deduplication mechanism. This RFC answers with a combined design and a survey of how the same problem is solved elsewhere.

Use case

• Stock E2B SDK (JS/Python) template builds with COPY/ADD against single-node POSIX and multi-node OSS deployments.
• Iterative template development: repeated builds where most steps and most context bytes are unchanged; the user expects unchanged prefixes to be skipped and unchanged context not to be re-uploaded. Scope note: rebuilding an existing alias in place is unsupported (parked with #70), so "iterative" here means repeated builds under new template names or IDs; the cache makes those cheap.
• Cluster operation: the upload-link request, the upload PUT, and the build start are routed independently by the gateway today (services/gateway/internal/schedule_hint.go recognizes only /sandboxes* routes), so any node may serve any of them.

Current behavior and workarounds

• Template builds run all steps in one sandbox and capture one snapshot at the end (src/template/runner.rs); there is no step caching, so every rebuild pays the full step cost.
• #71–#74 add the context store, upload API, COPY planning, and COPY execution. Pre-merge gaps found while preparing this RFC (fixes are folded into §Proposed approach): the OSS store's import is an exists check followed by an unconditional PUT, not an atomic first-write publication (src/snapshot/repository/backends/oss/build_files.rs); the OSS try_start_build transition is an unguarded read-modify-write, and the OSS client has no conditional-write primitive at all (src/snapshot/repository/backends/oss/{repository,client}.rs — an in-repo comment notes the S3-compatible path supports neither If-None-Match nor x-oss-forbid-overwrite); the gateway does not classify a bare upload PUT as streaming, so it inherits the short request deadline (services/gateway/internal/server.go, isStreamingRequest); and gateway error logs serialize the full upstream URL including the bearer query token.
• No workaround exists for execution reuse; force/cachedBuilds are dead schema fields.

Desired behavior

• Given an unchanged step prefix (same base, same steps, same bound context generations, same machine/runtime fingerprint), when a template build runs, then the prefix is not re-executed: the build resumes from the deepest cached checkpoint and executes only the suffix — and produces the same result a cold build would (cold-miss/resume equivalence).
• Given a context archive bound to a build, then that exact archive generation remains available to the build until it reaches a terminal state — regardless of GC, later uploads, or forceUpload by other builds; repeated link requests never silently rebind an existing binding to a newer generation.
• Given force on a step or build, when the build runs, then that step and everything after it re-execute, and refreshed results replace the stale cache entries without leaving mixed-generation chains observable.
• Given a node crash in any build state — including mid-promotion — then the build converges to a visible terminal state via lease expiry and reconcilers; no fenced-out worker can commit canonical state, and terminal states are monotonic (once Ready, no later write demotes it).
• Cached artifacts are bounded: retention and safe deletion exist in every phase where the cache can be enabled.

These are scoped guarantees, not absolutes: re-upload happens when retention expired before any binding or scope differs; re-execution happens on force, fingerprint change, or a lost cross-node publication race.

SDK constraints and current deployment assumptions

Observed behaviors of the stock SDKs (pinned: SDK @ cab27aa, infra @ b8bba83):

1. Call sequence. POST /v3/templates (register) → per-COPY hash computation → GET /templates/{templateID}/files/{hash} per COPY instruction → PUT <url> for each missing archive → only after all uploads complete, POST /v2/templates/{templateID}/builds/{buildID} → status polling. The JS SDK parallelizes link requests and uploads; the Python SDK is sequential in both its sync and async clients; all finish uploads before build start (index.ts, template_sync/main.py, template_async/main.py).
2. The hash is per-COPY-step, client-computed, and semantic. A sha256 digest seeded with a constructed COPY <src> <dest> string, then each file's context-relative path, mode, size, contents, and symlink targets; uid/gid/mtime deliberately excluded; files sorted for determinism (utils.ts calculateFilesHash). It is a digest of the step's semantic inputs, not of the uploaded tar bytes — the instruction string is not present in the tar, so the server cannot recompute or verify this hash from the uploaded archive.
3. The PUT is a bare HTTP client: no auth headers; the URL is the credential; exact Content-Length required (S3 presigned PUT rejects chunked encoding; see the in-code comment in buildApi.ts referencing e2b-dev/e2b#1243). The default client upload timeout is 1 hour (configurable).
4. present: true normally skips the upload, but a per-instruction forceUpload both re-uploads despite present and marks that COPY step force — and it does so using the URL from the same link response, without a second GET (upload loop, COPY construction). Every link response must therefore carry a usable upload URL, present or not.
5. At upload-link time the server has no recipe and no step arguments — only (templateID, hash) plus the request's auth context. No step-prefix computation is possible when answering present.

Deployment assumptions (properties of AgentENV today, not SDK law — changing them is part of the design space in §Alternatives considered):

• (D1) The gateway routes the register, upload-link, upload, and build-start requests independently (round_robin/random, no template pinning), so today the shared snapshot repository is the only thing making them coherent. AgentENV registration mints a fresh snapshot ID per build request (v3_templates_post → SnapshotId::generate()), so nothing about the protocol pins consecutive requests to one node.
• (D2) Context must remain available until the consuming step executes (or the build reaches a terminal state). Durable object storage is one way to provide that guarantee; leases over node-local staging are another.

Prior art

E2B upstream: a context store plus a per-step layer cache, joined by filesHash

E2B's build-cache bucket contains two kinds of objects (paths.go; the path helpers take a generic cacheScope — the public API passes the team ID, with a server-side fallback to the template ID when unset):

{scope}/files/{sha256}.tar     # build-context archives, addressed by the SDK hash
{scope}/index/{layerHash}      # tiny JSON: cache key -> build (snapshot) ID

• The upload-link handler stats the blob and mints a presigned URL; present = blob exists (template_layer_files_upload.go, upload_layer_files_template.go; 30-minute URL TTL). Self-hosted deployments can emulate the presigned URL with an HMAC-signed upload endpoint over filesystem storage; any-node coherence then requires that storage to be shared (a shared mount or object store) rather than a single node's disk.
• The builder is a phase engine running Firecracker VMs. Each step's cache key chains through its parent: H(parentHash ‖ stepType ‖ join(args) ‖ filesHash) (phases/steps/hash.go) — the "tree of cache keys with prefix matching" from the #71 review. All phase hashes are computable statically from the recipe; the engine computes and probes them phase by phase in forward order, continuing to probe after a miss (phase.go); a hit skips that phase's Build().
• A cached layer is a full pausable VM snapshot (memfile + rootfs, stored as diffs against parents) under a fresh build ID. The context tar is downloaded from the store only on a cache miss, by whatever node runs the build, streamed into the guest via envd and extracted in-guest (copy.go). The upload-link request and build-start request each independently pick a build node; nothing reconciles them — coherent only because both talk to the same shared store.
• Correctness details adopted below: the hash→buildID index entry is written only after every snapshot artifact has finished uploading (layer_executor.go); force is sticky (builder.go); two global invalidation levers exist (a hashing version folded into the base hash; a minimum cached-format version at lookup — cache.go).
• Gaps not to copy: uploads are never verified against the claimed hash (containment is scoping only), and there is no durable-object GC for context tars, index entries, or intermediate layer snapshots (node-local caches do have TTL cleanup).

BuildKit: transfer and caching are orthogonal, joined by a content digest

BuildKit (8f96667) has no durable context-archive store; it persists the client's context as a live snapshot keyed by a heuristic sharedKey, updated by an incremental stat-based diff that only transfers changed file bodies (source/local/source.go, fsutil receive.go @ 6d9dc2e). The local-context vertex is deliberately uncacheable across sessions (ops/source.go); all COPY cache reuse comes from a server-side content checksum attached as an alternative cache key on the consuming step (edge.go, contenthash).

Relevant lessons: the E2B protocol shape (client hash + whole-archive upload) trades away incremental transfer and gains a one-round-trip skip plus node independence; the cost — one changed byte re-uploads the whole archive — must be accepted explicitly, and in a VM builder the per-step capture cost (full-VM memory) may dominate overall build time regardless of transfer efficiency, so benchmarks decide. In BuildKit's remote-cache export, min mode exports the layers of the resulting image while max additionally exports intermediate-stage layers (cache mode docs); on OverlayBD the writable upper is the per-step rootfs diff, so exporting intermediate rootfs layers carries little marginal data cost (sealing is O(mapping count), not free — §Proposed approach). BuildKit's GC tiers by reproducibility — cheap-to-reproduce classes get short-TTL small-budget tiers (gcpolicy.go) — and identical in-flight work is deduplicated per vertex.

Industry survey (trade-off examples, not verdicts)

• Modal (a close peer: bespoke image format, own builder; sandboxes are secure containers by default, with VM sandboxes as an opt-in beta): a content-addressed file store (per-file sha256, batched existence negotiation, presigned blob upload) plus per-layer get-or-create where each layer names its parent image ID (modal-client @ bef1b70, Images guide: breaking one layer "causes cascading rebuilds for all subsequent layers"). Modal distinguishes force_build (rebuild and overwrite cache) from ignore_cache (rebuild without publishing).
• Daytona: context tars content-addressed in S3 with skip-if-exists; the platform cache key is a single hash of dockerfile + sorted(contextHashes) for the whole build; intermediate BuildKit state is runner-local, while completed snapshots propagate across runners (v0.190.0, snapshot manager). A coherent store-only + whole-build-cache design; reuse granularity is the whole build.
• Depot / Fly.io: Depot attaches ephemeral builder instances to a persistent per-project/per-architecture cache volume (Ceph-backed) and routes a project's builds to a shared builder to maximize hits (Depot architecture). Fly.io's earlier shared remote builders drew user reports of large context re-transfer in specific setups (flyctl#591 for --local-only, community reports of large COPY . . contexts); Fly now defaults to Depot-managed builders (announcement). These illustrate the operational coupling that builder-affinity approaches take on; they are not evidence that such approaches fail.
• kaniko (daemonless per-step caching; pinned @ 236ba56): rolling composite key seeded by the base-image digest, extended per instruction with content hashes of referenced files; the cache index is an OCI registry, shared across machines by construction; strict stop-after-first-miss (composite_cache.go, caching docs).
• OverlayBD upstream (pinned @ 4f5c597): BuildKit ≥ 0.22 with the overlaybd snapshotter builds native OverlayBD images; in the native writable-layer mode the snapshotter's Commit() seals the live writable layer into an immutable lower per committed filesystem snapshot (QUICKSTART, buildkit docs/overlaybd.md). The conversion path keeps a dedup database keyed by OCI chain_id — a cumulative parent-chain identity used for conversion dedup (overlaybd_builder.go). This validates per-step filesystem-diff sealing on this storage format; AgentENV's repeated full-VM checkpoint loop (Phase 1) goes beyond what upstream exercises and is validated by this RFC's own benchmarks and tests.

Proposed approach

Execution model

Adopt the #71 review's model. A template build becomes:

1. Freeze context bindings (below), then compute the full cache-key chain for the recipe — statically, before any sandbox exists.
2. Look up the final-result entry for final_key first; on a miss, probe resumable checkpoint keys from the last mutating step toward the base until the deepest valid entry is found (validation includes lineage, below); after choosing it, execute the full suffix without further cache probes. (This is a design choice for VM builders — resuming is the expensive operation, so we pick the resume point once; E2B's engine instead probes forward per phase and can skip later phases after a miss.)
3. Boot/resume from the chosen checkpoint — whose stored CommandContext restores the logical build state — verify it against the recipe-derived expected context, replay any metadata-only steps in the suffix, execute the mutating steps, checkpointing after each.
4. Promote the final checkpoint to the user-visible template snapshot via the promotion protocol below; intermediate checkpoints are hidden cache records.

This Phase-1 design needs no new artifact format (entries are committed snapshots; layers are ordinary managed layers), but it does need new repository metadata (context bindings, index records, attempt records, a hidden record purpose) and a metadata coordinator on multi-node OSS — it is not "zero new primitives."

Metadata coordination and fencing (required decision)

The design needs linearizable metadata transitions: context-generation publication, upload intents, build bindings and freeze, index create-if-absent and fenced replace, attempt acquisition/renewal, lease grants, GC barriers. On POSIX these map to link/rename/O_EXCL. On the OSS backend no such primitive exists today: the client is unconditional GET/PUT/DELETE (src/snapshot/repository/backends/oss/client.rs), and an in-repo comment records that the Alibaba-OSS S3-compatible path supports neither If-None-Match nor x-oss-forbid-overwrite (src/snapshot/repository/backends/oss/repository.rs). Coordinator options:

• (a) Native conditional writes where the object store supports them (AWS S3 added If-None-Match conditional puts in August 2024 and ETag-based If-Match conditional writes in November 2024; support varies across S3-compatibles and is absent on Alibaba OSS's S3 path). Capability-detected; simplest where available.
• (b) External coordinator — a Redis (or similar) instance owning the metadata transitions; precedent exists (scheduler.redis_addr already backs scheduler binding HA).
• (c) Restrict multi-node OSS operation of the upload/build surface to a single designated writer until (a)/(b) lands.

Recommendation: (b) for multi-node OSS, (a) opportunistically, (c) as the fallback posture. The coordinator is a Phase-0 requirement for multi-node OSS — generation publication, bindings, Waiting expiry, and atomic quota admission need it even with the execution cache disabled.

Coordinator durability contract. The scheduler's existing Redis usage is droppable routing state; this is not. Bindings, intents, attempts, promotion intents, paused-sandbox roots, and GC leases are correctness-grade: the coordinator must run with AOF/replication and no eviction, or the deployment must be able to rebuild coordinator state from durable repository records. After a coordinator loss, GC remains closed until a recovery barrier completes (state rebuilt or verified); "coordinator reachable again" alone does not reopen GC. This contract is part of the coordinator capability definition.

Fencing across systems. A coordinator check followed by an unconditional worker write to OSS or an OCI registry is not fencing: the worker can lose its lease between check and write. The rules:

• Coordinator transitions CAS on their transition-specific precondition (state, epoch, lease — see the state machine in §Failure contract) in one operation; a reconciler closes a fence atomically by transitioning the attempt out of its current state, after which no same-epoch CAS against the old state can succeed.
• Workers write shared storage only at immutable, uniquely-identified paths (fresh candidate snapshot IDs, intent-qualified promotion staging); such writes are harmless if the writer was fenced out, because nothing references them until a coordinator-CAS'd pointer selects them.
• Canonical mutable or user-visible writes — the target snapshot record, alias binding, registry tags, Ready visibility — are never performed as unconditional worker writes; they are steps of the promotion outbox (§Cache publication), each gated by an intent-state CAS, executable by any actor but only in intent order. (Today's OSS publish writes artifacts, then alias, then the committed record unconditionally — src/snapshot/repository/backends/oss/{layout,repository}.rs — which is exactly the pattern the outbox replaces for promotion.)

Cache identity and the trust boundary

filesHash is a client-computed semantic digest the server cannot verify (constraint 2), and current API auth accepts any non-empty credential with no identity (src/api/impls/auth.rs, in-code TODO). Cache identity therefore cannot rest on the client hash alone.

Context bindings (generational, with upload intents). The store keeps:

global:  (scope, filesHash) -> { generation, blob_digest, size }    # latest published generation
build:   (buildID, filesHash) -> { generation, blob_digest }        # binding + lease
intent:  (grant token) -> { scope, buildID, filesHash, reserved_generation,
                            state: Pending -> Completed(generation, blob_digest) | Expired }

• Every GET files/{hash} returns a usable upload URL backed by a created-or-reused UploadIntent with a reserved generation — including when present: true, because the stock SDK's forceUpload PUTs to the URL from that same response without a second GET (constraint 4). Binding is create-if-absent: if the build already has a binding for this hash it is renewed, never rebound; if a published generation exists and no binding does, present: true and the current generation is bound; if none exists, present: false. Intents that are never used simply expire.

• A PUT never writes the reserved generation's object key directly. The body streams to a unique temporary object; the server computes blob_digest and publishes the bytes as an immutable content-addressed blob. Intent completion and binding replacement are then one atomic coordinator transition, mutually exclusive with build-start freeze:

  CAS( intent = Pending  ∧  build = Waiting  ∧  binding = expected current )
    →  intent = Completed(generation, blob_digest)  ∧  binding = generation
  

  Only after this CAS does the server return 2xx. Replays of the same grant token: same digest against an already-Completed intent succeeds only if the build's current (or frozen) binding still is that generation — otherwise it returns a post-freeze conflict; a different digest conflicts outright. There is no window in which a completed intent can rebind a build that has since frozen, and concurrent replays can never overwrite each other's bytes.

• forceUpload therefore works with no special casing: the re-upload completes the reserved intent, producing a new generation bound to this build (and the SDK independently marks the step force); other builds' bindings keep their pinned generations.

• Build-start is one coordinator transition: freeze bindings + record request digest + acquire attempt atomically; the key chain is computed from the frozen (filesHash, blob_digest) pairs, and every recipe-referenced hash must have a frozen binding. After freeze, all PUTs and rebinds for the build are rejected. Registered-but-never-started builds carry a Waiting deadline; expiry releases bindings and intents.

• GC never deletes a generation with live bindings. A malformed or digest-mismatched object is treated as absent/repairable, never as permanently present. (blob_digest detects corruption and pins identity; it cannot prove semantic correspondence between tar and filesHash — containment for a poisoned upload is scoping plus the fact that the poisoned generation only feeds cache keys derived from it.)

Scoping. Archives, bindings, index records, and cache entries are namespaced by tenant scope, not template ID — registration mints a fresh snapshot ID per build (v3_templates_post), so per-template scoping would destroy exactly the cross-build reuse this design exists for. Until authenticated tenancy lands, deployments must explicitly set single_trust_domain = true to enable the upload/cache surface — an operator acknowledgment, not a silent default; the scope key is threaded through from day one so real tenancy is a configuration change, not a redesign.

Key structure. Keys are computed over a versioned, length-delimited canonical encoding (no ambiguous string concatenation). Step classification: RUN, COPY/ADD, and WORKDIR (it runs mkdir -p in the guest) are mutating; ENV, USER, EXPOSED_PORT, VOLUME, LABEL are context-only (USER subject to the execution-identity fix below). Two key kinds exist:

logical_step_key      # one per recipe step, chained through every step
resumable_checkpoint_key = logical_step_key of a mutating step   # has a CheckpointEntry

base_key  = H(v ‖ fingerprint ‖ base_identity)
step_key  = H(v ‖ parent_logical_key ‖ step_kind ‖ canonical(args) ‖ filesHash? ‖ blob_digest?)
final_key = H(v ‖ last_logical_key ‖ canonical(finalization inputs))   # has a FinalResultEntry

fingerprint is derived from the canonical effective launch configuration (not a hand-maintained field list); it must cover at least: resolved base image manifest/layer identity, architecture, disk size, vCPU/memory geometry, virtualization mode, CPU config/template, Firecracker/kernel/envd/tools-image versions, snapshot & cache format versions, and a builder-semantics version. canonical(finalization inputs) covers the normalized/defaulted ready command, the startup command and whether it was inherited or explicit, the effective final CommandContext, ensure_default_user behavior, and a finalization-semantics version. v is the global hashing-version lever; a minimum cached-format version is enforced at lookup (both per E2B).

Execution-identity exclusion (USER). Today USER only updates the host-side CommandContext (src/template/step_executor.rs), and RUN passes envd only env vars and cwd — not the user — so a cold build executes every RUN as envd's boot-time default user. But resuming from a checkpoint re-initializes envd with the snapshot's stored default_user (src/sandbox/firecracker/sandbox.rs, envd init), so the same suffix would execute under a different user than a cold build — a cold-miss/resume divergence. Phase 1 therefore excludes recipes containing USER from the execution cache entirely — they neither read nor write cache entries (they build exactly as today). The proper fix — exec/process requests carrying the effective user explicitly, which also makes cold builds honor USER per Docker semantics — is follow-up work; when it lands, the envd/runtime version enters the fingerprint so old checkpoints with the divergent semantics can never be resumed under the new ones. A cold-miss vs resume equivalence test for the cached step vocabulary is mandatory regardless.

Custom extension. The start-fresh hook may inject instance-specific boot args while cache hits resume via start-resume — two semantically different paths, and today's hook API has no way to declare cache equivalence. When [custom_extension].url is configured, the cache is disabled. A real opt-in would need a defined contract (contract version, cache salt, canonical effective params, and explicit authorization of skipped hooks) — future work, out of scope here.

force semantics. force (build-level, per-step, or induced by forceUpload) bypasses cache reads for the forced step and, stickily, everything downstream. Refreshing shared entries is done safely via lineage (below): forced results replace pointers with a generation-CAS fenced through the attempt state machine; displaced targets become delayed-GC orphans.

Index records, lineage, and validation

Two index record kinds exist:

CheckpointEntry:  {scope}/build-index/{resumable_checkpoint_key} ->
  { snapshot_id, parent: {checkpoint_key, snapshot_id} | {root: base_key, base_identity},
    pointer_generation, kind, cache_schema_version, fingerprint_version }

FinalResultEntry: {scope}/build-index/final/{final_key} ->
  { candidate_snapshot_id, build_base_checkpoint_id?,
    parent: {checkpoint_key, snapshot_id} | {root: base_key, base_identity},
    pointer_generation, versions... }

• CheckpointEntry.parent names the nearest resumable ancestor (skipping context-only steps), or a typed root edge {base_key, base_identity} for the first mutating step (a fresh image base has no parent snapshot). Metadata-only steps contribute to the key chain but never resolve to snapshots — their effects live in the next checkpoint's stored post-step CommandContext, and on resume the builder replays any metadata-only suffix steps deterministically.
• Lineage validation: a chain of pointers is only usable if each record's parent reference resolves consistently (parent snapshot_id matches the parent key's current entry, or the root edge matches the build's base identity). This applies to FinalResultEntry too — a final hit validates the full lineage back to the root, so a forced replace of any ancestor also invalidates stale final pointers. Descendants of a replaced ancestor fail lineage and are treated as misses (and queued for repair) instead of silently gluing mixed-generation chains together. build_base_checkpoint_id is optional on final entries: Phase 0.5 takes no pre-finalization capture, so its promoted templates lack the field and derived builds from them fall back to cache-disabled semantics.
• Records are written only after the checkpoint's publication fully completes; writes are create-if-absent, and invalid or lineage-broken records are replaceable via compare-delete / replace-if-observed-invalid (coordinator-mediated), so a crashed publisher cannot squat a key forever.
• Every lookup validates: record exists, purpose/kind matches, schema/fingerprint versions compatible, target record and artifacts present, lineage intact. Authoritative invalidity (missing target, broken lineage, version mismatch) is a miss plus a repair task; transient unavailability (network or service errors while validating) merely bypasses the cache for this build — it neither queues repair nor mutates any pointer. Neither case surfaces an error to the build.

Cache entry publication and promotion

Cache checkpoints are committed snapshots with a BuildCache record purpose and cache-schema version. Hidden means enforced, not cosmetic: every public resolve/get/status/delete/alias/launch path rejects purpose = BuildCache records (today GET/status/delete and sandbox launch accept any snapshot UUID — src/api/impls/template.rs, src/api/impls/sandbox.rs); only internal cache APIs may read them. To keep rollback safe, the cache catalog lives in a separate repository namespace that older binaries do not scan; alternatively a downgrade requires disabling admission, draining leases, purging cache/P2P state, and completing GC first.

Two publication paths, both required:

• publish_cache_candidate: always stores layers as managed/object-storage layers and never publishes source-registry OCI tags or takes publication ownership — regardless of snapshot_image_storage = source_registry. Candidate metadata retains per-disk origin repositories from the original image resolution: managed materialization rewrites layer URLs to AgentENV storage, and the current exporter derives the target repository from image.json's repoBlobUrl (src/snapshot/repository/backends/common/acr/source_image.rs) — without recorded provenance, promotion could no longer publish to the source registry.

• promote_committed_snapshot: produces the final user snapshot under the target build's snapshot ID, driven by a PromotionIntent that works as a transactional outbox, with per-disk plans (rootfs and each attached drive may publish to different repositories/tags):

  PromotionIntent { build_id, epoch, exact_source_snapshot_id, build_base_checkpoint_id?,
                    target_id,
                    disk_plans: [{ subject: rootfs | drive_id, repository, tag,
                                   layer_actions: [{source_object, digest, size, action}],
                                   config_bytes/digest, manifest_bytes/digest,
                                   ownership_nonce }],
                    state: staged -> registry_committed -> visible -> done }
  

  Entering Publishing (writing the intent) is the irreversible commit decision: from that point the intent converges forward to Ready — lease expiry hands it to the promotion reconciler (the named recovery actor for node death mid-promotion), it never triggers a Failed-plus-cleanup path, so no cleanup can race a late writer. Failed is declared only for authoritative, permanent errors, and its cleanup follows the ownership rules below. Two write classes are distinguished:

  • Repository-side results (target record, alias, Ready) are never unconditional external writes: the record and alias content are staged at intent-qualified immutable paths, and canonical visibility is a coordinator pointer flip — public reads resolve the target/alias through the coordinator-committed pointer, so a fenced-out worker's stale staging is unreachable by construction.
  • Registry tags are inherently external mutable writes; they are made safe by determinism, not CAS: registry_committed is reached only when every disk plan's tag push has completed, each push is idempotent-by-content (the plan carries exact config/manifest bytes with the per-disk ownership_nonce embedded in the OCI config), and duplicate executors of the same intent write identical bytes.

  Registry adoption and ownership. The current publisher requires the tag to be absent before pushing (ensure_manifest_absent, src/snapshot/repository/backends/common/acr/publisher.rs), so a crash between registry PUT and record write would make a naive retry fail forever. With the per-disk plans, a reconciler retry adopts an existing manifest only when its digest matches that disk plan — proving it is this promotion's own remnant, not another writer's tag — and a crash after publishing only some drives' tags resumes with the remaining plans. Any cleanup deletes a tag only after re-reading that it still points at the intent's digest; when ownership cannot be proven, the artifact is left as an orphan rather than deleted.

Each mutating checkpoint stores its exact post-step CommandContext — CommittedSnapshot already carries context, and the resume path already initializes envd from it (src/sandbox/firecracker/config.rs); the builder replays the recipe's metadata effects only to verify the stored context, never as the source of truth.

fromTemplate builds (Phase-1 requirement, not an option). The current runner captures the user snapshot after startup/finalization (src/template/runner.rs), so building fromTemplate on the final snapshot would resume a finalized VM and re-run startup on top of it. User templates therefore record build_base_checkpoint_id — the checkpoint taken after ensure_default_user and before startup/finalization — in snapshot metadata (a GC root, preserved across rollback). The final candidate, the singleflight shared result, and the PromotionIntent all carry and pin this ID, so every waiter's promotion writes it into its own user snapshot. Derived builds extend the build-base checkpoint (startup runs exactly once, in the finalization phase of whichever build publishes); sandboxes keep launching from the final snapshot. Templates predating the field fall back to today's semantics with the cache disabled for that build.

Build identity (MMDS). Every build VM currently receives its sandbox/snapshot ID via MMDS (src/sandbox/firecracker/mmds.rs). The contract:

• The host-side SandboxId remains unique per VM; only the MMDS payload presented to cache-eligible build VMs uses a stable, cache-neutral identity — from Phase 0.5 onward (any build whose result may be promoted under a different identity).
• At user launch, live MMDS serves the target sandbox/template identity.
• MMDS values persisted to disk or memory by build steps remain neutral in the cached state; promotion cannot and does not rewrite them. Templates whose steps persist build identity must opt out of caching entirely (deployment-level cache disable, or the template is documented as unsupported for cached builds) — note that force does not help here: forced builds still run in neutral-MMDS mode and refresh the shared cache with neutral values.

Publication, concurrency, and races

Concurrent identical builds are not assumed to produce identical bytes: RUN may depend on time/network/randomness, and independently sealed OverlayBD uppers embed distinct UUIDs, so layer digests normally differ even for equivalent work. Therefore:

• A builder always publishes its checkpoint as a complete candidate snapshot first, then attempts the index write (create-if-absent; fenced replace only under force). Exactly one candidate wins per key; losers are complete-but-unreferenced snapshots handled as delayed-GC orphans. (Managed-layer digest dedup still collapses genuinely identical layers to zero bytes.)
• Phase-1 singleflight is whole-suffix, per node, keyed by (scope, final_key) — scoped so distinct tenants never merge flights. The leader executes the remaining suffix and publishes the neutral cache candidate (with its build_base_checkpoint_id); each waiter then performs its own promotion under its own build ID and attempt epoch from that shared candidate. A leader's promotion failure (alias, registry, target publication) affects only the leader. Two builds never share a live mutable VM — only completed immutable checkpoints. Per-step rendezvous is deferred. Forced builds bypass flights entirely.
• Cross-node duplicate execution is tolerated (bounded waste, converges via create-if-absent); cross-node in-flight merging is explicitly out of scope.

Failure contract

Build execution today is a detached in-process task holding all of its state in memory (src/api/impls/template.rs), and the OSS Waiting → Building transition is an unguarded read-modify-write — node death leaves Building forever. This RFC requires:

• A durable attempt record per build with an explicit state machine and transition-specific fences:

  Attempt { build_id, attempt_epoch, owner_node, service_instance_id,
            lease_expiry, request_digest, frozen_binding_set_digest,
            selected_checkpoint_ids, publishing_source_candidate_id }
  
  Waiting    --CAS(freeze bindings + acquire)---------------> Active(epoch, lease)
  Active     --CAS(state=Active, epoch, lease valid)--------> Publishing(promotion intent)
  Active     --CAS(state=Active, epoch)---------------------> Failed      # worker error, or reconciler on lease expiry
  Publishing --CAS(state=Publishing, intent-step order)-----> Ready | Failed
  

  The owner renews the lease while Active. The attempt reconciler transitions expired-lease Active attempts to the existing Error status with reason NodeLost (no new schema status) and releases bindings/quota; expired-lease Publishing attempts are instead handed to the promotion reconciler, which drives the intent forward to Ready (§Cache publication) — Failed there is reserved for authoritative permanent errors, never for lease expiry. Terminal states are monotonic — Ready can never be overwritten by a late error; a closed attempt rejects all subsequent same-epoch CAS attempts by construction.

• The minimal attempt lifecycle ships in Phase 0: Waiting → Active(lease) → Ready | Error, with renewal, the expiry reconciler, and terminal release of bindings, intents, and quota — because Phase 0 already introduces freeze+acquire and durable bindings that would otherwise pin forever when a node dies. Phase 0.5 adds Publishing and the promotion outbox; Phase 1 adds checkpoint progress tracking.

• Phase-1 failure policy: fail visibly, do not requeue. A user retry is a new build that reuses all committed checkpoints, which is where the cache already does the heavy lifting. Durable automatic requeue would additionally require persisting the full canonical BuildJobSpec and is deferred — listed as an open question.

• Final promotion is idempotent by (build_id, attempt_epoch) through PromotionIntent; the promotion crash windows (registry tag written but record missing; candidate promoted but status not yet Ready) converge via the reconciler's adopt-by-digest-and-nonce retry, and tests must cover both.

Lifecycle and bounded storage

Per-step caching multiplies snapshot records and managed layers by roughly the step count, and managed-layers/ is today an append-only CAS that record deletion never touches. Bounded storage is therefore a requirement of every phase in which the cache can be enabled: the cache ships default-off, and enabling it requires retention to be implemented.

Roots and leases must be cluster-visible and object-exact:

• Layers are read lazily for a sandbox's whole life, so starting/running/paused sandboxes hold distributed exact-layer leases for the layers their stacks reference, released on delete or lifecycle transition — with acquire-before-release ordering on Running↔Paused transitions (the new state's roots are registered before the old state's are dropped). A node that cannot renew its leases must self-stop its VMs within the lease safety margin — otherwise a partitioned node could keep reading layers whose leases have lapsed and been swept.
• Paused sandboxes are currently persisted in node-local RocksDB (src/orchestrator/persistence/file_backed.rs), invisible to another node's GC. A coordinator-backed paused-root record is written before a pause is visible as successful.
• Deletion uses a mandatory per-object barrier, not best-effort revalidation: Live → Deleting → external DELETE → Deleted, held in the coordinator. Deleting blocks new leases and new references; a publisher that finds an object in Deleting waits for Deleted and re-uploads rather than adopting the doomed object.
• P2P release is durable unpublish work with retry (P2pTransport::unpublish on the owning node) — ForgetP2pArtifact alone only drops the scheduler hint.
• GC fails closed when the coordinator is unavailable, and stays closed after coordinator recovery until the durability-contract recovery barrier completes (§Metadata coordination).

Sweep order:

1. Retire the index pointer (expired, displaced, or lineage-broken).
2. Tombstone the cache record — retaining its manifest so layer membership stays enumerable.
3. Wait out exact-object reader leases (leases pin snapshot_id/blob_digest, never mutable keys).
4. Mark reachable objects from all roots: current index targets, live attempt/candidate/promotion-intent records, user-visible snapshots (including build_base_checkpoint_id references), coordinator-registered paused/running sandbox roots (including attached-drive and memory layers), and fixed artifacts.
5. Transition sweep targets through the Deleting barrier; delete; enqueue durable P2P unpublish.
6. Delete the tombstoned record last.

Source-registry publications never enter cache lifecycle by construction (publish_cache_candidate cannot create them; promotion creates them under the user snapshot's identity and ownership).

Depth budget and preflight. OverlayBD stacks have a hard 255-layer ceiling that binds each stack independently — rootfs, memory, and every writable attached drive (each Phase-1 checkpoint appends a memory layer too); captures whose staged runtime suffix exceeds 32 layers additionally trigger a merge-rewrite of that staged suffix (DEFAULT_MAX_OVERLAYBD_SNAPSHOT_LAYERS — already-published checkpoint layers are not rewritten). The per-build checkpoint budget is computed dynamically:

budget = min(rootfs remaining, memory remaining,
             min over writable attached drives of remaining)
         - reserved captures (pre-finalization checkpoint + final capture)
         - safety margin

Preflight: before build start, the builder verifies the base stack can accommodate at least the reserved captures; if not, it compacts/rebases first or rejects with an actionable error — "degrade to non-checkpointing" only works when the mandatory captures still fit. Past the budget, later steps execute without checkpointing and the build still completes. (The memory term is deliberately conservative: successive captures of one live execution are typically siblings against the inherited memory base rather than an ever-growing live stack; the budget can be relaxed with benchmark evidence without affecting correctness.)

Alternatives considered

The design space decomposes into independent dimensions — where uploaded bytes land (sandbox / node disk / gateway / shared store), how long they must survive (until consumption vs durable), placement coupling (pinned build node vs any node), and failure policy (retry, re-upload, fail). Points in that space we evaluated:

1. Direct upload into the build sandbox. Under the current stock-SDK request sequence, the final build VM cannot be the upload sink: at upload-link time neither the recipe nor the base is known (constraint 5), so there is nothing to boot yet. Uploading "into the sandbox" therefore decomposes into alternative 2 (pin a node early, stage bytes there). Note #74's execution path already streams the archive into the sandbox via envd — staging is the hop between the SDK's upload and that last mile, not a competing transport.
2. Registration-time build→node binding + node-local CAS/spool. Entirely viable without pre-creating a VM: bind the build to a node at POST /v3/templates, route upload-links/PUTs to that node (new gateway capability), stage in a node-local content-addressed spool. Trade-offs vs a shared store: adds scheduler/gateway coupling and exposes staged bytes to single-node loss before consumption; present: true across builds only holds for builds bound to the same node, unless the spool replicates — a node-local lossy CAS answers strictly fewer present queries than a shared durable CAS. On single-node POSIX deployments this is effectively the proposed design (the repository is node-local there). A reasonable future direction once placement pinning exists for other reasons.
3. P2P context CAS. Stage archives node-locally, advertise via the existing P2P artifact index, fetch on demand at build time. To back present: true it needs synchronous replication or a durable fallback — the current P2P index is best-effort and can only serve as a hint. Compatible with adopting later behind the store's staging role.
4. Gateway involvement — two distinct designs: a gateway spool (gateway persists uploads; makes the stateless gateway stateful — rejected for now) vs a stateless tee (gateway streams through to a chosen node and/or durable fallback; keeps the gateway stateless but adds a routing dependency — plausible as an optimization of 2/3).
5. Always present: false + execution caching only. Protocol-compliant and simplest; every build re-uploads all context (the SDK tolerates it; forceUpload semantics unaffected). Loses upload dedup entirely — expected to be costly for large contexts; benchmark required. Kept as a fallback posture if the store is descoped.
6. A1: context store without execution caching (the open stack as implemented, after Phase-0 hardening). Wire-compatible and correctness-complete — with no cache, every step executes, force is trivially honored, cachedBuilds may legitimately be empty. It fails this RFC's execution-reuse goal, not E2B compatibility; it is the natural first milestone on the way to the full design.
7. A1 + whole-build cache key (Daytona's shape; Phase 0.5 below). One key over (recipe, frozen bindings, fingerprint) short-circuiting identical rebuilds. Not subsumed by per-step caching in operational terms: it avoids per-step O(RAM) captures, per-step hidden-record proliferation (one hidden final candidate per build remains), and depth pressure entirely, and may remain the better mode for repetitive CI rebuilds, high-memory builds, or depth-capped builds. It also exercises the same key/index/promotion/GC machinery, which is why it is proposed as its own phase rather than an alternative.

Proposed composition: per-step layer caching (execution reuse) + the slimmed shared store (transport, present oracle, miss-time materialization). The narrow claim: under the current arbitrary request routing (D1), with cross-build/cross-node present: true semantics, and without SDK modifications, shared staging is the minimal increment. If placement pinning lands later, alternatives 2–4 can take over the staging role; placement, lease, and upload-routing interfaces would change, while the execution-cache key/checkpoint model carries over unchanged.

Compatibility and operational impact

API/config changes:

• No SDK-visible endpoint changes (endpoints are dictated by the E2B protocol). [template_build] gains: cache toggles (default-off), single_trust_domain acknowledgment gate, retention budgets/TTLs, depth margins, coordinator configuration; grant TTL/limits carry over from #72 with single-use claims removed.
• force and cachedBuilds become functional rather than ignored (later phases for the latter). Error gains a NodeLost reason (no new status value).
• Public template/sandbox APIs reject purpose = BuildCache snapshots on every direct-ID path.

Snapshot or storage format changes:

• Additive: BuildCache records in a separate cache-catalog namespace (invisible to older binaries); cache-schema version; build_base_checkpoint_id on user snapshots; context generation/binding/intent records; index/attempt/lease/promotion-intent records; server-computed blob_digest. Committed snapshot manifests and layer formats are unchanged in Phases 0/0.5/1; Phase 2's ColdBootRootfs entries would introduce a new tagged artifact kind (see rollout — not resolved by this RFC).

New host/runtime requirements:

• Single-node and POSIX deployments: none. Multi-node OSS deployments require a metadata coordinator from Phase 0 (Redis with AOF/replication/no-eviction — or rebuildable state — or a conditional-write-capable object store) — upload bindings and build-state transitions need it even with the execution cache disabled. TLS on public_base_url is required for the upload surface.

Upgrade and rollback considerations:

• "Cache disabled" means no-cache execution — Phase 0 still changes operational behavior (bindings, Waiting deadlines, quotas, TLS requirement, coordinator on multi-node OSS); it does not change build results. Hashing-version and minimum-format-version levers allow global or format-scoped invalidation without migration. A full cache purge is safe only after: draining exact-object reader leases, draining or failing active attempts, promotion intents, and publication writers, completing queued P2P unpublish work, and resolving build_base_checkpoint_id roots — checkpoints referenced by user snapshots are user-snapshot roots, not cache-owned, so the purge either preserves them or clears the field (dropping those templates to legacy fromTemplate semantics). The separate cache namespace keeps downgraded binaries from ever observing cache records as templates.

Security posture:

• Upload/cache surface requires single_trust_domain = true until authenticated tenant identity exists; scope key threaded now. Grants bind (scope, buildID, hash, reserved generation, method, exact size ceiling, expiry); quota admission is atomic and happens before the body is read; grant replay is idempotent by digest and cannot mint generations. Bearer query tokens are redacted from error logs, access logs, traces, metrics, and forwarded-URI headers.

Rollout plan

• Phase 0 — transport, hardened (revise #71/#72; keep #73/#74 as-is). Store with generational records + upload intents (reserved generation on every link response), temp-object PUT path with digest-idempotent replay, build bindings + leases created on present:true and on upload 2xx, TTL grants (single-use claims dropped), tenant-scope threading + single_trust_domain gate, archive retention subordinate to bindings, Waiting deadlines, atomic freeze+acquire build-start, minimal attempt lifecycle (Waiting → Active(lease) → Ready | Error) with renewal, the Error(NodeLost) expiry reconciler, and terminal release of bindings/intents/quota, coordinator for multi-node OSS, gateway timeout/redaction fixes. #73 (copy_plan.rs, zero store deps) and #74 (envd streaming + in-guest extraction) are unchanged — they are required under every alternative above.
• Phase 0.5 — final-key-only cache (experimental, default-off). One whole-build key over (recipe, frozen bindings, fingerprint) with a FinalResultEntry; hit ⇒ promotion from the cached final candidate via the PromotionIntent outbox and promotion reconciler; cache-neutral MMDS from this phase on. Exercises keys, index records, validation, promotion, retention, and the coordinator end-to-end with no per-step machinery and minimal storage growth.
• Phase 1 — per-step cache (experimental, default-off). Recipes containing USER are excluded from the cache (§Cache identity; the explicit effective-user exec API is follow-up work). Then: key chain + fingerprint; logical vs resumable keys with typed root edges; runner refactored from run-all-steps-then-capture-once (src/template/runner.rs) into a per-step checkpoint loop over the existing live pause→capture→in-place-resume primitive (SandboxBackend::snapshot()); post-step CommandContext stored per checkpoint (replay for verification and metadata-only suffixes); build_base_checkpoint_id (post-ensure_default_user, pre-startup) threaded through candidate, singleflight result, and promotion; candidate publication via publish_cache_candidate + lineage-validated index; (scope, final_key) singleflight with independent waiter promotion; checkpoint progress on the attempt record; mark-and-sweep retention with exact-object leases, Deleting barriers, coordinator-backed sandbox roots, and fail-closed behavior; dynamic depth budget with preflight. Memory cost is accepted and measured: with default mincore selection each checkpoint copies O(resident RAM); with opt-in dirty-page tracking (00ba6cb) the Firecracker query is logically non-clearing (/vm/dirty-memory-ranges), so successive checkpoints re-copy pages dirtied since VM start/resume — complete images, cumulative cost. Benchmarks on both paths gate the default.
• Phase 2 — cold-boot cache entries (follow-up design; not yet proven safe, not resolved by this RFC). A distinct ColdBootRootfs cache-entry kind (new tagged artifact kind; publisher/resolver branches). Known preconditions already identified: in-guest syncfs/fsfreeze before sealing is a prerequisite even for the pause-based variant — a paused VM's page cache is not flushed into the OverlayBD upper, so a completed RUN's file contents can exist only in memory, and background processes/tmpfs/mounts/IPC state vanish on cold boot. Cold-boot entries would be restricted to steps declared filesystem-only/hermetic; arbitrary RUN steps stay on Phase-1 entries; writable attached drives unsupported initially. This phase proceeds only with its own design note and safety evidence.
• Phase 3 — cache-aware scheduling. A node-local residency catalog (distinct from the shared index — cachedBuilds must reflect node-local materialization/page-cache warmth to be useful), heartbeat population of NodeDetail.cachedBuilds, and a template-build schedule hint.

Acceptance criteria

Moved to the first comment below (GitHub issue size limit), together with the full commit-pinned reference list.

Non-goals

• Rebuild-in-place of an existing alias (#70 stays parked pending the alias-system refactor); same-name template rebuild is unsupported.
• A general Dockerfile/LLB frontend; the step vocabulary stays the template API's.
• Cross-tenant cache sharing; cross-node in-flight execution merging; per-step waiter rendezvous.
• Per-file (Modal-style) context dedup granularity — per-archive granularity is accepted; per-file is future work compatible with the store interface.
• Artifact-set identity decoupled from snapshot-record identity (future RFC; promotion copies artifacts instead).
• Custom-extension cache opt-in (requires a contract API that does not exist yet).
• SDK modifications of any kind.

Open questions

1. Tenancy timing. Is the single_trust_domain = true gate acceptable until authenticated identity lands, or should authenticated tenant ownership be a Phase-0 prerequisite?
2. Coordinator choice (§Metadata coordination): Redis (with the stated durability contract) vs native conditional writes vs single-writer restriction — which do maintainers prefer as the supported multi-node OSS posture?
3. Staging home. Shared repository store now (proposed), or invest directly in placement pinning + node-local/P2P staging (alternatives 2–3)?
4. Requeue. Is Error(NodeLost) + user retry acceptable long-term, or is durable requeue (with a persisted BuildJobSpec) wanted — and in which phase?
5. Cache gating. What benchmark thresholds flip any cache default from off to on?
6. Depth margins and step-cap numbers — settle with benchmarks (including whether the conservative memory-stack term can be relaxed).
7. Staleness of non-hermetic RUN. Rely purely on force (proposed, Docker/E2B semantics) or add opt-in per-template cache TTL?

Contribution

I can implement it.

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

---

Full commit-pinned reference list is in the first comment below.

Originally created by @JoyboyBrian on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/141 > **Status (2026-08-09): superseded.** Maintainer review chose a different direction — a native single-stage Dockerfile workflow orchestrated by `aenv`, streaming context into the build VM via envd with no host-side persistence. The shared durable context store, the coordinator, and stock E2B SDK `COPY` compatibility are out of scope; per-step caching and multi-stage support are separate follow-ups. Focused design: #147. The original proposal below is kept for reference. **TL;DR:** Adopt the reviewer's per-step cache, while retaining a slim shared context-staging path required by stock-SDK upload timing and cross-node routing; validate it through Phase 0/0.5 before enabling per-step reuse. - Status: **open for discussion** - Related: #28 (original full PR and concept discussion), #71 #72 #73 #74 (active COPY/ADD stack), #70 (parked alias rebuild) - Prompted by review feedback on #71 proposing (a) uploading build context directly into the build sandbox via envd, and (b) deduplicating builds by snapshotting after each COPY/RUN step so each step yields a cacheable, reusable OverlayBD layer, found via a tree of cache keys with prefix matching. ## Affected component Snapshot / template builder (cross-cutting: API server / proxy, gateway / scheduler in later phases). ## RFC intent and possible closure The #71 review proposes an execution model — cache every mutating build step as an OverlayBD layer and resume the longest completed prefix — that **this RFC recommends adopting**, subject to the evidence and constraints gathered here. One stock-SDK constraint remains to be placed: context uploads complete before the build-start request, and with AgentENV's current independent request routing those bytes need a staging home until the consuming step runs. The RFC therefore proposes retaining a smaller context store as transport and a `present` oracle — not as the build cache — and asks for decisions on: 1. the per-step cache model (key structure, context bindings, publication semantics) in §Proposed approach; 2. the **metadata coordinator** for multi-node OSS deployments (§Metadata coordination) — required from Phase 0, because upload bindings and build-state transitions need linearizable writes that the OSS backend cannot express today; 3. the attempt state machine, promotion outbox, and the reconciler actors that make crashes converge (§Failure contract, §Cache publication); 4. the form of context staging (shared store vs the alternatives in §Alternatives considered); 5. the phasing (including a final-key-only Phase 0.5) and default-off gating of the cache; 6. how the current stack (#71–#74) should be reshaped to fit. Useful closure outcomes: agreement on the target model with the stack revised to match; agreement on a different staging design from §Alternatives; or a recorded decision that execution caching is deferred and the stack lands as a compatibility-only milestone. ## Problem statement The stock E2B SDK's `Template.build()` has one hard correctness requirement AgentENV does not meet, and one capability the protocol anticipates that AgentENV lacks: 1. **Context transport (correctness).** The SDK resolves every `COPY`/`ADD` through `GET /templates/{templateID}/files/{hash}` and `PUT`s a tar archive to the returned URL with a bare HTTP client, finishing **all uploads before it sends the build-start request** ([Template.build](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/index.ts)). Without a place to accept those uploads and make the bytes available to the build node, `COPY` builds cannot work at all. 2. **Execution reuse (performance and cache semantics).** Today's template builder re-executes every step of every build. The E2B protocol exposes per-step cache controls — `TemplateStep.force`, `TemplateBuildStartV2.force`, `NodeDetail.cachedBuilds` exist in AgentENV's `src/api/openapi.yml` — which AgentENV parses and ignores; implementing a step cache gives those fields their intended function (the schema exposes cache control, though it does not by itself mandate any particular cache granularity). The open PR stack (#71–#74) implements transport via a repository-backed archive store, host-side COPY planning, and envd-streamed execution — but no execution caching. The #71 review asks whether the durable store is necessary and proposes per-step layer caching as the deduplication mechanism. This RFC answers with a combined design and a survey of how the same problem is solved elsewhere. ## Use case - Stock E2B SDK (JS/Python) template builds with `COPY`/`ADD` against single-node POSIX and multi-node OSS deployments. - Iterative template development: repeated builds where most steps and most context bytes are unchanged; the user expects unchanged prefixes to be skipped and unchanged context not to be re-uploaded. **Scope note:** rebuilding an existing alias in place is unsupported (parked with #70), so "iterative" here means repeated builds under new template names or IDs; the cache makes those cheap. - Cluster operation: the upload-link request, the upload `PUT`, and the build start are routed independently by the gateway today (`services/gateway/internal/schedule_hint.go` recognizes only `/sandboxes*` routes), so any node may serve any of them. ## Current behavior and workarounds - Template builds run all steps in one sandbox and capture one snapshot at the end (`src/template/runner.rs`); there is no step caching, so every rebuild pays the full step cost. - #71–#74 add the context store, upload API, COPY planning, and COPY execution. Pre-merge gaps found while preparing this RFC (fixes are folded into §Proposed approach): the OSS store's `import` is an `exists` check followed by an unconditional `PUT`, not an atomic first-write publication (`src/snapshot/repository/backends/oss/build_files.rs`); the OSS `try_start_build` transition is an unguarded read-modify-write, and the OSS client has no conditional-write primitive at all (`src/snapshot/repository/backends/oss/{repository,client}.rs` — an in-repo comment notes the S3-compatible path supports neither `If-None-Match` nor `x-oss-forbid-overwrite`); the gateway does not classify a bare upload `PUT` as streaming, so it inherits the short request deadline (`services/gateway/internal/server.go`, `isStreamingRequest`); and gateway error logs serialize the full upstream URL including the bearer query token. - No workaround exists for execution reuse; `force`/`cachedBuilds` are dead schema fields. ## Desired behavior - Given an unchanged step prefix (same base, same steps, same bound context generations, same machine/runtime fingerprint), when a template build runs, then the prefix is not re-executed: the build resumes from the deepest cached checkpoint and executes only the suffix — **and produces the same result a cold build would** (cold-miss/resume equivalence). - Given a context archive bound to a build, then that exact archive generation remains available to the build until it reaches a terminal state — regardless of GC, later uploads, or `forceUpload` by other builds; repeated link requests never silently rebind an existing binding to a newer generation. - Given `force` on a step or build, when the build runs, then that step and everything after it re-execute, and refreshed results replace the stale cache entries without leaving mixed-generation chains observable. - Given a node crash in any build state — including mid-promotion — then the build converges to a visible terminal state via lease expiry and reconcilers; no fenced-out worker can commit canonical state, and terminal states are monotonic (once `Ready`, no later write demotes it). - Cached artifacts are bounded: retention and safe deletion exist in every phase where the cache can be enabled. These are scoped guarantees, not absolutes: re-upload happens when retention expired before any binding or scope differs; re-execution happens on `force`, fingerprint change, or a lost cross-node publication race. ## SDK constraints and current deployment assumptions Observed behaviors of the stock SDKs (pinned: SDK @ [`cab27aa`](https://github.com/e2b-dev/E2B/tree/cab27aa6fa), infra @ [`b8bba83`](https://github.com/e2b-dev/infra/tree/b8bba83444)): 1. **Call sequence.** `POST /v3/templates` (register) → per-COPY hash computation → `GET /templates/{templateID}/files/{hash}` per COPY instruction → `PUT <url>` for each missing archive → **only after all uploads complete**, `POST /v2/templates/{templateID}/builds/{buildID}` → status polling. The JS SDK parallelizes link requests and uploads; the Python SDK is sequential in both its sync and async clients; all finish uploads before build start ([index.ts](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/index.ts), [template_sync/main.py](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/python-sdk/e2b/template_sync/main.py), [template_async/main.py](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/python-sdk/e2b/template_async/main.py)). 2. **The hash is per-COPY-step, client-computed, and semantic.** A sha256 digest seeded with a constructed `COPY <src> <dest>` string, then each file's context-relative path, mode, size, contents, and symlink targets; uid/gid/mtime deliberately excluded; files sorted for determinism ([utils.ts `calculateFilesHash`](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/utils.ts)). It is a digest of the *step's semantic inputs*, not of the uploaded tar bytes — the instruction string is not present in the tar, so **the server cannot recompute or verify this hash from the uploaded archive**. 3. **The `PUT` is a bare HTTP client**: no auth headers; the URL is the credential; exact `Content-Length` required (S3 presigned PUT rejects chunked encoding; see the in-code comment in [buildApi.ts](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/buildApi.ts) referencing e2b-dev/e2b#1243). The default client upload timeout is 1 hour (configurable). 4. **`present: true` normally skips the upload**, but a per-instruction `forceUpload` both re-uploads despite `present` **and marks that COPY step `force`** — and it does so **using the URL from the same link response**, without a second `GET` ([upload loop](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/index.ts#L1096-L1137), [COPY construction](https://github.com/e2b-dev/E2B/blob/cab27aa6fa/packages/js-sdk/src/template/index.ts#L576-L614)). Every link response must therefore carry a usable upload URL, `present` or not. 5. **At upload-link time the server has no recipe and no step arguments** — only `(templateID, hash)` plus the request's auth context. No step-prefix computation is possible when answering `present`. Deployment assumptions (properties of AgentENV today, not SDK law — changing them is part of the design space in §Alternatives considered): - **(D1)** The gateway routes the register, upload-link, upload, and build-start requests independently (`round_robin`/`random`, no template pinning), so today the shared snapshot repository is the only thing making them coherent. AgentENV registration mints a fresh snapshot ID per build request (`v3_templates_post` → `SnapshotId::generate()`), so nothing about the protocol pins consecutive requests to one node. - **(D2)** Context must remain **available until the consuming step executes** (or the build reaches a terminal state). Durable object storage is one way to provide that guarantee; leases over node-local staging are another. ## Prior art ### E2B upstream: a context store plus a per-step layer cache, joined by `filesHash` E2B's build-cache bucket contains two kinds of objects ([paths.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/storage/paths/paths.go#L7-L16); the path helpers take a generic `cacheScope` — the public API passes the team ID, with a server-side fallback to the template ID when unset): ``` {scope}/files/{sha256}.tar # build-context archives, addressed by the SDK hash {scope}/index/{layerHash} # tiny JSON: cache key -> build (snapshot) ID ``` - The upload-link handler stats the blob and mints a presigned URL; `present` = blob exists ([template_layer_files_upload.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/api/internal/handlers/template_layer_files_upload.go), [upload_layer_files_template.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/server/upload_layer_files_template.go); 30-minute URL TTL). Self-hosted deployments can emulate the presigned URL with an HMAC-signed upload endpoint over filesystem storage; any-node coherence then requires that storage to be shared (a shared mount or object store) rather than a single node's disk. - The builder is a phase engine running Firecracker VMs. Each step's cache key chains through its parent: `H(parentHash ‖ stepType ‖ join(args) ‖ filesHash)` ([phases/steps/hash.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/phases/steps/hash.go#L14-L20)) — the "tree of cache keys with prefix matching" from the #71 review. All phase hashes are computable statically from the recipe; the engine computes and probes them **phase by phase in forward order, continuing to probe after a miss** ([phase.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/phases/phase.go#L129-L170)); a hit skips that phase's `Build()`. - A cached layer is a full pausable VM snapshot (memfile + rootfs, stored as diffs against parents) under a fresh build ID. **The context tar is downloaded from the store only on a cache miss, by whatever node runs the build**, streamed into the guest via envd and extracted in-guest ([copy.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/commands/copy.go)). The upload-link request and build-start request each independently pick a build node; nothing reconciles them — coherent only because both talk to the same shared store. - Correctness details adopted below: the hash→buildID index entry is written **only after every snapshot artifact has finished uploading** ([layer_executor.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/layer/layer_executor.go)); `force` is sticky ([builder.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/builder.go)); two global invalidation levers exist (a hashing version folded into the base hash; a minimum cached-format version at lookup — [cache.go](https://github.com/e2b-dev/infra/blob/b8bba83444/packages/orchestrator/pkg/template/build/storage/cache/cache.go)). - Gaps not to copy: uploads are never verified against the claimed hash (containment is scoping only), and there is no durable-object GC for context tars, index entries, or intermediate layer snapshots (node-local caches do have TTL cleanup). ### BuildKit: transfer and caching are orthogonal, joined by a content digest BuildKit ([`8f96667`](https://github.com/moby/buildkit/tree/8f96667173)) has no durable context-archive store; it persists the client's context as a live snapshot keyed by a heuristic `sharedKey`, updated by an incremental stat-based diff that only transfers changed file bodies ([source/local/source.go](https://github.com/moby/buildkit/blob/8f96667/source/local/source.go), [fsutil receive.go @ `6d9dc2e`](https://github.com/tonistiigi/fsutil/blob/6d9dc2ebad/receive.go)). The local-context vertex is deliberately uncacheable across sessions ([ops/source.go](https://github.com/moby/buildkit/blob/8f96667/solver/llbsolver/ops/source.go)); all COPY cache reuse comes from a server-side content checksum attached as an alternative cache key on the consuming step ([edge.go](https://github.com/moby/buildkit/blob/8f96667/solver/edge.go), [contenthash](https://github.com/moby/buildkit/blob/8f96667/cache/contenthash/checksum.go)). Relevant lessons: the E2B protocol shape (client hash + whole-archive upload) trades away incremental transfer and gains a one-round-trip skip plus node independence; the cost — one changed byte re-uploads the whole archive — must be accepted explicitly, and in a VM builder the per-step capture cost (full-VM memory) may dominate overall build time regardless of transfer efficiency, so benchmarks decide. In BuildKit's remote-cache export, `min` mode exports the layers of the resulting image while `max` additionally exports intermediate-stage layers ([cache mode docs](https://docs.docker.com/build/cache/backends/#cache-mode)); on OverlayBD the writable upper **is** the per-step rootfs diff, so exporting intermediate rootfs layers carries little marginal data cost (sealing is O(mapping count), not free — §Proposed approach). BuildKit's GC tiers by reproducibility — cheap-to-reproduce classes get short-TTL small-budget tiers ([gcpolicy.go](https://github.com/moby/buildkit/blob/8f96667/cmd/buildkitd/config/gcpolicy.go)) — and identical in-flight work is deduplicated per vertex. ### Industry survey (trade-off examples, not verdicts) - **Modal** (a close peer: bespoke image format, own builder; sandboxes are secure containers by default, with VM sandboxes as an opt-in beta): a content-addressed file store (per-file sha256, batched existence negotiation, presigned blob upload) **plus** per-layer get-or-create where each layer names its parent image ID ([modal-client @ `bef1b70`](https://github.com/modal-labs/modal-client/tree/bef1b706bc), [Images guide](https://modal.com/docs/guide/images): breaking one layer "causes cascading rebuilds for all subsequent layers"). Modal distinguishes `force_build` (rebuild and overwrite cache) from `ignore_cache` (rebuild without publishing). - **Daytona**: context tars content-addressed in S3 with skip-if-exists; the platform cache key is a single hash of `dockerfile + sorted(contextHashes)` for the whole build; intermediate BuildKit state is runner-local, while completed snapshots propagate across runners ([v0.190.0](https://github.com/daytonaio/daytona/tree/v0.190.0), [snapshot manager](https://github.com/daytonaio/daytona/blob/v0.190.0/apps/api/src/sandbox/managers/snapshot.manager.ts#L337-L450)). A coherent store-only + whole-build-cache design; reuse granularity is the whole build. - **Depot / Fly.io**: Depot attaches ephemeral builder instances to a persistent per-project/per-architecture cache volume (Ceph-backed) and routes a project's builds to a shared builder to maximize hits ([Depot architecture](https://depot.dev/blog/depot-magic-explained)). Fly.io's earlier shared remote builders drew user reports of large context re-transfer in specific setups ([flyctl#591](https://github.com/superfly/flyctl/issues/591) for `--local-only`, [community reports of large `COPY . .` contexts](https://community.fly.io/t/why-does-my-remote-build-have-a-huge-transferring-context/25790)); Fly now defaults to Depot-managed builders ([announcement](https://community.fly.io/t/depot-remote-builders-becoming-the-default/21756)). These illustrate the operational coupling that builder-affinity approaches take on; they are not evidence that such approaches fail. - **kaniko** (daemonless per-step caching; pinned @ [`236ba56`](https://github.com/GoogleContainerTools/kaniko/tree/236ba5690e)): rolling composite key seeded by the base-image digest, extended per instruction with content hashes of referenced files; the cache index is an OCI registry, shared across machines by construction; strict stop-after-first-miss ([composite_cache.go](https://github.com/GoogleContainerTools/kaniko/blob/236ba5690e/pkg/executor/composite_cache.go), [caching docs](https://github.com/GoogleContainerTools/kaniko#caching)). - **OverlayBD upstream** (pinned @ [`4f5c597`](https://github.com/containerd/accelerated-container-image/tree/4f5c597f83)): BuildKit ≥ 0.22 with the overlaybd snapshotter builds native OverlayBD images; in the native writable-layer mode the snapshotter's `Commit()` seals the live writable layer into an immutable lower per committed filesystem snapshot ([QUICKSTART](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83/docs/QUICKSTART.md), [buildkit docs/overlaybd.md](https://github.com/moby/buildkit/blob/8f96667/docs/overlaybd.md)). The conversion path keeps a dedup database keyed by OCI chain_id — a cumulative parent-chain identity used for conversion dedup ([overlaybd_builder.go](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83/cmd/convertor/builder/overlaybd_builder.go)). This validates **per-step filesystem-diff sealing** on this storage format; AgentENV's repeated full-VM checkpoint loop (Phase 1) goes beyond what upstream exercises and is validated by this RFC's own benchmarks and tests. ## Proposed approach ### Execution model Adopt the #71 review's model. A template build becomes: 1. Freeze context bindings (below), then compute the full cache-key chain for the recipe — statically, before any sandbox exists. 2. Look up the **final-result entry** for `final_key` first; on a miss, probe **resumable checkpoint keys** from the last mutating step toward the base until the deepest valid entry is found (validation includes lineage, below); after choosing it, execute the full suffix without further cache probes. (This is a design choice for VM builders — resuming is the expensive operation, so we pick the resume point once; E2B's engine instead probes forward per phase and can skip later phases after a miss.) 3. Boot/resume from the chosen checkpoint — whose stored `CommandContext` restores the logical build state — verify it against the recipe-derived expected context, replay any metadata-only steps in the suffix, execute the mutating steps, checkpointing after each. 4. Promote the final checkpoint to the user-visible template snapshot via the promotion protocol below; intermediate checkpoints are hidden cache records. This Phase-1 design needs **no new artifact format** (entries are committed snapshots; layers are ordinary managed layers), but it does need new repository metadata (context bindings, index records, attempt records, a hidden record purpose) and a metadata coordinator on multi-node OSS — it is not "zero new primitives." ### Metadata coordination and fencing (required decision) The design needs linearizable metadata transitions: context-generation publication, upload intents, build bindings and freeze, index create-if-absent and fenced replace, attempt acquisition/renewal, lease grants, GC barriers. On POSIX these map to `link`/`rename`/`O_EXCL`. On the OSS backend **no such primitive exists today**: the client is unconditional GET/PUT/DELETE (`src/snapshot/repository/backends/oss/client.rs`), and an in-repo comment records that the Alibaba-OSS S3-compatible path supports neither `If-None-Match` nor `x-oss-forbid-overwrite` (`src/snapshot/repository/backends/oss/repository.rs`). Coordinator options: - **(a) Native conditional writes** where the object store supports them (AWS S3 added `If-None-Match` conditional puts in [August 2024](https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/) and ETag-based `If-Match` conditional writes in [November 2024](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/); support varies across S3-compatibles and is absent on Alibaba OSS's S3 path). Capability-detected; simplest where available. - **(b) External coordinator** — a Redis (or similar) instance owning the metadata transitions; precedent exists (`scheduler.redis_addr` already backs scheduler binding HA). - **(c) Restrict** multi-node OSS operation of the upload/build surface to a single designated writer until (a)/(b) lands. Recommendation: (b) for multi-node OSS, (a) opportunistically, (c) as the fallback posture. **The coordinator is a Phase-0 requirement for multi-node OSS** — generation publication, bindings, `Waiting` expiry, and atomic quota admission need it even with the execution cache disabled. **Coordinator durability contract.** The scheduler's existing Redis usage is droppable routing state; this is not. Bindings, intents, attempts, promotion intents, paused-sandbox roots, and GC leases are **correctness-grade**: the coordinator must run with AOF/replication and no eviction, or the deployment must be able to rebuild coordinator state from durable repository records. After a coordinator loss, **GC remains closed until a recovery barrier completes** (state rebuilt or verified); "coordinator reachable again" alone does not reopen GC. This contract is part of the coordinator capability definition. **Fencing across systems.** A coordinator check followed by an unconditional worker write to OSS or an OCI registry is **not** fencing: the worker can lose its lease between check and write. The rules: - Coordinator transitions CAS on their **transition-specific** precondition (state, epoch, lease — see the state machine in §Failure contract) in one operation; a reconciler closes a fence atomically by transitioning the attempt out of its current state, after which no same-epoch CAS against the old state can succeed. - Workers write shared storage **only at immutable, uniquely-identified paths** (fresh candidate snapshot IDs, intent-qualified promotion staging); such writes are harmless if the writer was fenced out, because nothing references them until a coordinator-CAS'd pointer selects them. - Canonical **mutable or user-visible** writes — the target snapshot record, alias binding, registry tags, `Ready` visibility — are never performed as unconditional worker writes; they are steps of the promotion outbox (§Cache publication), each gated by an intent-state CAS, executable by any actor but only in intent order. (Today's OSS publish writes artifacts, then alias, then the committed record unconditionally — `src/snapshot/repository/backends/oss/{layout,repository}.rs` — which is exactly the pattern the outbox replaces for promotion.) ### Cache identity and the trust boundary `filesHash` is a client-computed semantic digest the server cannot verify (constraint 2), and current API auth accepts any non-empty credential with no identity (`src/api/impls/auth.rs`, in-code TODO). Cache identity therefore cannot rest on the client hash alone. **Context bindings (generational, with upload intents).** The store keeps: ``` global: (scope, filesHash) -> { generation, blob_digest, size } # latest published generation build: (buildID, filesHash) -> { generation, blob_digest } # binding + lease intent: (grant token) -> { scope, buildID, filesHash, reserved_generation, state: Pending -> Completed(generation, blob_digest) | Expired } ``` - **Every `GET files/{hash}` returns a usable upload URL backed by a created-or-reused `UploadIntent` with a reserved generation** — including when `present: true`, because the stock SDK's `forceUpload` PUTs to the URL from that same response without a second `GET` (constraint 4). Binding is create-if-absent: if the build already has a binding for this hash it is renewed, never rebound; if a published generation exists and no binding does, `present: true` and the current generation is bound; if none exists, `present: false`. Intents that are never used simply expire. - **A `PUT` never writes the reserved generation's object key directly.** The body streams to a unique temporary object; the server computes `blob_digest` and publishes the bytes as an immutable content-addressed blob. Intent completion and binding replacement are then **one atomic coordinator transition**, mutually exclusive with build-start freeze: ``` CAS( intent = Pending ∧ build = Waiting ∧ binding = expected current ) → intent = Completed(generation, blob_digest) ∧ binding = generation ``` Only after this CAS does the server return 2xx. Replays of the same grant token: same digest against an already-`Completed` intent succeeds **only if** the build's current (or frozen) binding still is that generation — otherwise it returns a post-freeze conflict; a different digest conflicts outright. There is no window in which a completed intent can rebind a build that has since frozen, and concurrent replays can never overwrite each other's bytes. - `forceUpload` therefore works with no special casing: the re-upload completes the reserved intent, producing a new generation bound to this build (and the SDK independently marks the step `force`); other builds' bindings keep their pinned generations. - **Build-start is one coordinator transition**: `freeze bindings + record request digest + acquire attempt` atomically; the key chain is computed from the frozen `(filesHash, blob_digest)` pairs, and every recipe-referenced hash must have a frozen binding. After freeze, all `PUT`s and rebinds for the build are rejected. Registered-but-never-started builds carry a `Waiting` deadline; expiry releases bindings and intents. - GC never deletes a generation with live bindings. A malformed or digest-mismatched object is treated as absent/repairable, never as permanently `present`. (`blob_digest` detects corruption and pins identity; it cannot prove semantic correspondence between tar and `filesHash` — containment for a poisoned upload is scoping plus the fact that the poisoned generation only feeds cache keys derived from it.) **Scoping.** Archives, bindings, index records, and cache entries are namespaced by **tenant scope**, not template ID — registration mints a fresh snapshot ID per build (`v3_templates_post`), so per-template scoping would destroy exactly the cross-build reuse this design exists for. Until authenticated tenancy lands, deployments must explicitly set `single_trust_domain = true` to enable the upload/cache surface — an operator acknowledgment, not a silent default; the scope key is threaded through from day one so real tenancy is a configuration change, not a redesign. **Key structure.** Keys are computed over a versioned, length-delimited canonical encoding (no ambiguous string concatenation). Step classification: `RUN`, `COPY`/`ADD`, and `WORKDIR` (it runs `mkdir -p` in the guest) are **mutating**; `ENV`, `USER`, `EXPOSED_PORT`, `VOLUME`, `LABEL` are **context-only** (`USER` subject to the execution-identity fix below). Two key kinds exist: ``` logical_step_key # one per recipe step, chained through every step resumable_checkpoint_key = logical_step_key of a mutating step # has a CheckpointEntry base_key = H(v ‖ fingerprint ‖ base_identity) step_key = H(v ‖ parent_logical_key ‖ step_kind ‖ canonical(args) ‖ filesHash? ‖ blob_digest?) final_key = H(v ‖ last_logical_key ‖ canonical(finalization inputs)) # has a FinalResultEntry ``` `fingerprint` is derived from the **canonical effective launch configuration** (not a hand-maintained field list); it must cover at least: resolved base image manifest/layer identity, architecture, disk size, vCPU/memory geometry, virtualization mode, CPU config/template, Firecracker/kernel/envd/tools-image versions, snapshot & cache format versions, and a builder-semantics version. `canonical(finalization inputs)` covers the normalized/defaulted ready command, the startup command **and whether it was inherited or explicit**, the effective final `CommandContext`, `ensure_default_user` behavior, and a finalization-semantics version. `v` is the global hashing-version lever; a minimum cached-format version is enforced at lookup (both per E2B). **Execution-identity exclusion (`USER`).** Today `USER` only updates the host-side `CommandContext` (`src/template/step_executor.rs`), and `RUN` passes envd only env vars and cwd — **not the user** — so a cold build executes every `RUN` as envd's boot-time default user. But resuming from a checkpoint re-initializes envd with the snapshot's stored `default_user` (`src/sandbox/firecracker/sandbox.rs`, envd `init`), so the same suffix would execute under a *different* user than a cold build — a cold-miss/resume divergence. **Phase 1 therefore excludes recipes containing `USER` from the execution cache entirely — they neither read nor write cache entries** (they build exactly as today). The proper fix — exec/process requests carrying the effective user explicitly, which also makes cold builds honor `USER` per Docker semantics — is follow-up work; when it lands, the envd/runtime version enters the fingerprint so old checkpoints with the divergent semantics can never be resumed under the new ones. A cold-miss vs resume equivalence test for the cached step vocabulary is mandatory regardless. **Custom extension.** The `start-fresh` hook may inject instance-specific boot args while cache hits resume via `start-resume` — two semantically different paths, and today's hook API has no way to declare cache equivalence. When `[custom_extension].url` is configured, the cache is **disabled**. A real opt-in would need a defined contract (contract version, cache salt, canonical effective params, and explicit authorization of skipped hooks) — future work, out of scope here. **`force` semantics.** `force` (build-level, per-step, or induced by `forceUpload`) bypasses cache **reads** for the forced step and, stickily, everything downstream. Refreshing shared entries is done safely via lineage (below): forced results replace pointers with a **generation-CAS fenced through the attempt state machine**; displaced targets become delayed-GC orphans. ### Index records, lineage, and validation Two index record kinds exist: ``` CheckpointEntry: {scope}/build-index/{resumable_checkpoint_key} -> { snapshot_id, parent: {checkpoint_key, snapshot_id} | {root: base_key, base_identity}, pointer_generation, kind, cache_schema_version, fingerprint_version } FinalResultEntry: {scope}/build-index/final/{final_key} -> { candidate_snapshot_id, build_base_checkpoint_id?, parent: {checkpoint_key, snapshot_id} | {root: base_key, base_identity}, pointer_generation, versions... } ``` - `CheckpointEntry.parent` names the **nearest resumable ancestor** (skipping context-only steps), or a **typed root edge** `{base_key, base_identity}` for the first mutating step (a fresh image base has no parent snapshot). Metadata-only steps contribute to the key chain but never resolve to snapshots — their effects live in the next checkpoint's stored post-step `CommandContext`, and on resume the builder replays any metadata-only suffix steps deterministically. - **Lineage validation:** a chain of pointers is only usable if each record's parent reference resolves consistently (parent `snapshot_id` matches the parent key's current entry, or the root edge matches the build's base identity). **This applies to `FinalResultEntry` too** — a final hit validates the full lineage back to the root, so a forced replace of any ancestor also invalidates stale final pointers. Descendants of a replaced ancestor fail lineage and are treated as misses (and queued for repair) instead of silently gluing mixed-generation chains together. `build_base_checkpoint_id` is optional on final entries: Phase 0.5 takes no pre-finalization capture, so its promoted templates lack the field and derived builds from them fall back to cache-disabled semantics. - Records are written only after the checkpoint's publication fully completes; writes are create-if-absent, and **invalid or lineage-broken records are replaceable via compare-delete / replace-if-observed-invalid** (coordinator-mediated), so a crashed publisher cannot squat a key forever. - Every lookup validates: record exists, purpose/kind matches, schema/fingerprint versions compatible, target record and artifacts present, lineage intact. **Authoritative invalidity** (missing target, broken lineage, version mismatch) is a miss plus a repair task; **transient unavailability** (network or service errors while validating) merely bypasses the cache for this build — it neither queues repair nor mutates any pointer. Neither case surfaces an error to the build. ### Cache entry publication and promotion Cache checkpoints are committed snapshots with a `BuildCache` record purpose and cache-schema version. **Hidden means enforced, not cosmetic:** every public resolve/get/status/delete/alias/launch path rejects `purpose = BuildCache` records (today `GET`/status/delete and sandbox launch accept any snapshot UUID — `src/api/impls/template.rs`, `src/api/impls/sandbox.rs`); only internal cache APIs may read them. To keep rollback safe, the cache catalog lives in a **separate repository namespace that older binaries do not scan**; alternatively a downgrade requires disabling admission, draining leases, purging cache/P2P state, and completing GC first. Two publication paths, both required: - `publish_cache_candidate`: **always** stores layers as managed/object-storage layers and **never** publishes source-registry OCI tags or takes publication ownership — regardless of `snapshot_image_storage = source_registry`. Candidate metadata **retains per-disk origin repositories** from the original image resolution: managed materialization rewrites layer URLs to AgentENV storage, and the current exporter derives the target repository from `image.json`'s `repoBlobUrl` (`src/snapshot/repository/backends/common/acr/source_image.rs`) — without recorded provenance, promotion could no longer publish to the source registry. - `promote_committed_snapshot`: produces the final user snapshot under the target build's snapshot ID, driven by a **`PromotionIntent` that works as a transactional outbox**, with per-disk plans (rootfs and each attached drive may publish to different repositories/tags): ``` PromotionIntent { build_id, epoch, exact_source_snapshot_id, build_base_checkpoint_id?, target_id, disk_plans: [{ subject: rootfs | drive_id, repository, tag, layer_actions: [{source_object, digest, size, action}], config_bytes/digest, manifest_bytes/digest, ownership_nonce }], state: staged -> registry_committed -> visible -> done } ``` Entering `Publishing` (writing the intent) is the **irreversible commit decision**: from that point the intent converges **forward** to `Ready` — lease expiry hands it to the **promotion reconciler** (the named recovery actor for node death mid-promotion), it never triggers a `Failed`-plus-cleanup path, so no cleanup can race a late writer. `Failed` is declared only for authoritative, permanent errors, and its cleanup follows the ownership rules below. Two write classes are distinguished: - **Repository-side results (target record, alias, `Ready`)** are never unconditional external writes: the record and alias content are staged at intent-qualified immutable paths, and canonical visibility is a **coordinator pointer flip** — public reads resolve the target/alias through the coordinator-committed pointer, so a fenced-out worker's stale staging is unreachable by construction. - **Registry tags** are inherently external mutable writes; they are made safe by determinism, not CAS: `registry_committed` is reached only when **every** disk plan's tag push has completed, each push is idempotent-by-content (the plan carries exact config/manifest bytes with the per-disk `ownership_nonce` embedded in the OCI config), and duplicate executors of the same intent write identical bytes. **Registry adoption and ownership.** The current publisher requires the tag to be absent before pushing (`ensure_manifest_absent`, `src/snapshot/repository/backends/common/acr/publisher.rs`), so a crash between registry PUT and record write would make a naive retry fail forever. With the per-disk plans, a reconciler retry **adopts** an existing manifest only when its digest matches that disk plan — proving it is this promotion's own remnant, not another writer's tag — and a crash after publishing only some drives' tags resumes with the remaining plans. Any cleanup deletes a tag only after re-reading that it still points at the intent's digest; when ownership cannot be proven, the artifact is left as an orphan rather than deleted. Each mutating checkpoint stores its exact **post-step `CommandContext`** — `CommittedSnapshot` already carries `context`, and the resume path already initializes envd from it (`src/sandbox/firecracker/config.rs`); the builder replays the recipe's metadata effects only to *verify* the stored context, never as the source of truth. **`fromTemplate` builds (Phase-1 requirement, not an option).** The current runner captures the user snapshot **after** startup/finalization (`src/template/runner.rs`), so building `fromTemplate` on the final snapshot would resume a finalized VM and re-run startup on top of it. User templates therefore record `build_base_checkpoint_id` — the checkpoint taken **after `ensure_default_user` and before startup/finalization** — in snapshot metadata (a GC root, preserved across rollback). The final candidate, the singleflight shared result, and the `PromotionIntent` all carry and pin this ID, so every waiter's promotion writes it into its own user snapshot. Derived builds extend the build-base checkpoint (startup runs exactly once, in the finalization phase of whichever build publishes); sandboxes keep launching from the final snapshot. Templates predating the field fall back to today's semantics with the cache disabled for that build. **Build identity (MMDS).** Every build VM currently receives its sandbox/snapshot ID via MMDS (`src/sandbox/firecracker/mmds.rs`). The contract: - The host-side `SandboxId` remains unique per VM; only the **MMDS payload** presented to cache-eligible build VMs uses a stable, cache-neutral identity — from Phase 0.5 onward (any build whose result may be promoted under a different identity). - At user launch, live MMDS serves the target sandbox/template identity. - **MMDS values persisted to disk or memory by build steps remain neutral in the cached state; promotion cannot and does not rewrite them.** Templates whose steps persist build identity must **opt out of caching entirely** (deployment-level cache disable, or the template is documented as unsupported for cached builds) — note that `force` does *not* help here: forced builds still run in neutral-MMDS mode and refresh the shared cache with neutral values. ### Publication, concurrency, and races Concurrent identical builds are **not** assumed to produce identical bytes: `RUN` may depend on time/network/randomness, and independently sealed OverlayBD uppers embed distinct UUIDs, so layer digests normally differ even for equivalent work. Therefore: - A builder always publishes its checkpoint as a complete **candidate** snapshot first, then attempts the index write (create-if-absent; fenced replace only under `force`). Exactly one candidate wins per key; losers are complete-but-unreferenced snapshots handled as delayed-GC orphans. (Managed-layer digest dedup still collapses genuinely identical layers to zero bytes.) - **Phase-1 singleflight is whole-suffix, per node, keyed by `(scope, final_key)`** — scoped so distinct tenants never merge flights. The leader executes the remaining suffix and publishes the **neutral cache candidate** (with its `build_base_checkpoint_id`); each waiter then performs its own promotion under its own build ID and attempt epoch from that shared candidate. A leader's promotion failure (alias, registry, target publication) affects only the leader. Two builds never share a live mutable VM — only completed immutable checkpoints. Per-step rendezvous is deferred. Forced builds bypass flights entirely. - Cross-node duplicate execution is tolerated (bounded waste, converges via create-if-absent); cross-node in-flight merging is explicitly out of scope. ### Failure contract Build execution today is a detached in-process task holding all of its state in memory (`src/api/impls/template.rs`), and the OSS `Waiting → Building` transition is an unguarded read-modify-write — node death leaves `Building` forever. This RFC requires: - A durable **attempt record** per build with an explicit state machine and transition-specific fences: ``` Attempt { build_id, attempt_epoch, owner_node, service_instance_id, lease_expiry, request_digest, frozen_binding_set_digest, selected_checkpoint_ids, publishing_source_candidate_id } Waiting --CAS(freeze bindings + acquire)---------------> Active(epoch, lease) Active --CAS(state=Active, epoch, lease valid)--------> Publishing(promotion intent) Active --CAS(state=Active, epoch)---------------------> Failed # worker error, or reconciler on lease expiry Publishing --CAS(state=Publishing, intent-step order)-----> Ready | Failed ``` The owner renews the lease while `Active`. The **attempt reconciler** transitions expired-lease `Active` attempts to the existing `Error` status with reason `NodeLost` (no new schema status) and releases bindings/quota; expired-lease `Publishing` attempts are instead handed to the **promotion reconciler**, which drives the intent **forward to `Ready`** (§Cache publication) — `Failed` there is reserved for authoritative permanent errors, never for lease expiry. **Terminal states are monotonic** — `Ready` can never be overwritten by a late error; a closed attempt rejects all subsequent same-epoch CAS attempts by construction. - **The minimal attempt lifecycle ships in Phase 0**: `Waiting → Active(lease) → Ready | Error`, with renewal, the expiry reconciler, and terminal release of bindings, intents, and quota — because Phase 0 already introduces freeze+acquire and durable bindings that would otherwise pin forever when a node dies. Phase 0.5 adds `Publishing` and the promotion outbox; Phase 1 adds checkpoint progress tracking. - **Phase-1 failure policy: fail visibly, do not requeue.** A user retry is a new build that reuses all committed checkpoints, which is where the cache already does the heavy lifting. Durable automatic requeue would additionally require persisting the full canonical `BuildJobSpec` and is deferred — listed as an open question. - Final promotion is idempotent by `(build_id, attempt_epoch)` through `PromotionIntent`; the promotion crash windows (registry tag written but record missing; candidate promoted but status not yet `Ready`) converge via the reconciler's adopt-by-digest-and-nonce retry, and tests must cover both. ### Lifecycle and bounded storage Per-step caching multiplies snapshot records and managed layers by roughly the step count, and `managed-layers/` is today an append-only CAS that record deletion never touches. Bounded storage is therefore a **requirement of every phase in which the cache can be enabled**: the cache ships default-off, and enabling it requires retention to be implemented. **Roots and leases must be cluster-visible and object-exact:** - Layers are read lazily for a sandbox's whole life, so **starting/running/paused sandboxes hold distributed exact-layer leases** for the layers their stacks reference, released on delete or lifecycle transition — with **acquire-before-release ordering** on Running↔Paused transitions (the new state's roots are registered before the old state's are dropped). A node that cannot renew its leases must **self-stop its VMs within the lease safety margin** — otherwise a partitioned node could keep reading layers whose leases have lapsed and been swept. - Paused sandboxes are currently persisted in node-local RocksDB (`src/orchestrator/persistence/file_backed.rs`), invisible to another node's GC. A **coordinator-backed paused-root record** is written before a pause is visible as successful. - Deletion uses a **mandatory per-object barrier**, not best-effort revalidation: `Live → Deleting → external DELETE → Deleted`, held in the coordinator. `Deleting` blocks new leases and new references; a publisher that finds an object in `Deleting` waits for `Deleted` and re-uploads rather than adopting the doomed object. - P2P release is **durable unpublish work with retry** (`P2pTransport::unpublish` on the owning node) — `ForgetP2pArtifact` alone only drops the scheduler hint. - **GC fails closed when the coordinator is unavailable**, and stays closed after coordinator recovery until the durability-contract recovery barrier completes (§Metadata coordination). Sweep order: 1. Retire the index pointer (expired, displaced, or lineage-broken). 2. Tombstone the cache record — **retaining its manifest** so layer membership stays enumerable. 3. Wait out exact-object reader leases (leases pin `snapshot_id`/`blob_digest`, never mutable keys). 4. Mark reachable objects from all roots: current index targets, live attempt/candidate/promotion-intent records, user-visible snapshots (including `build_base_checkpoint_id` references), coordinator-registered paused/running sandbox roots (including attached-drive and memory layers), and fixed artifacts. 5. Transition sweep targets through the `Deleting` barrier; delete; enqueue durable P2P unpublish. 6. Delete the tombstoned record last. Source-registry publications never enter cache lifecycle by construction (`publish_cache_candidate` cannot create them; promotion creates them under the user snapshot's identity and ownership). **Depth budget and preflight.** OverlayBD stacks have a hard 255-layer ceiling that binds **each stack independently — rootfs, memory, and every writable attached drive** (each Phase-1 checkpoint appends a memory layer too); captures whose staged runtime suffix exceeds 32 layers additionally trigger a merge-rewrite of that staged suffix (`DEFAULT_MAX_OVERLAYBD_SNAPSHOT_LAYERS` — already-published checkpoint layers are not rewritten). The per-build checkpoint budget is computed dynamically: ``` budget = min(rootfs remaining, memory remaining, min over writable attached drives of remaining) - reserved captures (pre-finalization checkpoint + final capture) - safety margin ``` **Preflight:** before build start, the builder verifies the base stack can accommodate at least the reserved captures; if not, it compacts/rebases first or rejects with an actionable error — "degrade to non-checkpointing" only works when the mandatory captures still fit. Past the budget, later steps execute without checkpointing and the build still completes. (The memory term is deliberately conservative: successive captures of one live execution are typically siblings against the inherited memory base rather than an ever-growing live stack; the budget can be relaxed with benchmark evidence without affecting correctness.) ## Alternatives considered The design space decomposes into independent dimensions — where uploaded bytes land (sandbox / node disk / gateway / shared store), how long they must survive (until consumption vs durable), placement coupling (pinned build node vs any node), and failure policy (retry, re-upload, fail). Points in that space we evaluated: 1. **Direct upload into the build sandbox.** Under the current stock-SDK request sequence, the final build VM cannot be the upload sink: at upload-link time neither the recipe nor the base is known (constraint 5), so there is nothing to boot yet. Uploading "into the sandbox" therefore decomposes into alternative 2 (pin a node early, stage bytes there). Note #74's execution path already streams the archive into the sandbox via envd — staging is the hop between the SDK's upload and that last mile, not a competing transport. 2. **Registration-time build→node binding + node-local CAS/spool.** Entirely viable without pre-creating a VM: bind the build to a node at `POST /v3/templates`, route upload-links/PUTs to that node (new gateway capability), stage in a node-local content-addressed spool. Trade-offs vs a shared store: adds scheduler/gateway coupling and exposes staged bytes to single-node loss before consumption; `present: true` across builds only holds for builds bound to the same node, unless the spool replicates — a node-local lossy CAS answers strictly fewer `present` queries than a shared durable CAS. On single-node POSIX deployments this *is* effectively the proposed design (the repository is node-local there). A reasonable future direction once placement pinning exists for other reasons. 3. **P2P context CAS.** Stage archives node-locally, advertise via the existing P2P artifact index, fetch on demand at build time. To back `present: true` it needs synchronous replication or a durable fallback — the current P2P index is best-effort and can only serve as a hint. Compatible with adopting later behind the store's staging role. 4. **Gateway involvement** — two distinct designs: a gateway **spool** (gateway persists uploads; makes the stateless gateway stateful — rejected for now) vs a stateless **tee** (gateway streams through to a chosen node and/or durable fallback; keeps the gateway stateless but adds a routing dependency — plausible as an optimization of 2/3). 5. **Always `present: false` + execution caching only.** Protocol-compliant and simplest; every build re-uploads all context (the SDK tolerates it; `forceUpload` semantics unaffected). Loses upload dedup entirely — expected to be costly for large contexts; benchmark required. Kept as a fallback posture if the store is descoped. 6. **A1: context store without execution caching** (the open stack as implemented, after Phase-0 hardening). Wire-compatible and correctness-complete — with no cache, every step executes, `force` is trivially honored, `cachedBuilds` may legitimately be empty. It fails this RFC's execution-reuse goal, not E2B compatibility; it is the natural **first milestone** on the way to the full design. 7. **A1 + whole-build cache key** (Daytona's shape; Phase 0.5 below). One key over `(recipe, frozen bindings, fingerprint)` short-circuiting identical rebuilds. **Not subsumed by per-step caching in operational terms**: it avoids per-step O(RAM) captures, per-step hidden-record proliferation (one hidden final candidate per build remains), and depth pressure entirely, and may remain the better mode for repetitive CI rebuilds, high-memory builds, or depth-capped builds. It also exercises the same key/index/promotion/GC machinery, which is why it is proposed as its own phase rather than an alternative. **Proposed composition:** per-step layer caching (execution reuse) + the slimmed shared store (transport, `present` oracle, miss-time materialization). The narrow claim: **under the current arbitrary request routing (D1), with cross-build/cross-node `present: true` semantics, and without SDK modifications, shared staging is the minimal increment.** If placement pinning lands later, alternatives 2–4 can take over the staging role; placement, lease, and upload-routing interfaces would change, while the execution-cache key/checkpoint model carries over unchanged. ## Compatibility and operational impact API/config changes: - No SDK-visible endpoint changes (endpoints are dictated by the E2B protocol). `[template_build]` gains: cache toggles (default-off), `single_trust_domain` acknowledgment gate, retention budgets/TTLs, depth margins, coordinator configuration; grant TTL/limits carry over from #72 with single-use claims removed. - `force` and `cachedBuilds` become functional rather than ignored (later phases for the latter). `Error` gains a `NodeLost` reason (no new status value). - Public template/sandbox APIs reject `purpose = BuildCache` snapshots on every direct-ID path. Snapshot or storage format changes: - Additive: `BuildCache` records in a **separate cache-catalog namespace** (invisible to older binaries); cache-schema version; `build_base_checkpoint_id` on user snapshots; context generation/binding/intent records; index/attempt/lease/promotion-intent records; server-computed `blob_digest`. Committed snapshot manifests and layer formats are unchanged in Phases 0/0.5/1; Phase 2's `ColdBootRootfs` entries would introduce a new tagged artifact kind (see rollout — not resolved by this RFC). New host/runtime requirements: - Single-node and POSIX deployments: none. **Multi-node OSS deployments require a metadata coordinator from Phase 0** (Redis with AOF/replication/no-eviction — or rebuildable state — or a conditional-write-capable object store) — upload bindings and build-state transitions need it even with the execution cache disabled. TLS on `public_base_url` is required for the upload surface. Upgrade and rollback considerations: - "Cache disabled" means no-cache execution — Phase 0 still changes operational behavior (bindings, `Waiting` deadlines, quotas, TLS requirement, coordinator on multi-node OSS); it does not change build results. Hashing-version and minimum-format-version levers allow global or format-scoped invalidation without migration. A full cache purge is safe only after: draining exact-object reader leases, draining or failing active attempts, promotion intents, and publication writers, completing queued P2P unpublish work, and resolving `build_base_checkpoint_id` roots — checkpoints referenced by user snapshots are user-snapshot roots, not cache-owned, so the purge either preserves them or clears the field (dropping those templates to legacy `fromTemplate` semantics). The separate cache namespace keeps downgraded binaries from ever observing cache records as templates. Security posture: - Upload/cache surface requires `single_trust_domain = true` until authenticated tenant identity exists; scope key threaded now. Grants bind `(scope, buildID, hash, reserved generation, method, exact size ceiling, expiry)`; quota admission is atomic and happens before the body is read; grant replay is idempotent by digest and cannot mint generations. Bearer query tokens are redacted from error logs, access logs, traces, metrics, and forwarded-URI headers. ## Rollout plan - **Phase 0 — transport, hardened (revise #71/#72; keep #73/#74 as-is).** Store with generational records + upload intents (reserved generation on every link response), temp-object PUT path with digest-idempotent replay, build bindings + leases created on `present:true` and on upload 2xx, TTL grants (single-use claims dropped), tenant-scope threading + `single_trust_domain` gate, archive retention subordinate to bindings, `Waiting` deadlines, atomic freeze+acquire build-start, **minimal attempt lifecycle (`Waiting → Active(lease) → Ready | Error`) with renewal, the `Error(NodeLost)` expiry reconciler, and terminal release of bindings/intents/quota**, coordinator for multi-node OSS, gateway timeout/redaction fixes. #73 (`copy_plan.rs`, zero store deps) and #74 (envd streaming + in-guest extraction) are unchanged — they are required under every alternative above. - **Phase 0.5 — final-key-only cache (experimental, default-off).** One whole-build key over `(recipe, frozen bindings, fingerprint)` with a `FinalResultEntry`; hit ⇒ promotion from the cached final candidate via the `PromotionIntent` outbox and promotion reconciler; cache-neutral MMDS from this phase on. Exercises keys, index records, validation, promotion, retention, and the coordinator end-to-end with no per-step machinery and minimal storage growth. - **Phase 1 — per-step cache (experimental, default-off).** Recipes containing `USER` are excluded from the cache (§Cache identity; the explicit effective-user exec API is follow-up work). Then: key chain + fingerprint; logical vs resumable keys with typed root edges; runner refactored from run-all-steps-then-capture-once (`src/template/runner.rs`) into a per-step checkpoint loop over the existing live pause→capture→in-place-resume primitive (`SandboxBackend::snapshot()`); post-step `CommandContext` stored per checkpoint (replay for verification and metadata-only suffixes); `build_base_checkpoint_id` (post-`ensure_default_user`, pre-startup) threaded through candidate, singleflight result, and promotion; candidate publication via `publish_cache_candidate` + lineage-validated index; `(scope, final_key)` singleflight with independent waiter promotion; checkpoint progress on the attempt record; mark-and-sweep retention with exact-object leases, `Deleting` barriers, coordinator-backed sandbox roots, and fail-closed behavior; dynamic depth budget with preflight. Memory cost is accepted and measured: with default mincore selection each checkpoint copies O(resident RAM); with opt-in dirty-page tracking (`00ba6cb`) the Firecracker query is **logically non-clearing** (`/vm/dirty-memory-ranges`), so successive checkpoints re-copy pages dirtied since VM start/resume — complete images, cumulative cost. Benchmarks on both paths gate the default. - **Phase 2 — cold-boot cache entries (follow-up design; not yet proven safe, not resolved by this RFC).** A distinct `ColdBootRootfs` cache-entry kind (new tagged artifact kind; publisher/resolver branches). Known preconditions already identified: **in-guest `syncfs`/`fsfreeze` before sealing is a prerequisite even for the pause-based variant** — a paused VM's page cache is not flushed into the OverlayBD upper, so a completed `RUN`'s file contents can exist only in memory, and background processes/tmpfs/mounts/IPC state vanish on cold boot. Cold-boot entries would be restricted to steps declared filesystem-only/hermetic; arbitrary `RUN` steps stay on Phase-1 entries; writable attached drives unsupported initially. This phase proceeds only with its own design note and safety evidence. - **Phase 3 — cache-aware scheduling.** A node-local residency catalog (distinct from the shared index — `cachedBuilds` must reflect node-local materialization/page-cache warmth to be useful), heartbeat population of `NodeDetail.cachedBuilds`, and a template-build schedule hint. ## Acceptance criteria Moved to the first comment below (GitHub issue size limit), together with the full commit-pinned reference list. ## Non-goals - Rebuild-in-place of an existing alias (#70 stays parked pending the alias-system refactor); same-name template rebuild is unsupported. - A general Dockerfile/LLB frontend; the step vocabulary stays the template API's. - Cross-tenant cache sharing; cross-node in-flight execution merging; per-step waiter rendezvous. - Per-file (Modal-style) context dedup granularity — per-archive granularity is accepted; per-file is future work compatible with the store interface. - Artifact-set identity decoupled from snapshot-record identity (future RFC; promotion copies artifacts instead). - Custom-extension cache opt-in (requires a contract API that does not exist yet). - SDK modifications of any kind. ## Open questions 1. **Tenancy timing.** Is the `single_trust_domain = true` gate acceptable until authenticated identity lands, or should authenticated tenant ownership be a Phase-0 prerequisite? 2. **Coordinator choice** (§Metadata coordination): Redis (with the stated durability contract) vs native conditional writes vs single-writer restriction — which do maintainers prefer as the supported multi-node OSS posture? 3. **Staging home.** Shared repository store now (proposed), or invest directly in placement pinning + node-local/P2P staging (alternatives 2–3)? 4. **Requeue.** Is `Error(NodeLost)` + user retry acceptable long-term, or is durable requeue (with a persisted `BuildJobSpec`) wanted — and in which phase? 5. **Cache gating.** What benchmark thresholds flip any cache default from off to on? 6. **Depth margins and step-cap numbers** — settle with benchmarks (including whether the conservative memory-stack term can be relaxed). 7. **Staleness of non-hermetic `RUN`.** Rely purely on `force` (proposed, Docker/E2B semantics) or add opt-in per-template cache TTL? ## Contribution I can implement it. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation. --- *Full commit-pinned reference list is in the first comment below.*

bsakaguchi closed this issue 2026-10-29 14:18:50 +00:00

bsakaguchi commented 2026-10-29 14:18:50 +00:00

Author

Owner

Copy link

@JoyboyBrian commented on GitHub (Aug 7, 2026):

Acceptance criteria

Functional:

• Stock JS + Python SDK E2E (fresh template, COPY + ADD, content verification, pause/resume) passes against Phase 0; identical E2E with cache enabled passes with an unchanged-prefix rebuild skipping re-execution (observable via build logs/metrics). Same-name alias rebuild remains explicitly unsupported and documented.
• Context binding: present:false → PUT → GC pressure → build-start and present:true → GC pressure → build-start both materialize the bound generation; repeated GETs renew without rebinding; present:true + forceUpload completes the reserved intent and rebinds only the uploading build; grant replay with identical digest is idempotent, with a different digest conflicts (and never overwrites published bytes); post-freeze PUTs are rejected.
• Cold-miss vs resume equivalence: a recipe containing ENV and WORKDIR between two RUN steps produces identical guest state whether built cold or resumed from each checkpoint; a recipe containing USER neither reads nor writes the execution cache and builds with today's semantics.
• Force-generation: forced rebuild of step k refreshes its pointer and lineage-invalidates stale descendants; no mixed-generation chain is servable mid-crash.
• Metadata-only lineage: several metadata steps interleaved between two mutating steps produce correct lineage, context, and resume behavior; the first mutating step's root edge validates against the base identity.
• MMDS, two parts: (a) live MMDS serves neutral identity during cached builds and target identity after user launch; (b) MMDS values persisted by a build step remain neutral in cached state — documented, not promised otherwise.

Concurrency and failure:

• Fencing: a worker whose lease expires after a coordinator check cannot commit canonical state (pointer/index/promotion steps all rejected); a reconciler-closed attempt rejects all subsequent same-epoch writes; Ready is never demoted by a late error.
• Promotion: killing the owner in Publishing leads to the promotion reconciler driving the intent forward to Ready — including the registry-tag-written-record-missing window (adopt by digest + ownership nonce), a crash after publishing only some disk plans' tags (remaining plans resume; registry_committed requires all), and the candidate-promoted-not-yet-Ready window; adoption never claims a tag whose digest does not match the disk plan.
• Singleflight: two concurrent identical builds in one scope on one node share one suffix execution and promote independently (each waiter's user snapshot carries the pinned build_base_checkpoint_id); identical builds in different scopes never share a flight; a leader's promotion failure does not fail waiters.
• Node takeover: killing the owner node mid-build yields Error(NodeLost) via lease expiry in Phase 0 already, releasing bindings, intents, and quota; a retry build reuses all committed checkpoints.

Lifecycle:

• Retention: expired cache records and unreferenced layers are swept in the specified order; leased objects survive; tombstoned manifests remain enumerable until sweep completes; an object in Deleting admits no new lease and forces publishers to wait-and-reupload; GC concurrent with starting/running/paused sandboxes never removes their layers; Running↔Paused transitions acquire new roots before releasing old ones; GC fails closed during coordinator loss and stays closed until the recovery barrier completes.
• Partition: a node that cannot renew leases self-stops its VMs within the safety margin.
• Depth: preflight rejects (or compacts before) builds whose base cannot fit the reserved captures; builds hitting the budget mid-way degrade to non-checkpointing execution and still complete.
• Invalidation matrix: a change to each key input class (base identity, fingerprint field, step args, context blob, finalization inputs) invalidates exactly the expected suffix.

Benchmarks (gate any cache default):

• Compare Phase 0 vs 0.5 vs 1 on early/middle/late-mutation rebuild scenarios, measuring wall time, steps executed, bytes uploaded, capture CPU/I-O, retained bytes, and layer depth — in addition to mincore vs dirty-tracking capture cost across RAM sizes and step counts.

References

E2B (pinned: infra @ b8bba834, SDK @ cab27aa6)

• SDK build flow: index.ts · buildApi.ts · utils.ts · Python sync main.py · Python async main.py
• Server store & upload: paths.go · template_layer_files_upload.go · upload_layer_files_template.go
• Builder & cache: phases/phase.go · phases/steps/hash.go · storage/cache/cache.go · layer/layer_executor.go · commands/copy.go · build/builder.go

BuildKit (pinned @ 8f966671; fsutil @ 6d9dc2eb)

• source/local/source.go · solver/llbsolver/ops/source.go · solver/edge.go · cache/contenthash/checksum.go · cmd/buildkitd/config/gcpolicy.go · docs/dev/solver.md · docs/overlaybd.md · cache mode docs · fsutil receive.go

OverlayBD upstream (pinned @ 4f5c597f)

• QUICKSTART · USERSPACE_CONVERTOR · overlaybd_builder.go · WRITABLE.md

Industry

• Modal (pinned @ bef1b706): modal-client · Images guide · container loading internals · sandbox snapshots
• Daytona: v0.190.0 · snapshot manager · declarative builder · snapshots
• Depot: architecture · build context
• Fly.io: remote builds · flyctl#591 · context re-transfer thread · Depot builders default
• kaniko (pinned @ 236ba569): composite_cache.go · caching docs
• Namespace cache volumes: docs
• CodeSandbox (whole-VM snapshot model): cloning a running VM in 2s
• AWS S3 conditional writes: If-None-Match (2024-08) · ETag If-Match (2024-11)

AgentENV internals referenced (branch e2b-build/03-build-file-store unless noted)

• Store: src/snapshot/repository/build_files.rs, src/snapshot/repository/backends/{posixfs,oss}/build_files.rs (#71); upload API src/api/build_files.rs, src/api/impls/template.rs (#72 branch); COPY planning/execution src/template/copy_plan.rs (#73), src/template/step_executor.rs (#74)
• Capture/resume: src/sandbox/firecracker/sandbox.rs (snapshot(), pause_to_dir, envd init on resume with stored user/workdir/env), src/sandbox/firecracker/instance.rs (in-place resume), src/orchestrator/service.rs (capture_snapshot), src/template/runner.rs (current single-capture flow; per-build MMDS identity; startup before capture)
• Step semantics: src/template/step_executor.rs (User updates host-side context only; Run passes env/cwd without user)
• Snapshot context & resume init: src/snapshot/types/snapshot.rs (CommittedSnapshot.context), src/sandbox/firecracker/config.rs (resume initializes envd from stored context), src/sandbox/firecracker/mmds.rs (instanceID/envID)
• Seal/restack: storage/overlaybd/src/lsmt/file/readwrite.rs (close_seal), storage/overlaybd/src/image/image_file.rs, storage/overlaybd/src/image/snapshot.rs (export_upper_as_snapshot_layer, cumulative copy), src/sandbox/firecracker/overlaybd_snapshot.rs (memory conversion; staged-suffix compaction at 32 layers), storage/overlaybd/src/lsmt/file/types.rs (255-layer ceiling, applies per stack incl. memory)
• Layer model & dedup: src/snapshot/repository/backends/posixfs/artifacts.rs; chained conversion cache src/image/oci_image.rs (LayerConversionKey)
• OSS coordination gap & publish ordering: src/snapshot/repository/backends/oss/client.rs (unconditional GET/PUT/DELETE), src/snapshot/repository/backends/oss/repository.rs (no conditional writes on the S3-compatible path; try_start_build RMW; artifacts → alias → record publish order; disk_publications), src/snapshot/repository/backends/oss/layout.rs (catalog/alias/record keys)
• Source-registry export provenance & idempotency: src/snapshot/repository/backends/common/acr/source_image.rs (repository derived from repoBlobUrl), src/snapshot/repository/backends/common/acr/publisher.rs (ensure_manifest_absent preflight)
• Public direct-ID surfaces to be purpose-gated: src/api/impls/template.rs, src/api/impls/sandbox.rs
• Paused-sandbox persistence (node-local): src/orchestrator/persistence/file_backed.rs
• Pre-merge gaps folded into Phase 0: src/snapshot/repository/backends/oss/build_files.rs (import exists-then-PUT), services/gateway/internal/server.go (isStreamingRequest coverage; upstream-URL token logging), src/api/impls/auth.rs (presence-only auth TODO)
• Protocol cache fields parsed-but-ignored: src/api/openapi.yml (TemplateStep.force, TemplateBuildStartV2.force, NodeDetail.cachedBuilds)
• Dirty-page tracking (opt-in, non-clearing query): main 00ba6cb, thirdparty/firecracker-client/firecracker.yaml (/vm/dirty-memory-ranges)
• Custom extension start-fresh/start-resume asymmetry: src/sandbox/custom_extension/client.rs, CLAUDE.md §Custom extension

<!-- gh-comment-id:5221387267 --> @JoyboyBrian commented on GitHub (Aug 7, 2026): ## Acceptance criteria Functional: - Stock JS + Python SDK E2E (fresh template, COPY + ADD, content verification, pause/resume) passes against Phase 0; identical E2E with cache enabled passes with an unchanged-prefix rebuild skipping re-execution (observable via build logs/metrics). Same-name alias rebuild remains explicitly unsupported and documented. - Context binding: `present:false → PUT → GC pressure → build-start` and `present:true → GC pressure → build-start` both materialize the bound generation; repeated `GET`s renew without rebinding; `present:true + forceUpload` completes the reserved intent and rebinds only the uploading build; grant replay with identical digest is idempotent, with a different digest conflicts (and never overwrites published bytes); post-freeze `PUT`s are rejected. - **Cold-miss vs resume equivalence**: a recipe containing `ENV` and `WORKDIR` between two `RUN` steps produces identical guest state whether built cold or resumed from each checkpoint; a recipe containing `USER` neither reads nor writes the execution cache and builds with today's semantics. - Force-generation: forced rebuild of step *k* refreshes its pointer and lineage-invalidates stale descendants; no mixed-generation chain is servable mid-crash. - Metadata-only lineage: several metadata steps interleaved between two mutating steps produce correct lineage, context, and resume behavior; the first mutating step's root edge validates against the base identity. - MMDS, two parts: (a) live MMDS serves neutral identity during cached builds and target identity after user launch; (b) MMDS values persisted by a build step remain neutral in cached state — documented, not promised otherwise. Concurrency and failure: - Fencing: a worker whose lease expires after a coordinator check cannot commit canonical state (pointer/index/promotion steps all rejected); a reconciler-closed attempt rejects all subsequent same-epoch writes; `Ready` is never demoted by a late error. - Promotion: killing the owner in `Publishing` leads to the promotion reconciler driving the intent forward to `Ready` — including the registry-tag-written-record-missing window (adopt by digest + ownership nonce), a crash after publishing only **some** disk plans' tags (remaining plans resume; `registry_committed` requires all), and the candidate-promoted-not-yet-`Ready` window; adoption never claims a tag whose digest does not match the disk plan. - Singleflight: two concurrent identical builds in one scope on one node share one suffix execution and promote independently (each waiter's user snapshot carries the pinned `build_base_checkpoint_id`); identical builds in **different scopes never share a flight**; a leader's promotion failure does not fail waiters. - Node takeover: killing the owner node mid-build yields `Error(NodeLost)` via lease expiry **in Phase 0 already**, releasing bindings, intents, and quota; a retry build reuses all committed checkpoints. Lifecycle: - Retention: expired cache records and unreferenced layers are swept in the specified order; leased objects survive; tombstoned manifests remain enumerable until sweep completes; an object in `Deleting` admits no new lease and forces publishers to wait-and-reupload; GC concurrent with starting/running/paused sandboxes never removes their layers; Running↔Paused transitions acquire new roots before releasing old ones; GC fails closed during coordinator loss and stays closed until the recovery barrier completes. - Partition: a node that cannot renew leases self-stops its VMs within the safety margin. - Depth: preflight rejects (or compacts before) builds whose base cannot fit the reserved captures; builds hitting the budget mid-way degrade to non-checkpointing execution and still complete. - Invalidation matrix: a change to each key input class (base identity, fingerprint field, step args, context blob, finalization inputs) invalidates exactly the expected suffix. Benchmarks (gate any cache default): - Compare Phase 0 vs 0.5 vs 1 on early/middle/late-mutation rebuild scenarios, measuring wall time, steps executed, bytes uploaded, capture CPU/I-O, retained bytes, and layer depth — in addition to mincore vs dirty-tracking capture cost across RAM sizes and step counts. ## References **E2B** (pinned: infra @ [`b8bba834`](https://github.com/e2b-dev/infra/tree/b8bba83444022eb1d95de56853be28c61e7dad62), SDK @ [`cab27aa6`](https://github.com/e2b-dev/E2B/tree/cab27aa6fabd53f759189328c4f74df2df1550ad)) - SDK build flow: [`index.ts`](https://github.com/e2b-dev/E2B/blob/cab27aa6fabd53f759189328c4f74df2df1550ad/packages/js-sdk/src/template/index.ts) · [`buildApi.ts`](https://github.com/e2b-dev/E2B/blob/cab27aa6fabd53f759189328c4f74df2df1550ad/packages/js-sdk/src/template/buildApi.ts) · [`utils.ts`](https://github.com/e2b-dev/E2B/blob/cab27aa6fabd53f759189328c4f74df2df1550ad/packages/js-sdk/src/template/utils.ts) · [Python sync `main.py`](https://github.com/e2b-dev/E2B/blob/cab27aa6fabd53f759189328c4f74df2df1550ad/packages/python-sdk/e2b/template_sync/main.py) · [Python async `main.py`](https://github.com/e2b-dev/E2B/blob/cab27aa6fabd53f759189328c4f74df2df1550ad/packages/python-sdk/e2b/template_async/main.py) - Server store & upload: [`paths.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/storage/paths/paths.go) · [`template_layer_files_upload.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/api/internal/handlers/template_layer_files_upload.go) · [`upload_layer_files_template.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/server/upload_layer_files_template.go) - Builder & cache: [`phases/phase.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/phases/phase.go) · [`phases/steps/hash.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/phases/steps/hash.go) · [`storage/cache/cache.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/storage/cache/cache.go) · [`layer/layer_executor.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/layer/layer_executor.go) · [`commands/copy.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/commands/copy.go) · [`build/builder.go`](https://github.com/e2b-dev/infra/blob/b8bba83444022eb1d95de56853be28c61e7dad62/packages/orchestrator/pkg/template/build/builder.go) **BuildKit** (pinned @ [`8f966671`](https://github.com/moby/buildkit/tree/8f966671738d20a4e05e6e0f7fd344b9299ea6cd); fsutil @ [`6d9dc2eb`](https://github.com/tonistiigi/fsutil/tree/6d9dc2ebad626eb645fb587f14225b3aa96ec010)) - [`source/local/source.go`](https://github.com/moby/buildkit/blob/8f96667/source/local/source.go) · [`solver/llbsolver/ops/source.go`](https://github.com/moby/buildkit/blob/8f96667/solver/llbsolver/ops/source.go) · [`solver/edge.go`](https://github.com/moby/buildkit/blob/8f96667/solver/edge.go) · [`cache/contenthash/checksum.go`](https://github.com/moby/buildkit/blob/8f96667/cache/contenthash/checksum.go) · [`cmd/buildkitd/config/gcpolicy.go`](https://github.com/moby/buildkit/blob/8f96667/cmd/buildkitd/config/gcpolicy.go) · [`docs/dev/solver.md`](https://github.com/moby/buildkit/blob/8f96667/docs/dev/solver.md) · [`docs/overlaybd.md`](https://github.com/moby/buildkit/blob/8f96667/docs/overlaybd.md) · [cache mode docs](https://docs.docker.com/build/cache/backends/#cache-mode) · [fsutil `receive.go`](https://github.com/tonistiigi/fsutil/blob/6d9dc2ebad626eb645fb587f14225b3aa96ec010/receive.go) **OverlayBD upstream** (pinned @ [`4f5c597f`](https://github.com/containerd/accelerated-container-image/tree/4f5c597f83ac3af8ff77ffdd97253581b8cebdc0)) - [QUICKSTART](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83ac3af8ff77ffdd97253581b8cebdc0/docs/QUICKSTART.md) · [USERSPACE_CONVERTOR](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83ac3af8ff77ffdd97253581b8cebdc0/docs/USERSPACE_CONVERTOR.md) · [`overlaybd_builder.go`](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83ac3af8ff77ffdd97253581b8cebdc0/cmd/convertor/builder/overlaybd_builder.go) · [WRITABLE.md](https://github.com/containerd/accelerated-container-image/blob/4f5c597f83ac3af8ff77ffdd97253581b8cebdc0/docs/WRITABLE.md) **Industry** - Modal (pinned @ [`bef1b706`](https://github.com/modal-labs/modal-client/tree/bef1b706bccb2088689b6fbc70bd5b8dee3d4a90)): [modal-client](https://github.com/modal-labs/modal-client) · [Images guide](https://modal.com/docs/guide/images) · [container loading internals](https://modal.com/blog/jono-containers-talk) · [sandbox snapshots](https://modal.com/docs/guide/sandbox-snapshots) - Daytona: [v0.190.0](https://github.com/daytonaio/daytona/tree/v0.190.0) · [snapshot manager](https://github.com/daytonaio/daytona/blob/v0.190.0/apps/api/src/sandbox/managers/snapshot.manager.ts) · [declarative builder](https://www.daytona.io/docs/en/declarative-builder/) · [snapshots](https://www.daytona.io/docs/en/snapshots/) - Depot: [architecture](https://depot.dev/blog/depot-magic-explained) · [build context](https://depot.dev/blog/build-context) - Fly.io: [remote builds](https://fly.io/blog/persistent-storage-and-fast-remote-builds/) · [flyctl#591](https://github.com/superfly/flyctl/issues/591) · [context re-transfer thread](https://community.fly.io/t/why-does-my-remote-build-have-a-huge-transferring-context/25790) · [Depot builders default](https://community.fly.io/t/depot-remote-builders-becoming-the-default/21756) - kaniko (pinned @ [`236ba569`](https://github.com/GoogleContainerTools/kaniko/tree/236ba5690eda2963cba9c6ee2b5c162a8f0c0d1b)): [composite_cache.go](https://github.com/GoogleContainerTools/kaniko/blob/236ba5690eda2963cba9c6ee2b5c162a8f0c0d1b/pkg/executor/composite_cache.go) · [caching docs](https://github.com/GoogleContainerTools/kaniko#caching) - Namespace cache volumes: [docs](https://namespace.so/docs/architecture/storage/cache-volumes) - CodeSandbox (whole-VM snapshot model): [cloning a running VM in 2s](https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-seconds) - AWS S3 conditional writes: [If-None-Match (2024-08)](https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/) · [ETag If-Match (2024-11)](https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/) **AgentENV internals referenced** (branch `e2b-build/03-build-file-store` unless noted) - Store: `src/snapshot/repository/build_files.rs`, `src/snapshot/repository/backends/{posixfs,oss}/build_files.rs` (#71); upload API `src/api/build_files.rs`, `src/api/impls/template.rs` (#72 branch); COPY planning/execution `src/template/copy_plan.rs` (#73), `src/template/step_executor.rs` (#74) - Capture/resume: `src/sandbox/firecracker/sandbox.rs` (`snapshot()`, `pause_to_dir`, envd `init` on resume with stored user/workdir/env), `src/sandbox/firecracker/instance.rs` (in-place resume), `src/orchestrator/service.rs` (`capture_snapshot`), `src/template/runner.rs` (current single-capture flow; per-build MMDS identity; startup before capture) - Step semantics: `src/template/step_executor.rs` (`User` updates host-side context only; `Run` passes env/cwd without user) - Snapshot context & resume init: `src/snapshot/types/snapshot.rs` (`CommittedSnapshot.context`), `src/sandbox/firecracker/config.rs` (resume initializes envd from stored context), `src/sandbox/firecracker/mmds.rs` (`instanceID`/`envID`) - Seal/restack: `storage/overlaybd/src/lsmt/file/readwrite.rs` (`close_seal`), `storage/overlaybd/src/image/image_file.rs`, `storage/overlaybd/src/image/snapshot.rs` (`export_upper_as_snapshot_layer`, cumulative copy), `src/sandbox/firecracker/overlaybd_snapshot.rs` (memory conversion; staged-suffix compaction at 32 layers), `storage/overlaybd/src/lsmt/file/types.rs` (255-layer ceiling, applies per stack incl. memory) - Layer model & dedup: `src/snapshot/repository/backends/posixfs/artifacts.rs`; chained conversion cache `src/image/oci_image.rs` (`LayerConversionKey`) - OSS coordination gap & publish ordering: `src/snapshot/repository/backends/oss/client.rs` (unconditional GET/PUT/DELETE), `src/snapshot/repository/backends/oss/repository.rs` (no conditional writes on the S3-compatible path; `try_start_build` RMW; artifacts → alias → record publish order; `disk_publications`), `src/snapshot/repository/backends/oss/layout.rs` (catalog/alias/record keys) - Source-registry export provenance & idempotency: `src/snapshot/repository/backends/common/acr/source_image.rs` (repository derived from `repoBlobUrl`), `src/snapshot/repository/backends/common/acr/publisher.rs` (`ensure_manifest_absent` preflight) - Public direct-ID surfaces to be purpose-gated: `src/api/impls/template.rs`, `src/api/impls/sandbox.rs` - Paused-sandbox persistence (node-local): `src/orchestrator/persistence/file_backed.rs` - Pre-merge gaps folded into Phase 0: `src/snapshot/repository/backends/oss/build_files.rs` (`import` exists-then-PUT), `services/gateway/internal/server.go` (`isStreamingRequest` coverage; upstream-URL token logging), `src/api/impls/auth.rs` (presence-only auth TODO) - Protocol cache fields parsed-but-ignored: `src/api/openapi.yml` (`TemplateStep.force`, `TemplateBuildStartV2.force`, `NodeDetail.cachedBuilds`) - Dirty-page tracking (opt-in, non-clearing query): main `00ba6cb`, `thirdparty/firecracker-client/firecracker.yaml` (`/vm/dirty-memory-ranges`) - Custom extension start-fresh/start-resume asymmetry: `src/sandbox/custom_extension/client.rs`, `CLAUDE.md` §Custom extension

bsakaguchi commented 2026-10-29 14:18:51 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Aug 9, 2026):

Thanks for the detailed RFC. The proposal combines several distinct concerns: context transport, execution caching, distributed coordination, snapshot promotion, and garbage collection. I would prefer to avoid introducing Redis or treating build context as part of the durable snapshot repository.

My preference is to stream build context through envd into a node-local build VM. This fits the OverlayBD architecture better: storage can be bounded and reclaimed with the build, context files can be transferred on demand, and P2P can accelerate reusable artifacts without making OSS or POSIX storage the upload bottleneck.

The main difficulty is the stock E2B SDK request ordering: it uploads all files before the build-start request provides the base image and build recipe.

I think AgentENV should also consider a native Dockerfile build workflow supporting COPY/ADD and multi-stage builds. Instead of modeling a build as one server-side API call, aenv could orchestrate it as a sequence of API operations: create the build VM, upload context directly, execute each step, and capture the resulting snapshot. This would avoid the E2B ordering constraint while keeping the initial server-side design relatively small.

<!-- gh-comment-id:5229597507 --> @yingdi-shan commented on GitHub (Aug 9, 2026): Thanks for the detailed RFC. The proposal combines several distinct concerns: context transport, execution caching, distributed coordination, snapshot promotion, and garbage collection. I would prefer to avoid introducing Redis or treating build context as part of the durable snapshot repository. My preference is to stream build context through envd into a node-local build VM. This fits the OverlayBD architecture better: storage can be bounded and reclaimed with the build, context files can be transferred on demand, and P2P can accelerate reusable artifacts without making OSS or POSIX storage the upload bottleneck. The main difficulty is the stock E2B SDK request ordering: it uploads all files before the build-start request provides the base image and build recipe. I think AgentENV should also consider a native Dockerfile build workflow supporting COPY/ADD and multi-stage builds. Instead of modeling a build as one server-side API call, `aenv` could orchestrate it as a sequence of API operations: create the build VM, upload context directly, execute each step, and capture the resulting snapshot. This would avoid the E2B ordering constraint while keeping the initial server-side design relatively small.

bsakaguchi commented 2026-10-29 14:18:51 +00:00

Author

Owner

Copy link

@JoyboyBrian commented on GitHub (Aug 9, 2026):

Thanks @yingdi-shan this makes the intended direction clear. I'll drop the shared durable context store and the new correctness-grade external coordinator from the design; before revising #141 or starting implementation, I'd like to confirm the resulting scope.

My reading of your proposal:

• A native Dockerfile workflow where aenv creates an ordinary sandbox as the build VM, streams the required context through envd, executes the steps, and captures the resulting snapshot. Existing sandbox routing keeps these operations on the sandbox's node, so this path requires no new cross-node coordination.
• Build context is ephemeral, build-session-scoped transport state and never enters the durable snapshot repository.
• Per-step OverlayBD checkpoints remain the reusable artifacts, with P2P accelerating those immutable artifacts rather than context bytes.
• The build-session and transport path lands first; per-step execution caching is a separate later milestone.

I propose the following initial scope:

1. Native workflow. Start with single-stage Dockerfiles, including COPY/ADD and RUN. Multi-stage support would follow after explicitly defining stage filesystem access and COPY --from semantics. Is that sequencing acceptable?

2. Stock E2B SDK compatibility. The stock SDK completes its uploads before build-start provides the base image and recipe, so those bytes cannot go directly into the final build VM. If stock-SDK COPY support remains in the initial scope, I propose pinning the registered build session to a node and staging uploads in a bounded, build-scoped local spool on that node. At build-start, the same node creates the VM and streams the staged files through envd.

   The spool would be removed when the build terminates or expires. Initially, no cross-build context deduplication would be promised (present may be false for each new build). If the pinned node is lost before consumption, the build fails and a retry re-uploads rather than transparently resuming elsewhere. This requires a small gateway routing extension, but no durable context store or new correctness-grade external coordinator.

   Alternatively, stock-SDK COPY compatibility can follow the native workflow as a separate milestone. Do you prefer it in the initial milestone or as a follow-up?

If this matches your intent, I'll update and split #141 accordingly, rework or close #71/#72 depending on (2), reuse the applicable host-side COPY planning and envd streaming pieces from #73/#74, and only then begin implementation.

<!-- gh-comment-id:5229837514 --> @JoyboyBrian commented on GitHub (Aug 9, 2026): Thanks @yingdi-shan this makes the intended direction clear. I'll drop the shared durable context store and the new correctness-grade external coordinator from the design; before revising #141 or starting implementation, I'd like to confirm the resulting scope. My reading of your proposal: - A native Dockerfile workflow where `aenv` creates an ordinary sandbox as the build VM, streams the required context through envd, executes the steps, and captures the resulting snapshot. Existing sandbox routing keeps these operations on the sandbox's node, so this path requires no new cross-node coordination. - Build context is ephemeral, build-session-scoped transport state and never enters the durable snapshot repository. - Per-step OverlayBD checkpoints remain the reusable artifacts, with P2P accelerating those immutable artifacts rather than context bytes. - The build-session and transport path lands first; per-step execution caching is a separate later milestone. I propose the following initial scope: 1. **Native workflow.** Start with single-stage Dockerfiles, including `COPY`/`ADD` and `RUN`. Multi-stage support would follow after explicitly defining stage filesystem access and `COPY --from` semantics. Is that sequencing acceptable? 2. **Stock E2B SDK compatibility.** The stock SDK completes its uploads before build-start provides the base image and recipe, so those bytes cannot go directly into the final build VM. If stock-SDK `COPY` support remains in the initial scope, I propose pinning the registered build session to a node and staging uploads in a bounded, build-scoped local spool on that node. At build-start, the same node creates the VM and streams the staged files through envd. The spool would be removed when the build terminates or expires. Initially, no cross-build context deduplication would be promised (`present` may be `false` for each new build). If the pinned node is lost before consumption, the build fails and a retry re-uploads rather than transparently resuming elsewhere. This requires a small gateway routing extension, but no durable context store or new correctness-grade external coordinator. Alternatively, stock-SDK `COPY` compatibility can follow the native workflow as a separate milestone. Do you prefer it in the initial milestone or as a follow-up? If this matches your intent, I'll update and split #141 accordingly, rework or close #71/#72 depending on (2), reuse the applicable host-side COPY planning and envd streaming pieces from #73/#74, and only then begin implementation.

bsakaguchi commented 2026-10-29 14:18:51 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Aug 9, 2026):

1. It's fine to start with single-stage Docker support.
2. I'd prefer that our initial milestone not include COPY compatibility. In general, I'd rather avoid uploading files directly to the host OS, as it may introduce security risks. Uploading into a VM should be much safer.

<!-- gh-comment-id:5229950113 --> @yingdi-shan commented on GitHub (Aug 9, 2026): 1. It's fine to start with single-stage Docker support. 2. I'd prefer that our initial milestone not include COPY compatibility. In general, I'd rather avoid uploading files directly to the host OS, as it may introduce security risks. Uploading into a VM should be much safer.

bsakaguchi commented 2026-10-29 14:18:51 +00:00

Author

Owner

Copy link

@JoyboyBrian commented on GitHub (Aug 9, 2026):

Thanks, understood. Recording the decision:

• The initial milestone is the native single-stage Dockerfile workflow orchestrated by aenv. COPY/ADD inputs will be streamed directly into the build VM through envd; the design will not persist build-context bytes on the host filesystem.
• Stock E2B SDK COPY compatibility and the host-side spool proposal are out of scope for this milestone. Any future compatibility design will preserve the VM isolation boundary for untrusted context handling.
• Multi-stage support and per-step execution caching remain separate follow-ups.

I'll now create a smaller focused design issue for the initial milestone and mark #141 as superseded. #71/#72 will be closed as superseded; I'll audit #73/#74 and carry the reusable COPY planning, envd streaming, and in-guest extraction pieces into the native path. Implementation will start from main after the focused design is posted.

<!-- gh-comment-id:5233889664 --> @JoyboyBrian commented on GitHub (Aug 9, 2026): Thanks, understood. Recording the decision: - The initial milestone is the native single-stage Dockerfile workflow orchestrated by `aenv`. `COPY`/`ADD` inputs will be streamed directly into the build VM through envd; the design will not persist build-context bytes on the host filesystem. - Stock E2B SDK `COPY` compatibility and the host-side spool proposal are out of scope for this milestone. Any future compatibility design will preserve the VM isolation boundary for untrusted context handling. - Multi-stage support and per-step execution caching remain separate follow-ups. I'll now create a smaller focused design issue for the initial milestone and mark #141 as superseded. #71/#72 will be closed as superseded; I'll audit #73/#74 and carry the reusable COPY planning, envd streaming, and in-guest extraction pieces into the native path. Implementation will start from `main` after the focused design is posted.

bsakaguchi referenced this issue 2026-10-29 14:20:07 +00:00

[PR #51] [CLOSED] feat(scheduler): add snapshot artifact locality preference #141

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#51

No description provided.