kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #31] Support attached drives when creating a sandbox from a template #14

New issue

Closed

opened 2026-10-29 14:18:21 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:18:21 +00:00

Owner

Originally created by @jiaoqsh on GitHub (Jul 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/31

Summary

attachedDrives is currently accepted only by POST /sandboxes-cold. Once a drive is
attached during a cold start, its driveID, mountPath, subPath and readOnly are
frozen into the snapshot manifest, and every sandbox later created from that snapshot
inherits them verbatim — there is no way to add, remove, or reconfigure a drive at launch
time.

In effect, "attach a drive" and "start fast from a snapshot" are mutually exclusive at the
API level. I'd like to propose adding attachedDrives to NewSandbox (POST /sandboxes),
and I'm happy to implement it — but the merge semantics need to be agreed first, which is
why this is an issue rather than a PR.

Current behavior

Endpoint	Can specify `attachedDrives`	Start latency
`POST /sandboxes-cold`	yes	"may pull and convert OCI layers on a cache miss and can take tens of seconds" (per its own API description)
`POST /sandboxes`	no	fast snapshot resume

Relevant code:

NewColdSandbox.attachedDrives — src/api/openapi.yml:605; NewSandbox has no
equivalent field — src/api/openapi.yml:510-548
SandboxLaunchSource::Image { .., extra_drives, .. } carries drives, while
SandboxLaunchSource::Snapshot(Box<RunnableSnapshot>) has nowhere to put them —
src/orchestrator/types.rs:13-25
Mount configuration is persisted into the snapshot manifest —
src/sandbox/firecracker/manifest.rs:52-64 (FirecrackerAttachedDriveArtifacts stores
drive_id, read_only, mount_path, sub_path, virtual_size)
On resume it is read back verbatim with no override path —
src/snapshot/types/snapshot.rs:472 and FirecrackerSnapshotManifest::extra_drives()

The runtime is already capable of this: snapshot-based launches prepare inherited drives
through the same prepare_extra_drives path as cold starts (via
ExtraDrivePrepareMode::Resume), and read-only and writable drives already share a single
device lifecycle (docs/src/internals/architecture.md:124). What is missing is the API
entry point, not the machinery.

Why this is limiting

Any workload where the environment and the data vary independently runs into this. With
N template environments and M versions of a shared dataset or config, you currently need
N×M templates, because the only way to get a drive onto a fast-starting sandbox is to bake
it in at cold-start time and snapshot it.

Updating a shared read-only dataset likewise forces a template rebuild, even though the
drive content is just an OCI image reference that could be selected at launch.

Prior art

Both Kubernetes and E2B separate what the data is from how it is mounted — exactly the
separation AgentENV currently collapses.

Kubernetes declares the volume source in spec.volumes and its mount configuration
per-container in containers[].volumeMounts (mountPath, subPath, readOnly). The same
volume can be mounted at different paths, with different readOnly settings, in different
containers. Kubernetes also has an image volume source that mounts an OCI image reference
as a read-only volume — essentially the same primitive as AttachedDriveSource.image, but
with the mount configuration supplied at pod creation rather than frozen into the artifact:

spec:
  containers:
    - name: app
      volumeMounts:
        - name: shared-data
          mountPath: /mnt/data
          subPath: v3
          readOnly: true
  volumes:
    - name: shared-data
      image:
        reference: ghcr.io/example/dataset:v3

E2B mounts volumes at sandbox creation time, so the mount path is a property of the
creation call rather than of the volume:

const sandbox = await Sandbox.create({
  volumeMounts: { '/mnt/my-data': volume },
})

In AgentENV today, both the source and the mount configuration are fixed at cold-start time
and inherited immutably thereafter.

Proposal (sketch)

Add an optional attachedDrives field to NewSandbox.
Give SandboxLaunchSource::Snapshot a place to carry launch-provided drives and thread
it through LaunchPlan::for_create_from_snapshot.
Merge launch-provided drives with RunnableSnapshot::attached_drives() under agreed
rules (see open questions).
Enforce MAX_EXTRA_DRIVES (24, src/sandbox/firecracker/config.rs:52) against the
merged set.
Document the feature: attachedDrives is currently not mentioned in
docs/src/concepts/sandboxes.md at all.

Open questions

Was the cold-start-only restriction deliberate? I suspect it follows from the
latency contract (see below) rather than being an intentional design constraint, but I'd
like to confirm before implementing.
Merge conflict rules. When a launch-provided drive collides with a snapshot-inherited
one by driveID or by mountPath — reject with 400, or allow override? My inclination
is to always reject on mountPath collision, and to allow driveID override only if we
decide to support question 3.
Should launch-time overrides of inherited drives be supported? That is, re-mounting an
inherited drive at a different mountPath / subPath / readOnly, the way Kubernetes
lets each container mount a shared volume differently. This is arguably the more valuable
half of the feature, but it is also the larger API commitment.
Should launch-provided read-only drives be captured into snapshots taken from that
sandbox? Current behavior captures all attached drive state. For a shared read-only
reference drive, that makes every downstream snapshot carry a dependency on that image.
Excluding read-only launch-provided drives from capture may be preferable, but it makes
snapshot content depend on how the sandbox was launched.
diskSizeMB semantics on this path. docs/src/concepts/sandboxes.md states that
resizing applies only when creating a fresh writable root filesystem — not to read-only
images, images with an existing upper, or snapshot resume. A drive newly attached on a
snapshot-based launch is none of those exactly. Reject diskSizeMB here, or honor it?
Device naming stability. Guest device letters are assigned by attachment order
(vd(c+i), src/sandbox/firecracker/sandbox.rs:971), not by driveID. Appending
launch-provided drives after inherited ones keeps existing devices stable, but any future
support for removing an inherited drive would shift every subsequent letter. Worth
fixing the ordering rule explicitly as part of this change.

Note on start latency

The most likely objection: resolve_attached_drives calls ImageResolver::resolve
(src/api/impls/attached_drives.rs:95), which on a cache miss performs a full OCI pull and
conversion. That is presumably why the field lives on /sandboxes-cold, whose description
already warns about tens-of-seconds latency. Adding it to POST /sandboxes as-is would
import unbounded latency into the fast path.

Two mitigations, which I'd propose combining:

Accept only images that resolve without a pull — already-cached images, and
overlaybd-native images (which skip blob download entirely and are read on demand through
registryfs_v2). Reject anything else with a 400 pointing at /sandboxes-cold or at a
pre-warm step.
Optionally allow the slow path behind an explicit opt-in field, for clients that accept
the latency.

This keeps the default latency characteristics of POST /sandboxes unchanged.

#9 wrestles with the same tension from the other direction (a synchronous timeout versus
202 Accepted with polling for large images), so whatever convention is settled on there
would be good to follow here.

#9 — Support direct OCI image in sandbox creation API. Same underlying theme: a
capability that today exists only on the cold path should be reachable from the regular
creation API. That issue covers the rootfs; this one covers attached drives. They are
independent but should probably agree on how OCI resolution latency is surfaced.
#5 — E2B API Compatibility Matrix. Lists E2B's volumeMounts on POST /sandboxes
and the /volumes CRUD endpoints as not implemented. This issue is deliberately not
a proposal to implement E2B Volumes — attached drives are OCI-image-backed and have no
independent lifecycle — but making mount configuration a property of the creation call
is a prerequisite for any future volume-style feature, and would close part of the
volumeMounts gap tracked there.

Willing to implement

Happy to send a PR once the semantics above are settled. tests/integration/snapshot_attached_drive.rs
looks like the natural place to extend coverage.

Originally created by @jiaoqsh on GitHub (Jul 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/31 ## Summary `attachedDrives` is currently accepted only by `POST /sandboxes-cold`. Once a drive is attached during a cold start, its `driveID`, `mountPath`, `subPath` and `readOnly` are frozen into the snapshot manifest, and every sandbox later created from that snapshot inherits them verbatim — there is no way to add, remove, or reconfigure a drive at launch time. In effect, "attach a drive" and "start fast from a snapshot" are mutually exclusive at the API level. I'd like to propose adding `attachedDrives` to `NewSandbox` (`POST /sandboxes`), and I'm happy to implement it — but the merge semantics need to be agreed first, which is why this is an issue rather than a PR. ## Current behavior | Endpoint | Can specify `attachedDrives` | Start latency | |---|---|---| | `POST /sandboxes-cold` | yes | "may pull and convert OCI layers on a cache miss and can take tens of seconds" (per its own API description) | | `POST /sandboxes` | **no** | fast snapshot resume | Relevant code: - `NewColdSandbox.attachedDrives` — `src/api/openapi.yml:605`; `NewSandbox` has no equivalent field — `src/api/openapi.yml:510-548` - `SandboxLaunchSource::Image { .., extra_drives, .. }` carries drives, while `SandboxLaunchSource::Snapshot(Box<RunnableSnapshot>)` has nowhere to put them — `src/orchestrator/types.rs:13-25` - Mount configuration is persisted into the snapshot manifest — `src/sandbox/firecracker/manifest.rs:52-64` (`FirecrackerAttachedDriveArtifacts` stores `drive_id`, `read_only`, `mount_path`, `sub_path`, `virtual_size`) - On resume it is read back verbatim with no override path — `src/snapshot/types/snapshot.rs:472` and `FirecrackerSnapshotManifest::extra_drives()` The runtime is already capable of this: snapshot-based launches prepare inherited drives through the same `prepare_extra_drives` path as cold starts (via `ExtraDrivePrepareMode::Resume`), and read-only and writable drives already share a single device lifecycle (`docs/src/internals/architecture.md:124`). What is missing is the API entry point, not the machinery. ## Why this is limiting Any workload where the environment and the data vary independently runs into this. With N template environments and M versions of a shared dataset or config, you currently need N×M templates, because the only way to get a drive onto a fast-starting sandbox is to bake it in at cold-start time and snapshot it. Updating a shared read-only dataset likewise forces a template rebuild, even though the drive content is just an OCI image reference that could be selected at launch. ## Prior art Both Kubernetes and E2B separate *what the data is* from *how it is mounted* — exactly the separation AgentENV currently collapses. **Kubernetes** declares the volume source in `spec.volumes` and its mount configuration per-container in `containers[].volumeMounts` (`mountPath`, `subPath`, `readOnly`). The same volume can be mounted at different paths, with different `readOnly` settings, in different containers. Kubernetes also has an `image` volume source that mounts an OCI image reference as a read-only volume — essentially the same primitive as `AttachedDriveSource.image`, but with the mount configuration supplied at pod creation rather than frozen into the artifact: ```yaml spec: containers: - name: app volumeMounts: - name: shared-data mountPath: /mnt/data subPath: v3 readOnly: true volumes: - name: shared-data image: reference: ghcr.io/example/dataset:v3 ``` **E2B** mounts volumes at sandbox creation time, so the mount path is a property of the creation call rather than of the volume: ```js const sandbox = await Sandbox.create({ volumeMounts: { '/mnt/my-data': volume }, }) ``` In AgentENV today, both the source and the mount configuration are fixed at cold-start time and inherited immutably thereafter. ## Proposal (sketch) 1. Add an optional `attachedDrives` field to `NewSandbox`. 2. Give `SandboxLaunchSource::Snapshot` a place to carry launch-provided drives and thread it through `LaunchPlan::for_create_from_snapshot`. 3. Merge launch-provided drives with `RunnableSnapshot::attached_drives()` under agreed rules (see open questions). 4. Enforce `MAX_EXTRA_DRIVES` (24, `src/sandbox/firecracker/config.rs:52`) against the merged set. 5. Document the feature: `attachedDrives` is currently not mentioned in `docs/src/concepts/sandboxes.md` at all. ## Open questions 1. **Was the cold-start-only restriction deliberate?** I suspect it follows from the latency contract (see below) rather than being an intentional design constraint, but I'd like to confirm before implementing. 2. **Merge conflict rules.** When a launch-provided drive collides with a snapshot-inherited one by `driveID` or by `mountPath` — reject with 400, or allow override? My inclination is to always reject on `mountPath` collision, and to allow `driveID` override only if we decide to support question 3. 3. **Should launch-time overrides of inherited drives be supported?** That is, re-mounting an inherited drive at a different `mountPath` / `subPath` / `readOnly`, the way Kubernetes lets each container mount a shared volume differently. This is arguably the more valuable half of the feature, but it is also the larger API commitment. 4. **Should launch-provided read-only drives be captured into snapshots taken from that sandbox?** Current behavior captures all attached drive state. For a shared read-only reference drive, that makes every downstream snapshot carry a dependency on that image. Excluding read-only launch-provided drives from capture may be preferable, but it makes snapshot content depend on how the sandbox was launched. 5. **`diskSizeMB` semantics on this path.** `docs/src/concepts/sandboxes.md` states that resizing applies only when creating a fresh writable root filesystem — not to read-only images, images with an existing upper, or snapshot resume. A drive newly attached on a snapshot-based launch is none of those exactly. Reject `diskSizeMB` here, or honor it? 6. **Device naming stability.** Guest device letters are assigned by attachment order (`vd(c+i)`, `src/sandbox/firecracker/sandbox.rs:971`), not by `driveID`. Appending launch-provided drives after inherited ones keeps existing devices stable, but any future support for *removing* an inherited drive would shift every subsequent letter. Worth fixing the ordering rule explicitly as part of this change. ## Note on start latency The most likely objection: `resolve_attached_drives` calls `ImageResolver::resolve` (`src/api/impls/attached_drives.rs:95`), which on a cache miss performs a full OCI pull and conversion. That is presumably why the field lives on `/sandboxes-cold`, whose description already warns about tens-of-seconds latency. Adding it to `POST /sandboxes` as-is would import unbounded latency into the fast path. Two mitigations, which I'd propose combining: - Accept only images that resolve without a pull — already-cached images, and overlaybd-native images (which skip blob download entirely and are read on demand through `registryfs_v2`). Reject anything else with a 400 pointing at `/sandboxes-cold` or at a pre-warm step. - Optionally allow the slow path behind an explicit opt-in field, for clients that accept the latency. This keeps the default latency characteristics of `POST /sandboxes` unchanged. #9 wrestles with the same tension from the other direction (a synchronous timeout versus `202 Accepted` with polling for large images), so whatever convention is settled on there would be good to follow here. ## Related issues - **#9 — Support direct OCI image in sandbox creation API.** Same underlying theme: a capability that today exists only on the cold path should be reachable from the regular creation API. That issue covers the rootfs; this one covers attached drives. They are independent but should probably agree on how OCI resolution latency is surfaced. - **#5 — E2B API Compatibility Matrix.** Lists E2B's `volumeMounts` on `POST /sandboxes` and the `/volumes` CRUD endpoints as not implemented. This issue is deliberately *not* a proposal to implement E2B Volumes — attached drives are OCI-image-backed and have no independent lifecycle — but making mount configuration a property of the creation call is a prerequisite for any future volume-style feature, and would close part of the `volumeMounts` gap tracked there. ## Willing to implement Happy to send a PR once the semantics above are settled. `tests/integration/snapshot_attached_drive.rs` looks like the natural place to extend coverage.

bsakaguchi closed this issue

2026-10-29 14:18:21 +00:00

bsakaguchi commented

2026-10-29 14:18:22 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Jul 28, 2026):

Thanks for putting this together. I agree with the underlying goal: users should be able to select and configure attached drives at launch time.

I’m hesitant to add this directly to POST /sandboxes. That endpoint is intended to remain a minimal, predictable fast path for restoring an existing snapshot. Supporting launch-time drives there could make its behavior and latency less predictable, particularly when a snapshot already contains attached drives.

POST /sandboxes-cold, on the other hand, always starts from a fresh OCI image and is expected to be slower.

Related to the direction discussed in #9, I’m considering a separate image-oriented endpoint, tentatively /from-images. It could dynamically choose whether to use a compatible snapshot for a warm start or fall back to a cold start. It would also be a better place for richer launch-time operations such as mounting drives, hot-plugging memory, and resizing disks, allowing more efficient use of automatically created templates.

This would keep the contract of POST /sandboxes simple while providing a flexible interface for both the rootfs use case in #9 and the attached-drive use case described here.

@yingdi-shan commented on GitHub (Jul 28, 2026): Thanks for putting this together. I agree with the underlying goal: users should be able to select and configure attached drives at launch time. I’m hesitant to add this directly to `POST /sandboxes`. That endpoint is intended to remain a minimal, predictable fast path for restoring an existing snapshot. Supporting launch-time drives there could make its behavior and latency less predictable, particularly when a snapshot already contains attached drives. `POST /sandboxes-cold`, on the other hand, always starts from a fresh OCI image and is expected to be slower. Related to the direction discussed in #9, I’m considering a separate image-oriented endpoint, tentatively `/from-images`. It could dynamically choose whether to use a compatible snapshot for a warm start or fall back to a cold start. It would also be a better place for richer launch-time operations such as mounting drives, hot-plugging memory, and resizing disks, allowing more efficient use of automatically created templates. This would keep the contract of `POST /sandboxes` simple while providing a flexible interface for both the rootfs use case in #9 and the attached-drive use case described here.

bsakaguchi commented

2026-10-29 14:18:22 +00:00

Author

Owner

@jiaoqsh commented on GitHub (Jul 28, 2026):

That makes sense — keeping POST /sandboxes as a predictable snapshot-restore path is a
better contract than what I proposed, and it removes the need for the cache-only
restriction I suggested, since /from-images wouldn't be promising a fast path in the
first place.

Two things I'd want to pin down about /from-images, both about whether it fully covers
the case in this issue:

1. Does the drive set participate in snapshot compatibility?

If the auto-template key in #9 ({IMAGE_DIGEST}-{CPU}c-{MEM}mb-{DISK}gb) is extended to
include attached drives, then every new (rootfs image, dataset version) combination
requires a cold start to materialize its template — which is the cost this issue is trying
to avoid, just relocated.

If instead drives are attached at launch onto a rootfs-only snapshot, the endpoint stays
clean and the template namespace stays small — but the runtime still has to attach drives
on a snapshot-backed launch, so the merge and device-ordering questions above still need
answers; they'd just live behind /from-images instead of POST /sandboxes.

I'd advocate for the second. Worth deciding early, since it determines whether the
underlying work is "resolve drives and attach on resume" or "bake drives into the template
key".

2. What about templates that aren't derived from a single image?

/from-images is image-oriented, but templates produced by the template builder
(RUN / ENV / WORKDIR steps) don't correspond to one OCI image reference. If a user has such
a template and wants to mount a dataset drive into it, is that in scope for /from-images,
or would it need something else? This is also where the "re-mount an inherited drive at a
different path" case from open question 3 would live, if that's wanted at all.

@jiaoqsh commented on GitHub (Jul 28, 2026): That makes sense — keeping `POST /sandboxes` as a predictable snapshot-restore path is a better contract than what I proposed, and it removes the need for the cache-only restriction I suggested, since `/from-images` wouldn't be promising a fast path in the first place. Two things I'd want to pin down about `/from-images`, both about whether it fully covers the case in this issue: **1. Does the drive set participate in snapshot compatibility?** If the auto-template key in #9 (`{IMAGE_DIGEST}-{CPU}c-{MEM}mb-{DISK}gb`) is extended to include attached drives, then every new (rootfs image, dataset version) combination requires a cold start to materialize its template — which is the cost this issue is trying to avoid, just relocated. If instead drives are attached at launch onto a rootfs-only snapshot, the endpoint stays clean and the template namespace stays small — but the runtime still has to attach drives on a snapshot-backed launch, so the merge and device-ordering questions above still need answers; they'd just live behind `/from-images` instead of `POST /sandboxes`. I'd advocate for the second. Worth deciding early, since it determines whether the underlying work is "resolve drives and attach on resume" or "bake drives into the template key". **2. What about templates that aren't derived from a single image?** `/from-images` is image-oriented, but templates produced by the template builder (RUN / ENV / WORKDIR steps) don't correspond to one OCI image reference. If a user has such a template and wants to mount a dataset drive into it, is that in scope for `/from-images`, or would it need something else? This is also where the "re-mount an inherited drive at a different path" case from open question 3 would live, if that's wanted at all.

bsakaguchi commented

2026-10-29 14:18:23 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Jul 28, 2026):

We should avoid including attached drives in the template key, as this could cause a combinatorial explosion of templates. Instead, we should keep template key rootfs-only, attach drives via Firecracker's PATCH /drives API, and mount them by running a command inside the sandbox.
Templates not derived from a single image are also a valid use case. To support this, we can build a general-purpose interface that combines PATCH /drives with drive mounting, then use that interface to implement the /from-images behavior described above.

@yingdi-shan commented on GitHub (Jul 28, 2026): 1. We should avoid including attached drives in the template key, as this could cause a combinatorial explosion of templates. Instead, we should keep template key rootfs-only, attach drives via Firecracker's PATCH /drives API, and mount them by running a command inside the sandbox. 2. Templates not derived from a single image are also a valid use case. To support this, we can build a general-purpose interface that combines PATCH /drives with drive mounting, then use that interface to implement the /from-images behavior described above.

bsakaguchi commented

2026-10-29 14:18:23 +00:00

Author

Owner

@jiaoqsh commented on GitHub (Jul 29, 2026):

Sounds good. Two points that might be worth noting:

1. PATCH /drives cannot change read-only-ness. The vendored Firecracker 1.15.1 spec
(thirdparty/firecracker-client/firecracker.yaml) defines PartialDrive as
{drive_id, path_on_host?, rate_limiter?} — there is no is_read_only. So whichever step
creates a drive fixes whether it is read-only, and a later PATCH can only re-point
path_on_host. Noting it since it affects which of the current AttachedDrive fields can
vary per launch.

2. Mounting currently lives in the guest init, and subPath with it. Today
tools-image/init parses the agentenv_drives=vd<letter>:<mountPath>[:<subPath>] boot arg
and mounts before pivot_root. For subPath it stages a mount, bind-mounts the requested
subdirectory onto the guest mount path, then lazy-detaches the staging mount. If mounting
moves to a command run inside the sandbox, that subPath behavior would need to be handled
somewhere else, and mount failures would surface after the sandbox is already running rather
than during boot.

Looking forward to seeing where this lands.

@jiaoqsh commented on GitHub (Jul 29, 2026): Sounds good. Two points that might be worth noting: **1. `PATCH /drives` cannot change read-only-ness.** The vendored Firecracker 1.15.1 spec (`thirdparty/firecracker-client/firecracker.yaml`) defines `PartialDrive` as `{drive_id, path_on_host?, rate_limiter?}` — there is no `is_read_only`. So whichever step creates a drive fixes whether it is read-only, and a later `PATCH` can only re-point `path_on_host`. Noting it since it affects which of the current `AttachedDrive` fields can vary per launch. **2. Mounting currently lives in the guest init, and `subPath` with it.** Today `tools-image/init` parses the `agentenv_drives=vd<letter>:<mountPath>[:<subPath>]` boot arg and mounts before `pivot_root`. For `subPath` it stages a mount, bind-mounts the requested subdirectory onto the guest mount path, then lazy-detaches the staging mount. If mounting moves to a command run inside the sandbox, that `subPath` behavior would need to be handled somewhere else, and mount failures would surface after the sandbox is already running rather than during boot. Looking forward to seeing where this lands.

bsakaguchi commented

2026-10-29 14:18:23 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Sep 1, 2026):

Resolved in PR #214.

@yingdi-shan commented on GitHub (Sep 1, 2026): Resolved in PR #214.

bsakaguchi referenced this issue

2026-10-29 14:19:55 +00:00

[PR #14] [MERGED] Expose host interaction IP in custom extension API #119