[GH-ISSUE #441] Restore a sandbox from a snapshot with its original sandbox ID #106

New issue

Open

opened 2026-10-29 14:19:40 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:19:40 +00:00

Owner

Originally created by @aniket-sarvam on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/441

Affected component

Orchestrator / sandbox lifecycle

Problem statement

As an operator running AgentENV as a multi-node sandbox platform,
I need a sandbox restored from a snapshot on another node to keep its original sandbox ID,
because every create, including a create from a snapshot, gets a new ID, so a sandbox cannot be moved between nodes and still look like the same sandbox to clients.

In E2B, a sandbox keeps its ID across pause and resume, and clients build data-plane addresses from it ({port}-{sandboxID}.{domain}, the E2b-Sandbox-Id header). In AgentENV, pause and resume keep the ID but are node-local (the paused state lives under persisted_sandbox_store_path on that node). The only way to resume a sandbox on a different node, or after its node is gone, is to snapshot it and create a new sandbox from the snapshot, which assigns a new ID.

Use case

We run AgentENV on Kubernetes with several runtime nodes and a shared S3 snapshot repository. Agent sessions sit idle for hours, so we free their compute by snapshotting the sandbox and deleting it ("hibernate"), and restore it from the snapshot when the session continues. Nodes are added and removed by an autoscaler, so the restore usually lands on a different node from the original.

This happens for most sessions, many times a day per session, across hundreds of concurrent sandboxes.

Current behavior and workarounds

OrchestratorService::create_sandbox always generates the ID:

github.com/kvcache-ai/AgentENV@abc027b/src/orchestrator/service.rs (L425-L435)

so a sandbox restored from a snapshot comes back with a new ID. Consequences for clients:

E2B SDK objects that hold the old ID (for example sbx.connect() on an existing Sandbox) stop working.
Clients that stored the ID, or built {port}-{sandboxID}.{domain} URLs from it, stop working.
The envd and traffic access tokens are derived from the sandbox ID (SandboxAccessTokenGenerator::generate / generate_traffic, github.com/kvcache-ai/AgentENV@abc027b/src/sandbox/access.rs (L71-L77)), so tokens that clients already hold become invalid too.

Workarounds, and why they fall short:

Always reconnect with the class method Sandbox.connect(id), which adopts the ID and tokens the server returns, and keep an ID mapping outside AgentENV. Breaks E2B semantics for any client that holds a Sandbox object or a stored ID.
Put a proxy in front of the gateway that rewrites IDs and access tokens on every data-plane request. Adds a hop to all command, file and streaming traffic, and duplicates routing the gateway already does.
Use pause and resume. Keeps the ID, but the paused state is lost if the node is lost or scaled away.

Desired behavior

Given a snapshot of sandbox X, and sandbox X no longer running anywhere,
When a privileged caller creates a sandbox from that snapshot and asks for ID X,
Then the new sandbox has ID X, the gateway routes X to its new node, and its envd and traffic access tokens are the same as before (given the same access_token_hash_seed across nodes).
If X is still live on the target node, the request fails with 409 Conflict.
Requests that don't ask for an ID behave exactly as today.

Proposed approach

The orchestrator already supports creating with a caller-chosen ID: create_sandbox_inner(sandbox_id, request, template_builder) takes the ID as a parameter, and create_template_builder passes its own (github.com/kvcache-ai/AgentENV@abc027b/src/orchestrator/service.rs (L437-L456)). The change would be:

Add an optional sandboxID to the create request (NewSandbox / /v2/sandboxes), accepted only from an admin-scoped caller and only when creating from a snapshot.
Plumb it through CreateSandboxRequest into create_sandbox_inner, keeping SandboxId::new() as the default.
Reject the request with 409 if the ID is live on the node, using the orchestrator's existing store.
The gateway already records the assignment from the create response (RecordAssignment), and node heartbeats refresh bindings, so the binding moves to the new node without scheduler changes. Worth a test that a stale binding on the old node cannot win.

Cluster-wide uniqueness (that X is not live on another node) would stay the caller's responsibility, which suits a control plane that already serialises lifecycle operations per sandbox. A gateway-side check through LookupNode could be added if you prefer it enforced inside AgentENV.

Compatibility and operational impact

API/config changes: one optional request field, admin-only. No change for existing callers.

Snapshot or storage format changes: none.

New host/runtime requirements: none.

Upgrade and rollback considerations: older servers don't know the field, so clients should only send it to servers that support it. Rolling back removes the ability, nothing else.

Alternatives considered

Durable, cross-node pause (#183, "distributed" snapshot type). Would also solve this, but is a much larger change; this request is a small step that works with today's snapshot repository.
VM migration (#11). Moves running VMs; this request only covers restoring a stopped sandbox.
The proxy and client-side workarounds listed above.

Non-goals

Live migration of running sandboxes.
Choosing arbitrary IDs for brand-new sandboxes (not created from a snapshot).
Changing how pause and resume work.

Contribution

I can implement it

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @aniket-sarvam on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/441 ### Affected component Orchestrator / sandbox lifecycle ### Problem statement As an operator running AgentENV as a multi-node sandbox platform, I need a sandbox restored from a snapshot on another node to keep its original sandbox ID, because every create, including a create from a snapshot, gets a new ID, so a sandbox cannot be moved between nodes and still look like the same sandbox to clients. In E2B, a sandbox keeps its ID across pause and resume, and clients build data-plane addresses from it (`{port}-{sandboxID}.{domain}`, the `E2b-Sandbox-Id` header). In AgentENV, pause and resume keep the ID but are node-local (the paused state lives under `persisted_sandbox_store_path` on that node). The only way to resume a sandbox on a different node, or after its node is gone, is to snapshot it and create a new sandbox from the snapshot, which assigns a new ID. ### Use case We run AgentENV on Kubernetes with several runtime nodes and a shared S3 snapshot repository. Agent sessions sit idle for hours, so we free their compute by snapshotting the sandbox and deleting it ("hibernate"), and restore it from the snapshot when the session continues. Nodes are added and removed by an autoscaler, so the restore usually lands on a different node from the original. This happens for most sessions, many times a day per session, across hundreds of concurrent sandboxes. ### Current behavior and workarounds `OrchestratorService::create_sandbox` always generates the ID: https://github.com/kvcache-ai/AgentENV/blob/abc027b/src/orchestrator/service.rs#L425-L435 so a sandbox restored from a snapshot comes back with a new ID. Consequences for clients: - E2B SDK objects that hold the old ID (for example `sbx.connect()` on an existing `Sandbox`) stop working. - Clients that stored the ID, or built `{port}-{sandboxID}.{domain}` URLs from it, stop working. - The envd and traffic access tokens are derived from the sandbox ID (`SandboxAccessTokenGenerator::generate` / `generate_traffic`, https://github.com/kvcache-ai/AgentENV/blob/abc027b/src/sandbox/access.rs#L71-L77), so tokens that clients already hold become invalid too. Workarounds, and why they fall short: 1. Always reconnect with the class method `Sandbox.connect(id)`, which adopts the ID and tokens the server returns, and keep an ID mapping outside AgentENV. Breaks E2B semantics for any client that holds a `Sandbox` object or a stored ID. 2. Put a proxy in front of the gateway that rewrites IDs and access tokens on every data-plane request. Adds a hop to all command, file and streaming traffic, and duplicates routing the gateway already does. 3. Use pause and resume. Keeps the ID, but the paused state is lost if the node is lost or scaled away. ### Desired behavior - Given a snapshot of sandbox `X`, and sandbox `X` no longer running anywhere, - When a privileged caller creates a sandbox from that snapshot and asks for ID `X`, - Then the new sandbox has ID `X`, the gateway routes `X` to its new node, and its envd and traffic access tokens are the same as before (given the same `access_token_hash_seed` across nodes). - If `X` is still live on the target node, the request fails with `409 Conflict`. - Requests that don't ask for an ID behave exactly as today. ### Proposed approach The orchestrator already supports creating with a caller-chosen ID: `create_sandbox_inner(sandbox_id, request, template_builder)` takes the ID as a parameter, and `create_template_builder` passes its own (https://github.com/kvcache-ai/AgentENV/blob/abc027b/src/orchestrator/service.rs#L437-L456). The change would be: 1. Add an optional `sandboxID` to the create request (`NewSandbox` / `/v2/sandboxes`), accepted only from an admin-scoped caller and only when creating from a snapshot. 2. Plumb it through `CreateSandboxRequest` into `create_sandbox_inner`, keeping `SandboxId::new()` as the default. 3. Reject the request with `409` if the ID is live on the node, using the orchestrator's existing store. 4. The gateway already records the assignment from the create response (`RecordAssignment`), and node heartbeats refresh bindings, so the binding moves to the new node without scheduler changes. Worth a test that a stale binding on the old node cannot win. Cluster-wide uniqueness (that `X` is not live on another node) would stay the caller's responsibility, which suits a control plane that already serialises lifecycle operations per sandbox. A gateway-side check through `LookupNode` could be added if you prefer it enforced inside AgentENV. ### Compatibility and operational impact API/config changes: one optional request field, admin-only. No change for existing callers. Snapshot or storage format changes: none. New host/runtime requirements: none. Upgrade and rollback considerations: older servers don't know the field, so clients should only send it to servers that support it. Rolling back removes the ability, nothing else. ### Alternatives considered - Durable, cross-node pause (#183, "distributed" snapshot type). Would also solve this, but is a much larger change; this request is a small step that works with today's snapshot repository. - VM migration (#11). Moves running VMs; this request only covers restoring a stopped sandbox. - The proxy and client-side workarounds listed above. ### Non-goals - Live migration of running sandboxes. - Choosing arbitrary IDs for brand-new sandboxes (not created from a snapshot). - Changing how pause and resume work. ### Contribution I can implement it ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi commented

2026-10-29 14:19:41 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Oct 28, 2026):

Thanks, this is a valid need, but we want to be careful with the API design. Creating a sandbox from a snapshot naturally gives it a new sandbox ID, because the original sandbox keeps running after the snapshot is taken. If a restored sandbox could reuse the source ID, nothing would stop the source, or a second restore of the same snapshot, from being live under that ID at the same time.

We think the underlying problem is a different one: how to remove a node and move its paused and running sandboxes to other nodes while keeping their sandbox IDs. Pause/resume already keeps the ID. The gap is that paused state is stored only on the node that paused it. So we'd prefer to solve this by migrating paused sandboxes, as part of VM migration (#11): the source node hands the sandbox off, and the target node resumes it under the same ID. Your hibernate workflow would then become pause → migrate → resume.

Would this solve your problem? Full VM migration for running sandboxes may take a while, but migrating paused sandboxes should be a much smaller effort. That said, v0.3 includes a significant refactor of pause, so we plan to implement this after the v0.3 release.

@yingdi-shan commented on GitHub (Oct 28, 2026): Thanks, this is a valid need, but we want to be careful with the API design. Creating a sandbox from a snapshot naturally gives it a new sandbox ID, because the original sandbox keeps running after the snapshot is taken. If a restored sandbox could reuse the source ID, nothing would stop the source, or a second restore of the same snapshot, from being live under that ID at the same time. We think the underlying problem is a different one: **how to remove a node and move its paused and running sandboxes to other nodes while keeping their sandbox IDs.** Pause/resume already keeps the ID. The gap is that paused state is stored only on the node that paused it. So we'd prefer to solve this by migrating paused sandboxes, as part of VM migration (#11): the source node hands the sandbox off, and the target node resumes it under the same ID. Your hibernate workflow would then become pause → migrate → resume. Would this solve your problem? Full VM migration for running sandboxes may take a while, but migrating paused sandboxes should be a much smaller effort. That said, v0.3 includes a significant refactor of pause, so we plan to implement this after the v0.3 release.

bsakaguchi referenced this issue

2026-10-29 14:20:43 +00:00

[PR #106] [MERGED] docs: remove control plane prototype label #182