[GH-ISSUE #11] Add support for VM migration #8

New issue

Open

opened 2026-10-29 14:18:18 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:18 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/11

Summary

Add the ability to move a sandbox from one AgentENV node to another. The core insight is that we already have fast snapshot and fast resume, so migration is mostly orchestration on top of existing primitives: capture (or reuse) a self-contained snapshot on the source node, make its artifacts reachable from the target node (P2P-first), resume on the target preserving sandbox identity, then tear down the source and hand off the scheduler binding so the gateway routes traffic to the new node.

Two cases, with different cost profiles:

Running VM migration: snapshot on source, resume on target. Reuses capture_snapshot plus create-from-snapshot.
Paused VM migration: the VM is already paused and its content sits node-local. We only need to publish that paused content into P2P so the target can pull and resume. No re-snapshot required.

Motivation

Migration unlocks node draining for maintenance, load rebalancing, and bin packing. Paused sandboxes in particular are cheap to move because their state is already materialized, so making them mobile is high value for low cost.

Proposed design

Introduce a migration flow (orchestrator + scheduler RPC) parameterized by source node, target node, and sandbox ID. Preserve the SandboxId end-to-end so the gateway keeps routing transparently after the binding flips.

Scenario A: running VM

On source: run capture_snapshot to produce a self-contained committed snapshot (recent fix(sandbox): capture self-contained snapshot during pause already moves us toward self-contained capture).
Ensure target reachability: rely on P2P advertisement of fixed artifacts + overlaybd layers.
On target: create-from-snapshot reusing the same SandboxId, wait for Running.
Flip scheduler binding to target (RecordAssignment), then delete the source sandbox.

Scenario B: paused VM (P2P only, no re-snapshot)

Add a publish-paused-to-P2P path: enumerate the paused VM's artifacts from its persisted FirecrackerSnapshotConfig (vm_state.bin, mem_image.json overlaybd layers including stacked ancestors, rootfs layers, writable drive layers) and transport.publish them. Reuse the content-addressed overlaybd-layer/v1/sha256:<digest> keys from src/overlaybd/p2p/artifact.rs for layers, and a paused/v1/... (or reuse snapshot/v1/artifacts/...) namespace for the fixed vm_state.bin + a reconstructable manifest.
On target: a resume-from-P2P resolver reconstructs FirecrackerSnapshotConfig from the published manifest and materializes every layer from P2P, then resumes.
Flip binding, then delete the paused source record and unpin/unpublish.

Key challenges to resolve

P2P consumption gap: P2P-first resolution today covers OSS fixed artifacts and the overlaybd layer facade. Migration without a shared repository backend (POSIX, or a paused VM that was never committed) needs the target to materialize all artifact classes from P2P. This is the main new capability for Scenario B.
Incremental memory + rootfs layers: mem_image.json and rootfs are stacked overlaybd layers with parent ancestry. All ancestors must be reachable on the target. Content-addressed layer keys help, but the publish step must walk the full layer chain, not just the top layer.
Writable attached drives: per-drive drives/<id>/image.json layers must migrate too.
Binding handoff atomicity: traffic must flip to the target only after it is Running, and the source must not be deleted before the binding moves, to avoid a routing gap. Define the ordering and a rollback path if target resume fails.
CPU compatibility: resume on a different host needs a compatible CPU config. The existing cluster-wide CPU config intersection (heartbeat cpu_config_json, applied via pre-boot PUT /cpu-config) is a precondition; migration should refuse or warn if the target cannot satisfy it.
Cleanup on both success and failure: unpin/unpublish source artifacts, delete source sandbox/paused record, and handle partial failure (target resume fails, source already torn down).

Suggested phasing

Paused VM publish-to-P2P + resume-from-P2P resolver (Scenario B), since paused content is already materialized and the cost/value ratio is best.
Running VM migration (Scenario A) layered on capture_snapshot + create-from-snapshot.
Scheduler/orchestrator migration RPC, binding handoff, and gateway transparency tests.

Acceptance criteria

A paused sandbox on node A can be resumed on node B with no shared repository backend, pulling all artifacts via P2P, preserving its SandboxId.
A running sandbox can be migrated A -> B; the gateway routes client traffic to B after handoff with no manual reconfiguration.
Migration failures leave exactly one live, routable copy (no orphaned source, no double bind).
Integration coverage for both scenarios, including stacked memory/rootfs ancestry and writable drives.

Originally created by @yingdi-shan on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/11 ## Summary Add the ability to move a sandbox from one AgentENV node to another. The core insight is that we already have fast snapshot and fast resume, so migration is mostly orchestration on top of existing primitives: capture (or reuse) a self-contained snapshot on the source node, make its artifacts reachable from the target node (P2P-first), resume on the target preserving sandbox identity, then tear down the source and hand off the scheduler binding so the gateway routes traffic to the new node. Two cases, with different cost profiles: 1. **Running VM migration**: snapshot on source, resume on target. Reuses `capture_snapshot` plus create-from-snapshot. 2. **Paused VM migration**: the VM is already paused and its content sits node-local. We only need to publish that paused content into P2P so the target can pull and resume. No re-snapshot required. ## Motivation Migration unlocks node draining for maintenance, load rebalancing, and bin packing. Paused sandboxes in particular are cheap to move because their state is already materialized, so making them mobile is high value for low cost. ## Proposed design Introduce a migration flow (orchestrator + scheduler RPC) parameterized by source node, target node, and sandbox ID. Preserve the `SandboxId` end-to-end so the gateway keeps routing transparently after the binding flips. **Scenario A: running VM** 1. On source: run `capture_snapshot` to produce a self-contained committed snapshot (recent `fix(sandbox): capture self-contained snapshot during pause` already moves us toward self-contained capture). 2. Ensure target reachability: rely on P2P advertisement of fixed artifacts + overlaybd layers. 3. On target: create-from-snapshot reusing the same `SandboxId`, wait for `Running`. 4. Flip scheduler binding to target (`RecordAssignment`), then delete the source sandbox. **Scenario B: paused VM (P2P only, no re-snapshot)** 1. Add a publish-paused-to-P2P path: enumerate the paused VM's artifacts from its persisted `FirecrackerSnapshotConfig` (`vm_state.bin`, `mem_image.json` overlaybd layers including stacked ancestors, rootfs layers, writable drive layers) and `transport.publish` them. Reuse the content-addressed `overlaybd-layer/v1/sha256:<digest>` keys from `src/overlaybd/p2p/artifact.rs` for layers, and a `paused/v1/...` (or reuse `snapshot/v1/artifacts/...`) namespace for the fixed `vm_state.bin` + a reconstructable manifest. 2. On target: a resume-from-P2P resolver reconstructs `FirecrackerSnapshotConfig` from the published manifest and materializes every layer from P2P, then resumes. 3. Flip binding, then delete the paused source record and unpin/unpublish. ## Key challenges to resolve - **P2P consumption gap**: P2P-first resolution today covers OSS fixed artifacts and the overlaybd layer facade. Migration without a shared repository backend (POSIX, or a paused VM that was never committed) needs the target to materialize *all* artifact classes from P2P. This is the main new capability for Scenario B. - **Incremental memory + rootfs layers**: `mem_image.json` and rootfs are stacked overlaybd layers with parent ancestry. All ancestors must be reachable on the target. Content-addressed layer keys help, but the publish step must walk the full layer chain, not just the top layer. - **Writable attached drives**: per-drive `drives/<id>/image.json` layers must migrate too. - **Binding handoff atomicity**: traffic must flip to the target only after it is `Running`, and the source must not be deleted before the binding moves, to avoid a routing gap. Define the ordering and a rollback path if target resume fails. - **CPU compatibility**: resume on a different host needs a compatible CPU config. The existing cluster-wide CPU config intersection (heartbeat `cpu_config_json`, applied via pre-boot `PUT /cpu-config`) is a precondition; migration should refuse or warn if the target cannot satisfy it. - **Cleanup on both success and failure**: unpin/unpublish source artifacts, delete source sandbox/paused record, and handle partial failure (target resume fails, source already torn down). ## Suggested phasing 1. Paused VM publish-to-P2P + resume-from-P2P resolver (Scenario B), since paused content is already materialized and the cost/value ratio is best. 2. Running VM migration (Scenario A) layered on `capture_snapshot` + create-from-snapshot. 3. Scheduler/orchestrator migration RPC, binding handoff, and gateway transparency tests. ## Acceptance criteria - A paused sandbox on node A can be resumed on node B with no shared repository backend, pulling all artifacts via P2P, preserving its `SandboxId`. - A running sandbox can be migrated A -> B; the gateway routes client traffic to B after handoff with no manual reconfiguration. - Migration failures leave exactly one live, routable copy (no orphaned source, no double bind). - Integration coverage for both scenarios, including stacked memory/rootfs ancestry and writable drives.

bsakaguchi added the

enhancement

label

2026-10-29 14:18:18 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:04 +00:00

[PR #39] [MERGED] perf: isolate benchmark and e2e dependencies #134

bsakaguchi referenced this issue

2026-10-29 14:22:07 +00:00

[PR #283] feat(p2p): make Iroh backend optional #315

bsakaguchi referenced this issue

2026-10-29 14:22:07 +00:00