kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #440] [Design Proposal] Online node upgrades without interrupting running sandboxes #108

New issue

Open

opened 2026-10-29 14:19:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:40 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/440

Affected component

Cross-cutting / other (Orchestrator / sandbox lifecycle, ublk / ublk-daemon, Firecracker runtime, API server / proxy, Network, Build / installation)

Problem statement

Upgrading AgentENV on a node interrupts every sandbox running on it: sandboxes are paused (or lost), and every envd stream, WebSocket, and egress connection breaks. Releases therefore cost users availability, and even a one-line ublk-daemon fix costs as much as a full release.

The cause is that all runtime state of a running sandbox belongs to the server container or ends with it:

State	Why it dies with the server	Location
Firecracker VMs	`kill_on_drop(true)`, and they live in the server container's cgroup	`src/privileges.rs:111`, `src/sandbox/firecracker/instance.rs:570`
ublk daemon (rootfs, drives, volumes, guest memory)	Exits with the server; devices have no user recovery; `Recovery` is `unimplemented!()`	`storage/ublk-daemon/src/main.rs:341`, `storage/ublk/src/main.rs:35`
Netns mounts, API and ublk sockets	Under `runtime_path` (`/run/aenv`), container-local in Kubernetes	`config/default.toml:13`, `src/sandbox/network/manager.rs:91`
Network namespaces and host rules	Removed on shutdown and by the `atexit` hook	`docs/src/internals/networking.md:200`
Egress proxy and sandbox ingress	Served by threads and the `:8000` listener inside the server	`src/sandbox/network/egress_proxy.rs`, `src/api/proxy.rs`
Running-sandbox metadata	In memory only; only paused sandboxes are persisted	`src/orchestrator/store/in_memory.rs`
Node ID	`AENV_NODE_ID` is the pod name, which changes on every pod replacement	`deploy/k8s/base/agentenv-daemonset.yaml`

Related: #365 (durable-data compatibility during rollout), #302 (pooled ublk devices), roadmap §7 (runtime stability).

Use case

Kubernetes deployments (deploy/k8s/base) hosting long-lived, interactive agent sandboxes that stream long commands through envd or expose WebSocket services through the proxy. Components change at different rates: the server every release, the ublk daemon / OverlayBD often and independently, Firecracker occasionally, the pod spec and host kernel rarely.

Current behavior and workarounds

The pod is deleted, the node becomes LINGERING, preStop waits (up to 3600 s) for sandboxes to finish, SIGTERM pauses the rest, and the replacement pod reloads them under a new node ID. Auto-resume sandboxes come back on their next request.

This falls short because:

every sandbox is unavailable for pause + restart + resume, and every connection breaks;
if preStop uses up the grace period, pause-all gets about 2 s and running sandboxes are lost;
sandboxes without auto-resume stay Paused;
lookups for the node fail until the new pod's first heartbeat;
with maxUnavailable: 1, a cluster rollout can take hours per node.

Desired behavior

Server upgrade: no Firecracker process restarts (same pid and boot ID), no sandbox changes state, existing connections keep flowing, the node ID stays the same, and API calls during the handover either succeed or return a retryable status.

ublk-daemon upgrade: devices move to the new daemon one at a time with a bounded stall (target p99 ≤ 100 ms per device), zero guest I/O errors, no data loss in writable uppers, and shared memory devices keep their reference counts and COW.

Firecracker upgrade: new VMs use the new binary and existing VMs keep theirs. Optional forced migration pauses and resumes sandboxes one at a time, keeping their IDs.

Failures: a kill -9 at any handover step converges to exactly one owner per component, no orphaned processes, devices, netns, or runtime dirs, and every sandbox in Running, Paused, or a terminal state. Each component can roll back from N to N-1 on its own.

Proposed approach

1. Pod layout

Kubernetes can change a container's image on a live pod and restart only that container, while the pod's netns stays alive. Split the pod so the VMs never live in a container that gets upgraded:

flowchart TB
  subgraph POD["agentenv-node Pod (privileged, shareProcessNamespace)"]
    vmm["vmm<br/>idle; owns the cgroup of all VMs"]
    subgraph S["server slots"]
      sa["server-a (active)"]
      sb["server-b (standby)"]
    end
    subgraph U["ublk slots"]
      ua["ublk-a (active)"]
      ub["ublk-b (standby)"]
    end
  end
  sa -- "spawns VMs into" --> vmm
  sa -- device RPC --> ua
  sa <-. "lease + listener fds" .-> sb
  ua <-. "per-device recovery" .-> ub

vmm is an idle container that owns the cgroup and mount namespace of every Firecracker process. The server still sets up each VM as today but creates it with clone3(CLONE_INTO_CGROUP) into vmm, so restarting the server no longer kills VMs. The pause process reaps orphans, and kubelet still accounts for VM resources.
Server and ublk slots are A/B container pairs. A release updates the standby slot's image, then hands over to it (§3, §4). Server releases never touch storage.
Shared state: /var/lib/aenv stays on hostPath; runtime_path moves to a shared emptyDir with Bidirectional propagation. The pod netns, veths, and iptables rules survive container restarts, and the pod name (the node ID) stops changing on routine releases.

2. Runtime records and adoption

When a sandbox reaches Running, persist a versioned RuntimeRecord (LocalKvStore, Sync) and update it on every transition. It holds lifecycle metadata, the Firecracker pid, start time, version, and socket, the envd address, netns slot, and egress policy, the ublk devices, and the volume mounts.
On startup the server adopts each record: verify the process, Firecracker API, devices, and netns; register healthy sandboxes as Running without calling extension start again; tear down or recover partial ones; sweep anything without a record, and discard warm-pool slots and pooled devices.
Shared memory device reference counts are derived from the records, not stored separately.
SIGTERM to the active server always means the pod is going away, so it keeps today's pause-all.

3. Server handover

Standby N+1 starts, loads the records, and reports ready.
N+1 asks active N to hand over through an admin socket.
N stops taking lifecycle operations, finishes or aborts in-flight ones (aborts restore Running), flushes the records, and passes its listener fds (:8000 and the egress listeners) over SCM_RIGHTS.
N releases the lease (flock + epoch). N+1 takes it, adopts the sandboxes, and accepts on the inherited listeners, so no connection is refused. Requests that arrive during adoption wait for a bounded time.
N keeps relaying its existing connections until they close, then becomes the standby.
Mutating requests carry the epoch, and stale epochs are rejected.
If N+1 fails before taking the lease, N stays active and the rollout stops at that node.

4. ublk handover

This uses the kernel's ublk server-upgrade support (UBLK_U_CMD_QUIESCE_DEV, documented in the 6.17 uapi for "upgrade ublk server application").

Create devices with UBLK_F_USER_RECOVERY | UBLK_F_USER_RECOVERY_REISSUE | UBLK_F_QUIESCE when supported. UBLK_F_QUIESCE must be added to the vendored libublk-rs-sys, as UBLK_F_UPDATE_SIZE is in storage/ublk/src/ublk_caps.rs.
Per device: the old daemon quiesces, completes in-flight I/O, fsyncs the upper and index, and closes /dev/ublkcN; the new daemon runs START_USER_RECOVERY … END_USER_RECOVERY. If that fails, the old daemon takes the device back and the handover aborts.
REISSUE can replay writes, so OverlayBD upper writes, flushes, and discards must be idempotent (verified in P0).
Fallbacks: without QUIESCE, restart a single daemon with one stall across all devices; without USER_RECOVERY, pause and resume sandboxes one at a time.
Crash recovery (roadmap §7) reuses the same path.

5. Firecracker and rollout

Firecracker binaries ship in the server image and are installed into versioned directories. Each VM records its version, and a binary is removed once no VM uses it, so the vmm image never changes for Firecracker releases.
Pod-spec, vmm, and kernel changes still recreate the pod using today's drain path.
The stock DaemonSet recreates pods on any template change, so in-place updates need either OnDelete plus a small rollout controller (possibly in services/) or OpenKruise Advanced DaemonSet. Settings in AENV_* env vars move to the mounted config file so routine changes don't recreate pods.

6. Delivery plan

Phase	Deliverable	Exit criteria
P0 Spikes	ublk per-device handover (incl. zero copy / `AUTO_BUF_REG`); spawning into `vmm`; in-place restarts and rollout mechanism	Stall p50/p95/p99, `fio --verify` clean, kernel floor, spawn path and rollout mechanism chosen
P1 Records + adoption	`RuntimeRecord`, adoption, orphan sweep, shared runtime dir	`kill -9` of the server at any lifecycle stage re-adopts VMs with no state change
P2 `vmm` container	VMs spawned into `vmm`; single server slot updated in place	Server restart keeps VM pids and boot IDs
P3 ublk slots	Daemon container, recovery-capable devices, A/B handover	ublk targets met; rollback; chaos tests converge
P4 Server slots	Lease + epoch, fd passing, connection drain	Connections survive upgrades; no refused connections
P5 Rollout	Rollout controller or OpenKruise, versioned Firecracker, gateway retries	Cluster rollout with no sandbox state changes
P6 Upgrade CI	N/N-1 policy and upgrade/downgrade matrix	Upgrade each component from N-1, then exec, pause/resume, snapshot, fork

Order: P0 → P1 → P2 → (P3 ∥ P4) → P5 → P6. P3 goes first if ublk releases are the priority.

Compatibility and operational impact

API/config changes: The public API is unchanged. A new [upgrade] section holds slot identity, deadlines, request hold time, and drain cap. runtime_path must point to the shared volume in Kubernetes, AENV_* settings move to the config file, and the gateway retries the handover status.

Snapshot or storage format changes: None to snapshots, templates, or repositories. RuntimeRecord is node-local derived state. Devices from earlier builds lack recovery flags, so the first rollout needs one drain per node; it also changes the pod spec.

New host/runtime requirements: A kernel with UBLK_F_QUIESCE for per-device handover (older kernels use the fallbacks); shareProcessNamespace, a Bidirectional emptyDir, and a rollout controller or OpenKruise; memory for one idle standby slot per component.

Upgrade and rollback considerations: Each component rolls back independently to N-1 by updating its standby slot. An aborted handover leaves the old slot in charge. vmm needs oom_score_adj=-1000, memory.oom.group=0, and no liveness probe, because restarting it kills every VM. preStop must stay shorter than the grace period so pause-all can finish.

Alternatives considered

Harden today's pause/restart/resume: simple, but every release interrupts every sandbox. Kept as the fallback for pod recreation.
Separate supervisor DaemonSet (aenv-runtimed): works, but adds a service, an RPC layer, and a second DaemonSet.
Host-anchored data plane (hostNetwork, hostPID, systemd scopes): one container, but ties AgentENV to host systemd, hides VMs from kubelet, and pod deletion no longer cleans up.
Fewer containers or no A/B slots: smaller, but server releases restart the ublk daemon or drop streams, and ublk releases freeze I/O for a whole container restart.
execve self-restart, CRIU, or passing protocol state: loses in-process HTTP/2 and WebSocket state, bypasses image rollout, or breaks with io_uring, ublk, and KVM fds.
Cross-node live migration: handles kernel and host maintenance but costs far more per upgrade; complementary.

Non-goals

Kernel upgrades, host reboots, or pod-spec changes without a drain.
Swapping the Firecracker binary of a running VM.
Upgrading envd or the guest kernel inside running sandboxes.
Surviving gateway rollouts, or scheduler HA beyond retries.
Carrying long operations (large captures, template builds) across a handover.

Contribution

I can provide more design details

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @yingdi-shan on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/440 ### Affected component Cross-cutting / other (Orchestrator / sandbox lifecycle, ublk / ublk-daemon, Firecracker runtime, API server / proxy, Network, Build / installation) ### Problem statement Upgrading AgentENV on a node interrupts every sandbox running on it: sandboxes are paused (or lost), and every envd stream, WebSocket, and egress connection breaks. Releases therefore cost users availability, and even a one-line ublk-daemon fix costs as much as a full release. The cause is that all runtime state of a running sandbox belongs to the server container or ends with it: | State | Why it dies with the server | Location | | --- | --- | --- | | Firecracker VMs | `kill_on_drop(true)`, and they live in the server container's cgroup | `src/privileges.rs:111`, `src/sandbox/firecracker/instance.rs:570` | | ublk daemon (rootfs, drives, volumes, guest memory) | Exits with the server; devices have no user recovery; `Recovery` is `unimplemented!()` | `storage/ublk-daemon/src/main.rs:341`, `storage/ublk/src/main.rs:35` | | Netns mounts, API and ublk sockets | Under `runtime_path` (`/run/aenv`), container-local in Kubernetes | `config/default.toml:13`, `src/sandbox/network/manager.rs:91` | | Network namespaces and host rules | Removed on shutdown and by the `atexit` hook | `docs/src/internals/networking.md:200` | | Egress proxy and sandbox ingress | Served by threads and the `:8000` listener inside the server | `src/sandbox/network/egress_proxy.rs`, `src/api/proxy.rs` | | Running-sandbox metadata | In memory only; only paused sandboxes are persisted | `src/orchestrator/store/in_memory.rs` | | Node ID | `AENV_NODE_ID` is the pod name, which changes on every pod replacement | `deploy/k8s/base/agentenv-daemonset.yaml` | Related: #365 (durable-data compatibility during rollout), #302 (pooled ublk devices), roadmap §7 (runtime stability). ### Use case Kubernetes deployments (`deploy/k8s/base`) hosting long-lived, interactive agent sandboxes that stream long commands through envd or expose WebSocket services through the proxy. Components change at different rates: the server every release, the ublk daemon / OverlayBD often and independently, Firecracker occasionally, the pod spec and host kernel rarely. ### Current behavior and workarounds The pod is deleted, the node becomes `LINGERING`, `preStop` waits (up to 3600 s) for sandboxes to finish, SIGTERM pauses the rest, and the replacement pod reloads them under a new node ID. Auto-resume sandboxes come back on their next request. This falls short because: - every sandbox is unavailable for pause + restart + resume, and every connection breaks; - if `preStop` uses up the grace period, pause-all gets about 2 s and running sandboxes are **lost**; - sandboxes without auto-resume stay `Paused`; - lookups for the node fail until the new pod's first heartbeat; - with `maxUnavailable: 1`, a cluster rollout can take hours per node. ### Desired behavior **Server upgrade:** no Firecracker process restarts (same pid and boot ID), no sandbox changes state, existing connections keep flowing, the node ID stays the same, and API calls during the handover either succeed or return a retryable status. **ublk-daemon upgrade:** devices move to the new daemon one at a time with a bounded stall (target p99 ≤ 100 ms per device), zero guest I/O errors, no data loss in writable uppers, and shared memory devices keep their reference counts and COW. **Firecracker upgrade:** new VMs use the new binary and existing VMs keep theirs. Optional forced migration pauses and resumes sandboxes one at a time, keeping their IDs. **Failures:** a `kill -9` at any handover step converges to exactly one owner per component, no orphaned processes, devices, netns, or runtime dirs, and every sandbox in `Running`, `Paused`, or a terminal state. Each component can roll back from N to N-1 on its own. ### Proposed approach #### 1. Pod layout Kubernetes can change a container's image on a live pod and restart only that container, while the pod's netns stays alive. Split the pod so the VMs never live in a container that gets upgraded: ```mermaid flowchart TB subgraph POD["agentenv-node Pod (privileged, shareProcessNamespace)"] vmm["vmm<br/>idle; owns the cgroup of all VMs"] subgraph S["server slots"] sa["server-a (active)"] sb["server-b (standby)"] end subgraph U["ublk slots"] ua["ublk-a (active)"] ub["ublk-b (standby)"] end end sa -- "spawns VMs into" --> vmm sa -- device RPC --> ua sa <-. "lease + listener fds" .-> sb ua <-. "per-device recovery" .-> ub ``` - **`vmm`** is an idle container that owns the cgroup and mount namespace of every Firecracker process. The server still sets up each VM as today but creates it with `clone3(CLONE_INTO_CGROUP)` into `vmm`, so restarting the server no longer kills VMs. The pause process reaps orphans, and kubelet still accounts for VM resources. - **Server and ublk slots** are A/B container pairs. A release updates the standby slot's image, then hands over to it (§3, §4). Server releases never touch storage. - **Shared state:** `/var/lib/aenv` stays on hostPath; `runtime_path` moves to a shared `emptyDir` with `Bidirectional` propagation. The pod netns, veths, and iptables rules survive container restarts, and the pod name (the node ID) stops changing on routine releases. #### 2. Runtime records and adoption - When a sandbox reaches `Running`, persist a versioned `RuntimeRecord` (`LocalKvStore`, `Sync`) and update it on every transition. It holds lifecycle metadata, the Firecracker pid, start time, version, and socket, the envd address, netns slot, and egress policy, the ublk devices, and the volume mounts. - On startup the server adopts each record: verify the process, Firecracker API, devices, and netns; register healthy sandboxes as `Running` without calling extension `start` again; tear down or recover partial ones; sweep anything without a record, and discard warm-pool slots and pooled devices. - Shared memory device reference counts are derived from the records, not stored separately. - SIGTERM to the active server always means the pod is going away, so it keeps today's pause-all. #### 3. Server handover 1. Standby N+1 starts, loads the records, and reports ready. 2. N+1 asks active N to hand over through an admin socket. 3. N stops taking lifecycle operations, finishes or aborts in-flight ones (aborts restore `Running`), flushes the records, and passes its listener fds (`:8000` and the egress listeners) over `SCM_RIGHTS`. 4. N releases the lease (`flock` + epoch). N+1 takes it, adopts the sandboxes, and accepts on the inherited listeners, so no connection is refused. Requests that arrive during adoption wait for a bounded time. 5. N keeps relaying its existing connections until they close, then becomes the standby. 6. Mutating requests carry the epoch, and stale epochs are rejected. 7. If N+1 fails before taking the lease, N stays active and the rollout stops at that node. #### 4. ublk handover This uses the kernel's ublk server-upgrade support (`UBLK_U_CMD_QUIESCE_DEV`, documented in the 6.17 uapi for "upgrade ublk server application"). - Create devices with `UBLK_F_USER_RECOVERY | UBLK_F_USER_RECOVERY_REISSUE | UBLK_F_QUIESCE` when supported. `UBLK_F_QUIESCE` must be added to the vendored `libublk-rs-sys`, as `UBLK_F_UPDATE_SIZE` is in `storage/ublk/src/ublk_caps.rs`. - Per device: the old daemon quiesces, completes in-flight I/O, fsyncs the upper and index, and closes `/dev/ublkcN`; the new daemon runs `START_USER_RECOVERY` … `END_USER_RECOVERY`. If that fails, the old daemon takes the device back and the handover aborts. - `REISSUE` can replay writes, so OverlayBD upper writes, flushes, and discards must be idempotent (verified in P0). - Fallbacks: without `QUIESCE`, restart a single daemon with one stall across all devices; without `USER_RECOVERY`, pause and resume sandboxes one at a time. - Crash recovery (roadmap §7) reuses the same path. #### 5. Firecracker and rollout - Firecracker binaries ship in the server image and are installed into versioned directories. Each VM records its version, and a binary is removed once no VM uses it, so the `vmm` image never changes for Firecracker releases. - Pod-spec, `vmm`, and kernel changes still recreate the pod using today's drain path. - The stock DaemonSet recreates pods on any template change, so in-place updates need either `OnDelete` plus a small rollout controller (possibly in `services/`) or OpenKruise Advanced DaemonSet. Settings in `AENV_*` env vars move to the mounted config file so routine changes don't recreate pods. #### 6. Delivery plan | Phase | Deliverable | Exit criteria | | --- | --- | --- | | P0 Spikes | ublk per-device handover (incl. zero copy / `AUTO_BUF_REG`); spawning into `vmm`; in-place restarts and rollout mechanism | Stall p50/p95/p99, `fio --verify` clean, kernel floor, spawn path and rollout mechanism chosen | | P1 Records + adoption | `RuntimeRecord`, adoption, orphan sweep, shared runtime dir | `kill -9` of the server at any lifecycle stage re-adopts VMs with no state change | | P2 `vmm` container | VMs spawned into `vmm`; single server slot updated in place | Server restart keeps VM pids and boot IDs | | P3 ublk slots | Daemon container, recovery-capable devices, A/B handover | ublk targets met; rollback; chaos tests converge | | P4 Server slots | Lease + epoch, fd passing, connection drain | Connections survive upgrades; no refused connections | | P5 Rollout | Rollout controller or OpenKruise, versioned Firecracker, gateway retries | Cluster rollout with no sandbox state changes | | P6 Upgrade CI | N/N-1 policy and upgrade/downgrade matrix | Upgrade each component from N-1, then exec, pause/resume, snapshot, fork | Order: P0 → P1 → P2 → (P3 ∥ P4) → P5 → P6. P3 goes first if ublk releases are the priority. ### Compatibility and operational impact **API/config changes:** The public API is unchanged. A new `[upgrade]` section holds slot identity, deadlines, request hold time, and drain cap. `runtime_path` must point to the shared volume in Kubernetes, `AENV_*` settings move to the config file, and the gateway retries the handover status. **Snapshot or storage format changes:** None to snapshots, templates, or repositories. `RuntimeRecord` is node-local derived state. Devices from earlier builds lack recovery flags, so **the first rollout needs one drain per node**; it also changes the pod spec. **New host/runtime requirements:** A kernel with `UBLK_F_QUIESCE` for per-device handover (older kernels use the fallbacks); `shareProcessNamespace`, a `Bidirectional` `emptyDir`, and a rollout controller or OpenKruise; memory for one idle standby slot per component. **Upgrade and rollback considerations:** Each component rolls back independently to N-1 by updating its standby slot. An aborted handover leaves the old slot in charge. `vmm` needs `oom_score_adj=-1000`, `memory.oom.group=0`, and no liveness probe, because restarting it kills every VM. `preStop` must stay shorter than the grace period so pause-all can finish. ### Alternatives considered - **Harden today's pause/restart/resume:** simple, but every release interrupts every sandbox. Kept as the fallback for pod recreation. - **Separate supervisor DaemonSet (`aenv-runtimed`):** works, but adds a service, an RPC layer, and a second DaemonSet. - **Host-anchored data plane** (`hostNetwork`, `hostPID`, systemd scopes): one container, but ties AgentENV to host systemd, hides VMs from kubelet, and pod deletion no longer cleans up. - **Fewer containers or no A/B slots:** smaller, but server releases restart the ublk daemon or drop streams, and ublk releases freeze I/O for a whole container restart. - **`execve` self-restart, CRIU, or passing protocol state:** loses in-process HTTP/2 and WebSocket state, bypasses image rollout, or breaks with io_uring, ublk, and KVM fds. - **Cross-node live migration:** handles kernel and host maintenance but costs far more per upgrade; complementary. ### Non-goals - Kernel upgrades, host reboots, or pod-spec changes without a drain. - Swapping the Firecracker binary of a running VM. - Upgrading envd or the guest kernel inside running sandboxes. - Surviving gateway rollouts, or scheduler HA beyond retries. - Carrying long operations (large captures, template builds) across a handover. ### Contribution I can provide more design details ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label

2026-10-29 14:19:40 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:45 +00:00

[PR #108] [MERGED] Fix sandbox lifecycle bugs and improve network policy handling #183