kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #102] Snapshot restore may stall when AgentENV reuses an envd bootstrap connection across sandbox generations #36

New issue

Closed

opened 2026-10-29 14:18:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:38 +00:00

Owner

Originally created by @leether on GitHub (Aug 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/102

Affected component

Orchestrator / sandbox lifecycle

AgentENV version

Version or commit: reproduced on leether/AgentENV@105fbbc, based on kvcache-ai/AgentENV@8174bd9
Local modifications: ARM64 assembly-line changes were required to run AgentENV on this host. Before the proposed fix, src/sandbox/envd.rs was unchanged from upstream at the affected shared-client code path.
First source revision known to contain the affected shared-client pattern: kvcache-ai/AgentENV@8f028b1 (initial open-source release; not separately runtime-tested for this report)
envd: 0.6.13, built from source tree e2b-dev/infra@bfdbb24

Host environment

OS: Ubuntu 24.04.4 LTS
Kernel (uname -a, hostname redacted): Linux <dgx-host> 6.17.0-1021-nvidia #21-Ubuntu SMP PREEMPT_DYNAMIC Wed May 27 19:14:05 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
Architecture: aarch64, 4 KiB pages
Bare metal, VM, or cloud: NVIDIA DGX Spark, bare metal
ublk support: /dev/ublk-control present; real uvm-ublk-daemon and OverlayBD paths used
CPU: 20 CPUs, Cortex-X925 / Cortex-A725
Memory: 121 GiB visible to the OS
Storage/filesystem: NVMe, root filesystem on ext4 (/dev/nvme0n1p2)
Firecracker: kvcache-ai/firecracker-next v1.17.0-next.1 lineage, exact downstream build b792de1c4618092b0d712c59e5d2a195c97ff5bd

Operation and configuration

The failure was found by running the repository snapshot benchmark repeatedly in fresh processes. The benchmark uses a 1-vCPU, 128-MiB guest and calls FirecrackerSandbox::resume_from_snapshot_config() directly.

Representative invocation shape, with isolated paths and test CIDRs:

# Build the repository benchmark first.
cargo bench --bench snapshot --no-run

# Run the built snapshot benchmark with an isolated AgentENV environment.
# The exact target hash depends on the build.
env \
  AENV_CONFIG_PATH=/tmp/agentenv-official-bench.toml \
  AENV_HOME_PATH=/tmp/agentenv-official-bench/home \
  AENV_RUNTIME_PATH=/tmp/agentenv-official-bench-run1/runtime \
  AENV_DEPS_PATH=/path/to/aenv-deps \
  AENV_FIRECRACKER_WORK_DIR=/tmp/agentenv-official-bench-run1/work \
  AENV_FIRECRACKER_SERIAL_DIR=/tmp/agentenv-official-bench-run1/serial \
  NO_PROXY=127.0.0.1,localhost,10.211.0.0/16,10.212.0.0/16,169.254.0.0/16 \
  no_proxy=127.0.0.1,localhost,10.211.0.0/16,10.212.0.0/16,169.254.0.0/16 \
  ./target/release/deps/snapshot-<build-hash> snapshot_resume

Relevant configuration characteristics:

[pool]
low_watermark = 2
high_watermark = 64

[pool.network]
maintenance_enabled = true

The test used dedicated home/runtime/work/serial/cache directories, an isolated ublk daemon and metrics port, and dedicated network CIDRs. Production AgentENV was not stopped or restarted.

Steps to reproduce

Run an AgentENV server process containing the process-wide ENVD_HTTP_CLIENT from src/sandbox/envd.rs.
Create a Firecracker snapshot after envd bootstrap has created guest TCP state.
Repeatedly restore the same snapshot in the same AgentENV process, allowing warm network slots and interaction IPs to be reused.
Ensure the restore sequence returns to the interaction IP/slot whose TCP tuple is present in the snapshot. The repository snapshot benchmark reaches this state through normal slot rotation; a one-slot/deterministic test setup can force it directly.
Observe one envd health/init request reuse the host-side idle connection from the previous runtime generation.

In three fresh benchmark processes, each cold and hot resume group contained one approximately 30-second sample. A separate deterministic packet-capture run reproduced the same generation sequence with timings of 163.237 ms, 30,995.648 ms, and 160.390 ms.

Expected behavior

Each sandbox runtime generation should bootstrap envd through a TCP connection that belongs to that generation. Restoring a snapshot into a reused interaction IP should not allow the host HTTP client to send health or init over an idle connection retained from a previous VM.

If an old flow is unusable, bootstrap should establish a fresh connection immediately rather than waiting for the TCP user timeout.

Actual behavior

The process-wide reqwest client pools by HTTP origin (scheme + IP + port) and outlives individual sandbox runtimes. When the same interaction IP is reused, it can check out an idle TCP connection from the previous runtime generation.

If the restored guest snapshot contains the matching TCP tuple, the guest does not immediately reject the packet. Instead, guest and host are in different TCP sequence spaces and make no forward progress. The bootstrap request waits approximately 30 seconds before reqwest opens a new connection.

Across three unforced benchmark rounds:

cold snapshot resume: 3/30 samples stalled for 30.91–31.04 s; the other 27 were 91.70–118.71 ms;
hot snapshot resume: 3/30 samples stalled for 30.52–30.54 s; the other 27 were 61.55–85.06 ms.

An ordinary-template control ran 10 starts in 95–179 ms. In that case the immutable snapshot did not contain the pooled connection's tuple, so the guest returned RST and reqwest reconnected quickly. Reusing the IP alone is therefore not sufficient; the snapshot must also contain the matching guest-side TCP state.

Logs and diagnostics

The deterministic trace used this tuple:

10.242.0.8:39158 -> 10.241.0.4:49983

Before the snapshot, the connection was at guest/host state G0/H0. The first restore reused the tuple successfully and advanced the host side to H1:

host health  seq 42026036:42026097 ack 1773910168
guest reply  seq 1773910168:1773910287 ack 42026097
host init    seq 42026097:42026255 ack 1773910287
guest reply  seq 1773910287:1773910414 ack 42026255

The second restore rewound only the guest to G0, while the pooled host socket remained at H1:

host:  seq 42026255:42026316 ack 1773910414
guest: seq 1773910041:1773910168 ack 42026036

The host sequence is ahead by exactly 219 bytes, matching the first restore's 61-byte health request plus 158-byte init request. Both sides continued ACK/SACK and retransmission without advancing the cumulative ACK.

The first stuck packet was captured at 13:32:46.595216; the fresh-connection SYN was sent at 13:33:17.457549, a 30.862333 s interval. The built reqwest 0.13.3 has a 30-second TCP user timeout. As a control, changing only that timeout to five seconds shortened the stall to approximately 5.5 seconds.

Further controls:

disabling idle pooling removed the stall;
constructing a client per runtime generation also removed the stall;
PAWS/timestamp rejection was excluded because both sides continued exchanging ACK/SACK and timestamp values advanced;
killing the previously observed unrelated ghost TCP flows did not remove the earlier 504 incident, so this report does not claim that every restored stale socket causes this bootstrap stall.

Reproduction frequency

Intermittently

With the unforced repository benchmark and normal slot rotation, it appeared in 3/30 cold and 3/30 hot samples. Once the same-slot generation sequence was made deterministic, that sequence reproduced it every time.

Regression information

Last known good: unknown
First known bad: unknown as a runtime regression
First source revision known to contain the affected process-wide client: kvcache-ai/AgentENV@8f028b1 (not separately runtime-tested)

Additional context

Root cause

The connection identity used by the host pool is the HTTP origin, but the actual endpoint identity is (origin, sandbox runtime generation). Snapshot restore rewinds guest TCP state; the host process and its idle connection pool are not part of the snapshot and continue forward. Warm network-slot reuse then assigns the same origin to a later guest runtime.

The resulting state transition is:

snapshot:       guest G0 / host H0
first restore:  guest G1 / host H1
second restore: guest G0 / host H1  -> sequence mismatch -> TCP user timeout

PR #56 addressed the same origin-versus-runtime-generation mismatch for the sandbox HTTP proxy client. The envd bootstrap path uses a separate generated reqwest client and was not covered by that change.

This was discovered while following up the unexplained approximately 30-second snapshot benchmark samples reported in #101. ARM64 is the environment in which it was found, but the mechanism is not ARM-specific: it requires snapshot restore, a host client that outlives the guest runtime, and reuse of the same interaction address.

Narrow fix evaluated

We tested keeping the shared bootstrap client configuration while preventing its connections from entering the idle pool:

Client::builder()
    .pool_max_idle_per_host(0)
    .build()

The scope is only envd GET /health and POST /init. Command execution continues to use a separately created gRPC ProcessClient; sandbox proxy traffic is unchanged.

A behavioral regression test keeps the health socket open and requires init to arrive through a second accept(). With pool_max_idle_per_host(1) the negative control times out because init reuses the first socket; with 0 it passes.

Validation of the narrow fix on the same isolated ARM64 host:

three rounds of 10 cold + 10 hot samples: 60/60 below 200 ms;
1,000 sequential hot restores: p50 130.00 ms, p99 152.91 ms, max 158.41 ms, 0/1,000 > 1 s;
two 50-concurrency x 10 runs: 1,000/1,000 restores completed;
library tests: 660 passed, 0 failed, 2 ignored;
short-lived sockets returned to zero orphan count within 20 seconds after the concurrent run; allocated file handles returned close to baseline.

The evaluated patch intentionally does not add .no_proxy() or .http1_only(), because proxy routing and protocol negotiation are unrelated behavior changes.

Would maintainers be open to a focused PR that applies pool_max_idle_per_host(0) only to the envd bootstrap client and adds the behavioral regression test? I can prepare it on a clean branch from current upstream main.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @leether on GitHub (Aug 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/102 ## Affected component Orchestrator / sandbox lifecycle ## AgentENV version - Version or commit: reproduced on [`leether/AgentENV@105fbbc`](https://github.com/leether/AgentENV/commit/105fbbc62cec0f00291de30a62038968a0e4dd11), based on [`kvcache-ai/AgentENV@8174bd9`](https://github.com/kvcache-ai/AgentENV/commit/8174bd9674806917a2c0e75d26a30b409459c51d) - Local modifications: ARM64 assembly-line changes were required to run AgentENV on this host. Before the proposed fix, `src/sandbox/envd.rs` was unchanged from upstream at the affected shared-client code path. - First source revision known to contain the affected shared-client pattern: [`kvcache-ai/AgentENV@8f028b1`](https://github.com/kvcache-ai/AgentENV/commit/8f028b1369a4ce9a66d140174d0a646e72398517) (initial open-source release; not separately runtime-tested for this report) - envd: `0.6.13`, built from source tree [`e2b-dev/infra@bfdbb24`](https://github.com/e2b-dev/infra/commit/bfdbb24130ede13b449331acab40d39981f9b329) ## Host environment - OS: Ubuntu 24.04.4 LTS - Kernel (`uname -a`, hostname redacted): `Linux <dgx-host> 6.17.0-1021-nvidia #21-Ubuntu SMP PREEMPT_DYNAMIC Wed May 27 19:14:05 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux` - Architecture: `aarch64`, 4 KiB pages - Bare metal, VM, or cloud: NVIDIA DGX Spark, bare metal - ublk support: `/dev/ublk-control` present; real `uvm-ublk-daemon` and OverlayBD paths used - CPU: 20 CPUs, Cortex-X925 / Cortex-A725 - Memory: 121 GiB visible to the OS - Storage/filesystem: NVMe, root filesystem on ext4 (`/dev/nvme0n1p2`) - Firecracker: `kvcache-ai/firecracker-next` `v1.17.0-next.1` lineage, exact downstream build `b792de1c4618092b0d712c59e5d2a195c97ff5bd` ## Operation and configuration The failure was found by running the repository snapshot benchmark repeatedly in fresh processes. The benchmark uses a 1-vCPU, 128-MiB guest and calls `FirecrackerSandbox::resume_from_snapshot_config()` directly. Representative invocation shape, with isolated paths and test CIDRs: ```shell # Build the repository benchmark first. cargo bench --bench snapshot --no-run # Run the built snapshot benchmark with an isolated AgentENV environment. # The exact target hash depends on the build. env \ AENV_CONFIG_PATH=/tmp/agentenv-official-bench.toml \ AENV_HOME_PATH=/tmp/agentenv-official-bench/home \ AENV_RUNTIME_PATH=/tmp/agentenv-official-bench-run1/runtime \ AENV_DEPS_PATH=/path/to/aenv-deps \ AENV_FIRECRACKER_WORK_DIR=/tmp/agentenv-official-bench-run1/work \ AENV_FIRECRACKER_SERIAL_DIR=/tmp/agentenv-official-bench-run1/serial \ NO_PROXY=127.0.0.1,localhost,10.211.0.0/16,10.212.0.0/16,169.254.0.0/16 \ no_proxy=127.0.0.1,localhost,10.211.0.0/16,10.212.0.0/16,169.254.0.0/16 \ ./target/release/deps/snapshot-<build-hash> snapshot_resume ``` Relevant configuration characteristics: ```toml [pool] low_watermark = 2 high_watermark = 64 [pool.network] maintenance_enabled = true ``` The test used dedicated home/runtime/work/serial/cache directories, an isolated ublk daemon and metrics port, and dedicated network CIDRs. Production AgentENV was not stopped or restarted. ## Steps to reproduce 1. Run an AgentENV server process containing the process-wide `ENVD_HTTP_CLIENT` from `src/sandbox/envd.rs`. 2. Create a Firecracker snapshot after envd bootstrap has created guest TCP state. 3. Repeatedly restore the same snapshot in the same AgentENV process, allowing warm network slots and interaction IPs to be reused. 4. Ensure the restore sequence returns to the interaction IP/slot whose TCP tuple is present in the snapshot. The repository `snapshot` benchmark reaches this state through normal slot rotation; a one-slot/deterministic test setup can force it directly. 5. Observe one envd health/init request reuse the host-side idle connection from the previous runtime generation. In three fresh benchmark processes, each cold and hot resume group contained one approximately 30-second sample. A separate deterministic packet-capture run reproduced the same generation sequence with timings of `163.237 ms`, `30,995.648 ms`, and `160.390 ms`. ## Expected behavior Each sandbox runtime generation should bootstrap envd through a TCP connection that belongs to that generation. Restoring a snapshot into a reused interaction IP should not allow the host HTTP client to send health or init over an idle connection retained from a previous VM. If an old flow is unusable, bootstrap should establish a fresh connection immediately rather than waiting for the TCP user timeout. ## Actual behavior The process-wide reqwest client pools by HTTP origin (`scheme + IP + port`) and outlives individual sandbox runtimes. When the same interaction IP is reused, it can check out an idle TCP connection from the previous runtime generation. If the restored guest snapshot contains the matching TCP tuple, the guest does not immediately reject the packet. Instead, guest and host are in different TCP sequence spaces and make no forward progress. The bootstrap request waits approximately 30 seconds before reqwest opens a new connection. Across three unforced benchmark rounds: - cold snapshot resume: 3/30 samples stalled for `30.91–31.04 s`; the other 27 were `91.70–118.71 ms`; - hot snapshot resume: 3/30 samples stalled for `30.52–30.54 s`; the other 27 were `61.55–85.06 ms`. An ordinary-template control ran 10 starts in `95–179 ms`. In that case the immutable snapshot did not contain the pooled connection's tuple, so the guest returned RST and reqwest reconnected quickly. Reusing the IP alone is therefore not sufficient; the snapshot must also contain the matching guest-side TCP state. ## Logs and diagnostics The deterministic trace used this tuple: ```text 10.242.0.8:39158 -> 10.241.0.4:49983 ``` Before the snapshot, the connection was at guest/host state `G0/H0`. The first restore reused the tuple successfully and advanced the host side to `H1`: ```text host health seq 42026036:42026097 ack 1773910168 guest reply seq 1773910168:1773910287 ack 42026097 host init seq 42026097:42026255 ack 1773910287 guest reply seq 1773910287:1773910414 ack 42026255 ``` The second restore rewound only the guest to `G0`, while the pooled host socket remained at `H1`: ```text host: seq 42026255:42026316 ack 1773910414 guest: seq 1773910041:1773910168 ack 42026036 ``` The host sequence is ahead by exactly 219 bytes, matching the first restore's 61-byte health request plus 158-byte init request. Both sides continued ACK/SACK and retransmission without advancing the cumulative ACK. The first stuck packet was captured at `13:32:46.595216`; the fresh-connection SYN was sent at `13:33:17.457549`, a `30.862333 s` interval. The built reqwest `0.13.3` has a 30-second TCP user timeout. As a control, changing only that timeout to five seconds shortened the stall to approximately 5.5 seconds. Further controls: - disabling idle pooling removed the stall; - constructing a client per runtime generation also removed the stall; - PAWS/timestamp rejection was excluded because both sides continued exchanging ACK/SACK and timestamp values advanced; - killing the previously observed unrelated ghost TCP flows did not remove the earlier 504 incident, so this report does not claim that every restored stale socket causes this bootstrap stall. ## Reproduction frequency Intermittently With the unforced repository benchmark and normal slot rotation, it appeared in 3/30 cold and 3/30 hot samples. Once the same-slot generation sequence was made deterministic, that sequence reproduced it every time. ## Regression information - Last known good: unknown - First known bad: unknown as a runtime regression - First source revision known to contain the affected process-wide client: [`kvcache-ai/AgentENV@8f028b1`](https://github.com/kvcache-ai/AgentENV/commit/8f028b1369a4ce9a66d140174d0a646e72398517) (not separately runtime-tested) ## Additional context ### Root cause The connection identity used by the host pool is the HTTP origin, but the actual endpoint identity is `(origin, sandbox runtime generation)`. Snapshot restore rewinds guest TCP state; the host process and its idle connection pool are not part of the snapshot and continue forward. Warm network-slot reuse then assigns the same origin to a later guest runtime. The resulting state transition is: ```text snapshot: guest G0 / host H0 first restore: guest G1 / host H1 second restore: guest G0 / host H1 -> sequence mismatch -> TCP user timeout ``` PR #56 addressed the same origin-versus-runtime-generation mismatch for the sandbox HTTP proxy client. The envd bootstrap path uses a separate generated reqwest client and was not covered by that change. This was discovered while following up the unexplained approximately 30-second snapshot benchmark samples reported in #101. ARM64 is the environment in which it was found, but the mechanism is not ARM-specific: it requires snapshot restore, a host client that outlives the guest runtime, and reuse of the same interaction address. ### Narrow fix evaluated We tested keeping the shared bootstrap client configuration while preventing its connections from entering the idle pool: ```rust Client::builder() .pool_max_idle_per_host(0) .build() ``` The scope is only envd `GET /health` and `POST /init`. Command execution continues to use a separately created gRPC `ProcessClient`; sandbox proxy traffic is unchanged. A behavioral regression test keeps the health socket open and requires init to arrive through a second `accept()`. With `pool_max_idle_per_host(1)` the negative control times out because init reuses the first socket; with `0` it passes. Validation of the narrow fix on the same isolated ARM64 host: - three rounds of 10 cold + 10 hot samples: 60/60 below 200 ms; - 1,000 sequential hot restores: p50 `130.00 ms`, p99 `152.91 ms`, max `158.41 ms`, `0/1,000 > 1 s`; - two 50-concurrency x 10 runs: 1,000/1,000 restores completed; - library tests: 660 passed, 0 failed, 2 ignored; - short-lived sockets returned to zero orphan count within 20 seconds after the concurrent run; allocated file handles returned close to baseline. The evaluated patch intentionally does not add `.no_proxy()` or `.http1_only()`, because proxy routing and protocol negotiation are unrelated behavior changes. Would maintainers be open to a focused PR that applies `pool_max_idle_per_host(0)` only to the envd bootstrap client and adds the behavioral regression test? I can prepare it on a clean branch from current upstream `main`. ## Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi closed this issue

2026-10-29 14:18:39 +00:00

bsakaguchi referenced this issue

2026-10-29 14:22:42 +00:00

[PR #351] [MERGED] fix(aenv): preserve queued input during recovery #371