[PR #56] [MERGED] fix(proxy): isolate connections across runtime generations #143

New issue

Closed

opened 2026-10-29 14:20:11 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:11 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/56
Author: @MasakiMu319
Created: 7/29/2026
Status: ✅ Merged
Merged: 7/29/2026
Merged by: @LSX-s-Software

Base: main ← Head: fix/proxy-runtime-generation-connection-reuse

📝 Commits (2)

14ac34e fix(proxy): isolate connections across runtime generations
9a3bae2 test(proxy): clarify runtime generation simulation

📊 Changes

1 file changed (+108 additions, -1 deletions)

View changed files

📝 src/api/proxy.rs (+108 -1)

📄 Description

Problem

Sandbox interaction IPs are recycled when runtimes stop, while envd always listens on the same port. The process-wide Hyper client pools idle connections by URI authority (IP:port), which does not encode the sandbox runtime generation.

This allows the following sequence:

A request to runtime A leaves an idle keep-alive connection in the pool.
Runtime A stops and its interaction IP is assigned to runtime B.
The next request for runtime B reuses A's stale TCP flow.
The new guest rejects that flow with an RST and Hyper returns client error (SendRequest); the proxy returns 502.

Retrying automatically is unsafe because proxied ConnectRPC requests can have side effects.

Fix

Disable idle per-host pooling for the server-to-sandbox proxy client with pool_max_idle_per_host(0).
Add a behavioral regression test that serves one request over a keep-alive connection, simulates the address moving to a new runtime generation, and verifies that the second request opens a fresh TCP connection.

Active requests and streaming connections are unchanged; only reuse after a connection becomes idle is disabled.

Validation

The regression test was run against the current main before the production change and failed with:

proxy client reused an idle TCP connection after the runtime generation changed

After the fix:

cargo test -p agentenv --lib api::proxy::tests — 32 passed
focused regression test repeated 50 times — 50 passed
cargo clippy -p agentenv --all-targets -- -D warnings
cargo fmt --all -- --check

A same-host lifecycle probe reproduced 15 initial proxy failures in 200 cycles before the fix and 0 in 1,200 cycles afterward, with no retries. Median latency remained effectively unchanged:

create: 117.18 ms → 117.25 ms
first exec: 81.26 ms → 81.46 ms

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/56 **Author:** [@MasakiMu319](https://github.com/MasakiMu319) **Created:** 7/29/2026 **Status:** ✅ Merged **Merged:** 7/29/2026 **Merged by:** [@LSX-s-Software](https://github.com/LSX-s-Software) **Base:** `main` ← **Head:** `fix/proxy-runtime-generation-connection-reuse` --- ### 📝 Commits (2) - [`14ac34e`](https://github.com/kvcache-ai/AgentENV/commit/14ac34e70b2f8051f40394d9b3e805466b09e713) fix(proxy): isolate connections across runtime generations - [`9a3bae2`](https://github.com/kvcache-ai/AgentENV/commit/9a3bae2f395e1fe9c6424c910280d21f8b60bfc4) test(proxy): clarify runtime generation simulation ### 📊 Changes **1 file changed** (+108 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `src/api/proxy.rs` (+108 -1) </details> ### 📄 Description ## Problem Sandbox interaction IPs are recycled when runtimes stop, while envd always listens on the same port. The process-wide Hyper client pools idle connections by URI authority (`IP:port`), which does not encode the sandbox runtime generation. This allows the following sequence: 1. A request to runtime A leaves an idle keep-alive connection in the pool. 2. Runtime A stops and its interaction IP is assigned to runtime B. 3. The next request for runtime B reuses A's stale TCP flow. 4. The new guest rejects that flow with an RST and Hyper returns `client error (SendRequest)`; the proxy returns 502. Retrying automatically is unsafe because proxied ConnectRPC requests can have side effects. ## Fix - Disable idle per-host pooling for the server-to-sandbox proxy client with `pool_max_idle_per_host(0)`. - Add a behavioral regression test that serves one request over a keep-alive connection, simulates the address moving to a new runtime generation, and verifies that the second request opens a fresh TCP connection. Active requests and streaming connections are unchanged; only reuse after a connection becomes idle is disabled. ## Validation The regression test was run against the current `main` before the production change and failed with: ```text proxy client reused an idle TCP connection after the runtime generation changed ``` After the fix: - `cargo test -p agentenv --lib api::proxy::tests` — 32 passed - focused regression test repeated 50 times — 50 passed - `cargo clippy -p agentenv --all-targets -- -D warnings` - `cargo fmt --all -- --check` A same-host lifecycle probe reproduced 15 initial proxy failures in 200 cycles before the fix and 0 in 1,200 cycles afterward, with no retries. Median latency remained effectively unchanged: - create: 117.18 ms → 117.25 ms - first exec: 81.26 ms → 81.46 ms --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:11 +00:00