[PR #65] [MERGED] perf: stabilize concurrent snapshot resume #150

New issue

Closed

opened 2026-10-29 14:20:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/65
Author: @yingdi-shan
Created: 7/29/2026
Status: ✅ Merged
Merged: 7/31/2026
Merged by: @yingdi-shan

Base: main ← Head: fix/firecracker-spawn-pre-exec

📝 Commits (1)

fcbd5a4 perf: stabilize concurrent snapshot resume

📊 Changes

8 files changed (+496 additions, -145 deletions)

View changed files

📝 crates/benchmarks/benches/snapshot_benchmark.rs (+56 -23)
📝 crates/warm-pool/src/lib.rs (+61 -7)
📝 src/privileges.rs (+238 -19)
📝 src/sandbox/envd.rs (+53 -4)
📝 src/sandbox/firecracker/instance.rs (+24 -25)
📝 src/sandbox/network/iptables_util.rs (+26 -29)
📝 src/sandbox/network/manager.rs (+6 -6)
📝 src/sandbox/network/slot.rs (+32 -32)

📄 Description

What

Stabilize high-concurrency snapshot resume by removing pre_exec hooks from Firecracker and privileged network command launches.

Also harden envd readiness checks, make warm-pool misses drive adaptive refill, and improve the concurrent resume benchmark warm-up and diagnostics.

Why

Firecracker currently uses pre_exec hooks to enter its network namespace and drop capabilities. These hooks make Rust Command ineligible for the posix_spawn fast path, forcing every launch through fork + exec.

AgentENV is a large, multithreaded process. Although fork uses copy-on-write, it must still duplicate process metadata and page tables. Starting 50 Firecracker processes concurrently amplifies that work and creates substantial contention and tail latency before the children reach exec.

Network slot reuse introduces two additional sources of variance:

an idle envd HTTP connection can still refer to the previous VM that owned the reused IP and stall until TCP detects the dead peer
warm-pool acquisition misses previously returned before increasing refill pressure, so a burst starting from an empty pool did not grow future capacity

Measured with make bench-snapshot using the concurrent_resume workload with 50 concurrent workers and 10 measured rounds:

Metric	Before	After	Improvement
Mean resume latency	851.81 ms	86.70 ms	9.8x faster / 89.8% lower
Maximum observed latency	2383.73 ms	166.72 ms	14.3x faster / 93.0% lower

These are end-to-end results for the complete patch and do not attribute the improvement to individual changes.

N/A - no linked issue yet.

Scope and non-goals

Included:

Firecracker and privileged network subprocess launch behavior
envd readiness connection and timeout handling
adaptive warm-pool refill behavior
concurrent snapshot resume benchmark behavior

Non-goals:

API, configuration, snapshot format, or storage layout changes
changes to configured pool watermarks or benchmark concurrency

Design and behavior changes

Launch Firecracker from a short-lived thread that enters the target network namespace, scopes its capabilities, and then uses normal command spawning. With no pre_exec callback, the command remains eligible for posix_spawn.
Pass the spawned child back through a one-shot channel. If the receiver is dropped, kill and reap the unclaimed child.
Run synchronous privileged network commands on short-lived threads with an exact capability set, leaving the calling server thread unchanged.
Disable idle envd connection reuse and bound each health probe to one second or the remaining readiness deadline, whichever is shorter.
Record acquisition pressure on warm-pool misses, request maintenance when a refill is needed, and prime the concurrent benchmark with one matching burst.

Compatibility and operations

Public API or generated protocol: N/A; no changes.
Configuration or defaults: N/A; no changes.
Snapshot manifest, artifact layout, or storage format: N/A; no changes.
Upgrade and rollback: no migration required; rollback only requires reverting to the previous binary.
Host requirements, permissions, ports, or dependencies: unchanged; existing Linux capability and network namespace requirements still apply.

Validation

Benchmarks or performance comparison completed

make bench-snapshot

concurrent_resume:
851.81 ms mean / 2383.73 ms max
  -> 86.70 ms mean / 166.72 ms max

Risks and reviewer notes

Each privileged subprocess launch creates a short-lived OS thread.
Disabling envd keep-alive adds connection setup, but avoids stale connections across reused network-slot IPs.
Repeated pool misses can raise the refill target faster and temporarily increase prewarmed resource usage.
Review should focus on capability inheritance, namespace isolation, and cleanup of children whose launch result is no longer claimed.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/65 **Author:** [@yingdi-shan](https://github.com/yingdi-shan) **Created:** 7/29/2026 **Status:** ✅ Merged **Merged:** 7/31/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `fix/firecracker-spawn-pre-exec` --- ### 📝 Commits (1) - [`fcbd5a4`](https://github.com/kvcache-ai/AgentENV/commit/fcbd5a4f29f1a3ab089a364830aec3f3bbd5503e) perf: stabilize concurrent snapshot resume ### 📊 Changes **8 files changed** (+496 additions, -145 deletions) <details> <summary>View changed files</summary> 📝 `crates/benchmarks/benches/snapshot_benchmark.rs` (+56 -23) 📝 `crates/warm-pool/src/lib.rs` (+61 -7) 📝 `src/privileges.rs` (+238 -19) 📝 `src/sandbox/envd.rs` (+53 -4) 📝 `src/sandbox/firecracker/instance.rs` (+24 -25) 📝 `src/sandbox/network/iptables_util.rs` (+26 -29) 📝 `src/sandbox/network/manager.rs` (+6 -6) 📝 `src/sandbox/network/slot.rs` (+32 -32) </details> ### 📄 Description ## What Stabilize high-concurrency snapshot resume by removing `pre_exec` hooks from Firecracker and privileged network command launches. Also harden envd readiness checks, make warm-pool misses drive adaptive refill, and improve the concurrent resume benchmark warm-up and diagnostics. ## Why Firecracker currently uses `pre_exec` hooks to enter its network namespace and drop capabilities. These hooks make Rust `Command` ineligible for the `posix_spawn` fast path, forcing every launch through `fork` + `exec`. AgentENV is a large, multithreaded process. Although `fork` uses copy-on-write, it must still duplicate process metadata and page tables. Starting 50 Firecracker processes concurrently amplifies that work and creates substantial contention and tail latency before the children reach `exec`. Network slot reuse introduces two additional sources of variance: - an idle envd HTTP connection can still refer to the previous VM that owned the reused IP and stall until TCP detects the dead peer - warm-pool acquisition misses previously returned before increasing refill pressure, so a burst starting from an empty pool did not grow future capacity Measured with `make bench-snapshot` using the `concurrent_resume` workload with 50 concurrent workers and 10 measured rounds: | Metric | Before | After | Improvement | | --- | ---: | ---: | ---: | | Mean resume latency | 851.81 ms | 86.70 ms | 9.8x faster / 89.8% lower | | Maximum observed latency | 2383.73 ms | 166.72 ms | 14.3x faster / 93.0% lower | These are end-to-end results for the complete patch and do not attribute the improvement to individual changes. ## Related issue N/A - no linked issue yet. ## Scope and non-goals Included: - Firecracker and privileged network subprocess launch behavior - envd readiness connection and timeout handling - adaptive warm-pool refill behavior - concurrent snapshot resume benchmark behavior Non-goals: - API, configuration, snapshot format, or storage layout changes - changes to configured pool watermarks or benchmark concurrency ## Design and behavior changes - Launch Firecracker from a short-lived thread that enters the target network namespace, scopes its capabilities, and then uses normal command spawning. With no `pre_exec` callback, the command remains eligible for `posix_spawn`. - Pass the spawned child back through a one-shot channel. If the receiver is dropped, kill and reap the unclaimed child. - Run synchronous privileged network commands on short-lived threads with an exact capability set, leaving the calling server thread unchanged. - Disable idle envd connection reuse and bound each health probe to one second or the remaining readiness deadline, whichever is shorter. - Record acquisition pressure on warm-pool misses, request maintenance when a refill is needed, and prime the concurrent benchmark with one matching burst. ## Compatibility and operations - Public API or generated protocol: N/A; no changes. - Configuration or defaults: N/A; no changes. - Snapshot manifest, artifact layout, or storage format: N/A; no changes. - Upgrade and rollback: no migration required; rollback only requires reverting to the previous binary. - Host requirements, permissions, ports, or dependencies: unchanged; existing Linux capability and network namespace requirements still apply. ## Validation - [x] Benchmarks or performance comparison completed ```text make bench-snapshot concurrent_resume: 851.81 ms mean / 2383.73 ms max -> 86.70 ms mean / 166.72 ms max ``` ## Risks and reviewer notes - Each privileged subprocess launch creates a short-lived OS thread. - Disabling envd keep-alive adds connection setup, but avoids stale connections across reused network-slot IPs. - Repeated pool misses can raise the refill target faster and temporarily increase prewarmed resource usage. - Review should focus on capability inheritance, namespace isolation, and cleanup of children whose launch result is no longer claimed. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:16 +00:00