[PR #189] fix(server): reap orphaned runtime processes and retry RocksDB lock on startup #238

New issue

Open

opened 2026-10-29 14:21:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:22 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/189
Author: @epicvinny
Created: 8/18/2026
Status: 🔄 Open

Base: main ← Head: upstream/orphan-process-reap

📝 Commits (1)

da5215b fix(server): reap orphaned runtime processes and retry RocksDB lock on startup

📊 Changes

4 files changed (+414 additions, -8 deletions)

View changed files

📝 src/bin/server.rs (+6 -0)
📝 src/local_store.rs (+136 -8)
📝 src/setup/mod.rs (+8 -0)
➕ src/setup/orphans.rs (+264 -0)

📄 Description

What

Makes node-agent startup resilient to leftover runtime processes and held RocksDB locks from a previous server incarnation:

Orphan reaper (src/setup/orphans.rs, wired in src/bin/server.rs): early in startup — before the ublk daemon is spawned and before any store is opened — the server scans /proc for processes whose executable is the configured Firecracker binary or uvm-ublk-daemon binary. Any match necessarily belongs to a previous incarnation (nothing has been spawned yet), so they are terminated: SIGTERM, 5s grace, then SIGKILL. Zombie entries are ignored (they hold no resources).
RocksDB lock retry (src/local_store.rs): LocalKvStore::open now retries lock-contention errors (No locks available / Resource temporarily unavailable on the LOCK file) with exponential backoff for up to 120s instead of failing immediately. This covers the case where the lock holder lives in a different PID namespace (e.g. a previous container still terminating on the same hostPath) and therefore cannot be seen or killed by the reaper.

Why

Observed in staging: after a pod was terminated (force-deleted after the preStop hang addressed in #187), firecracker and uvm-ublk-daemon processes from the old container were still alive on the host, and the replacement pod CrashLoopBackOffed trying to open the persisted-sandbox store at /var/lib/aenv/env/persisted-sandboxes/records.db — the RocksDB LOCK was still held. Recovery required kubectl debug node/... -- chroot /host and manually killing the leaked PIDs.

Two contributing gaps in the current code:

Firecracker (src/sandbox/firecracker/instance.rs) and the ublk daemon (storage/ublk-daemon/src/client.rs) are spawned in their own process groups (process_group(0)). Graceful teardown (orchestrator pause-all, pool shutdown, daemon shutdown) only runs on SIGTERM/Ctrl+C; on SIGKILL/OOM/force-delete nothing runs and the children persist with their resources (ublk devices, open image files, inherited fds).
LocalKvStore::open made a single DB::open attempt, so a still-terminating previous holder turned startup into an immediate fatal error → CrashLoopBackOff.

Scope and non-goals

Included: startup orphan reaper for the two runtime binaries, lock-contention retry in the shared LocalKvStore::open (benefits all three callers: persisted-sandbox records, image cache graph, P2P catalog), unit tests for both.
Excluded: changing the graceful-shutdown path itself (it already stops sandboxes, the warm pool, and the daemon); any changes to pool/manager code touched by #181/#182; PID-namespace or cgroup changes in the deployment manifests.

Compatibility and operations

No API, config, or storage-format changes. The reaper only signals processes whose executable matches the binaries this server is configured to use, and only within its own PID namespace. The lock retry is a no-op when the database opens cleanly on the first attempt.
Rollback: safe both ways; no persisted state involved.

Validation

cargo test -p agentenv --lib local_store (3 passed, incl. retry-until-release and budget-exhaustion cases with a real held RocksDB lock)
cargo test -p agentenv --lib setup::orphans (4 passed: exe matching, (deleted) binary suffix, non-numeric/missing exe entries, zombie state parsing)
cargo clippy -p agentenv --lib --bins -- -D warnings
cargo fmt --check on touched files
Relevant Rust integration tests (require root + KVM host)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/189 **Author:** [@epicvinny](https://github.com/epicvinny) **Created:** 8/18/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `upstream/orphan-process-reap` --- ### 📝 Commits (1) - [`da5215b`](https://github.com/kvcache-ai/AgentENV/commit/da5215bd317e6d3516b2922dc872640618d7d303) fix(server): reap orphaned runtime processes and retry RocksDB lock on startup ### 📊 Changes **4 files changed** (+414 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `src/bin/server.rs` (+6 -0) 📝 `src/local_store.rs` (+136 -8) 📝 `src/setup/mod.rs` (+8 -0) ➕ `src/setup/orphans.rs` (+264 -0) </details> ### 📄 Description ## What Makes node-agent startup resilient to leftover runtime processes and held RocksDB locks from a previous server incarnation: 1. **Orphan reaper** (`src/setup/orphans.rs`, wired in `src/bin/server.rs`): early in startup — before the ublk daemon is spawned and before any store is opened — the server scans `/proc` for processes whose executable is the configured Firecracker binary or `uvm-ublk-daemon` binary. Any match necessarily belongs to a previous incarnation (nothing has been spawned yet), so they are terminated: SIGTERM, 5s grace, then SIGKILL. Zombie entries are ignored (they hold no resources). 2. **RocksDB lock retry** (`src/local_store.rs`): `LocalKvStore::open` now retries lock-contention errors (`No locks available` / `Resource temporarily unavailable` on the `LOCK` file) with exponential backoff for up to 120s instead of failing immediately. This covers the case where the lock holder lives in a *different* PID namespace (e.g. a previous container still terminating on the same hostPath) and therefore cannot be seen or killed by the reaper. ## Why Observed in staging: after a pod was terminated (force-deleted after the preStop hang addressed in #187), `firecracker` and `uvm-ublk-daemon` processes from the old container were still alive on the host, and the replacement pod CrashLoopBackOffed trying to open the persisted-sandbox store at `/var/lib/aenv/env/persisted-sandboxes/records.db` — the RocksDB `LOCK` was still held. Recovery required `kubectl debug node/... -- chroot /host` and manually killing the leaked PIDs. Two contributing gaps in the current code: - Firecracker (`src/sandbox/firecracker/instance.rs`) and the ublk daemon (`storage/ublk-daemon/src/client.rs`) are spawned in their own process groups (`process_group(0)`). Graceful teardown (orchestrator pause-all, pool shutdown, daemon shutdown) only runs on SIGTERM/Ctrl+C; on SIGKILL/OOM/force-delete nothing runs and the children persist with their resources (ublk devices, open image files, inherited fds). - `LocalKvStore::open` made a single `DB::open` attempt, so a still-terminating previous holder turned startup into an immediate fatal error → CrashLoopBackOff. ## Scope and non-goals - Included: startup orphan reaper for the two runtime binaries, lock-contention retry in the shared `LocalKvStore::open` (benefits all three callers: persisted-sandbox records, image cache graph, P2P catalog), unit tests for both. - Excluded: changing the graceful-shutdown path itself (it already stops sandboxes, the warm pool, and the daemon); any changes to pool/manager code touched by #181/#182; PID-namespace or cgroup changes in the deployment manifests. ## Compatibility and operations - No API, config, or storage-format changes. The reaper only signals processes whose executable matches the binaries this server is configured to use, and only within its own PID namespace. The lock retry is a no-op when the database opens cleanly on the first attempt. - Rollback: safe both ways; no persisted state involved. ## Validation - [x] `cargo test -p agentenv --lib local_store` (3 passed, incl. retry-until-release and budget-exhaustion cases with a real held RocksDB lock) - [x] `cargo test -p agentenv --lib setup::orphans` (4 passed: exe matching, `(deleted)` binary suffix, non-numeric/missing exe entries, zombie state parsing) - [x] `cargo clippy -p agentenv --lib --bins -- -D warnings` - [x] `cargo fmt --check` on touched files - [ ] Relevant Rust integration tests (require root + KVM host) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 14:21:22 +00:00

bsakaguchi referenced this issue

2026-10-29 14:21:46 +00:00

[PR #238] feat(sandbox): bind running Firecracker threads to host CPUs #279