[PR #187] fix(k8s): bound agentenv preStop drain loop with retries and deadline #237

New issue

Open

opened 2026-10-29 14:21:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:20 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/187
Author: @epicvinny
Created: 8/18/2026
Status: 🔄 Open

Base: main ← Head: upstream/prestop-bounded-drain

📝 Commits (1)

32428cc fix(k8s): bound agentenv preStop drain loop with retries and deadline

📊 Changes

1 file changed (+20 additions, -5 deletions)

View changed files

📝 deploy/k8s/base/agentenv-daemonset.yaml (+20 -5)

📄 Description

What

Bounds the agentenv-node DaemonSet preStop drain loop in deploy/k8s/base/agentenv-daemonset.yaml:

At most 5 consecutive failures to query GET /sandboxes (with linear backoff), then the hook exits 0 — if the local API is gone there is nothing left to drain.
A global 240s deadline, after which the hook exits 0 so the kubelet can SIGTERM the agent and let its graceful shutdown persist sandboxes.
curl --max-time 5 so a hung connection cannot stall the loop.

Why

The hook polled the local API in an unbounded while true loop. When the endpoint was unreachable (or sandboxes never drained), the hook never exited and — with terminationGracePeriodSeconds: 3600 — the pod stayed Terminating for up to an hour.

Observed in staging: one agentenv-node pod stuck in Terminating, recovered only via kubectl delete pod --force --grace-period=0. The force-delete then raced the replacement pod on the shared hostPath (/var/lib/aenv) state, contributing to the RocksDB LOCK contention addressed separately in the orphan-process/lock-retry PR.

Scope and non-goals

Included: bounded retries with backoff, global drain deadline, curl timeout in the vendored base DaemonSet manifest.
Excluded: changing terminationGracePeriodSeconds or the graceful-shutdown semantics of the agent itself; the drain policy (who deletes sandboxes before termination) is unchanged.

Compatibility and operations

No API, config, or storage-format changes. Rollout only changes pod termination behavior: pods now terminate within minutes instead of hanging up to the full grace period.

Validation

YAML parses (yaml.safe_load) and the embedded script passes sh -n
Scripted dry-run of all three paths with stubbed curl/jq: clean drain (exit 0 immediately), unreachable API (5 attempts with backoff, exit 0), never-draining API (global deadline, exit 0)
make test-unit (N/A: manifest-only change, no Rust code touched)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/187 **Author:** [@epicvinny](https://github.com/epicvinny) **Created:** 8/18/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `upstream/prestop-bounded-drain` --- ### 📝 Commits (1) - [`32428cc`](https://github.com/kvcache-ai/AgentENV/commit/32428cc6aa1e420a9139b82e97325871c648f9bc) fix(k8s): bound agentenv preStop drain loop with retries and deadline ### 📊 Changes **1 file changed** (+20 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `deploy/k8s/base/agentenv-daemonset.yaml` (+20 -5) </details> ### 📄 Description ## What Bounds the `agentenv-node` DaemonSet `preStop` drain loop in `deploy/k8s/base/agentenv-daemonset.yaml`: - At most 5 consecutive failures to query `GET /sandboxes` (with linear backoff), then the hook exits 0 — if the local API is gone there is nothing left to drain. - A global 240s deadline, after which the hook exits 0 so the kubelet can SIGTERM the agent and let its graceful shutdown persist sandboxes. - `curl --max-time 5` so a hung connection cannot stall the loop. ## Why The hook polled the local API in an unbounded `while true` loop. When the endpoint was unreachable (or sandboxes never drained), the hook never exited and — with `terminationGracePeriodSeconds: 3600` — the pod stayed `Terminating` for up to an hour. Observed in staging: one `agentenv-node` pod stuck in `Terminating`, recovered only via `kubectl delete pod --force --grace-period=0`. The force-delete then raced the replacement pod on the shared `hostPath` (`/var/lib/aenv`) state, contributing to the RocksDB `LOCK` contention addressed separately in the orphan-process/lock-retry PR. ## Scope and non-goals - Included: bounded retries with backoff, global drain deadline, curl timeout in the vendored base DaemonSet manifest. - Excluded: changing `terminationGracePeriodSeconds` or the graceful-shutdown semantics of the agent itself; the drain policy (who deletes sandboxes before termination) is unchanged. ## Compatibility and operations - No API, config, or storage-format changes. Rollout only changes pod termination behavior: pods now terminate within minutes instead of hanging up to the full grace period. ## Validation - [x] YAML parses (`yaml.safe_load`) and the embedded script passes `sh -n` - [x] Scripted dry-run of all three paths with stubbed `curl`/`jq`: clean drain (exit 0 immediately), unreachable API (5 attempts with backoff, exit 0), never-draining API (global deadline, exit 0) - [ ] `make test-unit` (N/A: manifest-only change, no Rust code touched) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 14:21:20 +00:00

bsakaguchi referenced this issue

2026-10-29 14:21:46 +00:00

[PR #237] [MERGED] fix(privileges): make scoped-spawn capability tests root-aware #278