[GH-ISSUE #413] Node drain outlives the termination grace period; server is SIGKILLed mid-shutdown #103

New issue

Open

opened 2026-10-29 14:19:37 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:37 +00:00

Owner

Originally created by @dushulin on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/413

Summary

When a busy node is drained (DaemonSet rollout or pod deletion), the AgentENV server usually never gets to finish its graceful shutdown. It is SIGKILLed while it is still pausing sandboxes.

Two things combine to cause this:

The preStop hook in deploy/k8s/base/agentenv-daemonset.yaml has no deadline, so it can use up the whole termination grace period.
run_shutdown_cleanup pauses sandboxes one at a time, so the time the server needs after SIGTERM grows with the number of running sandboxes.

The result: running sandboxes that were not paused yet are lost, and uvm-ublk-daemon is killed while its ublk devices still have live queues and guest I/O in flight. The graceful daemon shutdown added in #266 and #335 never runs.

Affected component

Orchestrator / sandbox lifecycle (and the Kubernetes manifest under deploy/k8s/base)

AgentENV version

main @ b4ee1b7

Host environment

Kubernetes DaemonSet deployed from deploy/k8s/base (terminationGracePeriodSeconds: 3600).
Linux x86_64 hosts, virtualization_mode = "kvm", Ubuntu 24.04 with the 6.8 GA kernel.
The runtime evidence below comes from a downstream deployment with the same preStop loop and shutdown code.

Operation and configuration

Rolling update (or any pod deletion) of agentenv-node while the node still has running sandboxes. The deadline problem needs sandboxes whose timeout is longer than the grace period, for example sandboxes created with a 24h timeout.

Steps to reproduce

Deploy with deploy/k8s/base.
On one node, create several sandboxes with timeout > 3600s and keep them running.
Delete that node's agentenv-node pod, or roll the DaemonSet.

Expected behavior

The node stops receiving new sandboxes. This already works: the scheduler treats a terminating endpoint as lingering and excludes it from Schedule.
preStop waits for running sandboxes to finish, but only for part of the grace period.
The server then still has enough time after SIGTERM to:
- pause and persist every sandbox that is still running;
- stop the ublk devices gracefully (#266, #335);
- exit.

Actual behavior

1. preStop holds the pod for the whole grace period. The loop waits until GET /sandboxes reports 0 and has no other exit. Sandboxes with long timeouts never finish within the hour, so preStop runs for the full 3600s. When a preStop hook overruns the grace period, kubelet grants a one-off 2s extension, sends SIGTERM, and sends SIGKILL 2s later. The server gets about 2 seconds for its whole shutdown.

2. Shutdown time is linear in the sandbox count.

On SIGTERM, src/bin/server.rs calls orchestrator.shutdown().
run_shutdown_cleanup (src/orchestrator/service.rs) pauses Running sandboxes one by one, awaiting each pause before starting the next.
UblkDeviceManager::global().shutdown_daemon() runs only after that returns.
So even when SIGTERM arrives on time, a node with N running sandboxes needs roughly N × (pause time) before the ublk daemon is asked to stop.

What SIGKILL does. It takes out the whole container cgroup at once:

the server;
uvm-ublk-daemon;
the Firecracker processes of every sandbox that was not paused yet.

Consequences

Lost sandboxes: sandboxes that were not reached are never persisted. They do not come back as Paused after the restart.
Ungraceful ublk teardown: ublk devices go away with their queue threads exiting while Firecracker still has the block devices open. This is the path #266 was meant to avoid.

Host crash risk. On kernels without torvalds/linux@f40139fde5 ("ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd", merged for 6.15), a ublk server dying with outstanding uring_cmds while block I/O is in flight can crash the host. The Ubuntu 24.04 6.8 GA kernels we checked do not carry that fix. During a rollout we hit one host panic in that path: a NULL dereference in io_fallback_tw, raised from a ublk queue thread during do_exit. That happened when the daemon was killed with devices still registered. AgentENV cannot fix the kernel, but this makes an ungraceful daemon exit far more expensive than "only" losing sandboxes.

Logs and diagnostics

Observed during a DaemonSet rollout:

Nodes holding sandboxes created with a 24h timeout kept the old pod in preStop for the full hour, logging preStop: N sandbox(es) remaining every 3s until the grace period expired.
A node that had exactly one running sandbox at expiry came back with that sandbox Paused. 2s is enough for one pause, not for many.

Reproduction frequency

Always (whenever running sandboxes outlive the grace period, or there are more of them than the server can pause one by one in the time left)

Proposed fix

preStop: stop waiting at a deadline that leaves part of the grace period to the server's own SIGTERM shutdown, for example 3000s of the 3600s, leaving 10 minutes.
run_shutdown_cleanup: preserve sandboxes with bounded concurrency instead of one at a time, so those 10 minutes cover a busy node.

I have a PR for both.

Additional context

Related: #266 and #335 make the ublk daemon shutdown graceful; this issue is about keeping that path reachable.
Different from #372, which is about a resume racing shutdown.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @dushulin on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/413 ### Summary When a busy node is drained (DaemonSet rollout or pod deletion), the AgentENV server usually never gets to finish its graceful shutdown. It is SIGKILLed while it is still pausing sandboxes. Two things combine to cause this: 1. The preStop hook in `deploy/k8s/base/agentenv-daemonset.yaml` has no deadline, so it can use up the whole termination grace period. 2. `run_shutdown_cleanup` pauses sandboxes one at a time, so the time the server needs after SIGTERM grows with the number of running sandboxes. The result: running sandboxes that were not paused yet are lost, and `uvm-ublk-daemon` is killed while its ublk devices still have live queues and guest I/O in flight. The graceful daemon shutdown added in #266 and #335 never runs. ### Affected component Orchestrator / sandbox lifecycle (and the Kubernetes manifest under `deploy/k8s/base`) ### AgentENV version `main` @ `b4ee1b7` ### Host environment - Kubernetes DaemonSet deployed from `deploy/k8s/base` (`terminationGracePeriodSeconds: 3600`). - Linux x86_64 hosts, `virtualization_mode = "kvm"`, Ubuntu 24.04 with the 6.8 GA kernel. - The runtime evidence below comes from a downstream deployment with the same preStop loop and shutdown code. ### Operation and configuration Rolling update (or any pod deletion) of `agentenv-node` while the node still has running sandboxes. The deadline problem needs sandboxes whose timeout is longer than the grace period, for example sandboxes created with a 24h timeout. ### Steps to reproduce 1. Deploy with `deploy/k8s/base`. 2. On one node, create several sandboxes with `timeout` > 3600s and keep them running. 3. Delete that node's `agentenv-node` pod, or roll the DaemonSet. ### Expected behavior - The node stops receiving new sandboxes. This already works: the scheduler treats a terminating endpoint as lingering and excludes it from `Schedule`. - preStop waits for running sandboxes to finish, but only for part of the grace period. - The server then still has enough time after SIGTERM to: - pause and persist every sandbox that is still running; - stop the ublk devices gracefully (#266, #335); - exit. ### Actual behavior **1. preStop holds the pod for the whole grace period.** The loop waits until `GET /sandboxes` reports 0 and has no other exit. Sandboxes with long timeouts never finish within the hour, so preStop runs for the full 3600s. When a preStop hook overruns the grace period, kubelet grants a one-off 2s extension, sends SIGTERM, and sends SIGKILL 2s later. The server gets about 2 seconds for its whole shutdown. **2. Shutdown time is linear in the sandbox count.** - On SIGTERM, `src/bin/server.rs` calls `orchestrator.shutdown()`. - `run_shutdown_cleanup` (`src/orchestrator/service.rs`) pauses Running sandboxes one by one, awaiting each pause before starting the next. - `UblkDeviceManager::global().shutdown_daemon()` runs only after that returns. - So even when SIGTERM arrives on time, a node with N running sandboxes needs roughly N × (pause time) before the ublk daemon is asked to stop. **What SIGKILL does.** It takes out the whole container cgroup at once: - the server; - `uvm-ublk-daemon`; - the Firecracker processes of every sandbox that was not paused yet. **Consequences** - **Lost sandboxes:** sandboxes that were not reached are never persisted. They do not come back as `Paused` after the restart. - **Ungraceful ublk teardown:** ublk devices go away with their queue threads exiting while Firecracker still has the block devices open. This is the path #266 was meant to avoid. **Host crash risk.** On kernels without torvalds/linux@f40139fde527 ("ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd", merged for 6.15), a ublk server dying with outstanding uring_cmds while block I/O is in flight can crash the host. The Ubuntu 24.04 6.8 GA kernels we checked do not carry that fix. During a rollout we hit one host panic in that path: a NULL dereference in `io_fallback_tw`, raised from a ublk queue thread during `do_exit`. That happened when the daemon was killed with devices still registered. AgentENV cannot fix the kernel, but this makes an ungraceful daemon exit far more expensive than "only" losing sandboxes. ### Logs and diagnostics Observed during a DaemonSet rollout: - Nodes holding sandboxes created with a 24h timeout kept the old pod in preStop for the full hour, logging `preStop: N sandbox(es) remaining` every 3s until the grace period expired. - A node that had exactly one running sandbox at expiry came back with that sandbox `Paused`. 2s is enough for one pause, not for many. ### Reproduction frequency Always (whenever running sandboxes outlive the grace period, or there are more of them than the server can pause one by one in the time left) ### Proposed fix - **preStop:** stop waiting at a deadline that leaves part of the grace period to the server's own SIGTERM shutdown, for example 3000s of the 3600s, leaving 10 minutes. - **`run_shutdown_cleanup`:** preserve sandboxes with bounded concurrency instead of one at a time, so those 10 minutes cover a busy node. I have a PR for both. ### Additional context - Related: #266 and #335 make the ublk daemon shutdown graceful; this issue is about keeping that path reachable. - Different from #372, which is about a resume racing shutdown. ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi referenced this issue

2026-10-29 14:20:43 +00:00

[PR #103] [MERGED] fix(sandbox): isolate envd bootstrap connections across generations #180