[GH-ISSUE #312] bug(network): Debian ifupdown hotplug triggers ifquery process storm during large-scale sandbox churn, inflating pause/resume latency #83

New issue

Closed

opened 2026-10-29 14:19:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:21 +00:00

Owner

Originally created by @helloC-A on GitHub (Sep 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/312

Affected component

Network

AgentENV version

Version or commit: 2f48c9c (kvcache-ai/AgentENV main)
Local modifications: none required to reproduce — the storm is observable on stock main on Debian hosts

Host environment

OS: Debian GNU/Linux 12 (bookworm)
Kernel (uname -a): Linux 6.6.95+ #5 SMP Tue Aug 4 14:23:38 CST 2026 aarch64 GNU/Linux
Architecture: aarch64 (ARM64)
Bare metal, VM, or cloud: bare metal (systemd-detect-virt: none)
ublk support: ublk_drv module loaded and in active use (lsblk shows ~30 live ublkb* sandbox drives, 4G/64G)
CPU: 2 sockets × 96 cores × 2 threads = 384 logical CPUs (aarch64; lscpu reports no model name on this platform)
Memory: 1.5 TiB
Storage/filesystem: NVMe SSD (3× 2.9T + 1× 14T, all non-rotational)

Operation and configuration

Host provisioning and server start:

  sudo env AENV_HOME_PATH=/home/aenv API_ADDR=0.0.0.0:8000 AENV_RUN_USER=root \
       ./target/release/server --setup-host

Load test: an internal driver creates, pauses, resumes, and destroys hundreds of
sandboxes concurrently (thousands of lifecycle operations; each sandbox has a
dedicated veth pair on the host, prefix "veth-{slot}").

Observation command:

  watch -n 1 'ps -eo comm | grep -c "^ifquery$"'

Steps to reproduce

On a Debian host with the ifupdown package installed, confirm the hotplug rule is active:
cat /lib/udev/rules.d/85-ifupdown.rules
(SUBSYSTEM=="net", ACTION=="add|remove" -> RUN ifupdown-hotplug)
Provision and start AgentENV as shown in "Operation and configuration".
Drive concurrent sandbox lifecycle churn (create/destroy of hundreds of sandboxes,
or run a load test with repeated pause/resume cycles across many sandboxes).
While churn is running, observe host processes:
watch -n 1 'ps -eo comm | grep -c "^ifquery$"'
Compare sandbox pause/resume latency under churn against a quiet host.

Expected behavior

Creating or deleting an AENV NIC on the host should not spawn per-event helper
processes from unrelated subsystems: AENV NICs are not listed in
/etc/network/interfaces, so ifupdown has nothing to do for them.

Under large-scale sandbox churn, host CPU should remain available for sandbox
lifecycle operations, and pause/resume latency should stay at its intrinsic
level (tens of milliseconds).

Actual behavior

Every NIC add/remove udev event invokes /lib/udev/ifupdown-hotplug, which forks
ifquery once per event to reach the conclusion "not managed" — a check whose
outcome is always "no", yet costs a full process spawn (~3 forks per event:
udevd worker + hotplug shell + ifquery).

During churn, concurrent ifquery counts grow monotonically (arrival rate >
service rate) instead of staying flat:

110 -> 110 -> 198 -> 342 -> ...   (watch -n 1, seconds apart)

The storm saturates host CPUs and inflates sandbox lifecycle operations that
are short and host-CPU-mediated. Load test results, same workload before and
after suppressing the storm (n = 22434 pause/resume cycles):

Metric	Storm present	Storm suppressed	Change
pause_ms mean / min	1595.2 / 421.3	940.3 / 56.9	-41% / -86%
resume_ms mean / p95	1493.4 / 2469.4	662.1 / 1114.7	-56% / -55%
pause+resume per cycle (ms)	3088.7	1602.4	-48%
exec_ms / think_ms	unchanged	unchanged	—

No error is logged; the symptom is process-count growth and latency inflation.

Logs and diagnostics

ifquery process count during sandbox churn (1s interval):

$ watch -n 1 'ps -eo comm | grep -c "^ifquery$"'
110
110
198
342

Per-step timing aggregates across the load test (cross-task, n = 22434 cycles):

Before fix:
exec_ms   n=22434  mean 1635.3  min 17.5  max 14384.1  p95 10006.6
think_ms  n=22434  mean 8666.5  min 3000.1 max 13673.1 p95 10000.2
pause_ms  n=22434  mean 1595.2  min 421.3  max 15098.3  p95 2990.6
resume_ms n=22434  mean 1493.4  min 282.7  max 4891.8   p95 2469.4
pause+resume per cycle: 3088.7 ms

After fix:
exec_ms   n=22434  mean 1722.0  min 18.3  max 13591.5  p95 10004.4
think_ms  n=22434  mean 8637.7  min 3000.1 max 10011.5  p95 9979.0
pause_ms  n=22434  mean 940.3   min 56.9   max 13981.6  p95 2493.5
resume_ms n=22434  mean 662.1   min 92.0   max 1946.2   p95 1114.7
pause+resume per cycle: 1602.4 ms

Reproduction frequency

Always

Regression information

N/A — not a regression. The storm is inherent to how AENV NIC events interact
with Debian's ifupdown hotplug integration and is observable on stock main.
No known-good release exists where Debian hosts were unaffected.

Additional context

Root cause

Debian's ifupdown package ships a udev rule that runs /lib/udev/ifupdown-hotplug
for every net device add/remove. Its net_ifup() runs
ifquery --allow hotplug -l "$INTERFACE" to decide whether the interface is
hotplug-managed — the check happens before the judgment. AENV NICs are never in
/etc/network/interfaces, so the answer is always "no", but ifquery is fully
spawned each time. At sandbox scale this becomes a process storm: pause/resume
are short host-CPU-mediated operations (API socket round trips, process_vm_readv
snapshot copies, fork/exec on resume), and per-step scheduling queueing under
hundreds of runnable processes adds hundreds of ms per cycle.

Other distributions are unaffected: NetworkManager / systemd-networkd handle net
events in-process without forking external check processes. This is why the issue
is Debian-specific.

Proposed fix (design check before PR, per contributing guidelines)

Rename the host-side veth prefix veth- -> aenv- so AENV NICs are uniquely
identifiable (no false match with Docker/K8s NICs).
Add aenv-* to the existing early-exit whitelists in ifupdown-hotplug (both
add and remove paths). This whitelist already exists for NICs managed by other
software (ppp*, lo, ipsec*), so this is its intended use — ifupdown's effective
behavior for these NICs is unchanged (it did nothing before, it does nothing
now; only the wasteful fork is skipped).
Guard the patched script with dpkg-divert (no --rename) so ifupdown package
upgrades cannot overwrite it; keep a backup beside the script. Everything is
applied by --setup-host, idempotent, no manual steps.
AENV configures its NICs itself (netlink + ip= boot arg), so nothing else
depends on ifupdown for these interfaces.

Alternatives evaluated and rejected: OPTIONS+="ignore_device" (removed in udev
148, silently ignored), GOTO-skip udev rules (system rule override / upgrade
conflicts), disabling ifupdown hotplug globally (breaks other services),
whitelisting veth-* (collides with Docker/K8s).

With the fix applied, ifquery counts stay in single digits during the same churn,
and pause+resume per cycle improves from 3088.7 ms to 1602.4 ms (-48%, n=22434).

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @helloC-A on GitHub (Sep 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/312 ### Affected component Network ### AgentENV version - Version or commit: 2f48c9c (kvcache-ai/AgentENV main) - Local modifications: none required to reproduce — the storm is observable on stock main on Debian hosts ### Host environment - OS: Debian GNU/Linux 12 (bookworm) - Kernel (`uname -a`): Linux 6.6.95+ #5 SMP Tue Aug 4 14:23:38 CST 2026 aarch64 GNU/Linux - Architecture: aarch64 (ARM64) - Bare metal, VM, or cloud: bare metal (systemd-detect-virt: none) - ublk support: ublk_drv module loaded and in active use (lsblk shows ~30 live ublkb* sandbox drives, 4G/64G) - CPU: 2 sockets × 96 cores × 2 threads = 384 logical CPUs (aarch64; lscpu reports no model name on this platform) - Memory: 1.5 TiB - Storage/filesystem: NVMe SSD (3× 2.9T + 1× 14T, all non-rotational) ### Operation and configuration ```shell Host provisioning and server start: sudo env AENV_HOME_PATH=/home/aenv API_ADDR=0.0.0.0:8000 AENV_RUN_USER=root \ ./target/release/server --setup-host Load test: an internal driver creates, pauses, resumes, and destroys hundreds of sandboxes concurrently (thousands of lifecycle operations; each sandbox has a dedicated veth pair on the host, prefix "veth-{slot}"). Observation command: watch -n 1 'ps -eo comm | grep -c "^ifquery$"' ``` ### Steps to reproduce 1. On a Debian host with the `ifupdown` package installed, confirm the hotplug rule is active: `cat /lib/udev/rules.d/85-ifupdown.rules` (SUBSYSTEM=="net", ACTION=="add|remove" -> RUN ifupdown-hotplug) 2. Provision and start AgentENV as shown in "Operation and configuration". 3. Drive concurrent sandbox lifecycle churn (create/destroy of hundreds of sandboxes, or run a load test with repeated pause/resume cycles across many sandboxes). 4. While churn is running, observe host processes: `watch -n 1 'ps -eo comm | grep -c "^ifquery$"'` 5. Compare sandbox pause/resume latency under churn against a quiet host. ### Expected behavior Creating or deleting an AENV NIC on the host should not spawn per-event helper processes from unrelated subsystems: AENV NICs are not listed in /etc/network/interfaces, so ifupdown has nothing to do for them. Under large-scale sandbox churn, host CPU should remain available for sandbox lifecycle operations, and pause/resume latency should stay at its intrinsic level (tens of milliseconds). ### Actual behavior Every NIC add/remove udev event invokes /lib/udev/ifupdown-hotplug, which forks ifquery once per event to reach the conclusion "not managed" — a check whose outcome is always "no", yet costs a full process spawn (~3 forks per event: udevd worker + hotplug shell + ifquery). During churn, concurrent ifquery counts grow monotonically (arrival rate > service rate) instead of staying flat: 110 -> 110 -> 198 -> 342 -> ... (watch -n 1, seconds apart) The storm saturates host CPUs and inflates sandbox lifecycle operations that are short and host-CPU-mediated. Load test results, same workload before and after suppressing the storm (n = 22434 pause/resume cycles): | Metric | Storm present | Storm suppressed | Change | |----------------------------|---------------|------------------|----------| | pause_ms mean / min | 1595.2 / 421.3 | 940.3 / 56.9 | -41% / -86% | | resume_ms mean / p95 | 1493.4 / 2469.4 | 662.1 / 1114.7 | -56% / -55% | | pause+resume per cycle (ms)| 3088.7 | 1602.4 | -48% | | exec_ms / think_ms | unchanged | unchanged | — | No error is logged; the symptom is process-count growth and latency inflation. ### Logs and diagnostics ```text ifquery process count during sandbox churn (1s interval): $ watch -n 1 'ps -eo comm | grep -c "^ifquery$"' 110 110 198 342 Per-step timing aggregates across the load test (cross-task, n = 22434 cycles): Before fix: exec_ms n=22434 mean 1635.3 min 17.5 max 14384.1 p95 10006.6 think_ms n=22434 mean 8666.5 min 3000.1 max 13673.1 p95 10000.2 pause_ms n=22434 mean 1595.2 min 421.3 max 15098.3 p95 2990.6 resume_ms n=22434 mean 1493.4 min 282.7 max 4891.8 p95 2469.4 pause+resume per cycle: 3088.7 ms After fix: exec_ms n=22434 mean 1722.0 min 18.3 max 13591.5 p95 10004.4 think_ms n=22434 mean 8637.7 min 3000.1 max 10011.5 p95 9979.0 pause_ms n=22434 mean 940.3 min 56.9 max 13981.6 p95 2493.5 resume_ms n=22434 mean 662.1 min 92.0 max 1946.2 p95 1114.7 pause+resume per cycle: 1602.4 ms ``` ### Reproduction frequency Always ### Regression information N/A — not a regression. The storm is inherent to how AENV NIC events interact with Debian's ifupdown hotplug integration and is observable on stock main. No known-good release exists where Debian hosts were unaffected. ### Additional context ### Root cause Debian's ifupdown package ships a udev rule that runs /lib/udev/ifupdown-hotplug for every net device add/remove. Its net_ifup() runs `ifquery --allow hotplug -l "$INTERFACE"` to decide whether the interface is hotplug-managed — the check happens before the judgment. AENV NICs are never in /etc/network/interfaces, so the answer is always "no", but ifquery is fully spawned each time. At sandbox scale this becomes a process storm: pause/resume are short host-CPU-mediated operations (API socket round trips, process_vm_readv snapshot copies, fork/exec on resume), and per-step scheduling queueing under hundreds of runnable processes adds hundreds of ms per cycle. Other distributions are unaffected: NetworkManager / systemd-networkd handle net events in-process without forking external check processes. This is why the issue is Debian-specific. ### Proposed fix (design check before PR, per contributing guidelines) 1. Rename the host-side veth prefix `veth-` -> `aenv-` so AENV NICs are uniquely identifiable (no false match with Docker/K8s NICs). 2. Add `aenv-*` to the existing early-exit whitelists in ifupdown-hotplug (both add and remove paths). This whitelist already exists for NICs managed by other software (ppp*, lo, ipsec*), so this is its intended use — ifupdown's effective behavior for these NICs is unchanged (it did nothing before, it does nothing now; only the wasteful fork is skipped). 3. Guard the patched script with dpkg-divert (no --rename) so ifupdown package upgrades cannot overwrite it; keep a backup beside the script. Everything is applied by --setup-host, idempotent, no manual steps. 4. AENV configures its NICs itself (netlink + ip= boot arg), so nothing else depends on ifupdown for these interfaces. Alternatives evaluated and rejected: `OPTIONS+="ignore_device"` (removed in udev 148, silently ignored), GOTO-skip udev rules (system rule override / upgrade conflicts), disabling ifupdown hotplug globally (breaks other services), whitelisting `veth-*` (collides with Docker/K8s). With the fix applied, ifquery counts stay in single digits during the same churn, and pause+resume per cycle improves from 3088.7 ms to 1602.4 ms (-48%, n=22434). ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:19:21 +00:00