[GH-ISSUE #122] Idle busy-spin in kfd_wait_on_events (~80k ioctls/s, one core at 100%) on gfx1151 APU with 0.5.5; fixed by 0.5.6 upgrade #46

New issue

Closed

opened 2026-10-26 22:24:03 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:24:03 +00:00

Owner

Originally created by @LeonJi on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/122

Environment

Hardware: AMD Ryzen AI Max+ 395, iGPU gfx1151 (Strix Halo), 128 GB unified memory, no dGPU
OS: Linux, kernel 7.0.0-34-generic
torch 2.14.1+rocm7.2 (HIP 7.2.53211), Python 3.12.3
ComfyUI core e8665ca (2026-10-13), launched with --reserve-vram 32 --cache-ram 4 8 --highvram
comfy-aimdo 0.5.5 (reproduces) → 0.5.6 (no longer reproduces)

Symptom (comfy-aimdo 0.5.5)

Idle ComfyUI service burned ~100% of one full CPU core (~82% of it in kernel time) continuously for 5+ hours. A native thread created ~2-3 s after process start (thread comm inherited as "python", invisible to py-spy) was hot:

strace -c -p <tid>: 100% of samples in ioctl, ~80,000 ioctls/sec on /dev/kfd
perf record: [k] kfd_wait_on_events + [k] __raw_spin_lock_irqsave dominating
The spin started essentially right after startup (process CPU-time delta ≈ wall-clock), not tied to any running prompt
Occurred with and without DynamicVRAM active in the ComfyUI sense (--highvram was set, but fast_disk/model-mmap paths were loaded)

Result after upgrading to 0.5.6

Idle CPU 0%. I then re-ran the same full model matrix that used to sit next to the spinning service:

workload	result
Stable Audio 3 small_sfx + t5gemma TE (~6.6 GB)	idle stayed 0%
ACE-Step 1.5 XL turbo (~20 GB: UNet 10.3 GB + qwen_4b/qwen_0.6b TE + VAE)	idle stayed 0%
Breeze TTS 2 int8-hybrid	idle stayed 0%
Demucs htdemucs	idle stayed 0%
delayed monitor, 30 s samples × 16 min after the 20 GB ACE run	all 0%

So the 0.5.5 → 0.5.6 upgrade resolved this for me.

Why this is maybe still worth tracking

The hot thread was polling kfd_wait_on_events in a tight loop with no visible backoff on this APU — if the 0.5.6 fix simply switched that loop to sleep/interrupt mode, it would be good to confirm, since Strix Halo's KFD event behavior under interrupt-less polling has bitten other projects too.
On older cores (and per the other open issues here) similar "burns a core while idle" stories may go unreported because users don't correlate idle CPU with aimdo.

Questions

Is the 0.5.6 changelog intentionally covering this kfd busy-wait fix (I could not find release notes on PyPI)?
Is there a knob for the vbar/vram monitor poll interval for systems where KFD event interrupts are unreliable (APUs, SR-IOV guests)?

Originally created by @LeonJi on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/122 ## Environment - Hardware: AMD Ryzen AI Max+ 395, iGPU `gfx1151` (Strix Halo), 128 GB unified memory, no dGPU - OS: Linux, kernel 7.0.0-34-generic - torch 2.14.1+rocm7.2 (HIP 7.2.53211), Python 3.12.3 - ComfyUI core `e8665ca` (2026-10-13), launched with `--reserve-vram 32 --cache-ram 4 8 --highvram` - comfy-aimdo **0.5.5** (reproduces) → **0.5.6** (no longer reproduces) ## Symptom (comfy-aimdo 0.5.5) Idle ComfyUI service burned ~100% of one full CPU core (~82% of it in kernel time) continuously for 5+ hours. A native thread created ~2-3 s after process start (thread `comm` inherited as "python", invisible to py-spy) was hot: - `strace -c -p <tid>`: 100% of samples in `ioctl`, ~80,000 ioctls/sec on `/dev/kfd` - `perf record`: `[k] kfd_wait_on_events` + `[k] __raw_spin_lock_irqsave` dominating - The spin started essentially right after startup (process CPU-time delta ≈ wall-clock), not tied to any running prompt - Occurred with and without DynamicVRAM active in the ComfyUI sense (`--highvram` was set, but fast_disk/model-mmap paths were loaded) ## Result after upgrading to 0.5.6 Idle CPU 0%. I then re-ran the same full model matrix that used to sit next to the spinning service: | workload | result | |---|---| | Stable Audio 3 small_sfx + t5gemma TE (~6.6 GB) | idle stayed 0% | | ACE-Step 1.5 XL turbo (~20 GB: UNet 10.3 GB + qwen_4b/qwen_0.6b TE + VAE) | idle stayed 0% | | Breeze TTS 2 int8-hybrid | idle stayed 0% | | Demucs htdemucs | idle stayed 0% | | delayed monitor, 30 s samples × 16 min after the 20 GB ACE run | all 0% | So the 0.5.5 → 0.5.6 upgrade resolved this for me. ## Why this is maybe still worth tracking 1. The hot thread was polling `kfd_wait_on_events` in a tight loop with no visible backoff on this APU — if the 0.5.6 fix simply switched that loop to sleep/interrupt mode, it would be good to confirm, since Strix Halo's KFD event behavior under interrupt-less polling has bitten other projects too. 2. On older cores (and per the other open issues here) similar "burns a core while idle" stories may go unreported because users don't correlate idle CPU with aimdo. ## Questions - Is the 0.5.6 changelog intentionally covering this kfd busy-wait fix (I could not find release notes on PyPI)? - Is there a knob for the vbar/vram monitor poll interval for systems where KFD event interrupts are unreliable (APUs, SR-IOV guests)?

bsakaguchi closed this issue

2026-10-26 22:24:03 +00:00

bsakaguchi commented

2026-10-26 22:24:04 +00:00

Author

Owner

@LeonJi commented on GitHub (Oct 12, 2026):

Closing — my own follow-up investigation shows the 0.5.5 vs 0.5.6 delta is only the NULL-guard refactor in physical_page_unref (identical export sets, +72 byte .so), i.e. nothing on the event-wait path. The idle spin was most likely armed by runtime state rather than the version. I'm running a controlled replay (load-failure churn on 0.5.6) and will reopen with exact repro steps if it reproduces.

@LeonJi commented on GitHub (Oct 12, 2026): Closing — my own follow-up investigation shows the 0.5.5 vs 0.5.6 delta is only the NULL-guard refactor in `physical_page_unref` (identical export sets, +72 byte .so), i.e. nothing on the event-wait path. The idle spin was most likely armed by runtime state rather than the version. I'm running a controlled replay (load-failure churn on 0.5.6) and will reopen with exact repro steps if it reproduces.

bsakaguchi referenced this issue

2026-10-26 22:24:25 +00:00

[PR #46] [MERGED] Implement threaded loader components (CORE-43) #85