s #51

New issue

Open

opened 2026-10-26 22:24:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:24:04 +00:00

Owner

Originally created by @fengyangsi on GitHub (Oct 24, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/131

Summary

Two distinct problems on an AMD Radeon 890M (gfx1150, Strix Point APU) under ROCm 7.14:

DynamicVRAM enabled (default): the first CLIPTextEncode hangs forever inside lib.vbar_fault().
With --disable-dynamic-vram there is no hang, but a 6.5 GB SDXL checkpoint takes ~6 minutes to load, because HIP routes H2D copies of mmap-backed safetensors pages through host pinning.

Both were diagnosed by attaching gdb to the live backend.

Environment


ComfyUI	0.42.5 (Desktop standalone, linux-amd bundle v0.42.1-env1)
comfy-aimdo	0.5.6
PyTorch	2.12.0+rocm7.14.0
ROCm	7.14
GPU	AMD Radeon 890M, gfx1150, RDNA3.5, integrated=1, XNACK disabled
CPU / RAM	Ryzen AI 9 HX 370, 46 GB shared
OS	openSUSE Leap 16.0, kernel 6.12.0
Model	SDXL checkpoint, 6.5 GB, 1024x1024, 32 steps

Reproduced with a minimal core-only workflow (CheckpointLoaderSimple -> CLIPTextEncode -> KSampler -> VAEDecode -> SaveImage), no custom nodes involved.

Problem 1 - DynamicVRAM hangs inside vbar_fault()

600 s, never completes. Last log line:

Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached. Force pre-loaded 180 weights: 400 KB.

Execution thread, Python stack:

model_vbar.py:88    fault
model_vbar.py:137   vbar_fault
comfy/ops.py:184    cast_modules_with_vbar
comfy/ops.py:400    cast_bias_weight
comfy/ops.py:489    __init__
comfy/ops.py:821    forward_comfy_cast_weights
comfy/sd1_clip.py:213  process_tokens

Same thread, native stack:

#11 hipMemSetAccess ()  from libamdhip64.so.7
#12 three_stooges (size=33554432)  at /project/src/plat.h:207
#13 vbar_fault ()      at /project/src/plat.h:165

Symptoms match #124 exactly: one core pinned at 100%, VRAM stuck near 2.1 GB, GPU 3-5% busy, zero disk I/O, no further log lines, process alive indefinitely.

Also observed: rocclr's AsyncEventsLoop thread busy-spins in user space from process start and never stops (869 s of CPU time on a 14-minute-old process), while its sibling thread blocks correctly in hsaKmtWaitOnMultipleEvents. Possibly related to what #122 described for gfx1151, but this is already 0.5.6.

Problem 2 - ~6 min model load with --disable-dynamic-vram

The workflow completes, but 1024x1024 / 32 steps takes 476 s total, of which only 75 s is sampling. The remaining ~400 s is weight upload.

Native stack during the slow phase:

#0 ioctl ()
#1 hsakmt_ioctl ()
#2 fmm_map_mem_svm_api ()
#3 _fmm_map_to_gpu_userptr ()
#5 hsaKmtMapMemoryToGPUNodesCtx ()

Isolated benchmark - 60 real safetensors weight tensors (~167 MB), straight .to('cuda'):

Configuration	Time	Rate
baseline	28.2 s	5.9 MB/s (470 ms/tensor)
`GPU_PINNED_MIN_XFER_SIZE=65536`	0.24 s	699 MB/s
`GPU_PINNED_MIN_XFER_SIZE=1048576`	0.07 s	2370 MB/s
`.clone()` first, no env var	5.0 s / 300 tensors	313 MB/s

Controls: pure torch.randn memory, 500 x 2.6 MB tensors -> 0.09 s (6.8 GB/s). Single 256 MB pageable H2D -> 9.6 GB/s. So the hardware and torch are fine; only mmap-backed source pages are pathological.

This looks like the rocclr bug addressed by ROCm/rocm-systems#10962: getBuffer in rocclr/device/rocm/rocblit.cpp prefers host pinning over the staging copy, and pinning a userptr on an iGPU goes through fmm_map_mem_svm_api -> AMDKFD_IOC_SVM ACCESS_IN_PLACE, ~15x slower per chunk on XNACK=0. That PR was merged 2026-09-23 and tested on gfx1151 / ROCm 7.11, but it does not appear to take effect here on gfx1150 / ROCm 7.14.

Workaround

launch args: --disable-dynamic-vram
env:         GPU_PINNED_MIN_XFER_SIZE=1048576

Result: 1024x1024 / 32 steps in 75 s (was 476 s), 512x512 / 8 steps in 17.9 s.

--disable-mmap also avoids problem 2 (80 s) but requires an extra full copy of the model in RAM; the env var is cheaper.

Questions

Is gfx1150 a tested VBAR target for DynamicVRAM? #124 asked the same for gfx1200 and got no reply.
If PR #10962 is already in ROCm 7.14, is there a separate code path on gfx1150 that still pins mmap pages? Should problem 2 be filed against ROCm/rocm-systems instead?
Should the AsyncEventsLoop user-space spin on gfx1150 be tracked separately, or is it meant to be covered by the 0.5.6 fix mentioned in #122?

Originally created by @fengyangsi on GitHub (Oct 24, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/131 ## Summary Two distinct problems on an AMD Radeon 890M (gfx1150, Strix Point APU) under ROCm 7.14: 1. **DynamicVRAM enabled (default): the first `CLIPTextEncode` hangs forever inside `lib.vbar_fault()`.** 2. **With `--disable-dynamic-vram` there is no hang, but a 6.5 GB SDXL checkpoint takes ~6 minutes to load**, because HIP routes H2D copies of mmap-backed safetensors pages through host pinning. Both were diagnosed by attaching gdb to the live backend. ## Environment | | | |---|---| | ComfyUI | 0.42.5 (Desktop standalone, linux-amd bundle v0.42.1-env1) | | comfy-aimdo | 0.5.6 | | PyTorch | 2.12.0+rocm7.14.0 | | ROCm | 7.14 | | GPU | AMD Radeon 890M, gfx1150, RDNA3.5, integrated=1, XNACK disabled | | CPU / RAM | Ryzen AI 9 HX 370, 46 GB shared | | OS | openSUSE Leap 16.0, kernel 6.12.0 | | Model | SDXL checkpoint, 6.5 GB, 1024x1024, 32 steps | Reproduced with a minimal core-only workflow (`CheckpointLoaderSimple -> CLIPTextEncode -> KSampler -> VAEDecode -> SaveImage`), no custom nodes involved. ## Problem 1 - DynamicVRAM hangs inside vbar_fault() 600 s, never completes. Last log line: ``` Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached. Force pre-loaded 180 weights: 400 KB. ``` Execution thread, Python stack: ``` model_vbar.py:88 fault model_vbar.py:137 vbar_fault comfy/ops.py:184 cast_modules_with_vbar comfy/ops.py:400 cast_bias_weight comfy/ops.py:489 __init__ comfy/ops.py:821 forward_comfy_cast_weights comfy/sd1_clip.py:213 process_tokens ``` Same thread, native stack: ``` #11 hipMemSetAccess () from libamdhip64.so.7 #12 three_stooges (size=33554432) at /project/src/plat.h:207 #13 vbar_fault () at /project/src/plat.h:165 ``` Symptoms match #124 exactly: one core pinned at 100%, VRAM stuck near 2.1 GB, GPU 3-5% busy, zero disk I/O, no further log lines, process alive indefinitely. Also observed: rocclr's `AsyncEventsLoop` thread busy-spins in user space from process start and never stops (869 s of CPU time on a 14-minute-old process), while its sibling thread blocks correctly in `hsaKmtWaitOnMultipleEvents`. Possibly related to what #122 described for gfx1151, but this is already 0.5.6. ## Problem 2 - ~6 min model load with --disable-dynamic-vram The workflow completes, but 1024x1024 / 32 steps takes 476 s total, of which only 75 s is sampling. The remaining ~400 s is weight upload. Native stack during the slow phase: ``` #0 ioctl () #1 hsakmt_ioctl () #2 fmm_map_mem_svm_api () #3 _fmm_map_to_gpu_userptr () #5 hsaKmtMapMemoryToGPUNodesCtx () ``` Isolated benchmark - 60 real safetensors weight tensors (~167 MB), straight `.to('cuda')`: | Configuration | Time | Rate | |---|---|---| | baseline | 28.2 s | 5.9 MB/s (470 ms/tensor) | | `GPU_PINNED_MIN_XFER_SIZE=65536` | 0.24 s | 699 MB/s | | `GPU_PINNED_MIN_XFER_SIZE=1048576` | 0.07 s | 2370 MB/s | | `.clone()` first, no env var | 5.0 s / 300 tensors | 313 MB/s | Controls: pure `torch.randn` memory, 500 x 2.6 MB tensors -> 0.09 s (6.8 GB/s). Single 256 MB pageable H2D -> 9.6 GB/s. So the hardware and torch are fine; only mmap-backed source pages are pathological. This looks like the rocclr bug addressed by [ROCm/rocm-systems#10962](https://github.com/ROCm/rocm-systems/pull/10962): `getBuffer` in `rocclr/device/rocm/rocblit.cpp` prefers host pinning over the staging copy, and pinning a userptr on an iGPU goes through `fmm_map_mem_svm_api` -> `AMDKFD_IOC_SVM ACCESS_IN_PLACE`, ~15x slower per chunk on XNACK=0. That PR was merged 2026-09-23 and tested on gfx1151 / ROCm 7.11, but it does **not** appear to take effect here on gfx1150 / ROCm 7.14. ## Workaround ``` launch args: --disable-dynamic-vram env: GPU_PINNED_MIN_XFER_SIZE=1048576 ``` Result: 1024x1024 / 32 steps in 75 s (was 476 s), 512x512 / 8 steps in 17.9 s. `--disable-mmap` also avoids problem 2 (80 s) but requires an extra full copy of the model in RAM; the env var is cheaper. ## Questions 1. Is gfx1150 a tested VBAR target for DynamicVRAM? #124 asked the same for gfx1200 and got no reply. 2. If PR #10962 is already in ROCm 7.14, is there a separate code path on gfx1150 that still pins mmap pages? Should problem 2 be filed against ROCm/rocm-systems instead? 3. Should the `AsyncEventsLoop` user-space spin on gfx1150 be tracked separately, or is it meant to be covered by the 0.5.6 fix mentioned in #122?

bsakaguchi referenced this issue

2026-10-26 22:24:25 +00:00

[PR #51] [MERGED] Dev/sync prewarm #86