Comfy-Org/comfy-aimdo

Fork 0

mirror of https://github.com/Comfy-Org/comfy-aimdo.git synced 2026-11-04 23:37:21 +00:00

[GH-ISSUE #124] [ROCm/Linux][gfx1200 RDNA4] DynamicVRAM hangs forever inside lib.vbar_fault() when materializing int8_convrot text-encoder weights (one core at 100%, VRAM never grows) #48

New issue

Open

opened 2026-10-26 22:24:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:24:03 +00:00

Owner

Originally created by @ChenFeng-ai on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/124

Summary

With DynamicVRAM enabled (the default on ROCm >= 7.14), the first forward pass of a quantized text encoder hangs forever inside the native vbar_fault() call. One CPU core spins at 100%, VRAM never grows, and no further log output is produced. The process stays alive indefinitely (observed 8+ minutes, then killed). --disable-dynamic-vram avoids it entirely.

Environment

Steps to reproduce

Leave DynamicVRAM enabled (no flags; it is auto-enabled because ROCm >= 7.14).
Load a Qwen-Image 2.1 workflow:
- diffusion model: qwen-image-2.1-UC-fp8.safetensors (6.6 GB, per-layer float8_e4m3fn)
- text encoder: qwen3vl_8b_int8_convrot.safetensors (8.9 GB, {"format":"int8_tensorwise","convrot":true,"convrot_groupsize":256})
- VAE: qwen_image_2.1_vae_bf16.safetensors
Queue any prompt.

Observed

The log stops here and never advances:

代码块

[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load QwenImage21TEModel_
[INFO] Model QwenImage21TEModel_ prepared for dynamic VRAM loading. 8916MB Staged. 0 patches attached. Force pre-loaded 255 weights: 849 KB.

Measured over the following 8 minutes (15 s samples):

/sys/class/drm/card0/device/mem_info_vram_used frozen at ~2.0 GiB
process CPU time advances exactly +1.00 s per wall-clock second (one core pinned, 100%)
gpu_busy_percent 12-14% (desktop compositor only), i.e. no GPU work happening
/proc/<pid>/io: read_bytes 0, no repeated model reads
VmRSS 1.5 GB, no RAM thrashing, /dev/shm empty
no further log lines at all

--debug-hang + SIGINT dumps:

代码块

Thread 0x00007f16bc2ff6c0 (most recent call first):
  File ".../comfy_aimdo/model_vbar.py", line 88 in fault
  File ".../comfy_aimdo/model_vbar.py", line 137 in vbar_fault
  File ".../ComfyUI/comfy/ops.py", line 184 in cast_modules_with_vbar
  File ".../ComfyUI/comfy/ops.py", line 396 in cast_bias_weight
  File ".../ComfyUI/comfy/ops.py", line 485 in __init__
  File ".../ComfyUI/comfy/ops.py", line 1791 in forward_comfy_cast_weights
  File ".../ComfyUI/comfy/ops.py", line 829 in forward
  File ".../torch/nn/modules/module.py", line 1789 in _call_impl
  File ".../ComfyUI/comfy/sd1_clip.py", line 213 in process_tokens
  File ".../ComfyUI/comfy/text_encoders/qwen_image21.py", line 38 in process_tokens
  File ".../ComfyUI/comfy/sd1_clip.py", line 266 in forward
  File ".../ComfyUI/comfy/sd1_clip.py", line 306 in encode
  File ".../ComfyUI/comfy/sd.py", line 417 in encode_from_tokens
  File ".../ComfyUI/comfy_extras/nodes_qwen.py", line 176 in execute
  ...

model_vbar.py:88 is:

python

res = lib.vbar_fault(self._devctx, self._ptr, offset, size, signature)

There are no Python frames below it — the thread is blocked inside the compiled aimdo.so. Note it does not return VBAR_FAULT_OOM (1) or raise Fault failed: 2; the call simply never returns.

Ruled out

Second GPU / iGPU. This machine also has a Granite Ridge iGPU (AMD Radeon Graphics, gfx11). Hiding it with HIP_VISIBLE_DEVICES=0 ROCR_VISIBLE_DEVICES=0 (only one aimdo ... inited for GPU line, only cuda:0 present) still reproduces identically.
Slow dequantization. Calling torch.ops.comfy_kitchen.dequantize_int8_convrot_weight_dtype directly on this machine measures 15,561 M params/s on GPU and 262 M params/s on CPU, so materializing the whole ~4.5 G-parameter encoder should take <20 s even on CPU. The hang is not slow dequant.
Disk / RAM pressure. No disk reads, RSS 1.5 GB, no swapping.

Workaround that works

代码块

--disable-dynamic-vram --lowvram

With those two flags the identical workflow completes successfully: 25 sampling steps at 1.25 s/it, 140 s end-to-end, peak VRAM 9.9 GB, correct output image. --lowvram is additionally needed because with DynamicVRAM disabled the 8.9 GB text encoder and 6.8 GB DiT are both resident and VAE decode then OOMs on a 16 GB card.

Related issues

#95 — [ROCM/Linux]: VBAR allocation fails on RDNA1
#118 — similar symptom profile (long stall, 100% util, flat disk/RAM) but Windows/NVIDIA
#29 — HIP/ROCm working port on RX 9070 (gfx1201), sibling silicon to gfx1200
#122 — different bug (idle busy-spin, no prompt running, kernel-side kfd_wait_on_events), and we are already on 0.5.6

Questions

Is there any state that makes vbar_fault() block instead of returning VBAR_FAULT_SUCCESS / OOM / ERROR? Even a timeout that degrades to VBAR_FAULT_ERROR would turn this permanent hang into a recoverable error.
Is gfx1200 (RDNA4, Navi 44) a tested target for VBAR allocation? #95 suggests RDNA1 already has allocation problems on ROCm/Linux.
Since --disable-dynamic-vram is documented as temporary ("this argument will be removed soon"), what is the recommended path for RDNA4 + ROCm 7.14 users who hit this?

Happy to run any instrumented build, strace/perf capture, or provide more logs.

Originally created by @ChenFeng-ai on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-aimdo/issues/124 <h3>Summary</h3> <p>With DynamicVRAM enabled (the default on ROCm >= 7.14), the first forward pass of a quantized text encoder hangs <strong>forever</strong> inside the native <code>vbar_fault()</code> call. One CPU core spins at 100%, VRAM never grows, and no further log output is produced. The process stays alive indefinitely (observed 8+ minutes, then killed). <code>--disable-dynamic-vram</code> avoids it entirely.</p> <h3>Environment</h3> <div class="_tableScroll_1ypvv_192 _tableFill_1ypvv_239"> | -- | -- comfy-aimdo | 0.5.6 (latest on PyPI) ComfyUI | 0.40.0 (commit 3541246a) torch | 2.12.0+rocm7.14.0 (HIP 7.14.60850) ROCm | 7.14 GPU | AMD Radeon RX 9060 XT, gfx1200 (Navi 44, RDNA4), 16304 MB VRAM OS / kernel | Debian 13 (trixie), Linux 6.12.111+deb13-amd64 Python | 3.13.12 RAM | 31 GB Launch method | reproduces both via ComfyUI Desktop 1.1.7 and plain python main.py </div> <h3>Steps to reproduce</h3> <ol><li>Leave DynamicVRAM enabled (no flags; it is auto-enabled because ROCm >= 7.14).</li><li>Load a Qwen-Image 2.1 workflow: <ul><li>diffusion model: <code>qwen-image-2.1-UC-fp8.safetensors</code> (6.6 GB, per-layer <code>float8_e4m3fn</code>)</li><li>text encoder: <code>qwen3vl_8b_int8_convrot.safetensors</code> (8.9 GB, <code>{"format":"int8_tensorwise","convrot":true,"convrot_groupsize":256}</code>)</li><li>VAE: <code>qwen_image_2.1_vae_bf16.safetensors</code></li></ul> </li><li>Queue any prompt.</li></ol> <h3>Observed</h3> <p>The log stops here and never advances:</p> <div class="_block_7gxqk_4 md-code-block _card_7gxqk_107" data-code-wrap="true"><div class="_bannerWrap_7gxqk_24"><div class="_header_1pq26_10" data-code-block-banner="true"><div class="_heading_1pq26_23"><span class="_language_1pq26_30">代码块</span></div><div class="_actions_1pq26_43"></div></div></div><div class="_content_7gxqk_74" data-code-block-content="true"><pre class="_plain_7gxqk_103"><code>[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16 [INFO] Requested to load QwenImage21TEModel_ [INFO] Model QwenImage21TEModel_ prepared for dynamic VRAM loading. 8916MB Staged. 0 patches attached. Force pre-loaded 255 weights: 849 KB.</code></pre></div></div> <p>Measured over the following 8 minutes (15 s samples):</p> <ul><li><code>/sys/class/drm/card0/device/mem_info_vram_used</code> frozen at ~2.0 GiB</li><li>process CPU time advances exactly +1.00 s per wall-clock second (one core pinned, 100%)</li><li><code>gpu_busy_percent</code> 12-14% (desktop compositor only), i.e. no GPU work happening</li><li><code>/proc/<pid>/io</code>: <code>read_bytes</code> 0, no repeated model reads</li><li><code>VmRSS</code> 1.5 GB, no RAM thrashing, <code>/dev/shm</code> empty</li><li>no further log lines at all</li></ul> <p><code>--debug-hang</code> + SIGINT dumps:</p> <div class="_block_7gxqk_4 md-code-block _card_7gxqk_107" data-code-wrap="true"><div class="_bannerWrap_7gxqk_24"><div class="_header_1pq26_10" data-code-block-banner="true"><div class="_heading_1pq26_23"><span class="_language_1pq26_30">代码块</span></div><div class="_actions_1pq26_43"></div></div></div><div class="_content_7gxqk_74" data-code-block-content="true"><pre class="_plain_7gxqk_103"><code>Thread 0x00007f16bc2ff6c0 (most recent call first): File ".../comfy_aimdo/model_vbar.py", line 88 in fault File ".../comfy_aimdo/model_vbar.py", line 137 in vbar_fault File ".../ComfyUI/comfy/ops.py", line 184 in cast_modules_with_vbar File ".../ComfyUI/comfy/ops.py", line 396 in cast_bias_weight File ".../ComfyUI/comfy/ops.py", line 485 in __init__ File ".../ComfyUI/comfy/ops.py", line 1791 in forward_comfy_cast_weights File ".../ComfyUI/comfy/ops.py", line 829 in forward File ".../torch/nn/modules/module.py", line 1789 in _call_impl File ".../ComfyUI/comfy/sd1_clip.py", line 213 in process_tokens File ".../ComfyUI/comfy/text_encoders/qwen_image21.py", line 38 in process_tokens File ".../ComfyUI/comfy/sd1_clip.py", line 266 in forward File ".../ComfyUI/comfy/sd1_clip.py", line 306 in encode File ".../ComfyUI/comfy/sd.py", line 417 in encode_from_tokens File ".../ComfyUI/comfy_extras/nodes_qwen.py", line 176 in execute ...</code></pre></div></div> <p><code>model_vbar.py:88</code> is:</p> <div class="_block_7gxqk_4 md-code-block _card_7gxqk_107" data-code-wrap="true"><div class="_bannerWrap_7gxqk_24"><div class="_header_1pq26_10" data-code-block-banner="true"><div class="_heading_1pq26_23"><span class="_language_1pq26_30">python</span></div><div class="_actions_1pq26_43"></div></div></div><div class="_content_7gxqk_74" data-code-block-content="true"><div><pre class="shiki css-variables" style="background-color:var(--shiki-background);color:var(--shiki-foreground)" tabindex="0"><code><span class="line"><span style="color:var(--shiki-foreground)">res </span><span style="color:var(--shiki-token-keyword)">=</span><span style="color:var(--shiki-foreground)"> lib</span><span style="color:var(--shiki-token-punctuation)">.</span><span style="color:var(--shiki-token-function)">vbar_fault</span><span style="color:var(--shiki-token-punctuation)">(self._devctx, self._ptr, offset, size, signature)</span></span></code></pre></div></div></div> <p>There are no Python frames below it — the thread is blocked inside the compiled <code>aimdo.so</code>. Note it does <strong>not</strong> return <code>VBAR_FAULT_OOM (1)</code> or raise <code>Fault failed: 2</code>; the call simply never returns.</p> <h3>Ruled out</h3> <ul><li><strong>Second GPU / iGPU.</strong> This machine also has a Granite Ridge iGPU (<code>AMD Radeon Graphics</code>, gfx11). Hiding it with <code>HIP_VISIBLE_DEVICES=0 ROCR_VISIBLE_DEVICES=0</code> (only one <code>aimdo ... inited for GPU</code> line, only <code>cuda:0</code> present) <strong>still reproduces</strong> identically.</li><li><strong>Slow dequantization.</strong> Calling <code>torch.ops.comfy_kitchen.dequantize_int8_convrot_weight_dtype</code> directly on this machine measures <strong>15,561 M params/s on GPU</strong> and <strong>262 M params/s on CPU</strong>, so materializing the whole ~4.5 G-parameter encoder should take <20 s even on CPU. The hang is not slow dequant.</li><li><strong>Disk / RAM pressure.</strong> No disk reads, RSS 1.5 GB, no swapping.</li></ul> <h3>Workaround that works</h3> <div class="_block_7gxqk_4 md-code-block _card_7gxqk_107" data-code-wrap="true"><div class="_bannerWrap_7gxqk_24"><div class="_header_1pq26_10" data-code-block-banner="true"><div class="_heading_1pq26_23"><span class="_language_1pq26_30">代码块</span></div><div class="_actions_1pq26_43"></div></div></div><div class="_content_7gxqk_74" data-code-block-content="true"><pre class="_plain_7gxqk_103"><code>--disable-dynamic-vram --lowvram</code></pre></div></div> <p>With those two flags the identical workflow completes successfully: 25 sampling steps at 1.25 s/it, 140 s end-to-end, peak VRAM 9.9 GB, correct output image. <code>--lowvram</code> is additionally needed because with DynamicVRAM disabled the 8.9 GB text encoder and 6.8 GB DiT are both resident and VAE decode then OOMs on a 16 GB card.</p> <h3>Related issues</h3> <ul><li>#95 — [ROCM/Linux]: VBAR allocation fails on RDNA1</li><li>#118 — similar symptom profile (long stall, 100% util, flat disk/RAM) but Windows/NVIDIA</li><li>#29 — HIP/ROCm working port on RX 9070 (gfx1201), sibling silicon to gfx1200</li><li>#122 — different bug (idle busy-spin, no prompt running, kernel-side <code>kfd_wait_on_events</code>), and we are already on 0.5.6</li></ul> <h3>Questions</h3> <ol><li>Is there any state that makes <code>vbar_fault()</code> block instead of returning <code>VBAR_FAULT_SUCCESS / OOM / ERROR</code>? Even a timeout that degrades to <code>VBAR_FAULT_ERROR</code> would turn this permanent hang into a recoverable error.</li><li>Is <strong>gfx1200</strong> (RDNA4, Navi 44) a tested target for VBAR allocation? #95 suggests RDNA1 already has allocation problems on ROCm/Linux.</li><li>Since <code>--disable-dynamic-vram</code> is documented as temporary ("this argument will be removed soon"), what is the recommended path for RDNA4 + ROCm 7.14 users who hit this?</li></ol> <p>Happy to run any instrumented build, <code>strace</code>/<code>perf</code> capture, or provide more logs.</p>

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-aimdo#48

No description provided.

Rows
Columns