Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #226] [AMD RDNA3 Regression] CK Attention produces corrupted Qwen Image 2.1 output after recent update #66

New issue

Open

opened 2026-10-26 22:19:54 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:19:54 +00:00

Owner

Originally created by @dkspwndj on GitHub (Sep 30, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/226

Description

After updating ComfyUI and the related dependencies recently, I started getting severely corrupted image output when using CK Attention on an AMD Radeon RX 7900 XTX.

The same workflow, input image, prompt, seed, model, sampler settings, and memory settings produce a normal image when I switch only the attention backend from CK Attention to Quad Cross Attention.

CK Attention was working normally on the same system before the recent update.

This appears to be a regression in the CK Attention path, although I am not sure whether the cause is in ComfyUI itself or comfy-kitchen.

Environment

GPU: AMD Radeon RX 7900 XTX 24GB
Architecture: RDNA3 / gfx1100
OS: Windows 11
Backend: ROCm
Model: Qwen Image 2.1
Diffusion model: qwen_image_2.1_int8_convrot.safetensors
Text encoder: qwen3vl_8b_int8_convrot.safetensors
VAE: BF16 Qwen Image 2.1 VAE
ComfyUI version: [insert current version / commit]
comfy-kitchen version: 0.2.36
PyTorch / ROCm version: [insert from startup log]

Reproduction

With CK Attention:

python main.py --auto-launch --use-ck-attention --bf16-vae --disable-pinned-memory --disable-async-offload --disable-dynamic-vram --cache-none --reserve-vram 1.0

The generated image becomes severely corrupted.

The output contains many green and purple artifacts, broken textures, and heavy structural corruption, although the generation process itself finishes normally.

There is no obvious exception or fatal error in the console.

I then change only the attention backend to Quad Cross Attention:

python main.py --auto-launch --use-quad-cross-attention --bf16-vae --disable-pinned-memory --disable-async-offload --disable-dynamic-vram --cache-none --reserve-vram 1.0

The same workflow and seed then produce a normal image.

--disable-dynamic-vram and --disable-async-offload are explicitly enabled, so this does not appear to be caused by Dynamic VRAM eviction/reloading or async offloading.

Expected behavior

CK Attention should generate a valid image, as it did before the recent update.

Actual behavior

CK Attention completes generation without an exception, but produces a severely corrupted image.

Switching only to Quad Cross Attention immediately restores normal output.

Regression information

This setup was previously working correctly with CK Attention.

The corruption started only after the recent ComfyUI / comfy-kitchen update.

Because Quad Cross Attention still works correctly with the exact same workflow and seed, the model files, VAE, workflow, input image, prompt, sampler, and scheduler appear to be valid.

This may be related to the AMD/ROCm CK Attention path on RDNA3 (gfx1100).

There are older reports of invalid output with CK Attention on AMD GPUs, but my case appears to be a newer regression involving:

RX 7900 XTX / RDNA3
Qwen Image 2.1
comfy-kitchen 0.2.36
recent ComfyUI versions

I can provide the complete startup log, generation log, workflow JSON, and additional test results if needed.

Comparison images

Show CK Attention vs Quad Cross Attention comparison

CK Attention — Corrupted	Quad Cross Attention — Normal

Originally created by @dkspwndj on GitHub (Sep 30, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/226 ### Description After updating ComfyUI and the related dependencies recently, I started getting severely corrupted image output when using **CK Attention** on an AMD Radeon RX 7900 XTX. The same workflow, input image, prompt, seed, model, sampler settings, and memory settings produce a normal image when I switch **only the attention backend** from CK Attention to Quad Cross Attention. CK Attention was working normally on the same system before the recent update. This appears to be a regression in the CK Attention path, although I am not sure whether the cause is in ComfyUI itself or comfy-kitchen. --- ### Environment - **GPU:** AMD Radeon RX 7900 XTX 24GB - **Architecture:** RDNA3 / gfx1100 - **OS:** Windows 11 - **Backend:** ROCm - **Model:** Qwen Image 2.1 - **Diffusion model:** `qwen_image_2.1_int8_convrot.safetensors` - **Text encoder:** `qwen3vl_8b_int8_convrot.safetensors` - **VAE:** BF16 Qwen Image 2.1 VAE - **ComfyUI version:** `[insert current version / commit]` - **comfy-kitchen version:** `0.2.36` - **PyTorch / ROCm version:** `[insert from startup log]` --- ### Reproduction With CK Attention: ```bat python main.py --auto-launch --use-ck-attention --bf16-vae --disable-pinned-memory --disable-async-offload --disable-dynamic-vram --cache-none --reserve-vram 1.0 ``` The generated image becomes severely corrupted. The output contains many green and purple artifacts, broken textures, and heavy structural corruption, although the generation process itself finishes normally. There is no obvious exception or fatal error in the console. I then change **only the attention backend** to Quad Cross Attention: ```bat python main.py --auto-launch --use-quad-cross-attention --bf16-vae --disable-pinned-memory --disable-async-offload --disable-dynamic-vram --cache-none --reserve-vram 1.0 ``` The same workflow and seed then produce a normal image. `--disable-dynamic-vram` and `--disable-async-offload` are explicitly enabled, so this does not appear to be caused by Dynamic VRAM eviction/reloading or async offloading. --- ### Expected behavior CK Attention should generate a valid image, as it did before the recent update. ### Actual behavior CK Attention completes generation without an exception, but produces a severely corrupted image. Switching only to Quad Cross Attention immediately restores normal output. --- ### Regression information This setup was previously working correctly with CK Attention. The corruption started only after the recent ComfyUI / comfy-kitchen update. Because Quad Cross Attention still works correctly with the exact same workflow and seed, the model files, VAE, workflow, input image, prompt, sampler, and scheduler appear to be valid. This may be related to the AMD/ROCm CK Attention path on **RDNA3 (`gfx1100`)**. There are older reports of invalid output with CK Attention on AMD GPUs, but my case appears to be a newer regression involving: - RX 7900 XTX / RDNA3 - Qwen Image 2.1 - comfy-kitchen `0.2.36` - recent ComfyUI versions I can provide the complete startup log, generation log, workflow JSON, and additional test results if needed. --- ### Comparison images <details> <summary><b>Show CK Attention vs Quad Cross Attention comparison</b></summary> <br> <table> <tr> <th>CK Attention — Corrupted</th> <th>Quad Cross Attention — Normal</th> </tr> <tr> <td> <img width="480" height="720" alt="CK Attention corrupted output" src="https://github.com/user-attachments/assets/796d63ed-82f0-4126-8b7c-646a10b02683" /> </td> <td> <img width="480" height="720" alt="Quad Cross Attention normal output" src="https://github.com/user-attachments/assets/de7c15c3-15b9-44f6-8599-d9a16a9bba69" /> </td> </tr> </table> </details>

bsakaguchi commented

2026-10-26 22:19:54 +00:00

Author

Owner

@0xDELUXA commented on GitHub (Oct 4, 2026):

This looks like #214. comfy-kitchen 0.2.36 introduced a prepared dense-mask path in the HIP int8 attention kernel, and on gfx11 fast-math folded away the empty-tile guard, so key tiles in which every key is masked produced NaN. Qwen Image 2.1 runs its text tokens through attention with a causal mask, which hits exactly that case, and the NaN then spreads through the rest of the model. 0.2.35 was not affected.

It was fixed in 19ea55b and released in comfy-kitchen 0.2.37, which current ComfyUI master now requires. Could you update (pip install comfy-kitchen==0.2.37, or pull ComfyUI and reinstall requirements.txt) and confirm whether CK attention produces correct images again?

@0xDELUXA commented on GitHub (Oct 4, 2026): This looks like #214. comfy-kitchen 0.2.36 introduced a prepared dense-mask path in the HIP int8 attention kernel, and on gfx11 fast-math folded away the empty-tile guard, so key tiles in which every key is masked produced NaN. Qwen Image 2.1 runs its text tokens through attention with a causal mask, which hits exactly that case, and the NaN then spreads through the rest of the model. 0.2.35 was not affected. It was fixed in 19ea55b and released in comfy-kitchen 0.2.37, which current ComfyUI master now requires. Could you update (`pip install comfy-kitchen==0.2.37`, or pull ComfyUI and reinstall `requirements.txt`) and confirm whether CK attention produces correct images again?

bsakaguchi referenced this issue

2026-10-26 22:20:32 +00:00

[PR #66] [MERGED] Fix issue when overflowing with cuda int8 linear. #117

bsakaguchi referenced this issue

2026-10-26 22:20:38 +00:00

[PR #80] [CLOSED] Optimize quantize_int8_rowwise_convrot64 launch config (up to 4.9x on the generic path) #126