Comfy-Org/comfy-kitchen
1
0
Fork 0
mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00
Code Issues 118 Projects Packages Wiki Activity

[GH-ISSUE #230] [Bug] comfy kitchen attention (INT8) returns pure noise on AMD gfx1100 once text conditioning exceeds ~150 tokens (Qwen-Image 2.1 int8_convrot) #65

New issue
Open
opened 2026-10-26 22:19:53 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 22:19:53 +00:00
Owner
Copy link

Originally created by @snow930 on GitHub (Oct 1, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/230

[Bug] comfy kitchen attention (INT8 attention) returns pure noise on AMD gfx1100 when the text conditioning exceeds ~60–200 tokens (Qwen-Image 2.1 int8_convrot)

提交建议 / Where to file

  • 首选 Comfy-Org/comfy-kitchen（内核本体：torch.ops.comfy_kitchen.int8_attention，hip 后端）。
  • 也可报 Comfy-Org/ComfyUI（集成层 comfy/ldm/modules/attention.py + ModelAttentionBackend 节点），由维护者转给 comfy-kitchen。
  • 附件图片用 output/diag/ 下的：C15a_min_kitchen_00001_.png（坏）、C15b_min_default_00001_.png（好）、C16_min_neutral_00001_.png（中性文本坏）。

Environment

Item Value
ComfyUI 0.38.0 (v0.38.0-7-gfb2315f1, released 2026-09-30)
comfy-kitchen 0.2.36
comfy-aimdo 0.5.5
PyTorch 2.13.0+rocm10.0.0
ROCm / HIP ROCm (7, 15) reported by ComfyUI; HIP 7.16 runtime
GPU AMD Radeon RX 7900 XT, gfx1100 (RDNA3), 20 GB
OS Windows 11 (win32), Python 3.13.15
Launch args --reserve-vram 1.0 --enable-dynamic-vram --use-split-cross-attention --disable-xformers --auto-launch --port 8188
Diffusion model Qwen-Image/qwen_image_2.1_int8_convrot.safetensors
Text encoder Qwen-Image/qwen3vl_8b_int8_convrot.safetensors
VAE Qwen-Image/qwen_image_2.1_vae_bf16.safetensors

Summary

With the attention backend set to comfy kitchen attention (comfy_kitchen_int8), Qwen-Image 2.1 int8_convrot sampling produces a pure彩色噪点 image, at normal speed and without any error or warning.

The corruption depends only on the length of the text conditioning, with a sharp cutoff:

  • ≤ 142 tokens (160 Chinese characters) → correct image
  • ≥ 150 tokens (170 Chinese characters) → pure noise
  • the cutoff is independent of image resolution (identical at 512×512 and 1024×1024)
  • 449 tokens (a typical prompt-enhancer output) → noise; 1072 tokens → noise

Length sweep (neutral repeated Chinese sentence, seed 123456789, 8 steps, cfg 1.0, euler/simple):

Chinese chars tokens 512×512 1024×1024
40 52 ✅ –
80 82 ✅ ✅
120 112 ✅ ✅
140 – – ✅
160 142 ✅ ✅
170 150 ❌ noise ❌ noise
180 160 ❌ noise ❌ noise
200 172 ❌ noise ❌ noise
240 202 ❌ noise ❌ noise
280 232 ❌ noise –

Switching back to the default attention backend (remove the ModelAttentionBackend node, so attention_split is used) with the identical prompt, resolution, steps and seed produces a correct image.

Minimal reproduction — official nodes only, no custom nodes

  1. UNETLoader → qwen_image_2.1_int8_convrot.safetensors (weight_dtype: default)
  2. CLIPLoader → qwen3vl_8b_int8_convrot.safetensors (type: qwen_image)
  3. VAELoader → qwen_image_2.1_vae_bf16.safetensors
  4. ModelAttentionBackend with backend = comfy kitchen attention, inserted between UNETLoader and KSampler
  5. TextEncodeQwenImage21, resolution = 1024, prompt = the 232-token text below (or any prompt ≳ 200 tokens)
  6. EmptySD3LatentImage 512×512
  7. KSampler: seed 123456789, steps 8, cfg 1.0, euler, simple, denoise 1.0
  8. VAEDecode → SaveImage

→ pure noise in ~6–9 s, exit status success, no warning.

Now delete node 4 and reconnect KSampler.model to UNETLoader (default split attention), rerun → correct image in ~9 s.

Reproduction text (232 tokens, neutral content)

一位东亚女性成年模特坐在书桌旁，银灰色短发，微笑，正面半身，自然光，超写实细节，

repeated 7 times (280 Chinese characters = 232 tokens).

Full API-format graph (the failing variant)

{
  "1": {"class_type": "UNETLoader", "inputs": {"unet_name": "Qwen-Image/qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default"}},
  "2": {"class_type": "CLIPLoader", "inputs": {"clip_name": "Qwen-Image/qwen3vl_8b_int8_convrot.safetensors", "type": "qwen_image", "device": "default"}},
  "3": {"class_type": "VAELoader", "inputs": {"vae_name": "Qwen-Image/qwen_image_2.1_vae_bf16.safetensors"}},
  "4": {"class_type": "ModelAttentionBackend", "inputs": {"model": ["1", 0], "attention": "comfy kitchen attention"}},
  "5": {"class_type": "TextEncodeQwenImage21", "inputs": {"clip": ["2", 0], "prompt": "<232+ token text>", "negative_prompt": "", "resolution": 1024}},
  "6": {"class_type": "EmptySD3LatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}},
  "7": {"class_type": "KSampler", "inputs": {"seed": 123456789, "steps": 8, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": ["4", 0], "positive": ["5", 0], "negative": ["5", 1], "latent_image": ["6", 0]}},
  "8": {"class_type": "VAEDecode", "inputs": {"samples": ["7", 0], "vae": ["3", 0]}},
  "9": {"class_type": "SaveImage", "inputs": {"images": ["8", 0], "filename_prefix": "noise_repro"}}
}

For the working control: delete "4" and set "7".inputs.model = ["1", 0].

Isolation matrix (all with the same checkpoint/VAE/sampler settings)

# attention backend prompt image size steps result
A comfy kitchen 35 tokens (English) 1024×1024 30 ✅ correct
C6 comfy kitchen 15 tokens 1024×1024 30 ✅ correct
C7 comfy kitchen 25 tokens, JSON-wrapped 1024×1024 30 ✅ correct
C8 comfy kitchen 232 tokens 1024×1024 30 ❌ noise
C9 comfy kitchen 1072 tokens 1024×1024 30 ❌ noise
C4 comfy kitchen 449 tokens 1024×1024 30 ❌ noise
C5 comfy kitchen 449 tokens + official latent 1024×1024 30 ❌ noise (byte-identical to C4)
C10 comfy kitchen 449 tokens 512×512 30 ❌ noise
C12 comfy kitchen 449 tokens 512×512 8 ❌ noise
C11 default (split) 449 tokens 512×512 8 ✅ correct
C15a comfy kitchen (official node) 449 tokens 512×512 8 ❌ noise
C15b default (official, no node) 449 tokens 512×512 8 ✅ correct
C16 comfy kitchen (official node) 232 tokens (neutral) 512×512 8 ❌ noise
C13 default (split) 449 tokens 1024×1824 8 ✅ correct

Ruled out

  • Latent source / latent shape: replacing the custom-node latent with an official EmptySD3LatentImage produced a byte-identical PNG (same sha256) — with denoise=1.0 the latent only supplies the noise shape.
  • Resolution / total sequence length: 512×512 (1024 image tokens) fails exactly like 1024×1024 (4096) and 1024×1824. Image-side token count is irrelevant.
  • Prompt format: a JSON-wrapped short prompt is fine — not a special-character issue.
  • Text encoder: an offline probe of the same qwen3vl_8b_int8_convrot gives conditioning (1, 21, 4096) for the short prompt and (1, 435, 4096) for the long one; both are NaN-free and Inf-free, and neither carries an attention_mask (extra only contains pooled_output). The TE output looks healthy.
  • Model / VAE / quantized weights / sampler settings: identical settings under the default split attention produce correct images (C11, C13, C15b), and the same prompt renders the requested four-panel character sheet.

Localization hints

  • Failing path: attention_comfy_kitchen_int8 (comfy/ldm/modules/attention.py) → comfy_kitchen.int8_attention (comfy_kitchen/sage_attention.py) → torch.ops.comfy_kitchen.int8_attention, i.e. the hip backend INT8 attention kernel, not AOTriton/SDPA.
  • The trigger correlates with the DiT prefix (text) length, not with total sequence length: prefix_len goes from ~21 to ~435 while the image side changes from 4096 to 1024 tokens — and it breaks in both directions.
  • Worth checking in the hip kernel: the KV padding path (_sage_buffers: padded_k = -(-kv_length // cta_k) * cta_k, _SAGE_CTA_K = 64) and the K "stabilization detector" anchor sampling (anchor_indices, one key index per batch/KV head read by the K quantizer), for inputs whose K length is dominated by a long prefix rather than by the padded image grid.
  • Note the failure is silent — no exception, no NaN, status: success, and normal speed (2.4 s/it at 1024×1824 vs 55 s/it for split). This matches the "attention kernel returns wrong results without surfacing an error" pattern reported for other AMD attention paths (e.g. ROCm/aotriton#210).

Impact

Qwen-Image 2.1 int8_convrot + comfy kitchen attention is effectively unusable on AMD RDNA3 for prompt-enhancer workflows, whose enhanced prompts are typically 300–600 tokens. The only working alternative on this machine is the default attention_split, which is ~23× slower at 1024×1824.

Note: the pytorch attention option is not a workaround for this checkpoint either — it raises
RuntimeError: convrot global spill requires caller-provided workspace buffers.

Image Image Image
Originally created by @snow930 on GitHub (Oct 1, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/230 # [Bug] `comfy kitchen attention` (INT8 attention) returns pure noise on AMD gfx1100 when the text conditioning exceeds ~60–200 tokens (Qwen-Image 2.1 `int8_convrot`) > **提交建议 / Where to file** > - 首选 **Comfy-Org/comfy-kitchen**（内核本体：`torch.ops.comfy_kitchen.int8_attention`，hip 后端）。 > - 也可报 **Comfy-Org/ComfyUI**（集成层 `comfy/ldm/modules/attention.py` + `ModelAttentionBackend` 节点），由维护者转给 comfy-kitchen。 > - 附件图片用 `output/diag/` 下的：`C15a_min_kitchen_00001_.png`（坏）、`C15b_min_default_00001_.png`（好）、`C16_min_neutral_00001_.png`（中性文本坏）。 ## Environment | Item | Value | |---|---| | ComfyUI | 0.38.0 (`v0.38.0-7-gfb2315f1`, released 2026-09-30) | | comfy-kitchen | 0.2.36 | | comfy-aimdo | 0.5.5 | | PyTorch | 2.13.0+rocm10.0.0 | | ROCm / HIP | ROCm (7, 15) reported by ComfyUI; HIP 7.16 runtime | | GPU | AMD Radeon RX 7900 XT, `gfx1100` (RDNA3), 20 GB | | OS | Windows 11 (`win32`), Python 3.13.15 | | Launch args | `--reserve-vram 1.0 --enable-dynamic-vram --use-split-cross-attention --disable-xformers --auto-launch --port 8188` | | Diffusion model | `Qwen-Image/qwen_image_2.1_int8_convrot.safetensors` | | Text encoder | `Qwen-Image/qwen3vl_8b_int8_convrot.safetensors` | | VAE | `Qwen-Image/qwen_image_2.1_vae_bf16.safetensors` | ## Summary With the attention backend set to **`comfy kitchen attention`** (`comfy_kitchen_int8`), Qwen-Image 2.1 `int8_convrot` sampling produces a **pure彩色噪点 image**, at normal speed and **without any error or warning**. The corruption depends **only on the length of the text conditioning**, with a sharp cutoff: - **≤ 142 tokens (160 Chinese characters) → correct image** - **≥ 150 tokens (170 Chinese characters) → pure noise** - the cutoff is **independent of image resolution** (identical at 512×512 and 1024×1024) - 449 tokens (a typical prompt-enhancer output) → noise; 1072 tokens → noise Length sweep (neutral repeated Chinese sentence, seed 123456789, 8 steps, cfg 1.0, euler/simple): | Chinese chars | tokens | 512×512 | 1024×1024 | |---|---|---|---| | 40 | 52 | ✅ | – | | 80 | 82 | ✅ | ✅ | | 120 | 112 | ✅ | ✅ | | 140 | – | – | ✅ | | 160 | 142 | ✅ | ✅ | | 170 | 150 | ❌ noise | ❌ noise | | 180 | 160 | ❌ noise | ❌ noise | | 200 | 172 | ❌ noise | ❌ noise | | 240 | 202 | ❌ noise | ❌ noise | | 280 | 232 | ❌ noise | – | Switching back to the default attention backend (**remove the `ModelAttentionBackend` node**, so `attention_split` is used) with the **identical prompt, resolution, steps and seed** produces a correct image. ## Minimal reproduction — official nodes only, no custom nodes 1. `UNETLoader` → `qwen_image_2.1_int8_convrot.safetensors` (`weight_dtype: default`) 2. `CLIPLoader` → `qwen3vl_8b_int8_convrot.safetensors` (`type: qwen_image`) 3. `VAELoader` → `qwen_image_2.1_vae_bf16.safetensors` 4. **`ModelAttentionBackend`** with `backend = comfy kitchen attention`, inserted between `UNETLoader` and `KSampler` 5. `TextEncodeQwenImage21`, `resolution = 1024`, prompt = the 232-token text below (or any prompt ≳ 200 tokens) 6. `EmptySD3LatentImage` 512×512 7. `KSampler`: `seed 123456789, steps 8, cfg 1.0, euler, simple, denoise 1.0` 8. `VAEDecode` → `SaveImage` → **pure noise in ~6–9 s, exit status `success`, no warning.** Now delete node 4 and reconnect `KSampler.model` to `UNETLoader` (default split attention), rerun → **correct image in ~9 s**. ### Reproduction text (232 tokens, neutral content) ``` 一位东亚女性成年模特坐在书桌旁，银灰色短发，微笑，正面半身，自然光，超写实细节， ``` repeated **7 times** (280 Chinese characters = 232 tokens). ### Full API-format graph (the failing variant) ```json { "1": {"class_type": "UNETLoader", "inputs": {"unet_name": "Qwen-Image/qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default"}}, "2": {"class_type": "CLIPLoader", "inputs": {"clip_name": "Qwen-Image/qwen3vl_8b_int8_convrot.safetensors", "type": "qwen_image", "device": "default"}}, "3": {"class_type": "VAELoader", "inputs": {"vae_name": "Qwen-Image/qwen_image_2.1_vae_bf16.safetensors"}}, "4": {"class_type": "ModelAttentionBackend", "inputs": {"model": ["1", 0], "attention": "comfy kitchen attention"}}, "5": {"class_type": "TextEncodeQwenImage21", "inputs": {"clip": ["2", 0], "prompt": "<232+ token text>", "negative_prompt": "", "resolution": 1024}}, "6": {"class_type": "EmptySD3LatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}}, "7": {"class_type": "KSampler", "inputs": {"seed": 123456789, "steps": 8, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": ["4", 0], "positive": ["5", 0], "negative": ["5", 1], "latent_image": ["6", 0]}}, "8": {"class_type": "VAEDecode", "inputs": {"samples": ["7", 0], "vae": ["3", 0]}}, "9": {"class_type": "SaveImage", "inputs": {"images": ["8", 0], "filename_prefix": "noise_repro"}} } ``` For the working control: delete `"4"` and set `"7".inputs.model = ["1", 0]`. ## Isolation matrix (all with the same checkpoint/VAE/sampler settings) | # | attention backend | prompt | image size | steps | result | |---|---|---|---|---|---| | A | comfy kitchen | 35 tokens (English) | 1024×1024 | 30 | ✅ correct | | C6 | comfy kitchen | 15 tokens | 1024×1024 | 30 | ✅ correct | | C7 | comfy kitchen | 25 tokens, JSON-wrapped | 1024×1024 | 30 | ✅ correct | | C8 | comfy kitchen | **232 tokens** | 1024×1024 | 30 | ❌ noise | | C9 | comfy kitchen | **1072 tokens** | 1024×1024 | 30 | ❌ noise | | C4 | comfy kitchen | **449 tokens** | 1024×1024 | 30 | ❌ noise | | C5 | comfy kitchen | 449 tokens + official latent | 1024×1024 | 30 | ❌ noise (**byte-identical to C4**) | | C10 | comfy kitchen | 449 tokens | 512×512 | 30 | ❌ noise | | C12 | comfy kitchen | 449 tokens | 512×512 | 8 | ❌ noise | | **C11** | **default (split)** | **449 tokens** | **512×512** | **8** | ✅ **correct** | | **C15a** | comfy kitchen (official node) | 449 tokens | 512×512 | 8 | ❌ noise | | **C15b** | **default (official, no node)** | **449 tokens** | **512×512** | **8** | ✅ **correct** | | C16 | comfy kitchen (official node) | 232 tokens (neutral) | 512×512 | 8 | ❌ noise | | C13 | default (split) | 449 tokens | 1024×1824 | 8 | ✅ correct | ## Ruled out - **Latent source / latent shape**: replacing the custom-node latent with an official `EmptySD3LatentImage` produced a **byte-identical PNG** (same sha256) — with `denoise=1.0` the latent only supplies the noise shape. - **Resolution / total sequence length**: 512×512 (1024 image tokens) fails exactly like 1024×1024 (4096) and 1024×1824. Image-side token count is irrelevant. - **Prompt format**: a JSON-wrapped short prompt is fine — not a special-character issue. - **Text encoder**: an offline probe of the same `qwen3vl_8b_int8_convrot` gives conditioning `(1, 21, 4096)` for the short prompt and `(1, 435, 4096)` for the long one; **both are NaN-free and Inf-free, and neither carries an `attention_mask`** (`extra` only contains `pooled_output`). The TE output looks healthy. - **Model / VAE / quantized weights / sampler settings**: identical settings under the default split attention produce correct images (C11, C13, C15b), and the same prompt renders the requested four-panel character sheet. ## Localization hints - Failing path: `attention_comfy_kitchen_int8` (`comfy/ldm/modules/attention.py`) → `comfy_kitchen.int8_attention` (`comfy_kitchen/sage_attention.py`) → `torch.ops.comfy_kitchen.int8_attention`, i.e. the **hip backend INT8 attention kernel**, not AOTriton/SDPA. - The trigger correlates with the **DiT prefix (text) length**, not with total sequence length: `prefix_len` goes from ~21 to ~435 while the image side changes from 4096 to 1024 tokens — and it breaks in both directions. - Worth checking in the hip kernel: the KV padding path (`_sage_buffers`: `padded_k = -(-kv_length // cta_k) * cta_k`, `_SAGE_CTA_K = 64`) and the K "stabilization detector" anchor sampling (`anchor_indices`, one key index per batch/KV head read by the K quantizer), for inputs whose K length is dominated by a long prefix rather than by the padded image grid. - Note the failure is **silent** — no exception, no NaN, `status: success`, and normal speed (2.4 s/it at 1024×1824 vs 55 s/it for split). This matches the "attention kernel returns wrong results without surfacing an error" pattern reported for other AMD attention paths (e.g. ROCm/aotriton#210). ## Impact Qwen-Image 2.1 `int8_convrot` + `comfy kitchen attention` is effectively unusable on AMD RDNA3 for prompt-enhancer workflows, whose enhanced prompts are typically 300–600 tokens. The only working alternative on this machine is the default `attention_split`, which is ~23× slower at 1024×1824. Note: the `pytorch attention` option is **not** a workaround for this checkpoint either — it raises `RuntimeError: convrot global spill requires caller-provided workspace buffers`. <img width="1024" height="1024" alt="Image" src="https://github.com/user-attachments/assets/d85515dd-ab9a-40e8-b1be-776b8fe0aab0" /> <img width="512" height="512" alt="Image" src="https://github.com/user-attachments/assets/37a9b020-5ffe-4b53-8243-18ebfe5ea7b9" /> <img width="512" height="512" alt="Image" src="https://github.com/user-attachments/assets/57f98f01-d50f-48cd-af71-1a6cc5e8d86d" />
bsakaguchi commented 2026-10-26 22:19:54 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Oct 4, 2026):

This is the same bug as #214: comfy-kitchen 0.2.36 added a prepared dense-mask path to the HIP int8 attention kernel, and on gfx11 fast-math folded away the empty-tile guard, so fully masked key tiles produced NaN.

That also explains why only prompt length matters. In Qwen Image 2.1 the text tokens attend with a causal bool mask, while the image queries attend without any mask. A longer text segment sends that causal mask through the dense path and produces fully masked tiles; image resolution only changes the unmasked part, so it has no effect.

Fixed in 19ea55b and released in comfy-kitchen 0.2.37, which current ComfyUI master requires. Could you update (pip install comfy-kitchen==0.2.37) and rerun the 232-token reproducer to confirm?

<!-- gh-comment-id:5983063114 --> @0xDELUXA commented on GitHub (Oct 4, 2026): This is the same bug as #214: comfy-kitchen 0.2.36 added a prepared dense-mask path to the HIP int8 attention kernel, and on gfx11 fast-math folded away the empty-tile guard, so fully masked key tiles produced NaN. That also explains why only prompt length matters. In Qwen Image 2.1 the text tokens attend with a causal bool mask, while the image queries attend without any mask. A longer text segment sends that causal mask through the dense path and produces fully masked tiles; image resolution only changes the unmasked part, so it has no effect. Fixed in 19ea55b and released in comfy-kitchen 0.2.37, which current ComfyUI master requires. Could you update (`pip install comfy-kitchen==0.2.37`) and rerun the 232-token reproducer to confirm?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev/adaptive-w3w5
dev/w3a8-gateup
dev/relayout-combine
dev/hip-residual-inplace
dev/qwen-adaptive-tree-pr
dev/h3-attn-layout-ci-ccache
dev/h3-attn-layout
dev/h3-quantizer-perf
dev/amd-resident-linear-fusions
dev/ada-unified-tree
dev/flash-decode-review
dev/qwen-decode-series
dev/bf16-delta-state
dev/int8-kv-decode
dev/qwen38-w6a8
matt/be-3580-agents-md
chore/node-runtime-refresh-2026-09-28
dev/cache-registry
feature/147-sageattention-wheels
fix/cla-api-failure-handling
docs/torch-min-version-2.7
alexisrolland-patch-2
alexisrolland-patch-1
dev/optim_int4
add-cla-workflow
dev/hip1
refactor-1
v0.2.40
v0.2.39
v0.2.38
v0.2.37
v0.2.36
v0.2.35
v0.2.34
v0.2.33
v0.2.32
v0.2.31
v0.2.30
v0.2.29
v0.2.28
v0.2.27
v0.2.26
v0.2.25
v0.2.24
v0.2.23
v0.2.22
v0.2.21
v0.2.20
v0.2.19
v0.2.18
v0.2.17
v0.2.16
v0.2.15
v0.2.14
v0.2.13
v0.2.12
v0.2.11
v0.2.10
v0.2.9
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Comfy-Org/comfy-kitchen#65
No description provided.