Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[PR #250] [MERGED] qwen 3.x LLM bugfixes and performance improvement pack #256

New issue

Closed

opened 2026-10-26 22:22:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:22:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/250
Author: @rattus128
Created: 10/14/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @comfyanonymous

Base: main ← Head: prs/qwen-3x-general-fixes

📝 Commits (5)

22d843f Fix concurrent flash decode counter ownership
8580491 Remove fused flash decode split combine
94f9cf4 Add deferred DeltaNet kernels for Ampere and Ada
30b3931 Restore noncausal HIP flash decode performance
f845771 Add persistent INT8 decode cache on HIP

📊 Changes

15 files changed (+614 additions, -265 deletions)

View changed files

📝 comfy_kitchen/backends/cuda/dlpack_bindings.cpp (+7 -25)
📝 comfy_kitchen/backends/cuda/ops/flash_decode.cu (+8 -107)
📝 comfy_kitchen/backends/cuda/ops/gated_delta.cu (+283 -8)
📝 comfy_kitchen/backends/hip/CMakeLists.txt (+1 -0)
📝 comfy_kitchen/backends/hip/dlpack_bindings.cpp (+3 -0)
➕ comfy_kitchen/backends/hip/int8_decode_bindings.cpp (+49 -0)
📝 comfy_kitchen/backends/hip/ops/flash_decode.hip (+20 -18)
📝 comfy_kitchen/backends/hip/sage_attention/int8_attn.hip (+95 -11)
📝 comfy_kitchen/backends/hip/sage_attention/quant_qk_int8.hip (+57 -11)
📝 comfy_kitchen/backends/hip/sage_attention/quant_v_int8.hip (+23 -2)
📝 comfy_kitchen/flash_attention.py (+2 -23)
📝 comfy_kitchen/gated_delta.py (+3 -4)
📝 comfy_kitchen/int8_decode.py (+17 -11)
📝 tests/test_flash_attention.py (+45 -43)
📝 tests/test_int8_attention.py (+1 -2)

📄 Description

Two quick bugfixes:

A theoretical race condition when multiple of the same kernel are running
A fusion with regressive performance. Retest on blackwell indicated negative value accross the board so straight revert. May re-introduce with appropriate conditionals if entangled with later stage speedups.

Other than than, the previous qwen 3.x PRs were overtuned for blackwell so fix with bringup for:

Ampere
Ada
AMD

Example Test Conditions:

ComfyUI: https://github.com/Comfy-Org/ComfyUI/pull/17007

Each of:

Linux, AMD R9700, 96GB; or
Linux, RTX6000 Ada; or
Linux, A40 Ampere

qwen 3.8 w4a8 creating writing with image input

Before:

AMD R9700:

Generating tokens:  91%|█████████ | 3715/4096 [01:53<00:11, 32.73it/s]
[INFO] Comfy model compiler graph breaks: 0, rogues: 0
[INFO] Prompt executed in 164.80 seconds

A40:

Generating tokens: 100%|██████████| 4096/4096 [02:32<00:00, 26.86it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 186.78 seconds

RTX6000 Ada

Generating tokens: 100%|██████████| 4096/4096 [01:34<00:00, 43.25it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 117.02 seconds

After:

AMD R9700 (note shorter output, minor win on step rate):

Generating tokens:  72%|███████▏  | 2967/4096 [01:13<00:27, 40.38it/s]
[INFO] Comfy model compiler graph breaks: 0, rogues: 0
[INFO] Prompt executed in 118.88 seconds

A40:

Generating tokens: 100%|██████████| 4096/4096 [01:36<00:00, 42.27it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 127.24 seconds

RTX6000 Ada

Generating tokens:  98%|█████████▊| 4013/4096 [01:01<00:01, 65.76it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 82.05 seconds

Same GPUs, long context:

Before:

R9700

Generating tokens:  51%|█████     | 2099/4096 [01:25<01:21, 24.65it/s]
[INFO] Comfy model compiler graph breaks: 0, rogues: 0
[INFO] Prompt executed in 132.73 seconds

A40:

Generating tokens:  56%|█████▋    | 2307/4096 [01:15<00:58, 30.42it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 153.21 seconds

RTX6000PRO Ada

Generating tokens:  59%|█████▉    | 2416/4096 [00:49<00:34, 48.31it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 85.80 seconds

After:

AMD R9700 (note significantly longer output):

Generating tokens:  64%|██████▍   | 2618/4096 [01:04<00:36, 40.55it/s]
[INFO] Comfy model compiler graph breaks: 0, rogues: 0
[INFO] Prompt executed in 112.34 seconds

A40 (note significantly longer output):

Generating tokens:  72%|███████▏  | 2943/4096 [00:58<00:22, 50.23it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 130.76 seconds

RTX6000 Ada

Generating tokens:  58%|█████▊    | 2392/4096 [00:29<00:21, 79.76it/s]
[INFO] Comfy model compiler graph breaks: 2, rogues: 1
[INFO] Prompt executed in 64.80 seconds

Regression Tests:

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/250 **Author:** [@rattus128](https://github.com/rattus128) **Created:** 10/14/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@comfyanonymous](https://github.com/comfyanonymous) **Base:** `main` ← **Head:** `prs/qwen-3x-general-fixes` --- ### 📝 Commits (5) - [`22d843f`](https://github.com/Comfy-Org/comfy-kitchen/commit/22d843f0f25646af2764e14e81d08c471f98029d) Fix concurrent flash decode counter ownership - [`8580491`](https://github.com/Comfy-Org/comfy-kitchen/commit/8580491360c4d82cf6c38f6c3f1b3ab9207ef1e9) Remove fused flash decode split combine - [`94f9cf4`](https://github.com/Comfy-Org/comfy-kitchen/commit/94f9cf4bf04518b4ce4c993941610eaec3ec5301) Add deferred DeltaNet kernels for Ampere and Ada - [`30b3931`](https://github.com/Comfy-Org/comfy-kitchen/commit/30b39310b085fefe5094e772c19494074c406f07) Restore noncausal HIP flash decode performance - [`f845771`](https://github.com/Comfy-Org/comfy-kitchen/commit/f8457710cf08f7c976814a33a5c9ac541be1154d) Add persistent INT8 decode cache on HIP ### 📊 Changes **15 files changed** (+614 additions, -265 deletions) <details> <summary>View changed files</summary> 📝 `comfy_kitchen/backends/cuda/dlpack_bindings.cpp` (+7 -25) 📝 `comfy_kitchen/backends/cuda/ops/flash_decode.cu` (+8 -107) 📝 `comfy_kitchen/backends/cuda/ops/gated_delta.cu` (+283 -8) 📝 `comfy_kitchen/backends/hip/CMakeLists.txt` (+1 -0) 📝 `comfy_kitchen/backends/hip/dlpack_bindings.cpp` (+3 -0) ➕ `comfy_kitchen/backends/hip/int8_decode_bindings.cpp` (+49 -0) 📝 `comfy_kitchen/backends/hip/ops/flash_decode.hip` (+20 -18) 📝 `comfy_kitchen/backends/hip/sage_attention/int8_attn.hip` (+95 -11) 📝 `comfy_kitchen/backends/hip/sage_attention/quant_qk_int8.hip` (+57 -11) 📝 `comfy_kitchen/backends/hip/sage_attention/quant_v_int8.hip` (+23 -2) 📝 `comfy_kitchen/flash_attention.py` (+2 -23) 📝 `comfy_kitchen/gated_delta.py` (+3 -4) 📝 `comfy_kitchen/int8_decode.py` (+17 -11) 📝 `tests/test_flash_attention.py` (+45 -43) 📝 `tests/test_int8_attention.py` (+1 -2) </details> ### 📄 Description Two quick bugfixes: - A theoretical race condition when multiple of the same kernel are running - A fusion with regressive performance. Retest on blackwell indicated negative value accross the board so straight revert. May re-introduce with appropriate conditionals if entangled with later stage speedups. Other than than, the previous qwen 3.x PRs were overtuned for blackwell so fix with bringup for: - Ampere - Ada - AMD Example Test Conditions: ComfyUI: https://github.com/Comfy-Org/ComfyUI/pull/17007 Each of: - Linux, AMD R9700, 96GB; or - Linux, RTX6000 Ada; or - Linux, A40 Ampere qwen 3.8 w4a8 creating writing with image input <img width="1445" height="872" alt="image" src="https://github.com/user-attachments/assets/38eaf6a1-e774-4d6c-ae25-b6e29a224115" /> Before: AMD R9700: ``` Generating tokens: 91%|█████████ | 3715/4096 [01:53<00:11, 32.73it/s] [INFO] Comfy model compiler graph breaks: 0, rogues: 0 [INFO] Prompt executed in 164.80 seconds ``` A40: ``` Generating tokens: 100%|██████████| 4096/4096 [02:32<00:00, 26.86it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 186.78 seconds ``` RTX6000 Ada ``` Generating tokens: 100%|██████████| 4096/4096 [01:34<00:00, 43.25it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 117.02 seconds ``` After: AMD R9700 (note shorter output, minor win on step rate): ``` Generating tokens: 72%|███████▏ | 2967/4096 [01:13<00:27, 40.38it/s] [INFO] Comfy model compiler graph breaks: 0, rogues: 0 [INFO] Prompt executed in 118.88 seconds ``` A40: ``` Generating tokens: 100%|██████████| 4096/4096 [01:36<00:00, 42.27it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 127.24 seconds ``` RTX6000 Ada ``` Generating tokens: 98%|█████████▊| 4013/4096 [01:01<00:01, 65.76it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 82.05 seconds ``` ------------------------------------------------------ Same GPUs, long context: <img width="1318" height="807" alt="image" src="https://github.com/user-attachments/assets/80fb824b-75bd-4698-a5b3-15655c5b0b0f" /> Before: R9700 ``` Generating tokens: 51%|█████ | 2099/4096 [01:25<01:21, 24.65it/s] [INFO] Comfy model compiler graph breaks: 0, rogues: 0 [INFO] Prompt executed in 132.73 seconds ``` A40: ``` Generating tokens: 56%|█████▋ | 2307/4096 [01:15<00:58, 30.42it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 153.21 seconds ``` RTX6000PRO Ada ``` Generating tokens: 59%|█████▉ | 2416/4096 [00:49<00:34, 48.31it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 85.80 seconds ``` After: AMD R9700 (note significantly longer output): ``` Generating tokens: 64%|██████▍ | 2618/4096 [01:04<00:36, 40.55it/s] [INFO] Comfy model compiler graph breaks: 0, rogues: 0 [INFO] Prompt executed in 112.34 seconds ``` A40 (note significantly longer output): ``` Generating tokens: 72%|███████▏ | 2943/4096 [00:58<00:22, 50.23it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 130.76 seconds ``` RTX6000 Ada ``` Generating tokens: 58%|█████▊ | 2392/4096 [00:29<00:21, 79.76it/s] [INFO] Comfy model compiler graph breaks: 2, rogues: 1 [INFO] Prompt executed in 64.80 seconds ``` Regression Tests: <img width="1080" height="610" alt="image" src="https://github.com/user-attachments/assets/5555c606-846d-4e39-a35c-d06481130138" /> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:22:10 +00:00