Comfy-Org/comfy-kitchen
1
0
Fork 0
mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-05 07:47:40 +00:00
Code Issues 118 Projects Packages Wiki Activity

[GH-ISSUE #71] ComfyUI crashes using Flux Klein 9b with Triton backend - dequantize_nvfp4_kernel_tl #26

New issue
Open
opened 2026-10-26 22:19:18 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 22:19:18 +00:00
Owner
Copy link

Originally created by @waltercool on GitHub (Jul 10, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/71

Hi there,

As reported https://github.com/Comfy-Org/ComfyUI/issues/14510, ComfyUI crashes when rendering under the triton kernel.

The ComfyUI log is not much helpful, however, the MLIR dump blames dequantize_nvfp4_kernel_tl :

#di_subroutine_type = #llvm.di_subroutine_type<callingConvention = DW_CC_normal, types = #di_null_type, #di_derived_type, #di_derived_type1, #di_derived_type2, #di_derived_type2, #di_basic_type3, #di_basic_type3, #di_basic_type3, #di_basic_type3, #di_derived_type1, #di_derived_type1>
#di_subprogram = #llvm.di_subprogram<recId = distinct[0]<>, id = distinct[1]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "dequantize_nvfp4_kernel_tl", linkageName = "dequantize_nvfp4_kernel_tl", file = #di_file, line = 418, scopeLine = 418, subprogramFlags = "Definition|Optimized", type = #di_subroutine_type>
#di_lexical_block_file = #llvm.di_lexical_block_file<scope = #di_subprogram, file = #di_file, discriminator = 0>
#loc133 = loc(fused<#di_subprogram>[#loc])
#loc134 = loc(fused<#di_lexical_block_file>[#loc97])
#loc135 = loc(fused<#di_lexical_block_file>[#loc98])
#loc136 = loc(fused<#di_lexical_block_file>[#loc99])
#loc137 = loc(fused<#di_lexical_block_file>[#loc100])
#loc138 = loc(fused<#di_lexical_block_file>[#loc101])
#loc139 = loc(fused<#di_lexical_block_file>[#loc102])
#loc140 = loc(fused<#di_lexical_block_file>[#loc103])
#loc141 = loc(fused<#di_lexical_block_file>[#loc104])
#loc142 = loc(fused<#di_lexical_block_file>[#loc105])
#loc143 = loc(fused<#di_lexical_block_file>[#loc106])
#loc144 = loc(fused<#di_lexical_block_file>[#loc107])
#loc145 = loc(fused<#di_lexical_block_file>[#loc108])
#loc146 = loc(fused<#di_lexical_block_file>[#loc109])
#loc147 = loc(fused<#di_lexical_block_file>[#loc110])
#loc148 = loc(fused<#di_lexical_block_file>[#loc111])
#loc149 = loc(fused<#di_lexical_block_file>[#loc112])
#loc150 = loc(fused<#di_lexical_block_file>[#loc113])
#loc151 = loc(fused<#di_lexical_block_file>[#loc114])
#loc152 = loc(callsite(#loc134 at #loc135))
#loc153 = loc(callsite(#loc136 at #loc135))
#loc154 = loc(callsite(#loc137 at #loc135))
#loc155 = loc(callsite(#loc138 at #loc135))
#loc156 = loc(callsite(#loc139 at #loc135))
#loc157 = loc(callsite(#loc140 at #loc135))
#loc158 = loc(callsite(#loc141 at #loc135))
#loc159 = loc(callsite(#loc142 at #loc135))
#loc160 = loc(callsite(#loc143 at #loc135))
#loc161 = loc(callsite(#loc144 at #loc135))
#loc162 = loc(callsite(#loc145 at #loc135))
#loc163 = loc(callsite(#loc146 at #loc135))
#loc164 = loc(callsite(#loc147 at #loc135))
#loc165 = loc(callsite(#loc148 at #loc135))
#loc166 = loc(callsite(#loc149 at #loc135))
#loc167 = loc(callsite(#loc150 at #loc135))
#loc168 = loc(callsite(#loc136 at #loc151))
#loc169 = loc(callsite(#loc138 at #loc151))
#loc170 = loc(callsite(#loc142 at #loc151))
#loc171 = loc(callsite(#loc144 at #loc151))
#loc172 = loc(callsite(#loc145 at #loc151))
#loc173 = loc(callsite(#loc147 at #loc151))
#loc174 = loc(callsite(#loc148 at #loc151))
#loc175 = loc(callsite(#loc149 at #loc151))
#loc176 = loc(callsite(#loc150 at #loc151))


error: couldn't allocate input reg for constraint 'r'

Triton version: 3.7.1
Comfyui version: 0.27.1
Comfyui kitchen version: 0.2.16
GPU: gfx1151
ROCm version: 7.2.0

Originally created by @waltercool on GitHub (Jul 10, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/71 Hi there, As reported https://github.com/Comfy-Org/ComfyUI/issues/14510, ComfyUI crashes when rendering under the triton kernel. The ComfyUI log is not much helpful, however, the MLIR dump blames dequantize_nvfp4_kernel_tl : ``` #di_subroutine_type = #llvm.di_subroutine_type<callingConvention = DW_CC_normal, types = #di_null_type, #di_derived_type, #di_derived_type1, #di_derived_type2, #di_derived_type2, #di_basic_type3, #di_basic_type3, #di_basic_type3, #di_basic_type3, #di_derived_type1, #di_derived_type1> #di_subprogram = #llvm.di_subprogram<recId = distinct[0]<>, id = distinct[1]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "dequantize_nvfp4_kernel_tl", linkageName = "dequantize_nvfp4_kernel_tl", file = #di_file, line = 418, scopeLine = 418, subprogramFlags = "Definition|Optimized", type = #di_subroutine_type> #di_lexical_block_file = #llvm.di_lexical_block_file<scope = #di_subprogram, file = #di_file, discriminator = 0> #loc133 = loc(fused<#di_subprogram>[#loc]) #loc134 = loc(fused<#di_lexical_block_file>[#loc97]) #loc135 = loc(fused<#di_lexical_block_file>[#loc98]) #loc136 = loc(fused<#di_lexical_block_file>[#loc99]) #loc137 = loc(fused<#di_lexical_block_file>[#loc100]) #loc138 = loc(fused<#di_lexical_block_file>[#loc101]) #loc139 = loc(fused<#di_lexical_block_file>[#loc102]) #loc140 = loc(fused<#di_lexical_block_file>[#loc103]) #loc141 = loc(fused<#di_lexical_block_file>[#loc104]) #loc142 = loc(fused<#di_lexical_block_file>[#loc105]) #loc143 = loc(fused<#di_lexical_block_file>[#loc106]) #loc144 = loc(fused<#di_lexical_block_file>[#loc107]) #loc145 = loc(fused<#di_lexical_block_file>[#loc108]) #loc146 = loc(fused<#di_lexical_block_file>[#loc109]) #loc147 = loc(fused<#di_lexical_block_file>[#loc110]) #loc148 = loc(fused<#di_lexical_block_file>[#loc111]) #loc149 = loc(fused<#di_lexical_block_file>[#loc112]) #loc150 = loc(fused<#di_lexical_block_file>[#loc113]) #loc151 = loc(fused<#di_lexical_block_file>[#loc114]) #loc152 = loc(callsite(#loc134 at #loc135)) #loc153 = loc(callsite(#loc136 at #loc135)) #loc154 = loc(callsite(#loc137 at #loc135)) #loc155 = loc(callsite(#loc138 at #loc135)) #loc156 = loc(callsite(#loc139 at #loc135)) #loc157 = loc(callsite(#loc140 at #loc135)) #loc158 = loc(callsite(#loc141 at #loc135)) #loc159 = loc(callsite(#loc142 at #loc135)) #loc160 = loc(callsite(#loc143 at #loc135)) #loc161 = loc(callsite(#loc144 at #loc135)) #loc162 = loc(callsite(#loc145 at #loc135)) #loc163 = loc(callsite(#loc146 at #loc135)) #loc164 = loc(callsite(#loc147 at #loc135)) #loc165 = loc(callsite(#loc148 at #loc135)) #loc166 = loc(callsite(#loc149 at #loc135)) #loc167 = loc(callsite(#loc150 at #loc135)) #loc168 = loc(callsite(#loc136 at #loc151)) #loc169 = loc(callsite(#loc138 at #loc151)) #loc170 = loc(callsite(#loc142 at #loc151)) #loc171 = loc(callsite(#loc144 at #loc151)) #loc172 = loc(callsite(#loc145 at #loc151)) #loc173 = loc(callsite(#loc147 at #loc151)) #loc174 = loc(callsite(#loc148 at #loc151)) #loc175 = loc(callsite(#loc149 at #loc151)) #loc176 = loc(callsite(#loc150 at #loc151)) error: couldn't allocate input reg for constraint 'r' ``` Triton version: 3.7.1 Comfyui version: 0.27.1 Comfyui kitchen version: 0.2.16 GPU: gfx1151 ROCm version: 7.2.0
bsakaguchi commented 2026-10-26 22:19:19 +00:00
Author
Owner
Copy link

@sleppyrobot commented on GitHub (Aug 4, 2026):

Can AMD users get a fix for this?
I merged the PR #47 that adds a check to the master and it did allow nvfp4 TE for miniH3 to work with enable-triton-backend. Its been broken for quite a long time

<!-- gh-comment-id:5177715945 --> @sleppyrobot commented on GitHub (Aug 4, 2026): Can AMD users get a fix for this? I merged the PR #47 that adds a check to the master and it did allow nvfp4 TE for miniH3 to work with enable-triton-backend. Its been broken for quite a long time
bsakaguchi commented 2026-10-26 22:19:19 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Aug 11, 2026):

Triton is only a fallback now, use HIP instead.

<!-- gh-comment-id:5255248991 --> @0xDELUXA commented on GitHub (Aug 11, 2026): Triton is only a fallback now, use HIP instead.
bsakaguchi commented 2026-10-26 22:19:19 +00:00
Author
Owner
Copy link

@waltercool commented on GitHub (Aug 11, 2026):

Are they killing the Triton backend? Damn, it's a bit faster than HIP.

I do understand most people feel Triton as a workaround than a solution for AMD cards.

<!-- gh-comment-id:5255708699 --> @waltercool commented on GitHub (Aug 11, 2026): Are they killing the Triton backend? Damn, it's a bit faster than HIP. I do understand most people feel Triton as a workaround than a solution for AMD cards.
bsakaguchi commented 2026-10-26 22:19:19 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Aug 11, 2026):

Are they killing the Triton backend? Damn, it's a bit faster than HIP.

No, the Triton backend will continue to exist.

I do understand most people feel Triton as a workaround than a solution for AMD cards.

HIP is as native to AMD as CUDA is to NVIDIA, Triton is a fallback on both. It’s not just a matter of people considering it a workaround.

<!-- gh-comment-id:5256658518 --> @0xDELUXA commented on GitHub (Aug 11, 2026): > Are they killing the Triton backend? Damn, it's a bit faster than HIP. No, the Triton backend will continue to exist. > I do understand most people feel Triton as a workaround than a solution for AMD cards. HIP is as native to AMD as CUDA is to NVIDIA, Triton is a fallback on both. It’s not just a matter of people considering it a workaround.
bsakaguchi commented 2026-10-26 22:19:20 +00:00
Author
Owner
Copy link

@sleppyrobot commented on GitHub (Aug 11, 2026):

Triton has been the best option for a long time, assuming hip can match it then people probably wont use it.

The performance mainly affects Krea2 but it requires either int8 custom node or merging #68 to get the speed up on a 7900xtx.
For example, starting comfy with
python main.py --disable-dynamic-vram --async-offload 2 --vram-headroom 0.5 --disable-api-nodes --use-flash-attention for the eager test
and for Triton am merging pr #68 and disabling hip export COMFY_KITCHEN_DISABLE_HIP=1 using native comfy loader.

I had KJ Patch Sol-Attn node in the H3 test, i know it uses Triton somewhat but i think it uses the package and not anything in comfy-kitchen.

Model backend settings best of 3 inference speed
Kea2-int8 hip + eager cfg>1, 1008x1440 4.83 s/it
Kea2-int8 hip + enable-triton cfg>1, 1008x1440 4.83 s/it
Kea2-int8 enable-triton + #68 + hip disabled cfg>1, 1008x1440 4.51 s/it (winner)
H3-int8 eager cfg=1, 832x480, 4sec 11.55 s/it
H3-int8 enable-triton cfg=1, 832x480, 4sec 10.86 s/it (winner)
H3-int8 enable-triton + #68 + hip disabled cfg=1, 832x480, 4sec 13.85 s/it
H3-w4a8 enable-triton + #68 cfg=1, 832x480, 4sec 12.12 s/it
H3-w4a8 enable-triton + #68 + hip disabled cfg=1, 832x480, 4sec 13.85 s/it

H3 + triton + hip disabled was slowest and I noticed lower GPU usage for the test, i did another test with w4w8 model + triton with hip enabled and disabled.
Also doing this test with enabled-dynamic-vram will likely give slight better performance as there appears to be a bit of offloading with the H3 model.

<!-- gh-comment-id:5256848363 --> @sleppyrobot commented on GitHub (Aug 11, 2026): Triton has been the best option for a long time, assuming hip can match it then people probably wont use it. The performance mainly affects Krea2 but it requires either int8 custom node or merging #68 to get the speed up on a 7900xtx. For example, starting comfy with `python main.py --disable-dynamic-vram --async-offload 2 --vram-headroom 0.5 --disable-api-nodes --use-flash-attention` for the eager test and for Triton am merging pr #68 and disabling hip export `COMFY_KITCHEN_DISABLE_HIP=1` using native comfy loader. _I had KJ Patch Sol-Attn node in the H3 test, i know it uses Triton somewhat but i think it uses the package and not anything in comfy-kitchen._ |Model|backend|settings|best of 3 inference speed| |---|---|---|---| |Kea2-int8|hip + eager|cfg>1, 1008x1440|4.83 s/it| |Kea2-int8|hip + enable-triton|cfg>1, 1008x1440|4.83 s/it| |Kea2-int8|enable-triton + #68 + hip disabled |cfg>1, 1008x1440|4.51 s/it (winner)| |H3-int8|eager|cfg=1, 832x480, 4sec|11.55 s/it| |H3-int8|enable-triton|cfg=1, 832x480, 4sec|10.86 s/it (winner)| |H3-int8|enable-triton + #68 + hip disabled |cfg=1, 832x480, 4sec|13.85 s/it| |H3-w4a8|enable-triton + #68|cfg=1, 832x480, 4sec|12.12 s/it| |H3-w4a8|enable-triton + #68 + hip disabled |cfg=1, 832x480, 4sec|13.85 s/it| H3 + triton + hip disabled was slowest and I noticed lower GPU usage for the test, i did another test with w4w8 model + triton with hip enabled and disabled. Also doing this test with enabled-dynamic-vram will likely give slight better performance as there appears to be a bit of offloading with the H3 model.
bsakaguchi commented 2026-10-26 22:19:20 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Aug 11, 2026):

I'm not against improving the Triton backend on AMD at all. It's just that I'm not the person to decide which PRs to merge and which not to.

<!-- gh-comment-id:5256883236 --> @0xDELUXA commented on GitHub (Aug 11, 2026): I'm not against improving the Triton backend on AMD at all. It's just that I'm not the person to decide which PRs to merge and which not to.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev/adaptive-w3w5
dev/w3a8-gateup
dev/relayout-combine
dev/hip-residual-inplace
dev/qwen-adaptive-tree-pr
dev/h3-attn-layout-ci-ccache
dev/h3-attn-layout
dev/h3-quantizer-perf
dev/amd-resident-linear-fusions
dev/ada-unified-tree
dev/flash-decode-review
dev/qwen-decode-series
dev/bf16-delta-state
dev/int8-kv-decode
dev/qwen38-w6a8
matt/be-3580-agents-md
chore/node-runtime-refresh-2026-09-28
dev/cache-registry
feature/147-sageattention-wheels
fix/cla-api-failure-handling
docs/torch-min-version-2.7
alexisrolland-patch-2
alexisrolland-patch-1
dev/optim_int4
add-cla-workflow
dev/hip1
refactor-1
v0.2.41
v0.2.40
v0.2.39
v0.2.38
v0.2.37
v0.2.36
v0.2.35
v0.2.34
v0.2.33
v0.2.32
v0.2.31
v0.2.30
v0.2.29
v0.2.28
v0.2.27
v0.2.26
v0.2.25
v0.2.24
v0.2.23
v0.2.22
v0.2.21
v0.2.20
v0.2.19
v0.2.18
v0.2.17
v0.2.16
v0.2.15
v0.2.14
v0.2.13
v0.2.12
v0.2.11
v0.2.10
v0.2.9
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Comfy-Org/comfy-kitchen#26
No description provided.