Comfy-Org/comfy-kitchen
1
0
Fork 0
mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00
Code Issues 118 Projects Packages Wiki Activity

[GH-ISSUE #123] [HIP backend]Slow performance on Linux AMD. #38

New issue
Closed
opened 2026-10-26 22:19:33 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 22:19:33 +00:00
Owner
Copy link

Originally created by @jnolck on GitHub (Aug 18, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/123

I don't think it's intended to work this way but on a gfx1100 on the hip backend on Linux the eager backend is always taken.

With a w4a8 model.

[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend eager selected for w4a8_int8_linear
[DEBUG] Backend hip selected for apply_rope```

w4a4 model. 

[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear
[DEBUG] Backend eager selected for convrot_w4a4_linear

Int8 These are all krea models. 

[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend hip selected for apply_rope
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
[DEBUG] Backend eager selected for int8_linear
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:24<00:00,  3.05s/it]

Is there an env variable I'm missing for that sweet hand crafted wmma code to run on my card? Is it a bug? Is it intended?
Originally created by @jnolck on GitHub (Aug 18, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/123 I don't think it's intended to work this way but on a gfx1100 on the hip backend on Linux the eager backend is always taken. With a w4a8 model. ```[DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend eager selected for w4a8_int8_linear [DEBUG] Backend hip selected for apply_rope``` w4a4 model. [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear [DEBUG] Backend eager selected for convrot_w4a4_linear Int8 These are all krea models. [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend hip selected for apply_rope [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear [DEBUG] Backend eager selected for int8_linear 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:24<00:00, 3.05s/it] Is there an env variable I'm missing for that sweet hand crafted wmma code to run on my card? Is it a bug? Is it intended?
bsakaguchi commented 2026-10-26 22:19:34 +00:00
Author
Owner
Copy link

@jnolck commented on GitHub (Aug 18, 2026):

comfy-aimdo                                0.4.13
comfy-angle                                0.1.0
comfy-kitchen                              0.2.31
comfy-quants                               0.1.0
comfyui-embedded-docs                      0.5.10
rich                                       14.0.0
rocm                                       7.15.0a20260728
rocm-bootstrap                             0.1.0
rocm-profiler                              7.15.0a20260728
rocm-sdk-core                              7.15.0a20260728
rocm-sdk-devel                             7.15.0a20260728
rocm-sdk-device-gfx1100                    7.15.0a20260728
rocm-sdk-libraries                         7.15.0a20260728
rocm-sdk-libraries-gfx110x-all             7.13.0a20260423
rotary-embedding-torch                     0.9.1
rpds-py                                    2026.5.1
torch                                      2.13.0+rocm7.15.0a20260728
torchaudio                                 2.11.0.2+rocm7.15.0a20260728
torchdiffeq                                0.2.5
torchsde                                   0.2.6
torchvision                                0.28.0+rocm7.15.0a20260728
tqdm                                       4.67.1
trailrunner                                1.4.0
traitlets                                  5.15.1
trampoline                                 0.1.2
transformers                               5.13.0
trimesh                                    4.10.1
triton                                     3.8.0+git4cff872c.rocm7.15.0a20260728
triton-kernels                             1.0.0+amd.rocm7.0.0.gitd0d77a509
<!-- gh-comment-id:5329278941 --> @jnolck commented on GitHub (Aug 18, 2026): ``` comfy-aimdo 0.4.13 comfy-angle 0.1.0 comfy-kitchen 0.2.31 comfy-quants 0.1.0 comfyui-embedded-docs 0.5.10 rich 14.0.0 rocm 7.15.0a20260728 rocm-bootstrap 0.1.0 rocm-profiler 7.15.0a20260728 rocm-sdk-core 7.15.0a20260728 rocm-sdk-devel 7.15.0a20260728 rocm-sdk-device-gfx1100 7.15.0a20260728 rocm-sdk-libraries 7.15.0a20260728 rocm-sdk-libraries-gfx110x-all 7.13.0a20260423 rotary-embedding-torch 0.9.1 rpds-py 2026.5.1 torch 2.13.0+rocm7.15.0a20260728 torchaudio 2.11.0.2+rocm7.15.0a20260728 torchdiffeq 0.2.5 torchsde 0.2.6 torchvision 0.28.0+rocm7.15.0a20260728 tqdm 4.67.1 trailrunner 1.4.0 traitlets 5.15.1 trampoline 0.1.2 transformers 5.13.0 trimesh 4.10.1 triton 3.8.0+git4cff872c.rocm7.15.0a20260728 triton-kernels 1.0.0+amd.rocm7.0.0.gitd0d77a509 ```
bsakaguchi commented 2026-10-26 22:19:34 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Aug 18, 2026):

Try running:

import comfy_kitchen as ck
from comfy_kitchen.backends import hip
print("arches   :", hip._visible_gfx_arches())
print("available:", hip.is_available(), "| has_wmma:", hip.has_wmma())
caps = ck.list_backends()["hip"]["capabilities"]
for op in ("int8_linear", "w4a8_int8_linear", "convrot_w4a4_linear", "apply_rope"):
     print(f"  {op:22} advertised by hip: {op in caps}")

If arches shows more than one entry and has_wmma is False, hiding the non-WMMA device restores the kernels:

HIP_VISIBLE_DEVICES=0 python main.py

If instead it shows a single gfx1100 with has_wmma: True, then the ops are advertised and something is rejecting the individual calls, which is a different bug - in that case ck.registry.validate_backend_for_call("hip", "int8_linear", kwargs) returns the failing parameter and reason.

<!-- gh-comment-id:5329740088 --> @0xDELUXA commented on GitHub (Aug 18, 2026): Try running: ```python import comfy_kitchen as ck from comfy_kitchen.backends import hip print("arches :", hip._visible_gfx_arches()) print("available:", hip.is_available(), "| has_wmma:", hip.has_wmma()) caps = ck.list_backends()["hip"]["capabilities"] for op in ("int8_linear", "w4a8_int8_linear", "convrot_w4a4_linear", "apply_rope"): print(f" {op:22} advertised by hip: {op in caps}") ``` If arches shows more than one entry and `has_wmma` is False, hiding the non-WMMA device restores the kernels: `HIP_VISIBLE_DEVICES=0 python main.py` If instead it shows a single gfx1100 with `has_wmma: True`, then the ops are advertised and something is rejecting the individual calls, which is a different bug - in that case `ck.registry.validate_backend_for_call("hip", "int8_linear", kwargs)` returns the failing parameter and reason.
bsakaguchi commented 2026-10-26 22:19:34 +00:00
Author
Owner
Copy link

@jnolck commented on GitHub (Aug 19, 2026):

arches : ('gfx1100', 'gfx1036')
available: True | has_wmma: False
int8_linear advertised by hip: False
w4a8_int8_linear advertised by hip: False
convrot_w4a4_linear advertised by hip: False
apply_rope advertised by hip: True

Yup, that was it. starting with HIP_VISIBLE_DEVICES=0 solved the issue. Damn It's fast!!!

<!-- gh-comment-id:5346966883 --> @jnolck commented on GitHub (Aug 19, 2026): arches : ('gfx1100', 'gfx1036') available: True | has_wmma: False int8_linear advertised by hip: False w4a8_int8_linear advertised by hip: False convrot_w4a4_linear advertised by hip: False apply_rope advertised by hip: True Yup, that was it. starting with HIP_VISIBLE_DEVICES=0 solved the issue. Damn It's fast!!!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev/adaptive-w3w5
dev/w3a8-gateup
dev/relayout-combine
dev/hip-residual-inplace
dev/qwen-adaptive-tree-pr
dev/h3-attn-layout-ci-ccache
dev/h3-attn-layout
dev/h3-quantizer-perf
dev/amd-resident-linear-fusions
dev/ada-unified-tree
dev/flash-decode-review
dev/qwen-decode-series
dev/bf16-delta-state
dev/int8-kv-decode
dev/qwen38-w6a8
matt/be-3580-agents-md
chore/node-runtime-refresh-2026-09-28
dev/cache-registry
feature/147-sageattention-wheels
fix/cla-api-failure-handling
docs/torch-min-version-2.7
alexisrolland-patch-2
alexisrolland-patch-1
dev/optim_int4
add-cla-workflow
dev/hip1
refactor-1
v0.2.40
v0.2.39
v0.2.38
v0.2.37
v0.2.36
v0.2.35
v0.2.34
v0.2.33
v0.2.32
v0.2.31
v0.2.30
v0.2.29
v0.2.28
v0.2.27
v0.2.26
v0.2.25
v0.2.24
v0.2.23
v0.2.22
v0.2.21
v0.2.20
v0.2.19
v0.2.18
v0.2.17
v0.2.16
v0.2.15
v0.2.14
v0.2.13
v0.2.12
v0.2.11
v0.2.10
v0.2.9
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Comfy-Org/comfy-kitchen#38
No description provided.