Comfy-Org/comfy-kitchen
1
0
Fork 0
mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00
Code Issues 118 Projects Packages Wiki Activity

[GH-ISSUE #125] Segfault in HIP dequantize_per_tensor_fp8 on gfx1201 (RDNA4) #40

New issue
Open
opened 2026-10-26 22:19:35 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 22:19:35 +00:00
Owner
Copy link

Originally created by @fgheorghe on GitHub (Aug 21, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/125

Environment
GPU AMD Radeon AI PRO R9700, gfx1201 (RDNA4), 32 GB
comfy-kitchen 0.2.31
PyTorch 2.13.0+rocm7.2
ROCm 7.2
Python 3.12.13
OS Ubuntu 22.04 (Docker, rocm/dev-ubuntu-22.04:7.2.4-complete)
ComfyUI 0.21.1

Description

Hard segfault (no Python exception, process dies) in the HIP backend's
dequantize_per_tensor_fp8 when an fp8-quantized text encoder runs its first
forward pass.

Setting COMFY_KITCHEN_DISABLE_HIP=1 avoids the crash entirely, and the same
workflow then completes correctly via the eager backend - so the inputs
themselves are fine.

This is not an architecture mismatch. comfy-kitchen was rebuilt explicitly
for the local target:

COMFY_KITCHEN_BUILD_HIP=1 COMFY_HIP_ARCHS=gfx1201 \
  uv pip install --no-build-isolation git+https://github.com/Comfy-Org/comfy-kitchen.git

The crash reproduces identically before and after that rebuild.

Steps to reproduce

  1. ComfyUI on ROCm 7.2 / gfx1201, comfy-kitchen with the HIP backend active
  2. Load an fp8-scaled text encoder (qwen3vl_4b_fp8_scaled.safetensors)
  3. Run any workflow that encodes a prompt through it
  4. Segfault at the first token generation step

Backend detection at startup

HIP reports available and enabled, and lists dequantize_per_tensor_fp8 among
its capabilities - so this is a kernel that claims support, not a fallback path:

Found comfy_kitchen backend hip: {'available': True, 'disabled': False,
 'unavailable_reason': None, 'capabilities': [..., 'dequantize_per_tensor_fp8', ...]}

Stack trace

Generating tokens:   0%|          | 0/512 [00:00<?, ?it/s]Fatal Python error: Segmentation fault
 
Stack (most recent call first):
  File ".../comfy_kitchen/backends/hip/__init__.py", line 310 in dequantize_per_tensor_fp8
  File ".../comfy_kitchen/backends/eager/quantization.py", line 461 in _op_dequantize_fp8
  File ".../torch/_library/custom_ops.py", line 502 in wrapped_fn
  File ".../torch/_dynamo/eval_frame.py", line 1446 in _fn
  File ".../torch/_compile.py", line 54 in inner
  File ".../torch/_library/custom_ops.py", line 441 in backend_impl
  File ".../torch/_ops.py", line 1279 in __call__
  File ".../comfy_kitchen/__init__.py", line 273 in dequantize_per_tensor_fp8
  File ".../comfy_kitchen/tensor/fp8.py", line 71 in dequantize
  File ".../comfy_kitchen/tensor/base.py", line 291 in dequantize
  File ".../comfy_kitchen/tensor/base.py", line 382 in dequantize_args
  File ".../comfy_kitchen/tensor/base.py", line 386 in <genexpr>
  File ".../comfy_kitchen/tensor/base.py", line 386 in dequantize_args
  File ".../comfy_kitchen/tensor/fp8.py", line 145 in _handle_fp8_linear
  File ".../comfy_kitchen/tensor/base.py", line 362 in __torch_dispatch__
  File "/ComfyUI/comfy/ops.py", line 1328 in _forward
  File "/ComfyUI/comfy/ops.py", line 1345 in forward_comfy_cast_weights
  File "/ComfyUI/comfy/ops.py", line 1414 in forward
  File "/ComfyUI/comfy/text_encoders/llama.py", line 554 in forward
  File "/ComfyUI/comfy/text_encoders/llama.py", line 673 in forward
  File "/ComfyUI/comfy/text_encoders/llama.py", line 914 in core
  File "/ComfyUI/comfy/model_prefetch.py", line 66 in prefetch_queue_pop
  File "/ComfyUI/comfy/text_encoders/llama.py", line 924 in forward
  File "/ComfyUI/comfy/text_encoders/llama.py", line 1052 in generate
  File "/ComfyUI/comfy/text_encoders/qwen3vl.py", line 136 in generate
  File "/ComfyUI/comfy/sd1_clip.py", line 750 in generate
  File "/ComfyUI/comfy/sd.py", line 477 in generate
  File "/ComfyUI/comfy_extras/nodes_textgen.py", line 64 in execute
  ...

Workaround

COMFY_KITCHEN_DISABLE_HIP=1

Originally created by @fgheorghe on GitHub (Aug 21, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/125 ## Environment | | | |---|---| | GPU | AMD Radeon AI PRO R9700, gfx1201 (RDNA4), 32 GB | | comfy-kitchen | 0.2.31 | | PyTorch | 2.13.0+rocm7.2 | | ROCm | 7.2 | | Python | 3.12.13 | | OS | Ubuntu 22.04 (Docker, `rocm/dev-ubuntu-22.04:7.2.4-complete`) | | ComfyUI | 0.21.1 | ## Description Hard segfault (no Python exception, process dies) in the HIP backend's `dequantize_per_tensor_fp8` when an fp8-quantized text encoder runs its first forward pass. Setting `COMFY_KITCHEN_DISABLE_HIP=1` avoids the crash entirely, and the same workflow then completes correctly via the eager backend - so the inputs themselves are fine. **This is not an architecture mismatch.** comfy-kitchen was rebuilt explicitly for the local target: ``` COMFY_KITCHEN_BUILD_HIP=1 COMFY_HIP_ARCHS=gfx1201 \ uv pip install --no-build-isolation git+https://github.com/Comfy-Org/comfy-kitchen.git ``` The crash reproduces identically before and after that rebuild. ## Steps to reproduce 1. ComfyUI on ROCm 7.2 / gfx1201, comfy-kitchen with the HIP backend active 2. Load an fp8-scaled text encoder (`qwen3vl_4b_fp8_scaled.safetensors`) 3. Run any workflow that encodes a prompt through it 4. Segfault at the first token generation step ## Backend detection at startup HIP reports available and enabled, and lists `dequantize_per_tensor_fp8` among its capabilities - so this is a kernel that claims support, not a fallback path: ``` Found comfy_kitchen backend hip: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': [..., 'dequantize_per_tensor_fp8', ...]} ``` ## Stack trace ``` Generating tokens: 0%| | 0/512 [00:00<?, ?it/s]Fatal Python error: Segmentation fault Stack (most recent call first): File ".../comfy_kitchen/backends/hip/__init__.py", line 310 in dequantize_per_tensor_fp8 File ".../comfy_kitchen/backends/eager/quantization.py", line 461 in _op_dequantize_fp8 File ".../torch/_library/custom_ops.py", line 502 in wrapped_fn File ".../torch/_dynamo/eval_frame.py", line 1446 in _fn File ".../torch/_compile.py", line 54 in inner File ".../torch/_library/custom_ops.py", line 441 in backend_impl File ".../torch/_ops.py", line 1279 in __call__ File ".../comfy_kitchen/__init__.py", line 273 in dequantize_per_tensor_fp8 File ".../comfy_kitchen/tensor/fp8.py", line 71 in dequantize File ".../comfy_kitchen/tensor/base.py", line 291 in dequantize File ".../comfy_kitchen/tensor/base.py", line 382 in dequantize_args File ".../comfy_kitchen/tensor/base.py", line 386 in <genexpr> File ".../comfy_kitchen/tensor/base.py", line 386 in dequantize_args File ".../comfy_kitchen/tensor/fp8.py", line 145 in _handle_fp8_linear File ".../comfy_kitchen/tensor/base.py", line 362 in __torch_dispatch__ File "/ComfyUI/comfy/ops.py", line 1328 in _forward File "/ComfyUI/comfy/ops.py", line 1345 in forward_comfy_cast_weights File "/ComfyUI/comfy/ops.py", line 1414 in forward File "/ComfyUI/comfy/text_encoders/llama.py", line 554 in forward File "/ComfyUI/comfy/text_encoders/llama.py", line 673 in forward File "/ComfyUI/comfy/text_encoders/llama.py", line 914 in core File "/ComfyUI/comfy/model_prefetch.py", line 66 in prefetch_queue_pop File "/ComfyUI/comfy/text_encoders/llama.py", line 924 in forward File "/ComfyUI/comfy/text_encoders/llama.py", line 1052 in generate File "/ComfyUI/comfy/text_encoders/qwen3vl.py", line 136 in generate File "/ComfyUI/comfy/sd1_clip.py", line 750 in generate File "/ComfyUI/comfy/sd.py", line 477 in generate File "/ComfyUI/comfy_extras/nodes_textgen.py", line 64 in execute ... ``` ## Workaround `COMFY_KITCHEN_DISABLE_HIP=1`
bsakaguchi commented 2026-10-26 22:19:36 +00:00
Author
Owner
Copy link

@0xDELUXA commented on GitHub (Aug 22, 2026):

Upgrade ROCm/PyTorch to the latest TheRock nightly.

<!-- gh-comment-id:5379109198 --> @0xDELUXA commented on GitHub (Aug 22, 2026): Upgrade ROCm/PyTorch to the latest [TheRock](https://github.com/ROCm/TheRock) nightly.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev/adaptive-w3w5
dev/w3a8-gateup
dev/relayout-combine
dev/hip-residual-inplace
dev/qwen-adaptive-tree-pr
dev/h3-attn-layout-ci-ccache
dev/h3-attn-layout
dev/h3-quantizer-perf
dev/amd-resident-linear-fusions
dev/ada-unified-tree
dev/flash-decode-review
dev/qwen-decode-series
dev/bf16-delta-state
dev/int8-kv-decode
dev/qwen38-w6a8
matt/be-3580-agents-md
chore/node-runtime-refresh-2026-09-28
dev/cache-registry
feature/147-sageattention-wheels
fix/cla-api-failure-handling
docs/torch-min-version-2.7
alexisrolland-patch-2
alexisrolland-patch-1
dev/optim_int4
add-cla-workflow
dev/hip1
refactor-1
v0.2.40
v0.2.39
v0.2.38
v0.2.37
v0.2.36
v0.2.35
v0.2.34
v0.2.33
v0.2.32
v0.2.31
v0.2.30
v0.2.29
v0.2.28
v0.2.27
v0.2.26
v0.2.25
v0.2.24
v0.2.23
v0.2.22
v0.2.21
v0.2.20
v0.2.19
v0.2.18
v0.2.17
v0.2.16
v0.2.15
v0.2.14
v0.2.13
v0.2.12
v0.2.11
v0.2.10
v0.2.9
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.6
v0.1.5
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Comfy-Org/comfy-kitchen#40
No description provided.