Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #61] Fatal access violation (0xC0000005) running int8_tensorwise model on ROCm Windows (gfx1101 / RDNA3) #22

New issue

Open

opened 2026-10-26 22:19:14 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:19:14 +00:00

Owner

Originally created by @kleintech on GitHub (Jun 30, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/61

Environment

GPU: AMD Radeon RX 7700 XT (RDNA3, gfx1101)
ROCm: 7.2 (torch 2.9.1+rocm7.2.1, torch.version.hip = "7.2")
Python: 3.12.12
OS: Windows 11 (ComfyUI Desktop, standalone env)
ComfyUI: nightly (post int8_tensorwise merge — QUANT_ALGOS["int8_tensorwise"] present)
comfy-kitchen: 0.2.15
Model: Chroma1-HD, int8mixed_tensorwise quant (*_learned_int8mixed_tensorwise.safetensors)

Problem

Loading and running an int8_tensorwise mixed-precision diffusion model on ROCm/Windows (gfx1101) causes a fatal Windows access violation (0xC0000005) on the first sampling step. The model loads fully into VRAM without error; the crash happens the moment the int8 matmul path executes.

This is not a catchable Python exception — the process is killed by the OS before Python regains control, so no traceback reaches ComfyUI. The Windows fault dump shows only CPython interpreter frames (asyncio event loop driving node execution); the faulting frame is in the native int8 GEMM extension and is not symbolized.

Startup confirms the native int8 path is selected (and is the only quant op not emulated on this card):

Found comfy_kitchen backend cuda: {'available': True, 'disabled': True, ...}
Found comfy_kitchen backend triton: {'available': False, 'unavailable_reason': "ImportError: No module named 'triton'", ...}
Found comfy_kitchen backend eager: {'available': True, 'disabled': False, ...}
...
Detected mixed precision quantization
Native ops: int8_tensorwise , emulated ops: float8_e4m3fn, nvfp4, float8_e5m2, mxfp8
model_type FLUX
...
loaded completely; 10368.17 MB usable, 8767.38 MB loaded, full load: True
<process dies with 0xC0000005 on first step>

Root cause (analysis)

On ROCm, fp8 / nvfp4 / mxfp8 are correctly pushed to the emulated path. But the int8_tensorwise GEMM (TensorWiseINT8Layout → eager backend) runs a native HIP kernel that segfaults on gfx1101 — the same failure class as #32 (fp8 _scaled_mm_v2 on RDNA4) and #55 (native INT8 on CUDA SM75), but on a path neither of those covers:

#32 guarded the fp8 scaled_mm_v2 path with torch.version.hip is None and was fixed by pytorch PR #179138 — that fix does not cover the int8 layout path.
#55 is the int8 native backend, but CUDA SM75 — a different platform from ROCm.

So int8-on-ROCm currently has no working native kernel, no emulation fallback, and no guard. Relevant: in ComfyUI core comfy/ops.py::pick_operations, the disabled set is built from supports_fp8_compute / supports_nvfp4_compute / supports_mxfp8_compute. There is no supports_int8_compute check, so int8_tensorwise is never added to disabled, is always dispatched native, and has no CLI/config toggle to force emulation (unlike fp8/fp4).

Things tried (no effect)

TORCH_BLAS_PREFER_HIPBLASLT=0
Updating ComfyUI + comfy-kitchen to latest nightly (this is what enabled the format in the first place)

Works correctly on the same system

The fp8mixed variant of the same model (Chroma1-HD-fp8mixed.safetensors) — fp8 runs through the emulated path and generates fine.

Expected behavior

On hardware where the native int8 kernel is unavailable/broken, int8_tensorwise should either:

Route through a working path (analogous to the #32 hip-guard / torch._scaled_mm fallback), or
Fall back to emulation (which would require a supports_int8_compute-style gate in ComfyUI core so the format can be added to disabled), or
At minimum, raise an explicit "int8 unsupported on this device" error instead of a hard 0xC0000005 segfault.

Actual behavior

Hard native access violation (0xC0000005) on the first sampling step; the whole ComfyUI process dies and cannot be caught or recovered.

Originally created by @kleintech on GitHub (Jun 30, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/61 ## Environment - **GPU:** AMD Radeon RX 7700 XT (RDNA3, `gfx1101`) - **ROCm:** `7.2` (`torch 2.9.1+rocm7.2.1`, `torch.version.hip = "7.2"`) - **Python:** 3.12.12 - **OS:** Windows 11 (ComfyUI Desktop, standalone env) - **ComfyUI:** nightly (post `int8_tensorwise` merge — `QUANT_ALGOS["int8_tensorwise"]` present) - **comfy-kitchen:** 0.2.15 - **Model:** Chroma1-HD, `int8mixed_tensorwise` quant (`*_learned_int8mixed_tensorwise.safetensors`) ## Problem Loading and running an **`int8_tensorwise`** mixed-precision diffusion model on ROCm/Windows (gfx1101) causes a fatal Windows access violation (`0xC0000005`) on the **first sampling step**. The model loads fully into VRAM without error; the crash happens the moment the int8 matmul path executes. This is **not a catchable Python exception** — the process is killed by the OS before Python regains control, so no traceback reaches ComfyUI. The Windows fault dump shows only CPython interpreter frames (asyncio event loop driving node execution); the faulting frame is in the native int8 GEMM extension and is not symbolized. Startup confirms the native int8 path is selected (and is the only quant op *not* emulated on this card): ``` Found comfy_kitchen backend cuda: {'available': True, 'disabled': True, ...} Found comfy_kitchen backend triton: {'available': False, 'unavailable_reason': "ImportError: No module named 'triton'", ...} Found comfy_kitchen backend eager: {'available': True, 'disabled': False, ...} ... Detected mixed precision quantization Native ops: int8_tensorwise , emulated ops: float8_e4m3fn, nvfp4, float8_e5m2, mxfp8 model_type FLUX ... loaded completely; 10368.17 MB usable, 8767.38 MB loaded, full load: True <process dies with 0xC0000005 on first step> ``` ## Root cause (analysis) On ROCm, fp8 / nvfp4 / mxfp8 are correctly pushed to the **emulated** path. But the `int8_tensorwise` GEMM (`TensorWiseINT8Layout` → eager backend) runs a native HIP kernel that segfaults on `gfx1101` — the same failure *class* as #32 (fp8 `_scaled_mm_v2` on RDNA4) and #55 (native INT8 on CUDA SM75), but on a path neither of those covers: - **#32** guarded the **fp8** `scaled_mm_v2` path with `torch.version.hip is None` and was fixed by pytorch PR #179138 — that fix does **not** cover the int8 layout path. - **#55** is the **int8** native backend, but CUDA SM75 — a different platform from ROCm. So int8-on-ROCm currently has **no working native kernel, no emulation fallback, and no guard**. Relevant: in ComfyUI core `comfy/ops.py::pick_operations`, the `disabled` set is built from `supports_fp8_compute` / `supports_nvfp4_compute` / `supports_mxfp8_compute`. There is **no `supports_int8_compute` check**, so `int8_tensorwise` is never added to `disabled`, is always dispatched native, and has no CLI/config toggle to force emulation (unlike fp8/fp4). ## Things tried (no effect) - `TORCH_BLAS_PREFER_HIPBLASLT=0` - Updating ComfyUI + comfy-kitchen to latest nightly (this is what enabled the format in the first place) ## Works correctly on the same system - The **fp8mixed** variant of the same model (`Chroma1-HD-fp8mixed.safetensors`) — fp8 runs through the emulated path and generates fine. ## Expected behavior On hardware where the native int8 kernel is unavailable/broken, `int8_tensorwise` should either: 1. Route through a working path (analogous to the #32 hip-guard / `torch._scaled_mm` fallback), or 2. Fall back to emulation (which would require a `supports_int8_compute`-style gate in ComfyUI core so the format can be added to `disabled`), or 3. At minimum, raise an explicit "int8 unsupported on this device" error instead of a hard `0xC0000005` segfault. ## Actual behavior Hard native access violation (`0xC0000005`) on the first sampling step; the whole ComfyUI process dies and cannot be caught or recovered.

bsakaguchi commented

2026-10-26 22:19:15 +00:00

Author

Owner

@0xDELUXA commented on GitHub (Jul 2, 2026):

Please test this again on your system using the latest versions of ComfyUI/comfy-kitchen.

@0xDELUXA commented on GitHub (Jul 2, 2026): Please test this again on your system using the latest versions of ComfyUI/comfy-kitchen.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-kitchen#22

No description provided.

Rows
Columns