mirror of
https://github.com/Comfy-Org/comfy-kitchen.git
synced 2026-11-04 23:37:42 +00:00
[GH-ISSUE #64] Memory access fault using Krea 2 INT8 CONVROT with Triton backend on AMD gfx1151 #24
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/comfy-kitchen#24
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @waltercool on GitHub (Jul 7, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/64
Hi there,
I'm testing INT8 models using ROCm and Triton backend. In this scenario, using Krea 2 leads me to a memory fault crash. This only happens when enabling Triton backend.
Tried using Flash Attention, Pytorch Cross Attention and Sub Quad Optimization, same result.
Disabling the Triton backend aparently works fine, just terribly slow, painfully slower than FP8.
Model used: https://huggingface.co/Comfy-Org/Krea-2/blob/main/diffusion_models/krea2_turbo_int8_convrot.safetensors
[ 5603.532594] amdgpu 0000:c4:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:94)
[ 5603.532595] amdgpu 0000:c4:00.0: Process python3 pid 135257 thread python3 pid 135257
[ 5603.532596] amdgpu 0000:c4:00.0: in page starting at address 0x00007f4d8a007000 from client 10
[ 5603.532598] amdgpu 0000:c4:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801031
[ 5603.532598] amdgpu 0000:c4:00.0: Faulty UTCL2 client ID: TCP (0x8)
[ 5603.532599] amdgpu 0000:c4:00.0: MORE_FAULTS: 0x1
[ 5603.532600] amdgpu 0000:c4:00.0: WALKER_ERROR: 0x0
[ 5603.532601] amdgpu 0000:c4:00.0: PERMISSION_FAULTS: 0x3
[ 5603.532602] amdgpu 0000:c4:00.0: MAPPING_ERROR: 0x0
[ 5603.532603] amdgpu 0000:c4:00.0: RW: 0x0
ComfyUI logs:
comfyui.log
Any extra analysis, let me know, happy to provide information.
ROCM version: 7.2.0
Triton version: 3.7.1
Comfy Kitchen version: 0.2.16
ComfyUI version: 0.27.0
Footnote:
I'm also using INT8 CLIP for text generation, and this seems to work fine. Probably 20% faster than FP8. It doesn't crash.
INT8 Clip used: https://huggingface.co/Winnougan/Comfy-Qwen3-VL-INT8/blob/main/qwen3vl_4b_int8_convrot.safetensors
FP8 Clip used:
huggingface.co/Comfy-Org/Qwen3-VL@8bb77f7ad3/text_encoders/qwen3vl_4b_fp8_scaled.safetensors@jnolck commented on GitHub (Jul 7, 2026):
I have the same issue. I can anima in int8 on the triton backend but not krea2. It segfaults when it hits the apply_rope variation that krea2 calls.
@chejh-amd commented on GitHub (Jul 8, 2026):
Hi @waltercool,
Since jnolck saw the crash on the apply_rope variant Krea2 uses, it looks like the CONVROT rope kernel isn't handling Krea2's INT8 layout quite right, maybe an off index or stride calculation.
Try run with
AMD_SERIALIZE_KERNEL=3 AMD_LOG_LEVEL=3. That forces a sync after each kernel launch, so the fault tends to get pinned on the kernel that actually triggered it rather than whatever shows up later.@waltercool commented on GitHub (Jul 8, 2026):
Hi @chejh-amd ,
Thank you very much for your attention at this matter.
This is the whole output:
comfy.log.gz
And here is probably what matters from that log:
Kind regards
@jnolck commented on GitHub (Jul 8, 2026):
@chejh-amd commented on GitHub (Jul 9, 2026):
Both your logs stop at
ShaderName : apply_rope_kernel, so the fault is most likely that RoPE kernel. From what the AMD driver reports, it doesn't look like an out-of-memory issue. It looks more like something about the tensor shapes/layout Krea2 feeds into that kernel than the backend itself.@0xDELUXA commented on GitHub (Jul 9, 2026):
@waltercool Can you try this custom node? On our end, it works better than the native implementation, which even comfyanonymous said isn't really ready for AMD yet.
@0xDELUXA commented on GitHub (Jul 9, 2026):
Exactly. That's why I opened this PR. If you want, you can apply its changes locally.
@waltercool commented on GitHub (Jul 9, 2026):
This definitively fixes the eager INT8, yet still slower than FP8 and probably not related to your PR.
This works nice to me, considerably faster than Eager! Thank you!
Just a summary for anyone who read this, some quick benchmark generating a 1Mpx image:
Using eager FP8: ~60s
Using eager INT8 without patches: ~210s, probably 10-15 minutes if you have text generation with INT8 CONVROT clip.
Using eager INT8 with patches: ~67s
Using ComfyUI-INT8-Fast-ROCM custom node: ~49s
Using triton: CRASHES
Using triton with fp8 scaled: ALSO CRASHES
Worth to mention: Using eager INT8 without patches, it creates some input lag in the host machine.
@bitgamma commented on GitHub (Jul 10, 2026):
doesn't seem to depend on INT8, I have the same issue with BF16 version of Krea 2 and Triton backend and nightly ROCm 7.15:
@sleppyrobot commented on GitHub (Jul 18, 2026):
#77 Fyi
@waltercool commented on GitHub (Jul 18, 2026):
Thank you, this apparently works fine on my local using latest ComfyUI. I would need to do few other tests, but no problems detected so far.
@0xDELUXA commented on GitHub (Aug 11, 2026):
Triton is only a fallback now, use HIP instead.