[GH-ISSUE #120] [Bug] GPU LOST (driver device lost) with comfy kitchen attention (INT8) on NVIDIA RTX 5060 Ti (sm_120), Windows -- same workflow with native attention is stable #39

New issue

Open

opened 2026-10-26 22:19:33 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 22:19:33 +00:00

Owner

Originally created by @leimuxiao on GitHub (Aug 16, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/120

Summary

Using comfy kitchen attention (ModelAttentionBackend -> comfy_kitchen_int8) on an NVIDIA GeForce RTX 5060 Ti (Blackwell, sm_120) under Windows causes a driver-level GPU LOST (No devices were found / GPU is lost. Reboot the system to recover this GPU) ~2 seconds after compute starts.

The exact same workflow with pytorch attention completes successfully. This is a clean single-variable control: only the attention backend differs.

Environment

OS: Windows 11 Pro (build 26200)
GPU: NVIDIA GeForce RTX 5060 Ti 16GB (Blackwell, sm_120)
Driver / CUDA UMD: 610.88 / 13.3
torch: 2.12.0+cu130
triton-windows: 3.7.1.post27
comfy-kitchen: 0.2.31 (PyPI latest)
comfy-aimdo: 0.4.13
ComfyUI: started full-speed (main.py --listen --enable-manager, no stable-profile / offload args)

Reproduction

MiniMax H3 image-to-video workflow, 864x480, 243 frames, 6 steps (BasicScheduler simple / denoise 1 / euler), 8-step LoRA. Node ModelAttentionBackend attention = comfy kitchen attention (i.e. comfy_kitchen_int8).

The GPU LOST occurs ~2 s into the first compute burst (during MiniMaxH3TEModel weight staging / very first attention), before sampling begins.

Symptom

nvidia-smi then reports No devices were found / Unable to determine the device handle for GPU0: 0000:07:00.0: GPU is lost. Reboot the system to recover this GPU.
Not a 0x141 BSOD: no new minidump in C:\Windows\Minidump.
No Kernel-Power 41 / EventLog 6008: the recovery reboot was initiated normally by our watchdog (Event 1074), not an unclean power event.
comfyui.log tail shows aimdo: src/model-vbar.c:403:ERROR:VRAM Allocation failed — this appears after the device was lost (symptom, not cause).
At the moment of loss: 79 °C, 140 W / 180 W power limit, 100% GPU util, ~9.4 GB / 16 GB VRAM. Thermal / power / VRAM-exhaustion explanations are excluded.

Single-variable control

Same workflow, Node ModelAttentionBackend attention = pytorch attention (the only change, verified by diff):

Completes: Prompt executed in 337.10 seconds, MP4 produced.
Peak: 15331 MiB VRAM, 182.9 W (transient), 79 °C, sustained 100% util.
No GPU LOST.

So native attention is stable under higher memory/power load, while INT8 ck-attention loses the device under lighter load.

Questions / notes

Is NVIDIA / Blackwell (sm_120) a supported/tested target for ck-attention? Issue #110 notes the INT8 precondition is enforced identically in the CUDA backend, but our failure is a driver device-lost, not a Python exception.
Happy to run debug builds or share the full 1-second telemetry CSV, ComfyUI console log, and Windows event log if useful.

Evidence

1-second telemetry CSV capturing the loss moment (79 °C / 140 W / 100% at loss)
ComfyUI console log
Windows System event log (Event 1074 = watchdog-initiated reboot)

Originally created by @leimuxiao on GitHub (Aug 16, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/120 ### Summary Using `comfy kitchen attention` (`ModelAttentionBackend` -> `comfy_kitchen_int8`) on an NVIDIA GeForce RTX 5060 Ti (Blackwell, **sm_120**) under Windows causes a **driver-level GPU LOST** (`No devices were found` / `GPU is lost. Reboot the system to recover this GPU`) ~2 seconds after compute starts. The **exact same workflow** with `pytorch attention` completes successfully. This is a clean single-variable control: only the attention backend differs. ### Environment - OS: Windows 11 Pro (build 26200) - GPU: NVIDIA GeForce RTX 5060 Ti 16GB (Blackwell, sm_120) - Driver / CUDA UMD: 610.88 / 13.3 - torch: 2.12.0+cu130 - triton-windows: 3.7.1.post27 - comfy-kitchen: 0.2.31 (PyPI latest) - comfy-aimdo: 0.4.13 - ComfyUI: started full-speed (`main.py --listen --enable-manager`, no stable-profile / offload args) ### Reproduction MiniMax H3 image-to-video workflow, 864x480, **243 frames, 6 steps** (BasicScheduler simple / denoise 1 / euler), 8-step LoRA. Node `ModelAttentionBackend` attention = `comfy kitchen attention` (i.e. `comfy_kitchen_int8`). The GPU LOST occurs ~2 s into the **first compute burst** (during MiniMaxH3TEModel weight staging / very first attention), before sampling begins. ### Symptom - `nvidia-smi` then reports `No devices were found` / `Unable to determine the device handle for GPU0: 0000:07:00.0: GPU is lost. Reboot the system to recover this GPU`. - **Not** a 0x141 BSOD: no new minidump in `C:\Windows\Minidump`. - No `Kernel-Power 41` / `EventLog 6008`: the recovery reboot was initiated normally by our watchdog (Event 1074), not an unclean power event. - comfyui.log tail shows `aimdo: src/model-vbar.c:403:ERROR:VRAM Allocation failed` — this appears *after* the device was lost (symptom, not cause). - At the moment of loss: **79 °C, 140 W / 180 W power limit, 100% GPU util, ~9.4 GB / 16 GB VRAM**. Thermal / power / VRAM-exhaustion explanations are excluded. ### Single-variable control Same workflow, Node `ModelAttentionBackend` attention = `pytorch attention` (the only change, verified by diff): - Completes: `Prompt executed in 337.10 seconds`, MP4 produced. - Peak: **15331 MiB VRAM**, 182.9 W (transient), 79 °C, sustained 100% util. - **No GPU LOST.** So native attention is stable under *higher* memory/power load, while INT8 ck-attention loses the device under *lighter* load. ### Questions / notes - Is NVIDIA / Blackwell (sm_120) a supported/tested target for ck-attention? Issue #110 notes the INT8 precondition is enforced identically in the CUDA backend, but our failure is a **driver device-lost**, not a Python exception. - Happy to run debug builds or share the full 1-second telemetry CSV, ComfyUI console log, and Windows event log if useful. ### Evidence - 1-second telemetry CSV capturing the loss moment (79 °C / 140 W / 100% at loss) - ComfyUI console log - Windows System event log (Event 1074 = watchdog-initiated reboot)

bsakaguchi commented

2026-10-26 22:19:34 +00:00

Author

Owner

@coderabbitai[bot] commented on GitHub (Aug 16, 2026):

🔗 Related PRs

Comfy-Org/comfy-kitchen#103 - Implement optimized int8 attention (CORE-74) [merged]
Comfy-Org/comfy-kitchen#104 - HIP port of #103: optimized int8 attention [merged]

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

@coderabbitai[bot] commented on GitHub (Aug 16, 2026):  <details> <summary>🔗 Related PRs</summary> Comfy-Org/comfy-kitchen#103 - Implement optimized int8 attention (CORE-74) [merged] Comfy-Org/comfy-kitchen#104 - HIP port of `#103`: optimized int8 attention [merged] </details> --- <details> <summary> 🧪 Issue enrichment is currently in open beta.</summary> You can configure auto-planning by selecting labels in the issue_enrichment configuration. To disable automatic issue enrichment, add the following to your `.coderabbit.yaml`: ```yaml issue_enrichment: auto_enrich: enabled: false ``` </details> 💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit)!

bsakaguchi commented

2026-10-26 22:19:34 +00:00

Author

Owner

@0xDELUXA commented on GitHub (Aug 19, 2026):

Issue #110 isn't related, and it's also fixed now.

@0xDELUXA commented on GitHub (Aug 19, 2026): Issue #110 isn't related, and it's also fixed now.

bsakaguchi referenced this issue

2026-10-26 22:20:22 +00:00

[PR #39] [MERGED] build dev/** branches #96