[PR #176] [MERGED] Add w4a8 GEMV #198

New issue

Closed

opened 2026-10-26 22:21:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:21:29 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/176
Author: @kijai
Created: 9/14/2026
Status: ✅ Merged
Merged: 9/16/2026
Merged by: @comfyanonymous

Base: main ← Head: w4a8_gemv

📝 Commits (5)

a6289b9 w4a8 gemv
a73fc3e Fused GatedDeltaNet decode kernels
9d00247 Fix ruff
6b93c11 Small fixes
2441c00 device check

📊 Changes

11 files changed (+1106 additions, -31 deletions)

View changed files

📝 comfy_kitchen/__init__.py (+5 -0)
📝 comfy_kitchen/backends/cuda/CMakeLists.txt (+1 -0)
📝 comfy_kitchen/backends/cuda/__init__.py (+29 -0)
📝 comfy_kitchen/backends/cuda/dlpack_bindings.cpp (+204 -1)
📝 comfy_kitchen/backends/cuda/float_utils.cuh (+9 -0)
📝 comfy_kitchen/backends/cuda/ops/convrot_w4a4.cu (+2 -11)
➕ comfy_kitchen/backends/cuda/ops/gated_delta.cu (+338 -0)
📝 comfy_kitchen/backends/cuda/ops/int8_linear.cu (+174 -0)
📝 comfy_kitchen/backends/cuda/ops/w4a8_gemm.cu (+128 -19)
➕ comfy_kitchen/gated_delta.py (+101 -0)
➕ tests/test_gated_delta_fused.py (+115 -0)

📄 Description

Small-M kernels for LLM text-encoder decode.

W4A8 codebook GEMV (M <= 8): reads int4 weights once per step instead of the chunked GEMM tiles; ~870 GB/s effective.
Warp-level int8 rowwise convrot quantizer for the activation side.
int8 decode GEMV path in int8_linear.
Fused GatedDeltaNet decode: gate projections, gate math, q/k norm, S delta-rule steps and the gated RMSNorm in one kernel per head, plus a depthwise conv step kernel. Both write rollback snapshots for speculative decode.

Used by this core PR: https://github.com/Comfy-Org/ComfyUI/pull/15623

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/176 **Author:** [@kijai](https://github.com/kijai) **Created:** 9/14/2026 **Status:** ✅ Merged **Merged:** 9/16/2026 **Merged by:** [@comfyanonymous](https://github.com/comfyanonymous) **Base:** `main` ← **Head:** `w4a8_gemv` --- ### 📝 Commits (5) - [`a6289b9`](https://github.com/Comfy-Org/comfy-kitchen/commit/a6289b9d8a84e87bb3956318ccf247d73b4d8752) w4a8 gemv - [`a73fc3e`](https://github.com/Comfy-Org/comfy-kitchen/commit/a73fc3e776704b52d228a9ce58205b83de6d705b) Fused GatedDeltaNet decode kernels - [`9d00247`](https://github.com/Comfy-Org/comfy-kitchen/commit/9d00247fb9b7f852989bc44227d9ac46494c5bfa) Fix ruff - [`6b93c11`](https://github.com/Comfy-Org/comfy-kitchen/commit/6b93c1158fec4a41689ce8c1cf02fa58e83c8e8e) Small fixes - [`2441c00`](https://github.com/Comfy-Org/comfy-kitchen/commit/2441c0030e6e3e98f8ecf6f19a26f360037a6b78) device check ### 📊 Changes **11 files changed** (+1106 additions, -31 deletions) <details> <summary>View changed files</summary> 📝 `comfy_kitchen/__init__.py` (+5 -0) 📝 `comfy_kitchen/backends/cuda/CMakeLists.txt` (+1 -0) 📝 `comfy_kitchen/backends/cuda/__init__.py` (+29 -0) 📝 `comfy_kitchen/backends/cuda/dlpack_bindings.cpp` (+204 -1) 📝 `comfy_kitchen/backends/cuda/float_utils.cuh` (+9 -0) 📝 `comfy_kitchen/backends/cuda/ops/convrot_w4a4.cu` (+2 -11) ➕ `comfy_kitchen/backends/cuda/ops/gated_delta.cu` (+338 -0) 📝 `comfy_kitchen/backends/cuda/ops/int8_linear.cu` (+174 -0) 📝 `comfy_kitchen/backends/cuda/ops/w4a8_gemm.cu` (+128 -19) ➕ `comfy_kitchen/gated_delta.py` (+101 -0) ➕ `tests/test_gated_delta_fused.py` (+115 -0) </details> ### 📄 Description Small-M kernels for LLM text-encoder decode. - W4A8 codebook GEMV (M <= 8): reads int4 weights once per step instead of the chunked GEMM tiles; ~870 GB/s effective. - Warp-level int8 rowwise convrot quantizer for the activation side. - int8 decode GEMV path in int8_linear. - Fused GatedDeltaNet decode: gate projections, gate math, q/k norm, S delta-rule steps and the gated RMSNorm in one kernel per head, plus a depthwise conv step kernel. Both write rollback snapshots for speculative decode. Used by this core PR: https://github.com/Comfy-Org/ComfyUI/pull/15623 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:21:29 +00:00