[PR #191] [MERGED] Add w6a8 quantization support #208

New issue

Closed

opened 2026-10-26 22:21:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:21:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/191
Author: @kijai
Created: 9/22/2026
Status: ✅ Merged
Merged: 9/23/2026
Merged by: @comfyanonymous

Base: main ← Head: w6a8

📝 Commits (4)

f1d394d Add w6a8 quantization support
89653b7 ruff
6eef282 Bit faster
b10ac1f Reject sub-16 group sizes for 6-bit weights

📊 Changes

8 files changed (+678 additions, -231 deletions)

View changed files

📝 comfy_kitchen/backends/cuda/__init__.py (+77 -53)
📝 comfy_kitchen/backends/cuda/dlpack_bindings.cpp (+39 -19)
📝 comfy_kitchen/backends/cuda/ops/w4a8_gemm.cu (+102 -64)
📝 comfy_kitchen/backends/eager/w4a8_int8.py (+210 -60)
📝 comfy_kitchen/backends/hip/__init__.py (+24 -14)
📝 comfy_kitchen/backends/triton/w4a8_int8.py (+20 -12)
📝 comfy_kitchen/tensor/w4a8_int8.py (+32 -9)
➕ tests/test_w6a8.py (+174 -0)

📄 Description

W6A8: 6-bit weights on the INT8 GEMM path

Adds a 6-bit option to AsymW4A8Int8Layout. Same kernels and checkpoint plumbing.

6-bit is uniform (no codebook), symmetric, fp8 group scales, group size a multiple of 16.

On real MiniMax-H3 weights: ~3x lower weight error than W4A8 (0.024 vs 0.073 relL2) for
0.78 bytes/weight (int8: 1.0, W4A8: 0.56), at the same speed as W4A8, which is
int8 parity at H3 token counts.

Also switches the chunked W4A8/W6A8 linear to the faster 64-lane ConvRot
activation quantizer int8_linear already uses (~20% faster W4A8 at M=4096).

Verified on the real H3 out_proj layer (N=5376, K=7168) at M=4096, RTX 5090:

	int8	W4A8 g16	W6A8 g16	W6A8 g32	W6A8 g64
bytes/weight	1.00	0.56	0.81	0.78	0.77
weight relL2	0.0114	0.0732	0.0213	0.0237	0.0259
output relL2	0.0150	0.0738	0.0233	0.0256	0.0276
ms/forward	0.477	0.664	0.628	0.627	0.628

W6A8 timing is independent of group size (one fp8 scale per 16-column decode vector either way). Group sizes below 16 are not worth it and thus not supported.

Core side PR to use this: https://github.com/Comfy-Org/ComfyUI/pull/16483

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/191 **Author:** [@kijai](https://github.com/kijai) **Created:** 9/22/2026 **Status:** ✅ Merged **Merged:** 9/23/2026 **Merged by:** [@comfyanonymous](https://github.com/comfyanonymous) **Base:** `main` ← **Head:** `w6a8` --- ### 📝 Commits (4) - [`f1d394d`](https://github.com/Comfy-Org/comfy-kitchen/commit/f1d394d8853f313baa0be813f78fc0aaa88b55f7) Add w6a8 quantization support - [`89653b7`](https://github.com/Comfy-Org/comfy-kitchen/commit/89653b739754c123f6a4c5b3f917d3bc98fa9e90) ruff - [`6eef282`](https://github.com/Comfy-Org/comfy-kitchen/commit/6eef2829fbae7039ebdcbc0fe940d303f07032ee) Bit faster - [`b10ac1f`](https://github.com/Comfy-Org/comfy-kitchen/commit/b10ac1f1a95ceaa382b14be36dcde52b423f7a88) Reject sub-16 group sizes for 6-bit weights ### 📊 Changes **8 files changed** (+678 additions, -231 deletions) <details> <summary>View changed files</summary> 📝 `comfy_kitchen/backends/cuda/__init__.py` (+77 -53) 📝 `comfy_kitchen/backends/cuda/dlpack_bindings.cpp` (+39 -19) 📝 `comfy_kitchen/backends/cuda/ops/w4a8_gemm.cu` (+102 -64) 📝 `comfy_kitchen/backends/eager/w4a8_int8.py` (+210 -60) 📝 `comfy_kitchen/backends/hip/__init__.py` (+24 -14) 📝 `comfy_kitchen/backends/triton/w4a8_int8.py` (+20 -12) 📝 `comfy_kitchen/tensor/w4a8_int8.py` (+32 -9) ➕ `tests/test_w6a8.py` (+174 -0) </details> ### 📄 Description ## W6A8: 6-bit weights on the INT8 GEMM path Adds a 6-bit option to `AsymW4A8Int8Layout`. Same kernels and checkpoint plumbing. 6-bit is uniform (no codebook), symmetric, fp8 group scales, group size a multiple of 16. On real MiniMax-H3 weights: ~3x lower weight error than W4A8 (0.024 vs 0.073 relL2) for 0.78 bytes/weight (int8: 1.0, W4A8: 0.56), at the same speed as W4A8, which is int8 parity at H3 token counts. Also switches the chunked W4A8/W6A8 linear to the faster 64-lane ConvRot activation quantizer `int8_linear` already uses (~20% faster W4A8 at M=4096). Verified on the real H3 `out_proj` layer (N=5376, K=7168) at M=4096, RTX 5090: | | int8 | W4A8 g16 | W6A8 g16 | W6A8 g32 | W6A8 g64 | |---------------|-------:|---------:|---------:|---------:|---------:| | bytes/weight | 1.00 | 0.56 | 0.81 | 0.78 | 0.77 | | weight relL2 | 0.0114 | 0.0732 | 0.0213 | 0.0237 | 0.0259 | | output relL2 | 0.0150 | 0.0738 | 0.0233 | 0.0256 | 0.0276 | | ms/forward | 0.477 | 0.664 | 0.628 | 0.627 | 0.628 | W6A8 timing is independent of group size (one fp8 scale per 16-column decode vector either way). Group sizes below 16 are not worth it and thus not supported. Core side PR to use this: https://github.com/Comfy-Org/ComfyUI/pull/16483 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:21:35 +00:00