[PR #262] Optimize HIP SVDQuant W4A4 kernels #266

New issue

Open

opened 2026-10-26 22:22:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:22:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/262
Author: @0xDELUXA
Created: 10/19/2026
Status: 🔄 Open

Base: main ← Head: amd/hip-svdquant-perf

📝 Commits (1)

08216d9 Optimize HIP SVDQuant W4A4 kernels

📊 Changes

4 files changed (+305 additions, -81 deletions)

View changed files

📝 comfy_kitchen/backends/hip/__init__.py (+6 -1)
📝 comfy_kitchen/backends/hip/mma.h (+17 -0)
📝 comfy_kitchen/backends/hip/ops/svdquant_w4a4.hip (+251 -79)
📝 tests/test_hip_wmma.py (+31 -1)

📄 Description

Brings the HIP SVDQuant W4A4 path from below bf16 matmul speed to 1.1-2.1x faster than it at Flux-sized shapes, by restructuring its three kernels around how RDNA hides latency.

The LoRA-down now runs on the WMMA GEMM core (MmaBf16 gains the core's load, as MmaF16 already has), so the quantized paths still never reach hipBLASLt; fp32 inputs, and GPUs without matrix cores, use a small LDS-tiled kernel. The quantizer uses one wave per row with two elements per lane, as the CUDA kernel does. The GEMM stages each tile's activation and weight scales into LDS as fp32 alongside the tile, so the per-group flush reads them from LDS, and the LoRA-up correction reads its operands from LDS in rank chunks.

Measured on gfx1200, bf16, rank 32. Times are quantize plus GEMM.

M x K x N	bf16 matmul	Before	After	GEMM TOPS (before -> after)
4096 x 4096 x 4096	3.6 ms	24.3 ms	2.5 ms	12 -> 73
4096 x 4096 x 12288	12.3 ms	45.8 ms	5.8 ms	13 -> 77
4096 x 12288 x 4096	6.9 ms	63.5 ms	6.2 ms	17 -> 84
8192 x 4096 x 12288	24.2 ms	90.5 ms	11.4 ms	13 -> 79

The LoRA-down is within 10% of torch.mm.

Outputs match the previous kernels except for activation codes sitting on a rounding boundary, about 3 in 1e5 elements, where both are equally close to an fp64 reference. test_svdquant_w4a4_beats_eager_against_fp32_truth gains a case with a multi-chunk rank, partial N tile and an odd group count. Compile-verified for gfx1030, gfx1100 and gfx1200. Run-verified on gfx1200 only.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/262 **Author:** [@0xDELUXA](https://github.com/0xDELUXA) **Created:** 10/19/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `amd/hip-svdquant-perf` --- ### 📝 Commits (1) - [`08216d9`](https://github.com/Comfy-Org/comfy-kitchen/commit/08216d9c25fe932fc612cb4e7d148daa8efabd08) Optimize HIP SVDQuant W4A4 kernels ### 📊 Changes **4 files changed** (+305 additions, -81 deletions) <details> <summary>View changed files</summary> 📝 `comfy_kitchen/backends/hip/__init__.py` (+6 -1) 📝 `comfy_kitchen/backends/hip/mma.h` (+17 -0) 📝 `comfy_kitchen/backends/hip/ops/svdquant_w4a4.hip` (+251 -79) 📝 `tests/test_hip_wmma.py` (+31 -1) </details> ### 📄 Description Brings the HIP SVDQuant W4A4 path from below bf16 matmul speed to 1.1-2.1x faster than it at Flux-sized shapes, by restructuring its three kernels around how RDNA hides latency. The LoRA-down now runs on the WMMA GEMM core (`MmaBf16` gains the core's `load`, as `MmaF16` already has), so the quantized paths still never reach hipBLASLt; fp32 inputs, and GPUs without matrix cores, use a small LDS-tiled kernel. The quantizer uses one wave per row with two elements per lane, as the CUDA kernel does. The GEMM stages each tile's activation and weight scales into LDS as fp32 alongside the tile, so the per-group flush reads them from LDS, and the LoRA-up correction reads its operands from LDS in rank chunks. Measured on gfx1200, bf16, rank 32. Times are quantize plus GEMM. | M x K x N | bf16 matmul | Before | After | GEMM TOPS (before -> after) | |---|---|---|---|---| | 4096 x 4096 x 4096 | 3.6 ms | 24.3 ms | 2.5 ms | 12 -> 73 | | 4096 x 4096 x 12288 | 12.3 ms | 45.8 ms | 5.8 ms | 13 -> 77 | | 4096 x 12288 x 4096 | 6.9 ms | 63.5 ms | 6.2 ms | 17 -> 84 | | 8192 x 4096 x 12288 | 24.2 ms | 90.5 ms | 11.4 ms | 13 -> 79 | The LoRA-down is within 10% of `torch.mm`. Outputs match the previous kernels except for activation codes sitting on a rounding boundary, about 3 in 1e5 elements, where both are equally close to an fp64 reference. `test_svdquant_w4a4_beats_eager_against_fp32_truth` gains a case with a multi-chunk rank, partial N tile and an odd group count. Compile-verified for gfx1030, gfx1100 and gfx1200. Run-verified on gfx1200 only. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-26 22:22:16 +00:00

bsakaguchi referenced this issue

2026-10-26 22:22:17 +00:00

[PR #266] [MERGED] Implement attention masking and deltanet for tree MTP #269