[PR #129] [CLOSED] HIP: optimize SVDQuant INT4 activation quantization #165

New issue

Closed

opened 2026-10-26 22:21:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:21:02 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/129
Author: @tangzzycc
Created: 8/22/2026
Status: ❌ Closed

Base: main ← Head: opt/hip-svdquant-preprocess

📝 Commits (3)

2bf7b4b Optimize HIP SVDQuant INT4 quantization and WMMA LoRA-down
4ff75eb Address SVDQuant review feedback
71fd9ed perf(hip): use torch.mm for SVDQuant LoRA-down

📊 Changes

4 files changed (+155 additions, -67 deletions)

View changed files

📝 comfy_kitchen/backends/hip/__init__.py (+8 -5)
📝 comfy_kitchen/backends/hip/dlpack_bindings.cpp (+0 -24)
📝 comfy_kitchen/backends/hip/ops/svdquant_w4a4.hip (+102 -38)
📝 tests/test_hip_wmma.py (+45 -0)

📄 Description

Summary

This PR improves the HIP preprocessing path used by quantize_svdquant_w4a4:

Process each 64-element INT4 quantization group with 16 cooperating threads.
Reuse the smoothed activation values for reduction and packing, avoiding a second input read.
Use torch.mm for LoRA-down, matching the existing CUDA backend, and remove the dedicated scalar HIP binding.

The public Python API and output layout remain unchanged.

Implementation

Sixteen threads process each 64-element quantization group. Each thread loads and retains four smoothed values, participates in a 16-lane shuffle reduction, and writes four packed INT4 values through one 16-bit store. The existing scalar kernel remains available as a fallback for operands with different dtypes.

LoRA-down is dispatched through PyTorch's matrix multiplication path instead of the previous one-output-element-per-thread HIP kernel.

Performance

Measured on an AMD Radeon 8060S (gfx1151) with ROCm 7.2. Each result uses 20 warmup iterations followed by 100 measured iterations and reports the median kernel time.

dtype	shape	Before	After	Speedup
FP16	4096 x 4096	3.723 ms	0.214 ms	17.4x
FP16	4096 x 14336	15.363 ms	0.713 ms	21.5x
BF16	4096 x 4096	3.731 ms	0.212 ms	17.6x
BF16	4096 x 14336	15.250 ms	0.712 ms	21.4x
FP32	4096 x 4096	2.604 ms	0.350 ms	7.4x
FP32	4096 x 14336	11.105 ms	1.220 ms	9.1x

Based on the minimum required input/output traffic, the large FP16/BF16 cases reach approximately 198-209 GB/s of effective bandwidth.

Architecture verification

The extension was successfully compiled for all currently supported gfx11/gfx12 targets:

gfx1100 gfx1101 gfx1102 gfx1103
gfx1150 gfx1151 gfx1152 gfx1153
gfx1200 gfx1201

Code-object inspection confirmed packed quantizer stores and no private-memory spills in the optimized quantization kernel. Runtime performance was measured on gfx1151; the remaining targets were validated through compilation and code-object inspection.

Validation

CMAKE_BUILD_PARALLEL_LEVEL=16 .venv/bin/python setup.py \
  build_ext --inplace --no-cuda --hip --hip-archs=gfx1151

.venv/bin/pytest tests/test_svdquant_w4a4.py tests/test_hip_wmma.py -q

Result:

370 passed, 14 skipped

The tests cover signed and unsigned INT4 packing, quantization-group layout, transposed scales, padded rows, and LoRA-down results against torch.mm.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/129 **Author:** [@tangzzycc](https://github.com/tangzzycc) **Created:** 8/22/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `opt/hip-svdquant-preprocess` --- ### 📝 Commits (3) - [`2bf7b4b`](https://github.com/Comfy-Org/comfy-kitchen/commit/2bf7b4b3952b1ef0e1eeee0f927e6c580c496b4c) Optimize HIP SVDQuant INT4 quantization and WMMA LoRA-down - [`4ff75eb`](https://github.com/Comfy-Org/comfy-kitchen/commit/4ff75eb82e52eeb1e3f6055b3c94c60282080e8e) Address SVDQuant review feedback - [`71fd9ed`](https://github.com/Comfy-Org/comfy-kitchen/commit/71fd9edb197eda150fd288d6a45c12a6944b0b97) perf(hip): use torch.mm for SVDQuant LoRA-down ### 📊 Changes **4 files changed** (+155 additions, -67 deletions) <details> <summary>View changed files</summary> 📝 `comfy_kitchen/backends/hip/__init__.py` (+8 -5) 📝 `comfy_kitchen/backends/hip/dlpack_bindings.cpp` (+0 -24) 📝 `comfy_kitchen/backends/hip/ops/svdquant_w4a4.hip` (+102 -38) 📝 `tests/test_hip_wmma.py` (+45 -0) </details> ### 📄 Description ## Summary This PR improves the HIP preprocessing path used by `quantize_svdquant_w4a4`: - Process each 64-element INT4 quantization group with 16 cooperating threads. - Reuse the smoothed activation values for reduction and packing, avoiding a second input read. - Use `torch.mm` for LoRA-down, matching the existing CUDA backend, and remove the dedicated scalar HIP binding. The public Python API and output layout remain unchanged. ## Implementation Sixteen threads process each 64-element quantization group. Each thread loads and retains four smoothed values, participates in a 16-lane shuffle reduction, and writes four packed INT4 values through one 16-bit store. The existing scalar kernel remains available as a fallback for operands with different dtypes. LoRA-down is dispatched through PyTorch's matrix multiplication path instead of the previous one-output-element-per-thread HIP kernel. ## Performance Measured on an AMD Radeon 8060S (`gfx1151`) with ROCm 7.2. Each result uses 20 warmup iterations followed by 100 measured iterations and reports the median kernel time. | dtype | shape | Before | After | Speedup | |---|---:|---:|---:|---:| | FP16 | 4096 x 4096 | 3.723 ms | 0.214 ms | 17.4x | | FP16 | 4096 x 14336 | 15.363 ms | 0.713 ms | 21.5x | | BF16 | 4096 x 4096 | 3.731 ms | 0.212 ms | 17.6x | | BF16 | 4096 x 14336 | 15.250 ms | 0.712 ms | 21.4x | | FP32 | 4096 x 4096 | 2.604 ms | 0.350 ms | 7.4x | | FP32 | 4096 x 14336 | 11.105 ms | 1.220 ms | 9.1x | Based on the minimum required input/output traffic, the large FP16/BF16 cases reach approximately 198-209 GB/s of effective bandwidth. ## Architecture verification The extension was successfully compiled for all currently supported gfx11/gfx12 targets: ```text gfx1100 gfx1101 gfx1102 gfx1103 gfx1150 gfx1151 gfx1152 gfx1153 gfx1200 gfx1201 ``` Code-object inspection confirmed packed quantizer stores and no private-memory spills in the optimized quantization kernel. Runtime performance was measured on gfx1151; the remaining targets were validated through compilation and code-object inspection. ## Validation ```bash CMAKE_BUILD_PARALLEL_LEVEL=16 .venv/bin/python setup.py \ build_ext --inplace --no-cuda --hip --hip-archs=gfx1151 .venv/bin/pytest tests/test_svdquant_w4a4.py tests/test_hip_wmma.py -q ``` Result: ```text 370 passed, 14 skipped ``` The tests cover signed and unsigned INT4 packing, quantization-group layout, transposed scales, padded rows, and LoRA-down results against `torch.mm`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:21:02 +00:00