[GH-ISSUE #102] retune int8 GEMM tiles for gfx1103 and vectorize rowwise quantize #31

New issue

Closed

opened 2026-10-26 22:19:25 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:19:25 +00:00

Owner

Originally created by @LuXuxue on GitHub (Aug 9, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/102

ops/quantize_int8.hip should be useful for all cards.

ops/gemm_int8.hip changed tiles for gfx1103. This should ideally be automatically selected based on arch, but implementing it requires modifying more files, so it hasn't been implemented yet.

I locally compiled the wheel file, tested it with the Anima model, and confirmed it ran faster than before the modifications. However, my own compiled wheel file (before modifications) itself is slower than the official compilation, and after modifications it is as fast as the official compilation.
The test results are based on the following two scripts.

tile_scan.py
bench_int8.py

Change 1 — GEMM tile dispatch rework (ops/gemm_int8.hip)
Replaced the old three-tier shape heuristic

M >= 512 && N >= 128 && K > N -> 256x128 tile
M >= 96 && N >= 96 -> 128x128 tile
otherwise -> 64x64 tile

with a K-based heuristic

K >= 8192 -> 128x128 tile, BKB=64 (deep K: larger tile amortizes per-tile global reads)
K < 1024 -> 64x64 tile, BKB=32 (shallow K: smaller LDS footprint -> higher occupancy)
otherwise -> 64x64 tile, BKB=64

Why: on a 12-CU iGPU the large 256x128/128x128 blocks (256 threads, 128 accumulator VGPRs) leave too few resident warps per SIMD to hide DRAM latency (200–400 ns). The small 64x64 tile wins 22/28 scanned cases — its high occupancy hides latency, and the 2 MiB L2 already covers the cross-block reuse a bigger tile would otherwise buy with extra global traffic. Deep K is the exception: with enough K-steps, the larger tile's reduction in global reads outweighs the occupancy loss. A BKB=32 K-tile further helps shallow K (K ≤ 640: −13% to −25%) at the cost of more K-loop syncs, which is why it is only used for small K.

Also added a COMFY_KITCHEN_INT8_TILE environment hook (12 tile configs, off by default) so the heuristic can be re-scanned on other hardware without rebuilding.

Change 2 — vectorized rowwise quantize (ops/quantize_int8.hip)
The rowwise quantizer was the dominant cost for many shapes — e.g. Anima05 (8192×2048 bf16) spent 2.3 ms quantizing vs 0.74 ms in the GEMM, and it was 1.7–2.4x slower than triton's. The old kernel walked each row with scalar 2/4-byte load_in calls at stride 256, twice (absmax pass + write pass).

The new kernel gives each thread whole 16-byte chunks of a row (8 f16/bf16 or 4 f32): both passes use uint4 vector loads, and the int8 writeback is packed into uint2/uint32 stores. The old scalar kernel is kept as a fallback for rows that do not split into whole chunks (K % elems != 0) or whose base pointer is not 16-byte aligned (strided torch views).

Originally created by @LuXuxue on GitHub (Aug 9, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/102 [ops/quantize_int8.hip](https://github.com/user-attachments/files/30875595/quantize_int8.hip.txt) should be useful for all cards. [ops/gemm_int8.hip](https://github.com/user-attachments/files/30875597/gemm_int8.hip.txt) changed tiles for gfx1103. This should ideally be automatically selected based on arch, but implementing it requires modifying more files, so it hasn't been implemented yet. I locally compiled the wheel file, tested it with the Anima model, and confirmed it ran faster than before the modifications. However, my own compiled wheel file (before modifications) itself is slower than the official compilation, and after modifications it is as fast as the official compilation. The test results are based on the following two scripts. [tile_scan.py](https://github.com/user-attachments/files/30875762/tile_scan.py) [bench_int8.py](https://github.com/user-attachments/files/30875761/bench_int8.py) Change 1 — GEMM tile dispatch rework (ops/gemm_int8.hip) Replaced the old three-tier shape heuristic M >= 512 && N >= 128 && K > N -> 256x128 tile M >= 96 && N >= 96 -> 128x128 tile otherwise -> 64x64 tile with a K-based heuristic K >= 8192 -> 128x128 tile, BKB=64 (deep K: larger tile amortizes per-tile global reads) K < 1024 -> 64x64 tile, BKB=32 (shallow K: smaller LDS footprint -> higher occupancy) otherwise -> 64x64 tile, BKB=64 Why: on a 12-CU iGPU the large 256x128/128x128 blocks (256 threads, 128 accumulator VGPRs) leave too few resident warps per SIMD to hide DRAM latency (200–400 ns). The small 64x64 tile wins 22/28 scanned cases — its high occupancy hides latency, and the 2 MiB L2 already covers the cross-block reuse a bigger tile would otherwise buy with extra global traffic. Deep K is the exception: with enough K-steps, the larger tile's reduction in global reads outweighs the occupancy loss. A BKB=32 K-tile further helps shallow K (K ≤ 640: −13% to −25%) at the cost of more K-loop syncs, which is why it is only used for small K. Also added a COMFY_KITCHEN_INT8_TILE environment hook (12 tile configs, off by default) so the heuristic can be re-scanned on other hardware without rebuilding. Change 2 — vectorized rowwise quantize (ops/quantize_int8.hip) The rowwise quantizer was the dominant cost for many shapes — e.g. Anima05 (8192×2048 bf16) spent 2.3 ms quantizing vs 0.74 ms in the GEMM, and it was 1.7–2.4x slower than triton's. The old kernel walked each row with scalar 2/4-byte load_in calls at stride 256, twice (absmax pass + write pass). The new kernel gives each thread whole 16-byte chunks of a row (8 f16/bf16 or 4 f32): both passes use uint4 vector loads, and the int8 writeback is packed into uint2/uint32 stores. The old scalar kernel is kept as a fallback for rows that do not split into whole chunks (K % elems != 0) or whose base pointer is not 16-byte aligned (strided torch views).

bsakaguchi closed this issue

2026-10-26 22:19:25 +00:00

bsakaguchi commented

2026-10-26 22:19:26 +00:00

Author

Owner

@0xDELUXA commented on GitHub (Aug 11, 2026):

I'm splitting these, because they are not equally portable.

Change 1 (tile heuristic) should not become a global default. It is fitted on a 12 CU iGPU with ~89 GB/s shared DDR5 and no Infinity Cache, and the rationale (large tiles cannot hide 200-400 ns DRAM latency) is a property of that machine. gfx1200 measures 282 GB/s with far more CUs, so the occupancy versus reuse tradeoff lands elsewhere. As the issue itself says, this wants per arch selection.

Change 2 (vectorized rowwise quantize) is worth doing. Measured on gfx1200 against that card's own memcpy roofline of 282 GB/s, the current scalar kernel runs at 44-58% of roofline: 8192x2048 bf16 at 0.342 ms / 147 GB/s, 4096x4096 at 0.309 ms / 163 GB/s, 4096x1024 at 0.101 ms / 125 GB/s. So roughly 2x is on the table, which matches your 1.7-2.4x against Triton on gfx1103. That is headroom measured, not a cause identified: it does not follow that the scalar load path is what is holding it, and it should not be read as a prediction for what vector loads alone will recover on this card. The gap reproduces on discrete RDNA4, so it is not iGPU specific, and keeping the scalar path as a fallback for ragged K and unaligned views is the right shape. I think I'll open a separate PR for this in the future.

@0xDELUXA commented on GitHub (Aug 11, 2026): I'm splitting these, because they are not equally portable. Change 1 (tile heuristic) should not become a global default. It is fitted on a 12 CU iGPU with ~89 GB/s shared DDR5 and no Infinity Cache, and the rationale (large tiles cannot hide 200-400 ns DRAM latency) is a property of that machine. gfx1200 measures 282 GB/s with far more CUs, so the occupancy versus reuse tradeoff lands elsewhere. As the issue itself says, this wants per arch selection. Change 2 (vectorized rowwise quantize) is worth doing. Measured on gfx1200 against that card's own memcpy roofline of 282 GB/s, the current scalar kernel runs at 44-58% of roofline: 8192x2048 bf16 at 0.342 ms / 147 GB/s, 4096x4096 at 0.309 ms / 163 GB/s, 4096x1024 at 0.101 ms / 125 GB/s. So roughly 2x is on the table, which matches your 1.7-2.4x against Triton on gfx1103. That is headroom measured, not a cause identified: it does not follow that the scalar load path is what is holding it, and it should not be read as a prediction for what vector loads alone will recover on this card. The gap reproduces on discrete RDNA4, so it is not iGPU specific, and keeping the scalar path as a fallback for ragged K and unaligned views is the right shape. I think I'll open a separate PR for this in the future.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-kitchen#31

No description provided.

Rows
Columns