[PR #200] [CLOSED] [HIP] Add software tiled GEMM support to older HIP architectures (RDNA1 and Vega) #220

New issue

Closed

opened 2026-10-26 22:21:42 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:21:42 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-kitchen/pull/200
Author: @Zakhrov
Created: 9/25/2026
Status: ❌ Closed

Base: main ← Head: hip-bringup

📝 Commits (10+)

9507848 Add support for gfx90c and gfx1010 GPU architectures and optimize flash decode kernel
e0075df Refactor HIP backend for Sage attention kernels and enhance testing
83f53c3 Update error message for wave32 requirement
5e0ffef Refactor tests and improve code readability
da75655 Merge branch 'Comfy-Org:main' into main
54f400b Fix failing tests on gfx90c
a4cd6c0 Update README to reflect gfx90c and gfx1010 support
3750a88 Enable software GEMM tiling and attention for Vega, RDNA1 and RDNA2
7c226c3 Refactor HIP backend for Sage attention kernels and enhance testing
a9030ef Merge branch 'Comfy-Org:main' into main

📊 Changes

67 files changed (+5276 additions, -1156 deletions)

View changed files

📝 README.md (+237 -14)
📝 comfy_kitchen/__init__.py (+73 -6)
📝 comfy_kitchen/backends/cuda/__init__.py (+11 -1)
📝 comfy_kitchen/backends/eager/__init__.py (+20 -1)
📝 comfy_kitchen/backends/eager/conv3d.py (+35 -13)
📝 comfy_kitchen/backends/eager/quantization.py (+40 -3)
📝 comfy_kitchen/backends/eager/sol_attn.py (+7 -3)
📝 comfy_kitchen/backends/hip/CMakeLists.txt (+31 -3)
📝 comfy_kitchen/backends/hip/__init__.py (+613 -99)
📝 comfy_kitchen/backends/hip/architecture_config.h.in (+13 -0)
📝 comfy_kitchen/backends/hip/architectures.json (+7 -1)
📝 comfy_kitchen/backends/hip/dlpack_bindings.cpp (+261 -11)
📝 comfy_kitchen/backends/hip/epilogue.h (+14 -0)
➕ comfy_kitchen/backends/hip/gemm_simt.h (+694 -0)
📝 comfy_kitchen/backends/hip/gemm_wmma.h (+2 -1)
📝 comfy_kitchen/backends/hip/hadamard.h (+71 -21)
📝 comfy_kitchen/backends/hip/mma.h (+218 -58)
📝 comfy_kitchen/backends/hip/ops/conv3d_fp16.hip (+139 -9)
📝 comfy_kitchen/backends/hip/ops/convrot_w4a4.hip (+8 -1)
📝 comfy_kitchen/backends/hip/ops/flash_decode.hip (+6 -1)

...and 47 more files

📄 Description

This PR supersedes #185, and extends the HIP backend from its current RDNA2+ scope to validated Vega (gfx900, gfx906, gfx90c) and RDNA1 (gfx1010–gfx1012) targets.

The key point is that this is not a claim that these devices have WMMA/matrix-core hardware. They do not. Instead, the backend provides a software 16×16 tile policy that preserves the fragment contract used by the supported HIP kernels:

RDNA2 uses packed sdot4.
RDNA1 and Vega use vector-ALU arithmetic.
RDNA3/4 retain their existing native-WMMA paths.

That distinction lets us expand compatibility without changing the fast path or capability assumptions on newer hardware.

Why this matters

AMD has restored Rocm support for Vega/GCN5, and has brought full RDNA 1 support starting with Rocm 10.2 as shown here: https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md

While Vega and RDNA1 / RDNA2 don't have hardware level WMMA instructions, they are still perfectly capable of AI inference workloads and are currently only VRAM constricted when it comes to larger models, and the usual CUDA tricks of using Sageattention don't work on them for reducing VRAM usage.

This implementation aims to close that parity.

Falling through to eager attention has particularly poor memory behavior for long contexts:

eager Sol-Attn materializes a dense FP32 (B, H, T, T) score tensor and associated intermediate tensors;
the HIP Sol-Attn path instead uses packed INT8 carriers, pooled summaries, routing state, and bounded working storage;
Sage INT8 attention uses tiled online softmax and does not materialize a full score matrix;
NA3D retains scores, softmax state, and output accumulators in registers, using LDS only for small tiles.

Sol-Attn still has compact block-level quadratic routing storage, so this is not a claim of strictly linear memory. It is, however, a major reduction versus token-level FP32 quadratic score storage and makes long-context execution realistic on legacy AMD GPUs.

Scope and safety

The change is deliberately narrow and fail-closed:

only explicit, reviewed gfx targets are admitted;
unsupported or future architectures continue to be rejected;
existing dtype, alignment, layout, and shape constraints still route unsupported calls to another capable backend;
BF16 flash decode remains unavailable on Vega/RDNA1 because it does not use the software-tile path;
RDNA3/4 native-WMMA behavior is unchanged.

The dispatch tests cover the architecture grouping and ensure the software-tile targets advertise only the intended capabilities. pytest -q tests/test_hip_dispatch.py passes (67 passed, 2 skipped).

Expected tradeoff

This is a compatibility and memory-efficiency improvement, not a promise that Vega/RDNA1 will match RDNA3+ throughput. Software-tile attention will be slower than native WMMA. But the alternative is often eager execution with substantially higher peak VRAM, or no practical long-context execution at all.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-kitchen/pull/200 **Author:** [@Zakhrov](https://github.com/Zakhrov) **Created:** 9/25/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `hip-bringup` --- ### 📝 Commits (10+) - [`9507848`](https://github.com/Comfy-Org/comfy-kitchen/commit/9507848e58de940a821d90383f33b27f6f1c9492) Add support for gfx90c and gfx1010 GPU architectures and optimize flash decode kernel - [`e0075df`](https://github.com/Comfy-Org/comfy-kitchen/commit/e0075df9201df401fd89bad966860385e26d1081) Refactor HIP backend for Sage attention kernels and enhance testing - [`83f53c3`](https://github.com/Comfy-Org/comfy-kitchen/commit/83f53c3622b04597edf70c7af4473af85a365947) Update error message for wave32 requirement - [`5e0ffef`](https://github.com/Comfy-Org/comfy-kitchen/commit/5e0ffef2b098944ee1dbbcff79b4c733e050c524) Refactor tests and improve code readability - [`da75655`](https://github.com/Comfy-Org/comfy-kitchen/commit/da756553cb53a4fbc60ccb80a97cb637f27115d9) Merge branch 'Comfy-Org:main' into main - [`54f400b`](https://github.com/Comfy-Org/comfy-kitchen/commit/54f400b9f568cdceb6cb0f281cb996ecd587ec6b) Fix failing tests on gfx90c - [`a4cd6c0`](https://github.com/Comfy-Org/comfy-kitchen/commit/a4cd6c0e7c1d76176886d9d6b4b562a1e78c916e) Update README to reflect gfx90c and gfx1010 support - [`3750a88`](https://github.com/Comfy-Org/comfy-kitchen/commit/3750a8806675f3c4479153bcff4813350f156bdf) Enable software GEMM tiling and attention for Vega, RDNA1 and RDNA2 - [`7c226c3`](https://github.com/Comfy-Org/comfy-kitchen/commit/7c226c36858be335d4098614137953e038c44a19) Refactor HIP backend for Sage attention kernels and enhance testing - [`a9030ef`](https://github.com/Comfy-Org/comfy-kitchen/commit/a9030efb1f20b1b01c0b2cdcad6a9d0220dbc151) Merge branch 'Comfy-Org:main' into main ### 📊 Changes **67 files changed** (+5276 additions, -1156 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+237 -14) 📝 `comfy_kitchen/__init__.py` (+73 -6) 📝 `comfy_kitchen/backends/cuda/__init__.py` (+11 -1) 📝 `comfy_kitchen/backends/eager/__init__.py` (+20 -1) 📝 `comfy_kitchen/backends/eager/conv3d.py` (+35 -13) 📝 `comfy_kitchen/backends/eager/quantization.py` (+40 -3) 📝 `comfy_kitchen/backends/eager/sol_attn.py` (+7 -3) 📝 `comfy_kitchen/backends/hip/CMakeLists.txt` (+31 -3) 📝 `comfy_kitchen/backends/hip/__init__.py` (+613 -99) 📝 `comfy_kitchen/backends/hip/architecture_config.h.in` (+13 -0) 📝 `comfy_kitchen/backends/hip/architectures.json` (+7 -1) 📝 `comfy_kitchen/backends/hip/dlpack_bindings.cpp` (+261 -11) 📝 `comfy_kitchen/backends/hip/epilogue.h` (+14 -0) ➕ `comfy_kitchen/backends/hip/gemm_simt.h` (+694 -0) 📝 `comfy_kitchen/backends/hip/gemm_wmma.h` (+2 -1) 📝 `comfy_kitchen/backends/hip/hadamard.h` (+71 -21) 📝 `comfy_kitchen/backends/hip/mma.h` (+218 -58) 📝 `comfy_kitchen/backends/hip/ops/conv3d_fp16.hip` (+139 -9) 📝 `comfy_kitchen/backends/hip/ops/convrot_w4a4.hip` (+8 -1) 📝 `comfy_kitchen/backends/hip/ops/flash_decode.hip` (+6 -1) _...and 47 more files_ </details> ### 📄 Description This PR supersedes #185, and extends the HIP backend from its current RDNA2+ scope to validated Vega (`gfx900`, `gfx906`, `gfx90c`) and RDNA1 (`gfx1010`–`gfx1012`) targets. The key point is that this is not a claim that these devices have WMMA/matrix-core hardware. They do not. Instead, the backend provides a software 16×16 tile policy that preserves the fragment contract used by the supported HIP kernels: - RDNA2 uses packed `sdot4`. - RDNA1 and Vega use vector-ALU arithmetic. - RDNA3/4 retain their existing native-WMMA paths. That distinction lets us expand compatibility without changing the fast path or capability assumptions on newer hardware. ### Why this matters AMD has restored Rocm support for Vega/GCN5, and has brought full RDNA 1 support starting with Rocm 10.2 as shown here: https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md While Vega and RDNA1 / RDNA2 don't have hardware level WMMA instructions, they are still perfectly capable of AI inference workloads and are currently only VRAM constricted when it comes to larger models, and the usual CUDA tricks of using Sageattention don't work on them for reducing VRAM usage. This implementation aims to close that parity. Falling through to eager attention has particularly poor memory behavior for long contexts: - eager Sol-Attn materializes a dense FP32 `(B, H, T, T)` score tensor and associated intermediate tensors; - the HIP Sol-Attn path instead uses packed INT8 carriers, pooled summaries, routing state, and bounded working storage; - Sage INT8 attention uses tiled online softmax and does not materialize a full score matrix; - NA3D retains scores, softmax state, and output accumulators in registers, using LDS only for small tiles. Sol-Attn still has compact block-level quadratic routing storage, so this is not a claim of strictly linear memory. It is, however, a major reduction versus token-level FP32 quadratic score storage and makes long-context execution realistic on legacy AMD GPUs. ### Scope and safety The change is deliberately narrow and fail-closed: - only explicit, reviewed gfx targets are admitted; - unsupported or future architectures continue to be rejected; - existing dtype, alignment, layout, and shape constraints still route unsupported calls to another capable backend; - BF16 flash decode remains unavailable on Vega/RDNA1 because it does not use the software-tile path; - RDNA3/4 native-WMMA behavior is unchanged. The dispatch tests cover the architecture grouping and ensure the software-tile targets advertise only the intended capabilities. `pytest -q tests/test_hip_dispatch.py` passes (`67 passed, 2 skipped`). ### Expected tradeoff This is a compatibility and memory-efficiency improvement, not a promise that Vega/RDNA1 will match RDNA3+ throughput. Software-tile attention will be slower than native WMMA. But the alternative is often eager execution with substantially higher peak VRAM, or no practical long-context execution at all. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:21:42 +00:00