mirror of
https://github.com/ROCm/composable_kernel.git
synced 2026-07-18 01:28:27 +00:00
9d8386b920c5fa84b853fe8066520ebd0458cbff
3462 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JiaLuo-CAN
|
9d8386b920 |
[rocm-libraries] ROCm/rocm-libraries#9461 (commit 0f33948)
Revert PR #6526 "feat: [CK Tile] mxfp8 support for qr async pipeline (#6526)" (#9461) This reverts commit 3e0171014f5d0e31152376271b6208c6587ef786. ## Motivation The PR is identified breaking the CI RUN_CK_TILE_FMHA_TESTS. http://micimaster.amd.com/blue/organizations/jenkins/rocm-libraries-folder%2fComposable%20Kernel/detail/develop/1762/pipeline/305 and Aiter test_mha.py http://micimaster.amd.com/blue/organizations/jenkins/rocm-libraries-folder%2FComposable%20Kernel/detail/users%2Fjialuo%2Fck%2Ffix_ci_aiter_error/33/pipeline/246 ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Muhammed Emin Ozturk
|
6648115aed |
[rocm-libraries] ROCm/rocm-libraries#9000 (commit 9faa8de)
feat(ck-tile): add grouped GEMM variant to TE to dispatcher bridge (#9000) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > Re-opened from #8130 with a policy-compliant branch name (`users/muozturk/ck-tile/dispatcher-te-bridge-grouped-gemm`). Supersedes #8130. ## What this PR does Routes the **grouped_gemm** variant through the Tile Engine (TE) → Dispatcher **bridge**: TE only generates configs and benchmarks; the Dispatcher owns codegen, build, and runtime. This is the grouped counterpart of the regular-GEMM bridge (#8123/#8479), the fp8/bf8/int8 bridge (#8887), and the Stream-K bridge (#8136). **This PR now also contains the grouped Dispatcher codegen** that previously lived in #8075 — that PR has been **closed in favor of this one** to keep the grouped codegen in a single place (it was otherwise duplicated across both). ## Why grouped needs special handling Grouped GEMM is **multi-problem**: one launch runs a *list* of `(M, N, K)` sub-problems with arrays of A/B/C device pointers. 1. The single-problem run path (`g_dispatcher->run` / `GemmHostArgs`) cannot express a list of problems. 2. The generated registry wrapper (`generated_tile_backend.hpp::run()`) hard-codes the single-problem launch and won't compile against a grouped `SelectedKernel`. So the grouped path **bypasses the registry**: a dedicated ctypes lib calls the generated `SelectedKernel::launch(descs, stream)` directly and reports the name from the compile-time `KERNEL_NAME` macro. ## Changes **Codegen (absorbed from #8075)** - `codegen/arch_filter.py` — `GEMM_GROUPED` operator tile constraints. - `codegen/unified_gemm_codegen.py` — `GemmVariant.GROUPED`, the grouped launch generator (DeviceMem internal workspace via `MakeKargs`, persistent/non-persistent grid), `grouped` in `--variants`. - `examples/gemm/cpp/02_grouped_gemm_driver.cpp` — standalone, layout/dtype-generic grouped driver with per-group reference verification. - `codegen/README.md` + `examples/gemm/cpp/README.md` — grouped sections. **Bridge** - `bindings/ctypes/grouped_gemm_ctypes_lib.cpp` — multi-problem, registry-bypass C ABI; per-group device alloc/copy; strides derived from the compile-time `ALayout/BLayout/CLayout`; warmup/repeat timing matched to Old-TE (`CK_TILE_BENCH_WARMUP/REPEAT`). - `python/gemm_utils.py` — `GroupedGemmProblem`/`GroupedGemmResult`, `GpuGroupedGemmRunner`, `run_grouped`, fp16/bf16/fp8(E4M3 FNUZ)/bf8(E5M2 FNUZ) codecs, output-dtype-aware C buffer. - `tile_engine/ops/gemm/grouped_gemm_full_benchmark.py` + `run_one_grouped_gemm_kernel.py` — TE driver + worker for the parity sweep. - `bindings/ctypes/GROUPED_GEMM_BRIDGE.md` — design README. ## Coverage (= Old-TE grouped runnable set on develop) | Layout \ Dtype | fp16 | bf16 | fp8 (E4M3) | bf8 (E5M2) | |---|---|---|---|---| | rcr / rrr / ccr / crr | ✓ | ✓ | ✓ | ✓ | C is always row-major. `int8` (rejected by the TE grouped builder) and `fp32`/`fp64` (no MFMA warp tiles) are excluded on both sides. ## Parity vs Old-TE (MI300X / gfx942) Apples-to-apples (same warmup=50/repeat=100 both sides, A/B interleaved, single GPU, both engines rebuilt fresh, stale-`.so` guard, matched compile flags): - **Correctness: 64/64 PASS.** - **Performance: 64/64 within ±15%.** - The 5 small-shape (1024³ fp8/bf8) rows that initially read >15% were proven by `rocprof` to be a **measurement-harness artifact** (Old-TE's JSON `latency(ms)` rounded to 2 decimals → 30–50% TFLOPS swing on ~0.02 ms kernels), **not** a kernel/codegen difference — bridge and Old-TE launch byte-identical kernels (same grid/VGPR/SGPR, duration ≤3.22%); full-precision re-measure collapses all 5 to <3%. ## Notes - Targets `develop`. Depends on #8997 (fp16/bf16 bridge) and #8998 (fp8/bf8/int8 bridge) merging to `develop` first; until then this PR's diff also shows their content, after which it reduces to the grouped-only files. - Supersedes #8075 (closed). |
||
|
Rostyslav Geyyer
|
a6028c883b |
[rocm-libraries] ROCm/rocm-libraries#9359 (commit 0cb05ba)
fix(ck-tile): Fix compiler issue MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Bug causes miopen_composable_kernel_ut to fail due to numerical correctness mismatches in Composable Kernel GEMM/MX/blockscale/contraction paths. The failures are not build failures; the tests run, but device output does not match CPU/reference output, including large mismatches and NaN outputs. ## Technical Details __builtin_amdgcn_sched_barrier(0) with mask 0 = "no instruction of any kind may be reordered across this point, in either direction". It lowers to no hardware instruction — purely a compile-time scheduling boundary. ## Test Plan Confirm with the Compiler team ## Test Result TBD ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID : AICK-1556 |
||
|
Muhammed Emin Ozturk
|
5d3380aa30 |
[rocm-libraries] ROCm/rocm-libraries#8985 (commit 3d4cbef)
feat(ck-tile): add stream_k variant to GEMM Dispatcher codegen (#8985) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > Supersedes #8094 (closed when its branch was renamed to a policy-compliant path). Same commits, same head SHA. ## Motivation This is the next slice of the Tile Engine → Dispatcher consolidation, following the same pattern as the grouped_gemm PR (#8075). It adds the **stream-K** GEMM variant to the unified GEMM codegen, implemented **the dispatcher way** (workspace owned internally via `DeviceMem`, clean `launch(args, stream)` signature), and proves numeric + performance parity against Tile Engine. Branch is based on `develop` and contains **only** the stream-K work (no grouped_gemm commits). ## Technical Details - **`codegen/arch_filter.py`** — added `OperatorType.GEMM_STREAMK` and its tile constraints. - **`codegen/unified_gemm_codegen.py`**: - Added `GemmVariant.STREAM_K`, made it reachable from the CLI (`--variants stream_k`), wired naming (`_streamk` suffix), includes, and the variant→operator map. - New `_launch_function_streamk`: builds a single `StreamKHostArgs`, `MakeKernelArgs` → `GetWorkSpaceSize` → allocate `DeviceMem` workspace **internally** + `SetZero` → `SetWorkSpacePointer` → `IsSupportedArgument` check → `make_kernel` via `launch_kernel_time_mask` with an Atomic-reduction preprocess that zeros C between timed iterations. No external `kargs_ptr` (not the Tile Engine way). - Exported `A/B/CLayout` in the `CK_TILE_SINGLE_KERNEL_INCLUDE` block so a single-kernel driver is layout-generic. - Restricted stream_k configs to the `cshuffle` epilogue (only one the kernel supports). - **`examples/gemm/cpp/03_streamk_gemm_driver.cpp`** (NEW) — minimal standalone driver: `-include`s one generated stream-K header, builds a single A/B/C tensor, calls `SelectedKernel::launch(args, stream)`, verifies against `ck_tile::reference_gemm`, prints TFLOPS/GB/s. The generated GPU kernel (`StreamKKernel<StreamKTilePartitioner, GemmPipeline, GemmEpilogue>`) is identical to TE's; only host-side workspace ownership differs (internal `DeviceMem` vs TE's external pointer). Numerics match. ## Test Plan - **Config:** `fp16_rcr_compv3_cshuffle_intrawave_..._128x128x64_2x2x1_32x32x16` (atomic reduction; exists identically in TE and the dispatcher). - **Shape:** `M=3840, N=4096, K=2048`, `warmup=10`, `repeat=50`, MI300X (gfx942), ROCm 7.1.1. - Run the `03_streamk_gemm_driver` and verify against `ck_tile::reference_gemm`; compare latency/TFLOPS/GB/s against the matching Tile Engine config. > Methodology note: TE's benchmark forces `repeat=1, warmup=0` whenever `verify=1` (the atomic kernel accumulates into C, so it can only verify a single run). A `verify=1` invocation therefore reports a single cold iteration (~0.30 ms), which is **not** a representative perf number. The table below uses TE `verify=0` (so warmup/repeat are honored) for the perf row and a separate TE `verify=1` run for correctness. The dispatcher driver times (warmup=10/repeat=50) and verifies in the same run because it re-zeros C between timed iterations via the masked preprocess. ## Test Result Performance + numerical verification (Dispatcher vs Tile Engine): | | latency (ms) | TFLOPS | GB/s | verify | |---|---|---|---|---| | **Tile Engine** (warmup=10, repeat=50) | 0.24 | 266.7 | 264.8 | correct | | **Dispatcher** (warmup=10, repeat=50) | 0.242 | 266.1 | 264.2 | PASS | | **Δ** | ~0% | ~0% | ~0% | identical | ## Next - Once signed off, delete `tile_engine/ops/gemm_streamk/`. - Continue toward a first-class `dispatcher` GEMM interface folder (roadmap step 5). |
||
|
Yung-sheng Tu
|
8c5870f962 |
[rocm-libraries] ROCm/rocm-libraries#8020 (commit fa9cdd1)
=?UTF-8?q?feat:=20[CK=20TILE]=20Unification=20Work=20?= =?UTF-8?q?=E2=80=93=20Add=20WMMA=20Scale=20Mixed=20Types=20Support=20(#80?= =?UTF-8?q?20)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation This PR adds mixed data types and mixed scale types support in scale WMMA specialisations. ## Technical Details - Replaced `ScaleDataTypeToFlag_v` with `PackedDataTypeToFlag_v` and adjusted WMMA scale selector to the way as MFMA one. - Simplified scale_gfx125.hpp with macros. - Added WMMA scale specialisations with mixed data types and scale types. - Introduced `is_legal_combination` to restrict which combinations of ADataType/BDataType/ScaleAValueType/ScaleBValueType the `exec()` can be instantiated. - Pleased the compiler on gfx1250 emulator to make compilation working. ## Test Plan All the new wrappers were added to the test suite in test_amdgcn_mma_layout.inc. ## Test Result Test should pass. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. close #8949 |
||
|
Emily Martins
|
6667a90217 |
[rocm-libraries] ROCm/rocm-libraries#8595 (commit 7fbebbb)
[CK] Clean up Stream-K remnants in old CK and fix static_assert in CK Tile (#8595) ## Motivation This PR is cleans up some Stream-K documentation (e.g., typos, unneeded docs) and fixes an error in a Stream-K static assert. This PR adds no new functionality. ## Technical Details Exact details about the clean-up changes are as follows: - Removes Stream-K documentation from old CK that were missed in PR #8141. - Fixes `static_assert` syntax in Stream-K kernel by adding `constexpr` to if-else condition and correcting empty message parameter. - Cleans up some typos in some Doxygen comments in the CK Tile Stream-K kernel file. ## Test Plan Ran existing tests locally; no new functionality was added, so no new tests were added. ## Test Result Tests passed. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Wojciech Laskowski
|
327dd55f46 |
[rocm-libraries] ROCm/rocm-libraries#7852 (commit 8f6a245)
feat: [CK Tile] Adding gfx1250 wrappers for dense and scale builtins (#7852) ## Motivation This PR is part of the [WMMA/MFMA] unification work. It adds all the necessary dense and scale MMA gfx1250 builtins as amdgcn_mma structs. ## Technical Details This change adds 5 new specializations for WMMA gfx1250 scale builtins and 25 new for dense. On top of that we: - dispatch between scale and scale16 builtin call - add scale abstraction for WMMA - small refactor for scale traits (data type to flag will also be needed for dense builtins with packed data types) - fix layout test (we store scale values in every byte) - create gfx1250 target family ## Test Plan All the new wrappers were added to the test suite in `test_amdgcn_mma_layout.inc`. ## Test Result Test pass locally, waiting for the CI. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Closes https://github.com/ROCm/rocm-libraries/issues/8957 |
||
|
Zoltán Lakatos
|
cb859854a7 |
[rocm-libraries] ROCm/rocm-libraries#6526 (commit 3e01710)
feat: [CK Tile] mxfp8 support for qr async pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes: #6526 ## Motivation Adds MXFP8 support to the QR async FMHA forward pipeline by enabling MX quant scale paths, loosening padding constraints, and updating tests/codegen to cover the new combinations. ## Technical Details Changes: - Extend BlockFmhaPipelineQRKSVSAsync to support MX quant scales (Q/K/V/P scale tiles) and adjust padding/alignment behavior accordingly. - Update FMHA fwd tests to include head-dim adjustment helpers and add a new “General” parameterized test suite. - Update codegen pipeline constraints/selection for qr_async, and adjust tile-window vectorization logic for packed types. ## Test Plan Running CI with extended set of tests ## Test Result CI passing ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Harkirat Gill
|
4975bd0c8e |
[rocm-libraries] ROCm/rocm-libraries#9232 (commit 2d25e39)
fix: guard arg.Print() in WMMA bwd data grouped conv with log level check (#9232) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Resolves https://github.com/ROCm/ROCm/issues/6403. ## Technical Details Stray `arg.Print()` was causing a flood of debug lines when running a torch reproducer that triggers backward-data WMMA grouped-conv path on gfx1151. For example, ``` a_grid_desc_m_ak_container{7744, 7520} b_grid_desc_n_bk_container_{64, 7520} e_grid_desc_mblock_mperblock_nblock_nperblock_container_{7744, 64} a_grid_desc_m_ak_container_{7808, 5024} b_grid_desc_n_bk_container_{128, 5024} e_grid_desc_mblock_mperblock_nblock_nperblock_container_{7808, 128} ``` Guarded the `arg.Print()` with a log level check `stream_config.log_level_ > 0`, matching the logic found in other ``arg.Print()`` statements, [ref](https://github.com/ROCm/rocm-libraries/blob/develop/projects/composablekernel/include/ck/tensor_operation/gpu/device/impl/device_grouped_conv_bwd_data_multiple_d_wmma_cshuffle_v3.hpp#L1311-L1314). ## Test Plan Compile example (`example/38_grouped_conv_bwd_data_multiple_d/grouped_conv_bwd_data_wmma_v3_fp16.cpp`) and run binary against both patched and unpatched header. Check output to see that debug logging is no longer present with patched version. ## Test Result Unpatched output (flood present): ``` out: dim 5, lengths {32, 4, 192, 28, 28}, strides {602112, 150528, 1, 5376, 192} wei: dim 5, lengths {32, 192, 192, 3, 3}, strides {331776, 1728, 1, 576, 192} in: dim 5, lengths {32, 4, 192, 28, 28}, strides {602112, 150528, 1, 5376, 192} a_grid_desc_m_ak_container_{3136, 1728} b_grid_desc_n_bk_container_{192, 1728} e_grid_desc_mblock_mperblock_nblock_nperblock_container_{3136, 192} Perf: 0 ms, inf TFlops, inf GB/s ``` Patched output (flood gone, conv still runs): ``` out: dim 5, lengths {32, 4, 192, 28, 28}, strides {602112, 150528, 1, 5376, 192} wei: dim 5, lengths {32, 192, 192, 3, 3}, strides {331776, 1728, 1, 576, 192} in: dim 5, lengths {32, 4, 192, 28, 28}, strides {602112, 150528, 1, 5376, 192} Perf: 0 ms, inf TFlops, inf GB/s ``` ## Submission Checklist - [X] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Anton Gorenko
|
7a710c3542 |
[rocm-libraries] ROCm/rocm-libraries#8423 (commit 8ad1035)
feat(ck): Extend and optimize Quant Gemm Kernel for Aiter a8w8 (#8423) ## Motivation JIRA ID : ROCM-20362 JIRA ID : ROCM-26313 The main purpose of this PR is to allow using CK and CK Tile GEMM for Aiter a8w8 (row-col quantization) and improve its performance. ## Technical Details ### Multiple D for Aiter a8w8 with bias * Support multiple D (bias) in Quant GEMM kernel * Extend CShuffleEpilogue * support int8 -> int32 WarpGemms with row-col quantization * allow shuffling in fp32 before applying multiple D to prevent precision loss ### Large tensors support * Support large tensors in the Quant GEMM kernel by offsetting pointers of matrices A, D and C. This feature can be used when M is large, N and K are relatively small and layout is RCR, it's currently enabled only row-col quantization. * Allow broadcasting of D column vectors in the old CK's `DeviceGemmMultiD_Xdl_CShuffle_V3` with large tensors, this case is used to implement row-col quant scales in Aiter. ### Optimization and workarounds * Use literal 0 as scales for unscaled 16x16x128 and 32x32x64 mfma: llvm uses v_mfma for `__builtin_amdgcn_mfma_scale_f32_..._f8f6f4` instead of v_mfma_scale only if scales are literal 0 values. These instruction don't need loading scales and save vector registers. See https://github.com/ROCm/llvm-project/blob/therock-7.13/llvm/lib/Target/AMDGPU/SIInstrInfo.td#L317-L327 * Add workaround for inefficient buffer_load to lds on 7.2 The 3rd argument of buffer_load_dwordx4 is a scalar register. But the compiler generates a waterfall loop as if lanes can have a different value, even though the original values comes from as scalar register. * Use buffer_store_dwordx4 to store 8 bf16 values in epilogue instead of two buffer_store_dwordx2 * Optimize eight waves pipeline: * Improve instruction scheduling * Remove unneeded barriers * Use nontemporal store/load for C and D matrices in the Quant GEMM kernel (they are used once per block but may consume cache that can be better used for matrices A and B) * Use more efficient padding in epilogue with CTransposed ## Test Plan A new test is added for multiple D Quant Gemm (`TestCkTileGemmRowColQuantMultiD/*.RowColQuantMultiDTest`): ``` ninja test_tile_gemm_quant_rowcol && bin/test_tile_gemm_quant_rowcol ``` Testing the large tensor support is not feasible with the current testing infrastructure because of reference calculations on CPU - it takes several minutes to run a single test case. Such cases are tested manually in Aiter. ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Muhammed Emin Ozturk
|
1c455b1bf5 |
[rocm-libraries] ROCm/rocm-libraries#8998 (commit 5501ef1)
feat(ck-tile): TE to dispatcher GEMM bridge for fp8/bf8/int8 (all layouts) (#8998) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Extends the Tile Engine ↔ Dispatcher GEMM **bridge** to the remaining data types TE's plain GEMM has MFMA warp tiles for, beyond the fp16/bf16 surface of #8479: - **fp8** (E4M3) and **bf8** (E5M2) → fp16 output, fp32 accumulate - **int8** → int32 output and accumulate (gfx942) All four A/B layout combinations per dtype (row-major C only, matching #8479). `fp32`/`fp64` are intentionally **excluded** — they appear in TE's dtype-string map but have no MFMA warp tiles in `GEMM_WARP_TILE_SUPPORTED_COMBINATIONS`, so no kernel can be generated/run. **Depends on the fp16/bf16 bridge in #8997** (`users/muozturk/ck-tile/gemm-bridge-all-layout-bf16-fp16`), which carries the bridge infrastructure and is not yet merged. This PR targets `develop`, so until #8997 merges its diff also includes the base bridge changes; please merge #8997 first. ## Changes - **Codegen** (`codegen_common.py`, `unified_gemm_codegen.py`): add `int32` to the dtype maps; `get_output_dtype` int8→int32; new `get_acc_dtype` (int8→int32, else fp32); derive `AccDataType`/`CDataType`, the `GEMM_KEY_DTYPE_{C,ACC}` macros, and the registry `dtype_c`/`dtype_acc` from the dtype instead of hard-coding `float`/`fp32`. - **Host harness** (`gemm_utils.py`): fp8/bf8 **FNUZ** (gfx942) uint8 codecs — exact decode (matches device `fp8_t`/`bf8_t`), nearest-representable saturating encode (same pattern as the existing bf16 helper); `GpuGemmRunner.run` encodes A/B and sizes the C buffer per dtype; `expand_sweep` sets `dtype_c`/`dtype_acc`. - **Tests**: `test_gemm_utils.py` adds CPU-only fp8/bf8 codec + output-dtype tests (all green); `test_gemm_parity.py` adds fp8/bf8/int8 cases with dtype-aware inputs/references/tolerances (int8 is bit-exact), GPU-gated like the existing cases. ## Verification done - `test_gemm_utils.py` + `test_codegen_common.py`: **54 passed** (CPU). - Codegen smoke: fp8/int8/fp16 each generate 1 kernel + 1 wrapper, 0 failed; emitted `ADataType/CDataType/AccDataType` and `GEMM_KEY_*` macros are correct (int8→int32_t acc/C; fp8→fp16_t C). - `test_gemm_parity.py` collects 60 cases and skips cleanly without a GPU. - The 16 unrelated failures in `test_examples_integration` / `test_grouped_conv_codegen` / `test_library_caching` are **pre-existing** (verified identical on the base branch; they require a built dispatcher `.a` / GPU). ## Test plan - [x] Merge #8997 (fp16/bf16 bridge), then this reduces to just the fp8/bf8/int8 delta on `develop`. - [x] On an MI300X (gfx942) node: run `python3 tests/test_gemm_parity.py` and confirm fp8/bf8/int8 parity; tune the fp8/bf8 tolerances if needed (current values are first-cut headroom). - [x] FNUZ vs OCP: the fp8/bf8 host codec targets the gfx942 FNUZ format; validate / extend for gfx950 (OCP) before enabling there. |
||
|
Thrupti Raj Lakshmana Gowda
|
b0f200713a |
[rocm-libraries] ROCm/rocm-libraries#8519 (commit 9637390)
feat(ck-tile): add block-scale GEMM operators (aquant, bquant, abquant) (#8519) JIRA ID - AICK-1289 Motivation Adds three new block-scale quantized GEMM operators to the CK Tile Engine for FP8/BF8 inference workloads. Technical Details gemm_aquant: A-matrix quantized GEMM with per-row-group scale tensor [M, K/group_size_k] gemm_bquant: B-matrix quantized GEMM with per-column-group scale tensor [K/group_size_k, N] gemm_abquant: Both A and B quantized with independent group-scale tensors Each operator includes CMakeLists, Python instance builder with tier sampling, C++ benchmark/profiler with host reference verification, and config JSONs. Supporting changes to gemm_instance_builder.py, gemm_validation_utils.py, sampling infra, and the operation support matrix. Test Plan Build and run all three operators with fp8/bf8 on gfx942/gfx950 Verify correctness against CPU reference Verify CI config builds pass |
||
|
Kiefer van Teutem
|
05615558c6 |
[rocm-libraries] ROCm/rocm-libraries#8315 (commit bda7d43)
feat: CK Tile unification - swizzle support + gfx950 mixed prec scale + misc (#8315) ISSUE ID #8960 https://github.com/ROCm/rocm-libraries/issues/8960 ## Motivation This MR is about adding Swizzle support to the Tile Distribution Encoding Calculator and Mma Pipelines in the Unification framework. Swizzle is a modifier for Tile Distribution Encodings that effectively performs a permutation in the M dimension. This means that it affects the Tile Distribution Encodings of A and C. When combined with CTranspose, it affects the Encodings of B and C instead. In principle, for a regular gemm, the Swizzle factor does not affect the correctness of the kernel, since matrix multiplication is symmetric under permutations of rows and columns (M). However, this is only true if the same Encodings are used for the loading and storing of the data. For consecutive matrix multiplications, we may be in a situation where we use Swizzle to account for the effective layout of an intermediate result, so that it can immediately be used in another matrix operation without additional shuffling. In these cases, the Swizzle factor is crucial for correctness. As far as I know, this seems most likely to occur in attention kernels. ### Changes - I adapted the Tile Distribution Encoding Calculator to accept any Swizzle modifier, and use this to modify the layouts just like in CK Tile. Note that Swizzle is only compatible with certain intrinsics, due to the restriction that the Swizzle factor divides kCMNumAccess. This is possible for 32x32 MFMA instructions with SFactor 2 or 4, and for gfx11 WMMA instructions with SFactor 2, 4, or 8, although this is not used in CK Tile. - I adapted the layout test to check the correctness of layout *with* Swizzle modification, for all possible Swizzle factors for each intrinsic. - I adapted the Unification Dispatcher to take a Swizzle Factor and pass it on to the MmaPipelines. Note that the original dispatcher takes a boolean instead, which I convert to an SFactor of 2 when true. I believe this is correct since in all cases where CK Tile previously used the old dispatcher, and SFactor of 2 ended up being used. However, there are two named WarpGemms (WarpGemmMfmaFp8Fp8F32M32N32K32SwizzleBTransposedCDistribution and WarpGemmMfmaI8I8I32M32N32K32SwizzleBTransposedCDistribution) which can support any Swizzle factor, and are actually used with Swizzle factors up to 4. These were not used in the old dispatcher but instead always used directly in CK Tile pipelines. - I added custom named WarpGemms in case the Unification flag is ON, for the named WarpGemms using Swizzle that are directly used in CK Tile pipelines. There are only two of them and they are the ones mentioned in the previous point. ### Changes part 2 While trying to get a swizzle example to work, I ended up having to add a lot of other changes which would have normally been their own issue. We have: - Adding all mixed precision gfx950 scale intrinsics (50 in total) - Adding these intrinsics to the layout test - Tile distribution encoding tweak: Allow for simplified C layouts in blockless cases - MmaPipelines tweaks: Make pretty much all old-style layout params available ### Note on AttrNumAccess For the scale gfx950 intrinsics, the "canonical" layouts for A and B have NumAccess 1 or 2, depending on the A and B types. The 8-bit types have a canonical NumAccess of 2, and the others 1. So overall we may have (1, 1), (2, 1), (1, 2), or (2, 2). This is reflected in the intrinsic definitions. However, for the fully 8-bit intrinsics I still define them with (1, 1). The reason for this is that it is in principle possible to use these intrinsics with (1, 1) as long as you don't use scale. This may actually happen in CK Tile. Furthermore, there are some pipelines that instantiate a WarpGemm with (1, 1) just to peek at some parameters. Note that the (1, 2) and (2, 1) cases MUST have these NumAccess values or the base MMA does not work (regardless of scale). This is because you can't just permute K for A without doing the same for B and vice versa. ### Tests Layout tests with swizzle work. tile_example_fmha_fwd and tile_example_fmha_bwd now compile and run, with correct verification for default settings. With fp8bf16 and init=3, get 5% wrong results on both this branch and develop, and this one is definitely sensitive to swizzle, because without swizzle it's 50% wrong. Better test: test_ck_tile_fmha_fwd_fp8bf16. This one behaves as expected and confirms that swizzle is genuinely necessary for correctness and working properly in the unification framework. It passed on develop and on my this branch with unification on, and failed when I forced a swizzlefactor of 1 (failed 40 out of 43 unskipped tests). |
||
|
Muhammed Emin Ozturk
|
de292a24f9 |
[rocm-libraries] ROCm/rocm-libraries#8997 (commit 6e9bfd9)
feat(ck-tile): TE to dispatcher GEMM bridge (fp16/bf16, all layouts) (#8997) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > Re-opened from #8479 with a compliant branch name (users/muozturk/ck-tile/gemm-bridge-all-layout-bf16-fp16). Supersedes #8479. ## Summary This PR routes the **Tile Engine (TE) regular-GEMM sweep through the Dispatcher**, making the Dispatcher the single source of truth for **codegen → build → runtime** while the Tile Engine keeps only the **config search space** and the **benchmark loop**. It is the consolidated, **single-commit** GEMM bridge covering **all four layouts (`rcr`/`rrr`/`crr`/`ccr`)** and **both `fp16` and `bf16`**. It is a clean re-roll of the earlier bridge work (previously split across #8123 + the stacked key/bf16/layouts/parity/example PRs and consolidated in #8261). Those branches accumulated unrelated cross-project commits through repeated `develop` merges; **this branch is a single clean commit off the latest `develop`** containing only the GEMM-bridge files. It supersedes and replaces #8123 / #8261. ## Motivation The Tile Engine historically owned its own codegen/build/runtime for GEMM (`tile_engine/ops/gemm/gemm_universal/`). The consolidation goal is for the **Dispatcher** to own all of that — exactly as it already does for **FMHA** and **Grouped Conv** — so there is one kernel-generation/build/runtime path and the TE shrinks to a config+benchmark frontend. This PR brings regular GEMM in line with that reference binding. ## The binding (mirrors the FMHA/Conv reference, six stages) 1. **Config JSON (TE side)** — the sweep search space lives in `tile_engine/ops/gemm/configs/` (flat op-root layout, matching the `fmha/` and `grouped_conv/` bridges). 2. **Codegen (Dispatcher)** — `dispatcher/codegen/unified_gemm_codegen.py` emits one fully-typed `.hpp` per kernel; `GemmKernelConfig.name` reproduces `KERNEL_NAME` **byte-for-byte** (the thread tying config → kernel → runtime). 3. **Compile to `.so`** — a single static `gemm_ctypes_lib.cpp` is force-included (`-include <kernel.hpp>`); one `.so` per kernel. 4. **Flat `extern "C"` ABI** — `dispatcher_run_gemm(A, B, C, M, N, K, time_ms)` + the kernel-name enumeration entry points. **Host-pointer** memory model (the C lib `hipMalloc`s internally) — the FMHA-forward branch of the reference. 5. **Python ctypes wrapper** — `dispatcher/python/gemm_utils.py` (`GemmDispatcherLib` + `GpuGemmRunner`). 6. **TE driver (3 phases)** — `gemm_full_benchmark.py` (parallel codegen+build → `expand_sweep` → subprocess-isolated benchmark) + the disposable per-kernel worker `run_one_gemm_kernel.py`. ## What's included **Bridge core** - `dispatcher/codegen/unified_gemm_codegen.py` — GEMM codegen, byte-exact naming. - `dispatcher/bindings/ctypes/gemm_ctypes_lib.cpp` — flat C ABI, host-pointer model. - `dispatcher/python/gemm_utils.py` — `GemmKernelConfig`, multi-kernel build (`setup_multiple_gemm_dispatchers`), `expand_sweep`, one-`.so`-per-kernel. - `tile_engine/ops/gemm/gemm_full_benchmark.py` + `run_one_gemm_kernel.py` — 3-phase, multi-GPU, subprocess-isolated driver/worker. **Feature surface (the point of this PR)** - **All four layouts** `rcr`/`rrr`/`crr`/`ccr` (row-major C only — ck_tile rejects column-major C at build) with layout-aware host transpose. - **`fp16` + `bf16`** (bf16 via uint16 byte-encoding; dtype derived from kernel name). - **Trait-derived registry `KernelKey`** — replaces the earlier hard-coded fp16/rcr key so the registry path generalizes across dtype/layout/tile. **Correctness & performance hygiene** - **`--verify`** opt-in fp32 numpy-reference gate (global `max|out-ref|/max|ref|`), `verified`/`max_rel` columns in the CSV; a mismatch counts as a failure. - **Tile Engine AMDGPU `-mllvm` codegen-flag parity** (without these the kernel builds with different occupancy and the timing diverges) and **arch-validated tile filtering** against the real pipeline/scheduler. - **Multi-GPU** fan-out across all visible GPUs (`--devices`, device-pinned `HIP_VISIBLE_DEVICES` workers). **Example & tests** - `dispatcher/examples/gemm/python/12_te_bridge.py` — runnable end-to-end example. - `dispatcher/tests/test_gemm_parity.py`, `test_gemm_utils.py`, and a parity regression harness. **Cleanup** - Removes the legacy standalone `gemm_universal` build path (`gemm_universal_instance_builder.py`, `*_benchmark*.{py,cpp,hpp}`, `gemm_universal/CMakeLists.txt`) and the old `test/ck_tile/gemm_tile_engine/` harness; promotes the sweep configs to the flat op-root `configs/`. ## Design decisions (consistent with the reference) - **Host-pointer memory ownership** (C lib owns device memory) — matches FMHA-forward; the Python runner passes host numpy arrays straight through. - **One `.so` per kernel** — packaging choice; the multi-kernel name ABI is retained (`get_kernel_name_at(0)` reports the single kernel), so the Python enumeration path is unchanged from FMHA/Conv. - **Flat `configs/`** at the op root — matches the `fmha/`/`grouped_conv/` convention; the not-yet-bridged variants keep their per-variant `configs/` dirs, selected by `--variant`. ## Validation (gfx942 / MI300X) - Bridge build + benchmark + `--verify` across **`fp16` and `bf16`** and **all four layouts**, checked against an fp32 numpy reference (`A @ B`). - **Name parity** holds end-to-end: each `.so`'s reported runtime name equals `GemmKernelConfig(...).name`. - bf16 passes under a widened fp16/bf16 tolerance; fp16 within the standard `max_rel` gate. ## Test plan - [ ] `gemm_full_benchmark.py --verify` over `configs/default_ci_config.json` for `fp16` and `bf16`, each of `rcr`/`rrr`/`crr`/`ccr`. - [ ] `unified_gemm_codegen.py` emits a header whose stem == `GemmKernelConfig.name`. - [ ] `setup_multiple_gemm_dispatchers` builds + links each config against `gemm_ctypes_lib.cpp`. - [ ] `pytest dispatcher/tests/test_gemm_parity.py dispatcher/tests/test_gemm_utils.py`. - [ ] `examples/gemm/python/12_te_bridge.py` runs end to end. ## Notes - Single clean commit off the latest `develop`; the diff is **35 files, all under `projects/composablekernel/`** (dispatcher + tile_engine/ops/gemm + test/ck_tile). - **Supersedes #8123 and #8261**, which will be closed. - Stream-K (#8136) and grouped GEMM are separate bridge efforts, not in this PR. |
||
|
sonle5
|
52d291af1a |
[rocm-libraries] ROCm/rocm-libraries#8639 (commit eaf0131)
fix(ck): fix gfx950 bpreshuffle determinism and illegal-type codegen (#8639) ## Motivation Fix the CK `gemm_a8w8_blockscale_bpreshuffle` path behind: - ROCm/aiter#3261: nondeterministic `gemm_a8w8_blockscale_bpreshuffle` results on gfx950/MI350/MI355 for fixed inputs and launch shapes. - sgl-project/sglang#28685: GLM-5.2-FP8 can silently produce wrong rows on gfx950/ROCm 7.2 when out-of-allowlist block-FP8 linear shapes route to AITER's CK bpreshuffle path. The issue is isolated to the bpreshuffle CK kernel path on gfx950/ROCm 7.2, not general FP8 numerics. This PR updates the CK source so the kernel no longer depends on the removed LLVM illegal-type coercion workaround. ## Technical Details - Replace the packed 2-lane post-scale accumulator FMA update in bpreshuffle v1/v3 with scalar `AccDataType` updates. - Add a VGPR read/write accumulator anchor after each post-scale scalar update to enable deterministic repeated launches. - Add profiler/test support for: - repeated-run bytewise determinism checks - row-relative error checks to catch sparse bad-row failures near tile boundaries - GLM-5.2 representative/out-of-allowlist shapes Relevant issues: - https://github.com/ROCm/aiter/issues/3261 - https://github.com/sgl-project/sglang/issues/28685 ## Test Plan Validated on gfx950 with ROCm 7.2 using the CK profiler/test path. Coverage includes: - ROCm/aiter#3261 WKV determinism sweep: - `M=8192`, `N=512`, `K={384,640,3968,4096,4224}` - GLM-5.2-FP8 representative shapes: - `q_a_proj`, `q_b_proj`, `mlp.down_proj`, and `kv_b_proj` - row-boundary sweep including `M={32,48,56,64,72,96,128}`, `N=28672`, `K=512` - CPU reference comparison for correctness cases - row-relative error reporting for sparse bad-row failures - repeated-run determinism checks - before/after perf regression sweep - manual pre-commit run on changed CK files ## Test Result - Kernel passed the tests: - `ReportedWkvDeterminism` - `Glm52OutOfAllowlistAccuracyAndDeterminism` - Build/test compile command validation showed `COERCE_COUNT 0`, confirming the tests ran without `-mllvm -amdgpu-coerce-illegal-types=1`. - Perf A/B showed no material regression; worst measured slowdown was about `+2.47%`, with other representative WKV/GLM shapes flat or faster. - Manual pre-commit on changed CK files passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Bartłomiej Kocot
|
076f232181 |
[rocm-libraries] ROCm/rocm-libraries#8983 (commit 80c884a)
revert(ck): magic division for long_index_t This reverts commit c25787bf08274860e3cdd9197cd88b161752b567. ## Motivation Memfaults problems after this implementation. ## Technical Details revert magic division for large tensors. Return to standard division path ## Test Plan test_grouped_convnd_bwd_data ## Test Result passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID : ROCM-27282 JIRA ID : ROCM-27360 |
||
|
Bartłomiej Kocot
|
590c734233 |
[rocm-libraries] ROCm/rocm-libraries#9034 (commit 537389e)
revert(ck): not needed cmake change from test grouped conv bwd data (#9034) ## Motivation Revert not needed cmake change ## Technical Details Restore previous state of cmake ## Test Plan test_grouped_convnd_bwd_data ## Test Result passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID: AICK-1459 |
||
|
Aviral Goel
|
7d604924b2 |
[rocm-libraries] ROCm/rocm-libraries#8531 (commit 6851169)
[CK_TILE] Use launched block size for GEMM occupancy query (#8531) The grouped, grouped-quant, and stream-k GEMM kernels were asking the occupancy query about `kBlockSize`, but on wave32 (gfx1250) we actually launch `kBlockSize/2`. So the occupancy came back too low and the persistent/stream-k grid ended up undersized. Just pass `BlockSize().x` like the universal and flatmm kernels already do. No-op on wave64. Verified it builds + runs correctly on gfx1250 (grouped gemm) and builds on gfx950 (stream-k). |
||
|
Thrupti Raj Lakshmana Gowda
|
6c02d45356 |
[rocm-libraries] ROCm/rocm-libraries#8799 (commit f36ce12)
fix(ck-tile): add missing fp8/bf8 warp_gemm dispatcher entries for gfx950 (#8799) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit JIRA ID : AICK-1534 ## Motivation Enable asymmetric A/B access width (e.g. `EDouble` for A, `ESingle` for B) for fp8/bf8/f16/bf16/i8 warp gemms on gfx950, and fix related dispatcher bugs. ## Technical Details - Split `AttrNumAccess` into `AttrNumAccessA`/`AttrNumAccessB` for fp8/bf8/f16/bf16 warp gemm aliases (default `B = A`, backward compatible). - Added `__gfx950__`-guarded asymmetric dispatcher entries for fp8/bf8 at 16×16×64 and 32×32×32. - Fixed missing `bf8×bf8` 16×16×64 `EDouble` dispatcher on non-gfx950/non-gfx125 platforms. - Fixed `WarpGemmMfmaBf16Bf16F32M16N16K32` silently dropping `AttrNumAccessB` on non-gfx950. - Fixed duplicate definition of `WarpGemmMfma_i32_16x16x64_i8_i8_CTransposed`. - Fixed `IsScale16=true` dispatcher incorrectly routing to non-scale16 variant. ## Test Plan Build and run existing CK Tile GEMM unit tests for fp8/bf8/f16/bf16/i8 types. |
||
|
Linjun-AMD
|
f4e6fad973 |
[rocm-libraries] ROCm/rocm-libraries#8944 (commit 7be2dbb)
feat(ck): Add swiglu_oai (OAI SwiGLU) activation to XDL 2-stage MoE epilogue. (#8944) ## Motivation Enable the OAI-form SwiGLU activation (`swiglu_oai`, `gate * sigmoid(1.702 * gate) * (up + 1)`, gpt-oss style) in the Composable Kernel XDL 2-stage MoE path. The MoE gridwise kernel epilogue currently supports only silu/gelu; this adds swiglu_oai so OAI-style MoE models can use this path. JIRA ID : ROCM-27213 ## Technical Details - `gridwise_gemm_xdl_cshuffle_common.hpp`: add `Activation::swiglu_oai_and_mul = 3`. - `gridwise_moe_gemm.hpp`: add the `apply_swiglu_oai_activation` helper (`gate * sigmoid(1.702 * gate) * (up + 1)`, clamp `gate <= 7` and `up in [-7, 7]`, OAI/gpt-oss form) and wire it into all 4 epilogue paths (quant + non-quant x `Run` / `Run_2Lds`). - The activation is applied in fp32 in the epilogue and is orthogonal to the GEMM compute (MFMA/tile/pipeline untouched) and to quantization (existing per-token dequant reused). Only the non-blockscale gridwise kernel is changed. - Consumed by aiter via ROCm/aiter#3886 (dispatch + codegen); review/merge together. ## Test Plan Validate the new epilogue branch against a torch fp32 OAI-SwiGLU reference through the aiter per-token fp8 MoE path (op-isolate on gfx942 / MI308X). ## Test Result cos_sim = 0.999993 vs the torch fp32 OAI-SwiGLU reference; no NaN. Confirmed the per-token fp8 path dispatches to this `GridwiseMoeGemm` kernel (rocprofv3) and runs the swiglu_oai epilogue branch. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Bartłomiej Kocot
|
a65244b86c |
[rocm-libraries] ROCm/rocm-libraries#8841 (commit 2421b65)
feat(rocke): cshuffle support for fp32 and bf16 ## Motivation Add support for bf16 and fp32 cshuffle in ck dsl. ## Technical Details -expand instructions to support bf16 and fp32 cshuffle -fix initializaiton of bf16 ## Test Plan Tested on 2640 cases ## Test Result passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. JIRA ID : AICK-1459 |
||
|
Enrico Degregori
|
d559ec00a8 |
[rocm-libraries] ROCm/rocm-libraries#8554 (commit be9af54)
refactor(ck): mx gemm kernel unification ## Motivation CK tile currently has two separate MX GEMM kernels for gfx950 and gfx1250. This pull request refactors and modernizes the MX GEMM kernel and example to use new scale tensor handling, improved kernel argument structures, and updated pipeline and kernel APIs. The changes simplify the interface and improve type safety. JIRA ID ROCM-26313 ## Technical Details - Add support for gfx950 in MX GEMM kernel for gfx1250 and remove unused kernel - Unify comp async pipeline for GEMM and MX GEMM - Unify eight waves pipeline for GEMM and MX GEMM - Move preshuffle MX GEMM pipeline to gemm ops and remove gemm_mx ops - Unify testing framework for MX GEMM - Add gfx950 tests for grouped MX GEMM ## Test Plan - `test_mx_gemm_async.cpp` for MX GEMM on gfx950 - `test_mx_grouped_gemm_comp_async.cpp` for grouped MX GEMM on gfx950 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Yung-sheng Tu
|
604c56bc0e |
[rocm-libraries] ROCm/rocm-libraries#7850 (commit e8f2756)
=?UTF-8?q?style:=20[CK=20TILE]=20Unification=20Work=20?= =?UTF-8?q?=E2=80=93=20Unify=20format=20MFMA=20part=20(#7850)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation This PR unifies the parameter comments and simplifies the docs for `amdgcn_mma` specialisations of `MfmaOp`. ## Technical Details Except for the two things mentioned above, it also simplifies the sparse traits, unifies the usages of `enable_if_target_id_t`, and cleans up the files in [include/ck_tile/core/arch/mma](https://github.com/ROCm/rocm-libraries/tree/users/yungshengtu/ck/unification/unify_format_mfma/projects/composablekernel/include/ck_tile/core/arch/mma). **NOTE: The first commit is not in the scope of this PR.** ## Test Plan Test has existed. ## Test Result Test should pass. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. close #8907 |
||
|
Brock Hargreaves
|
3719bf05c2 |
[rocm-libraries] ROCm/rocm-libraries#8644 (commit 8b2545e)
[CK][CI] Expand other stages to use healthy-node retry logic. (#8644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation CI stages bound to a single node via the declarative `agent{ label }` cannot recover when that node is unhealthy — a `NodeFault` (e.g. an exhausted `gitNetRetry` after a persistent `Could not resolve host: github.com`, or a missing GPU) just fails the whole build. The existing `runOnHealthyNode` wrapper already reroutes such faults to a different node, but only the "Build CK and run Tests" stages used it. This PR brings the remaining node-bound stages under that wrapper so a bad node reroutes instead of failing the build. ## Technical Details - `runOnHealthyNode`/`preflight` (`ck.groovy`): `preflight` now takes a `requireGpu` flag that gates the GPU-only checks (`driverUp`/`devicesUp`); `daemonUp`/`diskOk` always run. `runOnHealthyNode` derives it from the node label (`!label.contains('nogpu')`), so no call-site argument is needed and nogpu stages skip the GPU checks automatically. - `Jenkinsfile`: migrated 19 `agent{ label }` stages to `agent none` + `ck.runOnHealthyNode(...)`: - 12 GPU test stages: Pytorch, AITER ×2, FA ×2, 3 grouped-conv (gfx90a), TILE_ENGINE_BASIC, TILE_ENGINE_GEMM ×3. - 4 FMHA stages: the per-stage `environment{}` block was dissolved and `build_and_run_fmha(arch)` is now evaluated on-node inside the closure (required since `agent none` evaluates `environment{}` off-node). - 3 nogpu stages: Determine CI Execution, Build Docker, Static checks (reroute via the label-derived `requireGpu=false`). - The commented-out `gfx908`/`gfx1010` stages were intentionally left untouched. ## Test Plan - Validated the Jenkinsfile against the pipeline linter (`pipeline-model-converter/validate`). - Manual pipeline run to confirm no regressions - Key checks: nogpu stages pass `preflight` (no false `gpu-devices-missing`); migrated stages still allocate, build, and test as before; on a `NodeFault` the `Node attempt N/3 … on <node>` reroute lines appear. ## Test Result - Linter: **"Jenkinsfile successfully validated."** - Groovy shared-library pipeline — not locally executable; the manual build above is required to fully validate. Check CI. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Kiefer van Teutem
|
2089713f94 |
[rocm-libraries] ROCm/rocm-libraries#8227 (commit 75c30d5)
=?UTF-8?q?[CK=20TILE]=20Unification=20Work=20=E2=80=93=20?= =?UTF-8?q?Remove=20unification=20Flag=20structs=20in=20favor=20of=20new?= =?UTF-8?q?=20WarpGemmParams=20(#8227)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Recently, the way flags are sent down to the intrinsics was changed in CK Tile. At the point where the WarpGemm is invoked, an arbitrary number of template parameters can be passed, and these are passed down all the way to the lowest level intrinsics wrappers. Here `WarpGemmParamsParser<>` is used to extract flags for the intrinsics. In this MR we adapt the the unification framework (amdgcn_mma struct and MmaPipelines) to work in the same way. By doing this, there is no longer a point in our custom intrinsic Flag structs, so these are removed. Unrelated but I also tried removing the MmaPipeline flags because they arn't used for anything except CTranspose, which is already available. This also make test_amdgcn_mma_pipeline completely redundant so removed that as well. |
||
|
Illia Silin
|
621697af8c |
[rocm-libraries] ROCm/rocm-libraries#8723 (commit e2f28c1)
[CK] [Security] pin getopt library to specific commit ## Motivation Making sure that FetchContent_Declare is pulling third-party code from a specific commit hash instead of the tip of main branch to avoid accidentally pulling potentially harmful code. ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
spolifroni-amd
|
713f1fbf46 |
[rocm-libraries] ROCm/rocm-libraries#8739 (commit 833c182)
[hipcub, rocthrust, rocprim, ck] updating changelogs for 7.14 (#8739) ## Motivation The 7.14 change logs have a different format than the ones for 7.13. This changes the format and also sets the right component versions. |
||
|
chris-tsiaousis-hpc
|
e503e6277a |
[rocm-libraries] ROCm/rocm-libraries#8762 (commit fe88750)
[CK] Fix flaky test `test_batched_gemm_b_scale_wmma` Loosen up the tolerance and add better logging in case of failure TLDR: `profile_batched_gemm_b_scale_impl.hpp` (lines 390-392) uses `rtol = atol = 1e-1`, while the sibling non-batched B-scale profiler uses `2e-2` in `profile_gemm_b_scale_impl.hpp` (lines 361-364). `KBatch > 1` adds an F16 atomic accumulation step, so the batched path is stricter on the noisier arithmetic path. Loosening up the tolerances should fix the flaky test. I've also added better loging in case of failures. ## Motivation `test_batched_gemm_b_scale_wmma` can fail on gfx1201 when `KBatch > 1` because that path uses split-K F16 atomic accumulation. The existing batched B-scale tolerance was stricter than the non-batched B-scale profiler, even though the batched split-K path has additional rounding noise from native F16 atomics. This PR aligns the batched tolerance with the comparable non-batched path and improves failure diagnostics so future mismatches identify the exact shape, KBatch value, instance, strides, and operator. ## Technical Details Updated [projects/composablekernel/profiler/include/profiler/profile_batched_gemm_b_scale_impl.hpp](projects/composablekernel/profiler/include/profiler/profile_batched_gemm_b_scale_impl.hpp) to: - Change non-FP8 verification tolerance from `1e-2` to `2e-2`. - Match the tolerance already used by the non-batched B-scale profiler. - Build a detailed failure message with: - instance index - `kbatch` - `M`, `N`, `K`, and `BatchSize` - `StrideA`, `StrideB`, and `StrideC` - selected operator name - Move operator name construction earlier so it is available for diagnostics. ## Test Plan Configured the CK gfx1201 build with device instances enabled and XDL disabled to isolate WMMA coverage. Built and ran the focused batched GEMM B-scale WMMA test target: ```bash ninja -C projects/composablekernel/build-gfx950-gfx1201 test_batched_gemm_b_scale_wmma ``` Ran the focused `MidLargeM` case across WMMA instances and then ran the full WMMA-only gtest target. ## Test Result The WMMA-only gfx1201 validation passed. - `MidLargeM` passed across 21 WMMA instances. - Full `test_batched_gemm_b_scale_wmma` passed all 3 gtests. - No incorrect-result lines were reported. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Signed-off-by: Chris Tsiaousis <chris.tsiaousis@streamhpc.com> |
||
|
Kiefer van Teutem
|
137f2a9a10 |
[rocm-libraries] ROCm/rocm-libraries#7407 (commit 0b79e05)
[CK TILE] Initial integration of MFMA / WMMA unification framework into CK Tile (#7407) (locked behind flag) Note: Everything works but this is still a draft MR because I want to do some more cleanup and maybe do some testing for MX fp6. Also please don't trigger copilot, I will do this once I feel it is clean enough, otherwise I'll get a bunch of comments about stuff I already know. ## Motivation The point of this MR is to finally use our unification MmaPipelines to replace the existing WarpGemms in CK tile and make sure everything works. I focused on gfx908 and gfx950 for now, dense and scale intrinsics, fp16, fp8, and fp4. I managed to get CK tests / examples working for all of these scenarios, so the basic implementation should be correct. I expect some more tweaks will be required to get full support, some of which I already anticipated in the section "New issues". ## Big switch: USE_NEW_UNIFIED_FRAMEWORK When USE_NEW_UNIFIED_FRAMEWORK is 1, we replace all WarpGemms with MmaPipelines from the new unified framework. This means WarpGemmDispatcher will use the UnificationDispatcher instead of the regular Dispatcher. Furthermore, named WarpGemms like WarpGemmMfmaF32F32F32M16N16K4 will also get rerouted to the UnificationDispatcher. The latter is necessary because some pipelines bypass the WarpGemmDispatcher in favor of directly using named WarpGemms. For now the switch is turned on for easier testing, so don't expect the CI to pass. When off, this MR should not affect any of the CK tile tests at all so I *would* expect the CI to pass. ## Simplification of MmaPipelineBase I found that the structure of MmaPipelineBase was a bit complex and I was able to reduce it a lot. The only thing an Mma Pipeline does (currently) is provide a wrapper around amdgcn structs that allows k iteration and sparse compression. We don't allow M and N composition for now for simplicity and since this is not expected from WarpGemms in CK Tile currently. ## Re-interpretation of tile distribution encodings for packed datatypes Tile distributions for packed types are expected to describe mathematical elements, not datatype elements! This distinction is why the gfx950 fp4 CK_tile tests were not working. Updated the interpretation in amdgcn_mma, tile distribution calculator, and layout test, along with comments. Tested on all architectures. ## getCMakeCompilerTarget() for configuration time target architecture This is a workaround because there are a lot of cases in CK Tile where the host code inspects Device constructions like WarpGemm, and we need to get the version that *will* be used on the device. This is a big kludge and we need to figure out a better solution. Also this util will always pick the *first* cmakelists target arch, so there will be issues when compiling for multiple target architectures. Ideally, the host code should not touch the WarpGemms at all, and there would be no issue. This has been a point of friction in CK for a long time. We can discuss this with Chris Millette. ## Tests I was able to verify that the following CK Tile tests and examples work with the new unified framework: tile_tutorial_mfma_16x16x16 (gfx9, fp16, uses transpose) tile_example_gemm_basic (gfx9, fp16) test_ck_tile_mx_gemm_async (gfx950, microscaling fp8 and fp4) Within the tile tutorial I was also able to use WarpGemmMfmaF16F16F32M16N16K32TransposedCDistribution instead of WarpGemmMfmaF16F16F32M16N16K16TransposedCDistribution to verify that basic K iteration also works. A little while ago I also verified that the performance did not change in a measurable way, and the compile did not change *much* but did see some swings up to 20% each way (faster or slower). We will need some broader and more accurate tests for this going forward. ## Moving forward To confidently be able to replace the existing Dispatcher and WarpGemm framework with our own, we need to make sure that all existing tests and examples work on all platforms. Furthermore, we should pay attention to performance and compile time of all these tests. Performance should definitely not change, as all we're doing is refactoring the support structure around the intrinsics, which should melt away during compilation. ## New issues (I will make new issues with descriptions for these but here is a short list (incomplete): Test RDNA CK Tile pipelines Test Sparse Ck Tile pipeline (does not exist but we can make one) Remove MmaOp flags from unification framework and update it to work with new WarpGemmParamsParser instead. Add Swizzle support and test in CK Tile pipelines. Test Scale + transpose Ck Tile pipelines. Coherent strategy for attrnumaccess for dense, scale, default, packed, wmma, gfx1250, etc in CK tile. It's messy now. Dispatcher should not be determining scale-ness of intrinsics based on MNK sizes. Try adding back the MN composition in MmaPipelines Why is test_amdgcn_wavewise_mma only compiled for CDNA? Investigate NOP and AGPR flags Maybe get rid of WmmaTag in dispatcher. Find a coherent strategy for dealing with host vs device compile passes, and the host sneaking a peak at WarpGemm internals. Related to getCMakeCompilerTarget(). ## TODO before merge Some changes exist just for ease of testing, and will be reverted before merging: - gemm_basic.cpp has a lot of datatypes disabled because otherwise compile time is huge for testing - USE_NEW_UNIFIED_FRAMEWORK is set to 1 for easier testing |
||
|
Illia Silin
|
bd3713c710 |
[rocm-libraries] ROCm/rocm-libraries#8716 (commit 8230b20)
[CK] [Security] remove allow-unauthenticated flag from dockerfile (#8716) ## Motivation Dockerfile uses apt-get install --allow-unauthenticated which disables APT GPG signature verification, allowing package installation without cryptographic validation. An attacker who can perform a man-in-the-middle attack on the build network (via corporate proxy, CI egress, or compromised mirror) can serve trojaned .deb packages such as libc6, cmake, or git that become embedded in published ROCm container images distributed to users. ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Enrico Degregori
|
55e30feac6 |
[rocm-libraries] ROCm/rocm-libraries#8637 (commit a1a7f5f)
[CK] Fix compilation ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Adel Johar
|
01bad4c3d9 |
[rocm-libraries] ROCm/rocm-libraries#8205 (commit f58120c)
[Docs] Standardize precision support reference pages across components (#8205) ## Motivation The goal of this PR is to standardize the precision support reference page format across all components, while also reducing the maintenance of burden of having to manually update the YAML data file in https://rocm.docs.amd.com/en/latest/reference/precision-support.html ## Technical Details - Each component maintains its own YAML file which will be eventually used in https://rocm.docs.amd.com/en/latest/reference/precision-support.html - A new precision support reference page is introduced which will not override existing data type/precision support content; it will serve as the overview/summary that will be linked in the ROCm reference page ## Test Plan - Built locally, viewed each component manually ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Bartłomiej Kocot
|
7c2b979de2 |
[rocm-libraries] ROCm/rocm-libraries#8573 (commit 04c9f1d)
[CK][CK Tile] Drop profiler for experimental builder codegen (#8573) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Switch to dispatcher profiler for ck tile conv. ## Technical Details - Switch to dispatcher profiler for ck tile conv. - Drop profiler for experimental codegen - Minor fixes for bwd data printing - Minor fixes for 3d conv in dispatcher codegen ## Test Plan test_grouped_conv*tile ## Test Result Passed ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Enrico Degregori
|
2733e75900 |
[rocm-libraries] ROCm/rocm-libraries#6565 (commit d41715e)
[CK Tile] Async support pipeline V3 ## Motivation Optimize pipeline V3 for gfx950 by enabling buffer load to lds (async pipeline) ## Technical Details - Add `Async` bool to `Problem` struct to enable async pipeline in existing one - Add `static_move_ys` to load transpose. This generates offset in assembly instructions saving registers - Add `is_valid` to `async_get_vectorized_elements`. Before hard coded to true. It allows to support padding - Remove unnecessary restrictions to `is_a_load_tr` and `is_b_load_tr` (wider use of lds load transpose on gfx950) - Integrate async support in existing V3 pipeline (avoid pipelines duplication) - Create policy to support both async and default cases. This could be used by any async pipeline (next steps) - Define `wg_attr_num_access` separately for A and B. This allows to optimize ds_read instruction width for cases when one matrix is transposed and the other is not. Before in such cases, `ds_read_b64` was used instead of `ds_read_b128` - Add test for V3 async. Currently only supporting cases with A and B having the same type ## Test Plan New test `test_ck_tile_gemm_pipeline_compv3_async` ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Brock Hargreaves
|
081fe18c1c |
[rocm-libraries] ROCm/rocm-libraries#8558 (commit ccfa08b)
[CK][CI] Retry git network ops to survive transient DNS blips (#8558) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation CI builds intermittently fail on transient git DNS blips (e.g. `Could notresolve host: github.com`). These surface as an untyped `exit code 1`, which the existing node/transient-fault retry doesn't catch — so a momentary glitch fails the whole build. ## Technical Details Added `gitNetRetry(label, body)` (3 attempts, 15s backoff) and wrapped every github.com-touching git step: ref-repo clone/update, `checkout scm`, and the hipTensor clone. All are idempotent on retry. Docker pulls are left to the existing `pullImage()` path. ## Test Plan - Mapped the failing build's `git remote update` DNS error to a now-wrapped call. - Confirmed no existing code retries git host-resolution failures. ## Test Result Groovy shared-library — not locally executable; needs a pipeline run to fully validate. Check CI. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Brock Hargreaves
|
8864dcc3a4 |
[rocm-libraries] ROCm/rocm-libraries#8560 (commit f8362a1)
[CK][CI] Post failure GitHub status on stage build errors (#8560) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Failed CI stages (e.g. Static checks) were left stuck on a `pending` GitHub status instead of reporting `failure`, so PRs showed an overall failure with no indication of which check actually failed. ## Technical Details `buildAndTest` posted `pending`/`success` statuses but its catch only rethrew, deferring failure reporting to `runOnHealthyNode` — which deferred right back. Neither posted `failure`. This adds a `failure` status post for real build errors in `buildAndTest`, while letting node-reroute signals (`NodeFault`/`TransientFault`) and aborts (`FlowInterruptedException`) propagate untouched so retries still work. Since every stage routes through `buildAndTest`, this fixes both the directly-called `Static checks` stage and the `runOnHealthyNode`-wrapped per-arch build stages. ## Test Plan Trigger a stage failure (e.g. introduce a clang-format violation) and confirm the corresponding GitHub status context transitions `pending` → `failure` rather than remaining `pending`. ## Test Result Pending CI run on a branch with a deliberate failure to confirm the status transition. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Brock Hargreaves
|
bad7870830 |
[rocm-libraries] ROCm/rocm-libraries#8508 (commit 5cc3bef)
[CK][CI] Make gfx1250 build compile-only ## Motivation gfx1250 has no CI hardware, so its build piggybacks on gfx90a nodes where gfx1250 binaries can be compiled but not run. The build currently fails because post-build runtime tests fire on the gfx90a node. This PR makes the gfx1250 build compile + install only. ## Technical Details The post-build test block in `buildAndTest` (`ck.groovy`) keys off the physical node arch (`gfx90a`), so runtime tests run for gfx1250. Gated that block off for gfx1250. Body-only change with no signature changes, so it's backward compatible with the develop-pinned shared library and doesn't affect other archs. ## Test Plan Trigger the gfx1250 build with `USE_CURRENT_BRANCH_FOR_CK_GROOVY=true` and confirm it compiles/installs with no runtime test steps; confirm gfx90a builds are unchanged. ## Test Result Check CI. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Sami Remes
|
a3a12b8945 |
[rocm-libraries] ROCm/rocm-libraries#5813 (commit 18b43cf)
[CK_TILE] Enable full transpose layout support for MX GEMM pipeline (#5813) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Enable full transpose layout support for MX GEMM pipeline (32x32x64 MFMA) ### Summary This PR enables all four matrix layout combinations (Row/Col, Row/Row, Col/Col, Col/Row) for the MX GEMM pipeline with `32x32x64` MFMA warp tiles, using `ds_read_tr` transposed LDS loads on gfx950. Previously, only the canonical `A=RowMajor, B=ColumnMajor` layout was supported. ### Changes **Kernel-side transpose support:** - **`warp_gemm_attribute_mfma.hpp`**: Introduce `kSplitFactor` logic in `get_warp_dstr_encoding` to split the K-dimension distribution encoding when `kPerLane` exceeds the `ds_read_tr` subtile minor dimension. This satisfies the `TransposeTileDistributionTraits` suffix validation required by `load_tile_transpose`. The distribution encoding now also receives the `DataType` template parameter to compute the split factor based on packed element size. - **`gemm_pipeline_ag_bg_cr_comp_async.hpp`**: Uncomment and enable the `InputTileDistributionTraits` logic to properly transform LDS load tile distributions for transposed reads. Add `static_assert`s to catch misconfigurations where a layout requires transpose loads but the warp tile size disables them (e.g. `KWarpTile=128` exceeds `ds_read_tr` limits). - **`load_tile_transpose.hpp`**: Fix `DataVec` sizing for packed types (`pk_fp4_t`) — divide `vecLoadSize` by `PackedSize` to prevent buffer overflow when each physical element contains multiple logical values. - **`warp_gemm_attribute_mfma_impl.hpp`**: Set `kDefaultScale` to `0x7F7F7F7F` (unity in e8m0 format) for the unscaled `operator()` overloads of `WarpGemmAttributeMfmaImpl_f32_32x32x64_f8f6f4`, ensuring correct behavior with `mfma_scale_f32_32x32x64_f8f6f4`. - **`warp_gemm.hpp` / `warp_gemm_dispatcher.hpp`**: Add generic `WarpGemmMfma_f32_32x32x64_f8f6f4<A, B>` alias and dispatcher specialization to support arbitrary MX data type combinations (fp4, fp6, fp8) with the 32x32x64 MFMA, consolidating the existing type-specific aliases. - **`gemm_pipeline_ag_bg_cr_comp_async_default_policy.hpp`**: Simplify `wg_attr_num_access` determination — `Double` for fp8, `Single` otherwise. **Reference implementation fix:** - **`reference_gemm.hpp`**: Fix nibble selection for packed 4-bit types (`pk_fp4_t`, `pk_int4_t`) in `reference_mx_gemm`, `reference_gemm`, and `reference_gemm_abquant`. The previous logic used `k % 2` or `index[K_DIM] & 1` to select which nibble to extract, which assumed K was always the fast (contiguous) memory dimension. This is only true for `A=RowMajor` / `B=ColumnMajor`. For other layouts, the fix computes the flat memory offset via `mDesc.GetOffsetFromMultiIndex(...)` and uses its parity to correctly select the nibble regardless of layout. **Test infrastructure:** - **`test_mx_gemm_config.hpp`**: Add `MxGemmConfig32` base and `MXfp4_GemmConfig32` / `MXfp8_GemmConfig32` configs for the 32x32x64 warp tile. - **`test_mx_gemm_fp4.cpp` / `test_mx_gemm_fp8.cpp`**: Add `Config32` test suites covering all four layout combinations. Restrict `Config16` (16x16x128) to `A=Row, B=Col` only, since `KWarpTile=128` exceeds `ds_read_tr` limits. - **`test_mx_gemm_util.hpp`**: Fix scale tensor layout — scales are always row-major `[M, K/32]` and column-major `[K/32, N]`, independent of A/B data layout. ### Test plan - [x] `test_ck_tile_mx_gemm_fp4` — 5/5 passed (16x16x128 Row/Col + 32x32x64 all 4 layouts) - [x] `test_ck_tile_mx_gemm_fp8` — 5/5 passed (16x16x128 Row/Col + 32x32x64 all 4 layouts) - [x] `test_ck_tile_mx_gemm_fp6` — 1/1 passed (16x16x128 Row/Col) |
||
|
Illia Silin
|
e2deaaba64 |
[rocm-libraries] ROCm/rocm-libraries#8591 (commit 5210ae6)
[CK] fix daily hipTensor tests. ## Motivation Had to change the way hipTensor is cloned to make sure it doesn't erase CK installation and uses the correct path for the installation. Also added the "install" target every time we build and test everything, so we could use CK for testing third-party libs that depend on it. ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Enrico Degregori
|
1762eaeaec |
[rocm-libraries] ROCm/rocm-libraries#8535 (commit a0f47eb)
[CK Tile] EightWaves pipeline int8 support ## Motivation EightWaves pipeline currently is supporting only FP types ## Technical Details - Enable 16x16x64 int8 instruction for gfx950 in dispatcher - Enable int8 in EightWaves pipeline - Add tests - Fix bug in `warp_gemm_attribute_mfma_impl.hpp` ## Test Plan Tests have been added for int8 GEMM using EightWaves pipeline ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Ville Pietilä
|
60b276647b |
[rocm-libraries] ROCm/rocm-libraries#8157 (commit b0d9d39)
[CK Tile] Rule-based configuration generation in CK Dispatcher codegen (#8157) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation The CK Tile Dispatcher code generation for CK Tile Profiler relies on flat JSON files to list the generated configurations. This approach has the following problems - The JSON files are verbose - The JSON files get easily out of sync with the CK Builder .config files from which they were generated from. - The JSON file based configuration make it hard to list explicitly the rules that govern the instance generation. ## Technical Details Replaced the JSON files with a rule based configuration. To preserve the existing functionality, the `profiler` and the `tests` instance sets are generated directly from the CK Builder config files. The JSON config files are removed from source control, and the "on-the-fly" generation guarantees that the Dispatcher codegen uses up to date configurations. This is PR introduces six different rule sets for the CK Tile Dispatcher code generation 1. `profiler`: matches with the old JSON set of profiler configurations. 2. `tests`: matches with the old JSON set of tests configurations. 3. `full`: full configuration set created from a rule-based config selection 4. `full-tests`: a subset of `full` for generating configurations for convolution integration tests. 5. `tiny`: a subset of `full-tests` to produce the minimal set of configurations to test the Dispatcher codegen. 6. `default`: the default rules, which corresponds to the existing heuristic rules for configuration selection. This ensures that ML based kernel selection doesn't get broken. The main use of the `full` rule set is to define a reasonable solution space for the possible implicit GEMM configurations. We start from the configurations that allowed by the device architecture. The `full` rule set defines the relevant tile sizes for each convolution direction. From the tile size we have a curated mapping to the number of waves over the different GEMM axes, i.e., we describe how many waves each GEMM dimensions corresponds to. The GEMM-K wave tile dimension can be computed from the other parameters and does not need to be listed explicitly. An orthogonal axis to the tiling strategy is the vectorization strategy. This mainly defined by the data type and hardware as in general, we want to use the maximum possible load widths. The maximum sizes for each convolution direction variant are defined by the implicit GEMM matrix dimensions. For cases where have a low number of channels per convolution group, we need smaller vector load sizes. These are captured by the `VecStrategy` enumeration in the codegen rules. The problem with the rule based configuration selection is that we "over generate" configurations. The old JSON configurations compose approximately 25% of all configuration that the `full` rule set creates. The additional configurations are valid, but they many not provide any performance benefits. Hence, we keep the `profiler` and `tests` rule set for now to avoid building an excessive amount configurations by default. The `full` rule set can be taken into use by specifying CMake configuration flag `-D DISPATCHER_RULE_SET=full`. By default, the `tests` rule set is used, i.e., we don't change the existing bahaviour. ## Test Plan Added a new stage in the CI/CD pipeline that ensures the Dispatcher codegen rules are up to date. Otherwise the functionality is covered by the existing CI/CD tests. There are no functional changes to the convolution kernels. Only how the different instances are generated. ## Test Result If the CK Tile conv instances build without errors, the Dispatcher codegen is generating valid code. If all tests in CI/CD pipeline are passing, the Dispatcher codegen generates valid instances. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Aviral Goel
|
c43b550206 |
[rocm-libraries] ROCm/rocm-libraries#8202 (commit 0911fa0)
[GFX1250][CK_TILE] Add scale16 (ScaleBlockSize=16) support to MX GEMM TDM pipeline (#8202) Enables `ScaleBlockSize=16` end-to-end for the FP8/BF8 MX GEMM TDM pipeline, building on the scale16 warp-gemm layer already in develop. - **warp gemm:** add the 32x32x128 f8f6f4 scale16 traits and alias (2x2 grid of 16x16x128 scale16 intrinsic calls with per-subtile `SCALE_OPSEL`), and route 32x32 f8f6f4 through the dispatcher's `IsScale16` path. - **default policy:** select the warp gemm via the dispatcher with `IsScale16=(ScaleBlockSize==16)` so `WarpTile=16` and `WarpTile=32` each pick the matching scale16 path; guard WarpTile M/N to 16 or 32; scale-tile distribution for the scale16 layout. - **pipeline V1/V2:** thread `Problem::ScaleBlockSize` through the scale-window setup (replacing the hardcoded 32); expose `ScaleBlockSize` for the kernel. - **block gemm:** extract int64 (scale16) / int32 (scale32) scales by width. - **kernel:** scale16 descriptor order; reject unsupported `BlockScaleSize`. Test coverage for this path is in the stacked follow-up PR. |
||
|
jakpiase
|
65bef78383 |
[rocm-libraries] ROCm/rocm-libraries#8518 (commit 1ad69c3)
[CK] Add support for large tensor index handling into conv bwd data (#8518) ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Illia Silin
|
b5713be6cd |
[rocm-libraries] ROCm/rocm-libraries#8501 (commit 54eb5dc)
[CK] disable DPP kernels by default
## Motivation
The dpp8 instruction has been disabled in the upstream llvm-project in
the latest compiler version, so we're hitting compilation errors with
staging compiler:
<inline asm>:2:33: error: not a valid operand.
v_dot2c_f32_f16_dpp v6, v8, v7 dpp8:[0, 0, 0, 0, 0, 0, 0, 0]
^
error: cannot compile inline asm
These instructions are used for fp16 gemms that are slightly faster than
dl gemms on gfx10, but are not critical.
Going to disable these kernels for now, until a better solution is
available, to unblock the builds with staging compiler.
## Technical Details
<!-- Explain the changes along with any relevant GitHub links. -->
## Test Plan
<!-- Explain any relevant testing done to verify this PR. -->
## Test Result
<!-- Briefly summarize test outcomes. -->
## Submission Checklist
- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
|
||
|
SamiAario-AMD
|
39182b50eb |
[rocm-libraries] ROCm/rocm-libraries#8487 (commit 06a73ba)
Skip tests on gfx11 that have intermittent failures ## Motivation On gfx11, skip sporadic failures for any load_and_convert_tile case where X and Y differ. Same-type tuples (half/half, bf16/bf16, fp8/fp8) have been stable. ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result Stress-tested on gfx11, gfx12, and gfx950 with 10000 iterations of the tests. No remaining test failures were detected. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
damien-lejeune
|
5bebfd460f |
[rocm-libraries] ROCm/rocm-libraries#8492 (commit 46b6a06)
Add tile size for FMHA batch prefill bf16 for MI308X ## Motivation Adding a tile size adapted to MI308X, for the FMHA Batch Prefill BF16 input type case ## Technical Details N/A ## Test Plan Benchmarking from the Aiter side with: ``` python3 op_tests/test_batch_prefill.py -s 8000 -p 1 -q 4 -k 1 --head_dim 256 -c true -d bf16 --input_dtype bf16 --quant_method none --kv_layout linear -t sglang -l 0.0 --return_lse false --profile ``` ## Test Result We see an improvement with the new tile size on MI308X (both with PLT mode OFF and ON) ## Submission Checklist - [X] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Co-authored-by: Damien Lejeune <damien.lejeune@amd.com> |
||
|
damien-lejeune
|
2c0b7cbb0a |
[rocm-libraries] ROCm/rocm-libraries#8424 (commit debb669)
Add missing constraint in the FMHA qr async pipeline to enforce bk0=bk1 (#8424) ## Motivation The purpose of this change is to add a guardrail to what values bk0 and bk1 can take. This is to avoid ill defined sizes, silently failing and generating NaN (or other error) at runtime. An example of such failure can be obtained using the tile engine: ``` cd rocm-libraries/projects/composablekernel/tile_engine/ops/fmha python fmha_benchmark.py configs/batch_prefill.json \ --problems "1,4,1,8000,8000,256" \ --filter "c.data_type=='bf16' and c.hdim_q==256 and c.pipeline=='qr_async' and c.mode=='group' and c.tile_n0==32 and c.tile_k0==64" ``` ## Technical Details The qr_async pipeline stages data in the K dimensions into LDS using a bk1-descriptor, while the (Q*K^T) gemm0 consumes bk0 ## Test Plan See command above ## Test Result Before the change: (invalid) generate instances, error at runtime After this change: no instance generated ## Submission Checklist - [X] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. Co-authored-by: Damien Lejeune <damien.lejeune@amd.com> |
||
|
Brock Hargreaves
|
1b649a8d4b |
[rocm-libraries] ROCm/rocm-libraries#8332 (commit 48c389c)
[CK][CI] Retry builds on node failure with automatic rerouting (#8332) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation When a Jenkins node enters a bad state (missing GPU driver, dead Docker daemon, full disk), every PR scheduled onto it fails the same way until a human manually takes it offline. Some failures are also transient and would pass on a simple retry. Today the pipeline does neither — every failure goes straight to red on the same node. ## Technical Details Two new retry behaviors based on failure type: - **Different node** for persistent node faults (driver missing, daemon down, disk full, container won't start) - **Retry in place** for transient glitches (registry pull, DNS), then a different node if retries are exhausted Real build/compile failures and aborted builds are never retried. **New:** `src/org/ck/NodeFault.groovy`, `TransientFault.groovy` — typed exceptions in the shared library `src/` for stable classloader identity under dynamic library loading. **`vars/ck.groovy`:** adds `preflight()` (host health checks before build), `pullImage()` (classifying pull failures at the call site, replacing `getDockerImage()`), `runOnHealthyNode()` (outer reroute loop, up to 3 nodes), `runInPlace()` (same-node transient retries). GitHub failure status is only set once all retries are exhausted. **`Jenkinsfile`:** all active `Build CK and run Tests` stages converted to `agent none` + `ck.runOnHealthyNode(…)`. ## Test Plan Tested on `users/brockhargreaves-amd/ck/node-failure-retry-logic` with `USE_CURRENT_BRANCH_FOR_CK_GROOVY=true`. Verified preflight logging, reroute on node fault, attempt counter in logs, no retry on aborts, and single failure status report after budget exhausted. ## Test Result Retry logic working as expected. Three bugs found and fixed during testing: false `NodeFault` from host-level sccache probe (sccache is in-container), `null` node name in catch logging, and `sh` calls outside `node()` context in status reporting. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Andriy Roshchenko
|
b8440b3aeb |
[rocm-libraries] ROCm/rocm-libraries#8325 (commit 559eaf6)
[GFX1250][MX GEMM] Unified FLATMM GroupedGemm Implementation for MX Data Types (#8325) ## Motivation Design and test a unified FLATMM GroupedGemm interface so that it supports all MX FP8, FP6, and FP4 data types on both the gfx950 and gfx1250 architectures and works seamlessly across these platforms. ## Technical Details Implementation exposes Grouped Gemm interface for MX FLATMM and MX TDM FLATMM pipelines. ## Test Plan Add the following tests: - ck_tile/grouped_gemm_mx/test_grouped_gemm_mx_flatmm_non_tdm.cpp - ck_tile/grouped_gemm_mx/test_grouped_gemm_mx_flatmm_tdm.cpp - ck_tile/flatmm/test_mx_flatmm_persistent.cpp Verify on the gfx950 and gfx1250 architectures. ## Test Result All tests pass. Verified on A0 hardware with rocm-7.14.0a20260517 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |
||
|
Sami Remes
|
c1f7104852 |
[rocm-libraries] ROCm/rocm-libraries#6663 (commit f19fc01)
[CKTile] Fix MX GEMM: num_loop==3 dispatch, split-K, unsupported-shape guard (#6663) Three independent MX GEMM correctness bugs reported against example/ck_tile/42_mx_gemm (fp8xfp8, A=Row/B=Col) on MI350X, plus one host-side atomic-add accumulation bug in the example's repeat loop. - Pipeline (gemm_pipeline_ag_bg_cr_comp_async.hpp): BlockHasHotloop required num_loop > PrefetchStages, which let num_loop == 3 enter a hot loop that produced 5 gemm accumulations instead of 3 (K == 3*K_Tile, e.g. K=768, deterministically wrong). Require num_loop >= 4 instead: pre-pipeline + TailNumber::Three already totals exactly 3. - Kernel (gemm_mx_kernel.hpp): split-K was silently broken because GridSize did not thread k_batch into blockIdx.z and the scale tile windows were anchored at K=0 for every k_id. Every k_id >= 1 therefore read the wrong packed scales. Fix: * GridSize returns dim3(grid_x, 1, k_batch) (persistent and non-persistent). * MakeScaleA/BBlockWindows accept a k_elem_offset and translate it to a packed-scale K offset (also apply pad_tensor_view so OOB scale loads return zero, matching A/B padding). * operator() derives k_id from blockIdx.z, uses GetSplitKElemOffset (matches Underlying::SplitKBatchOffset's K1-aligned formula), and dispatches the epilogue with memory_operation_enum::atomic_add for k_batch > 1, set for k_batch == 1. Same fp16/bf16 even-vector-size guard as UniversalGemmKernel. * MakeCBlockWindows templated on DstInMemOp; unconditionally applies pad_tensor_view using kPadM/kPadN so partial trailing M/N tiles are handled correctly. - Compile- and runtime unsupported-shape guards (gemm_mx_kernel.hpp): add IsSupportedArgument and a static_assert for configurations that produce silent wrong results: * static_assert(!kPadK) -- the MX comp-async pipeline uses async_load_tile whose OOB check is per-vector-start, so a vector straddling the K pad boundary reads garbage. Until the async path learns per-element pad masking, reject kPadK at compile time. * Runtime: k_batch >= 1; M/N multiples of MPerBlock/NPerBlock when kPadM/kPadN are false; M >= MPerBlock and N >= NPerBlock always (CShuffleEpilogue cannot safely run with a single partial tile); K % (KPerBlock * k_batch) == 0; and for k_batch > 1, K must be a multiple of WarpTile_K * k_batch so every split lands on a packed-scale boundary. * All error paths log under CK_TILE_LOGGING with actionable messages. - Example (example/ck_tile/42_mx_gemm/mx_gemm_instance.hpp): * Call Kernel::IsSupportedArgument up front and throw a clear runtime_error for rejected shapes (was silently launching an unsupported kernel). * Switch to launch_kernel_time_mask with a clear_gemm_output preprocess that zeroes C between iterations when k_batch > 1 (mirrors universal_gemm_invoker). Without this the default -warmup=50 -repeat=100 accumulated 150 atomic_adds into C after the kernel-side split-K fix. Tests (test/ck_tile/gemm_mx/): - Add MXfp8_GemmConfig16_PadMN (kPadM = kPadN = true). - test_mx_gemm_fp8.cpp: HotLoopTailNumLoopThree (K=768 regression), SplitK (k_batch=2,4 across full_k/partial_k paths), TestMxGemmFp8PadMN::{MNPaddingAligned, MPadding, NPadding, MNPadding} covering trailing partial tiles along M, N, or both. - Run(...) now takes k_batch. - packScalesMNxK: guard against OOB (mn, k) reads from src and initialise e8m0 bytes to the zero exponent (0x00) instead of the default-constructed NaN (0xFF), so padded lanes don't poison the packed int32_t shared with in-range lanes. - test_mx_gemm_instance.hpp: call IsSupportedArgument before launch. Verification on gfx950, ROCm 7.2.0: - ctest -R test_ck_tile_mx_gemm -> 100% (2/2). - Example sweep over the original bug-report shapes: all K-aligned shapes now validate correct (including 4096^3 sk=2 and the K=768 cases); all K=128 shapes cleanly rejected with the new error message instead of producing silent wrong results. Made-with: Cursor ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Technical Details <!-- Explain the changes along with any relevant GitHub links. --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. |