mirror of https://github.com/ROCm/composable_kernel.git synced 2026-05-20 21:09:08 +00:00

Files

Anton Gorenko 220bd7a9bb [CK_TILE] Support WMMA (gfx12) in FMHA (#2528 )

* Pass hdim to tile_example_fmha_fwd in fp8 tests

* Add WMMA support to fwd FMHA pipelines

* Tune tile sizes a bit for less spilling

fp16 256 is still quite slow

* Fix Q grad tile distribution for warp size = 32 and hdim >= 256

With AccDataType = float and warp size = 32, K0 becomes 0, K repeat is required to correcty distribute the tile.

* Use code based on BlockDropout in BlockDropoutBwd

* Fix split KV combine kernel for gfx12 (warp size 32) and make it more universal

* Fix LSE LDS tensor descriptors: kMaxSplits and kM0 were swapped, it worked on gfx9
  because they both equal to 8 while on gfx12 they are 8 and 4;
* Fix Oacc LDS tensor descriptor: it was transposed even though its shape=[4 * kM0, kN1],
  it worked on gfx9 because 4 * kM == kN1 == 32;
* Removing these hidden dependecies allows to support:
    * any number of warps (power-of-2), not only 4;
    * kN1 = 16, not only 32;
    * any number of splits;

* Rename ids like o_acc_4 and Oacc4 to eliminate confusion: kNumWarps doesn't have to be 4 now

* Replace hard-coded kN1 in dispatch code with the requested tile size

* Add gfx12-specific tile sizes for split KV

* Pass GPU architecture to kernel generation scripts

This is still a temporary solution.

* Build and run FMHA CI tests for gfx12

* Fix issue after merging

* Fix bwd tile sizes

The current pipelines always read only one tile K and V tile, this
requires bk0 == bhdq and bk2 == bhdv (kK0 == kQKHeaddim and
kK2 == kVHeaddim).

* Use hardware f32->f8 on gfx12, remove v_perm

__builtin_amdgcn_perm is not needed because
__builtin_amdgcn_cvt_pk_fp8_f32 allows to specify which word (16 bit of
 32-bit dword) is used to store results (two f8 values).

* Update changelog

* Add WMMA support to pagedkv

* Fix scripts after rebasing

* Support 16x16 (MFMA, WMMA) and 32x32 (MFMA) tiles in fwd and bwd BlockDropout

Add comments with dropout implementation details

Fix performance regression of fwd+dropout

    * Remove some usage of type punning (reinterpret_cast with ref or ptr) in Philox;
    * "scalarize" seed and offset, they may come either from kernel args or from device memory
      (presumably loaded with vector loads).

    These changes help the compiler to procude more optimal code and reduce register spilling.

Use WarpGemmDispatcher instead of explicit WarpGemmMfma... to get  CWarpDstrEncoding

Use code based on BlockDropout in BlockDropoutBwd

Refactor BlockDropout (fwd)

Implement BlockDropout (fwd) for WMMA

    Originally BlockDropout only supported 32x32 tiles (IsWG32 = true),
    this version supports 16x16 tiles.
    If MPerBlock > MWarp * 16, it can generate numbers for two 16x16 tiles, similarly
    to BlockDropoutBwd.

Implement BlockDropoutBwd for WMMA

Remove MakeRandValLds* functions unused in BlockDropoutBwd

Remove unused Run overload from BlockDropoutBwd

* Fix regression with philox seed and offset when they exceed 32-bit int

__builtin_amdgcn_readfirstlane works with 32-bit values, seed and offset
are 64-bit so they get truncated.

* Fix names after cherry-picking

* Fix selection of a fallback tile based on bm0

The assumption that the largest bm0 == 128 is not always true for
current fp32 tiles.

* Do not use filters related to qr_async_trload

They disable tiles/pipelines which are valid for gfx12.

* Use different dstr encoding when C is transposed

* Do not call GetQKBlockGemm (and hence WarpGemmDispatcher) in host code

Some WarpGemmDispatcher instantiations are defined only
for specific archs and undefined on host.
Calculations related to sched barriers are moved from Pipeline's public
fields into pipeline's operator().

* Fix incorrect name WarpGemmMfmaFp8Fp8F32M32N32K16SwizzleBTransposedCDistribution

Correct name is WarpGemmMfmaFp8Fp8F32M32N32K32SwizzleBTransposedCDistribution
because it's 32x32x16 with IterateK = 2 so K = 32, also all tiles used
in codegen scripts are 32, 32, 32.

* Generalize usages of WarpGemmDispatcher for MFMA and WMMA

WarpGemmMfmaFp8Fp8F32M32N32K32SwizzleBTransposedCDistribution is still
used explicitly becaus of swizzle factor = 4.

* Mark has_load_tr as maybe_unused

There are no transpose loading for RDNA.

* Remove CK_TILE_USE_MFMA/WMMA from fmha-related code

* Detect BlockSize on host based on warp size of the current device

If kBlockSize == kNumWarps * get_warp_size(), the kernel is launched with
kBlockSize / 2 because on host get_warp_size() == 64 always.

* Fix calculation of grid size for combine kernel with warp size = 32

* Add missing includes and header

* Support multiple archs in one binary for fwd

* Support multiple archs in one binary for fwd_splitkv, fwd_appendkv, pagedkv_prefill

* Support multiple archs in one binary for bwd

* trload kernels are compiled only for gfx950;
* instances with padding are checked after instances without padding so
  they can be used as fallbacks (similarly to fwd);

* Extract common code from register_traits

* Revert "Fix regression with philox seed and offset when they exceed 32-bit int"

To simplify merging , the proper fix is in develop already.

* Support new numerical d paddings in trait ordering checks

* Build fp32 tests only on gfx9

* Do not use hardcoded M0 = 64 for dot bwd kernel

* Use textwrap.indent from standard library

* Make fp8 pipelines on gfx12 consistent with gfx9

* Update tests for current pipelines

* Make ninja check more responsive in CI

ninja buffers output so this job looks hanging.

* Support fp8fp32 by limiting O vector size

The fp32 output type requires storing 8 * sizeof(float) = 32 bytes,
which is not implemented (here 8 is the number of C values per lane for
v_wmma_f32_16x16x16...).

* Remove unused cmake options

* Unify including  amd_buffer_addressing.hpp/_builtins.hpp

* Temporarily use amd_buffer_addressing.hpp on >=gfx10

amd_buffer_addressing_builtins.hpp uses inline asm for loads/stores
which is not compatible with >=gfx10:
 * 1 scalar for exec masks instead of 2,
 * gfx12 uses different instruction names etc.

* Update asm in bf16 conversions to work with warp 32

* Do not generate splitkv/appendkv with vlayout=col for consistency with fwd

* Add arch tags to kernels/host funcs, compile for each arch separately

* Add kM0 to fmha_bwd_dot_do_o kernel name to match filename

* Add workaround for miscompilation of bwd with padded hdim

SWDEV-559729: v_wmma instructions can be incorrectly placed in divergent
branches used to store padded tensors (when some lanes are inactive due
to padding). Inline asm with dummy dependencies on VGPRs of the tensors
prevents the compiler doing this.

* Fix add_gtest_executable for absolute paths

Some tests (like gemm_tile_engine) pass absolute paths to source files.
In CI the branch name is a part of the root dir, and if the branch name
contains "wmma", "xdl" etc., files can be incorrectly excluded.

* Run only hdim 128 smoke tests for fp8fp32

There are no instances for hdim 64 and 256.

* Format py with ruff to simplify merging develop

* Fix incorrect var name

* Codegen for gfx9,gfx950 when --targets is not specified

Aiter and Pytorch require changes for passing their targets to the codegen scripts.
With this temporary solution the files are generated but not all of them
have to be really built (depending on the used --offload-arch=).

* Combine arch-related values into ArchTrait

This more centralized approach removes duplication of various formatting templates.

* Try a workaround for Jenkins error "groovyjarjarasm.asm.MethodTooLargeException: Method too large"

Some code is extracted into a function.

[ROCm/composable_kernel commit: 1e77695fe8]

2025-10-29 13:31:08 -07:00

6.6 KiB

Raw Blame History

Changelog for Composable Kernel

Documentation for Composable Kernel available at https://rocm.docs.amd.com/projects/composable_kernel/en/latest/.

(Unreleased) Composable Kernel for ROCm

Added

Added a compute async pipeline in the CK TILE universal GEMM on gfx950
Added support for B Tensor type pk_int4_t in the CK TILE weight preshuffle GEMM.
Added the new api to load different memory sizes to SGPR.
Added support for B Tensor Preshuffle in CK TILE Grouped GEMM.
Added a basic copy kernel example and supporting documentation for new CK Tile developers.
Added support for grouped_gemm kernels to perform multi_d elementwise operation.
Added support for Multiple ABD GEMM
Added benchmarking support for tile engine GEMM Multi D.
Added block scaling support in CK_TILE GEMM, allowing flexible use of quantization matrices from either A or B operands.
Added the row-wise column-wise quantization for CK_TILE GEMM & CK_TILE Grouped GEMM.
Added support for f32 to FMHA (fwd/bwd).
Added tensor-wise quantization for CK_TILE GEMM.
Added support for batched contraction kernel.
Added WMMA (gfx12) support for FMHA.
Added pooling kernel in CK_TILE
Added top-k sigmoid kernel in CK_TILE

Changed

Removed BlockSize in make_kernel and CShuffleEpilogueProblem to support Wave32 in CK_TILE (#2594)
Added an optional template parameter Arch (gfx9_t, gfx12_t etc.) to make_kernel to support linking multiple object files that have the same kernel compiled for different architectures.
FMHA examples and tests can be built for multiple architectures (gfx9, gfx950, gfx12) at the same time.

Composable Kernel 1.1.0 for ROCm 7.1.0

Added

Added support for hdim as a multiple of 32 for FMHA (fwd/fwd_splitkv/bwd)
Added support for elementwise kernel.

Upcoming changes

Non-grouped convolutions are deprecated. Their functionality is supported by grouped convolution.

Composable Kernel 1.1.0 for ROCm 7.0.0

Added

Added support for bf16, f32, and f16 for 2D and 3D NGCHW grouped convolution backward data
Added a fully asynchronous HOST (CPU) arguments copy flow for CK grouped GEMM kernels.
Added support GKCYX layout for grouped convolution forward (NGCHW/GKCYX/NGKHW, number of instances in instance factory for NGCHW/GKYXC/NGKHW has been reduced).
Added support for GKCYX layout for grouped convolution forward (NGCHW/GKCYX/NGKHW).
Added support for GKCYX layout for grouped convolution backward weight (NGCHW/GKCYX/NGKHW).
Added support for GKCYX layout for grouped convolution backward data (NGCHW/GKCYX/NGKHW).
Added support for Stream-K version of mixed fp8/bf16 GEMM
Added support for Multiple D GEMM
Added GEMM pipeline for microscaling (MX) FP8/FP6/FP4 data types
Added support for FP16 2:4 structured sparsity to universal GEMM.
Added support for Split K for grouped convolution backward data.
Added logit soft-capping support for fMHA forward kernels.
Added support for hdim as a multiple of 32 for FMHA (fwd/fwd_splitkv)
Added benchmarking support for tile engine GEMM.
Added Ping-pong scheduler support for GEMM operation along the K dimension.
Added rotating buffer feature for CK_Tile GEMM.
Added int8 support for CK_TILE GEMM.

Optimized

Optimize the gemm multiply multiply preshuffle & lds bypass with Pack of KGroup and better instruction layout.
Added Vectorize Transpose optimization for CK Tile
Added the asynchronous copy for gfx950

Changed

Removed support for gfx940 and gfx941 targets (#1944)
Replaced the raw buffer load/store intrinsics with Clang20 built-ins (#1876)
DL and DPP kernels are now enabled by default.
Number of instances in instance factory for grouped convolution forward NGCHW/GKYXC/NGKHW has been reduced.
Number of instances in instance factory for grouped convolution backward weight NGCHW/GKYXC/NGKHW has been reduced.
Number of instances in instance factory for grouped convolution backward data NGCHW/GKYXC/NGKHW has been reduced.

Composable Kernel 1.1.0 for ROCm 6.1.0

Additions

Added generic instances for GEMM XDL operations (#1161)
Added gamma and beta parameters for the layernorm and groupnorm bwd operations (#1133)
Introduced wrapper sublibrary (limited functionality). (#1071, #1098, #1108, #1126)
Added an option to vary the number of warm-up cycles and iterations for ckProfiler (#1124)

Optimizations

New performance optimizations for GEMM operations on MI200 and MI300 architectures (#1135)

Fixes

Reduced the build time for most GPU architectures (#1084)
Fixed some conversion issues for fp8 data type (#1099)

Changes

None

Known issues

None

Composable Kernel 1.1.0 for ROCm 6.0.0

Fixes

Fixed a hazard associated with inline v_dot (#808)
Fixed two bugs in grouped convolution backward data without K padding (#848 #876)

Optimizations

None

Additions

Added an image to a column kernel (#867)
Added a column to an image kernel (#930)
Support for 3D grouped convolution on RDNA 3 GPUs (#935, #950, #985)
Grouped convolution support for small K and C (#822 #879 #897)
Support for NHWGC (2D and 3D) grouped convolution backward weight (#769 #804)
Support for bf16/f32/f16 and NHWGC (2D and 3D) grouped convolution backward data (#757 #799)
Support for Batched GEMM DL (#732)

Changes

Changed the grouped convolution API to maintain consistency with other convolution kernels (#817)

Composable Kernel 0.2.0 for ROCm 5.7.0

Fixes

Fixed a bug in 6-dimensional kernels (#555)
Fixed a test case failure with grouped convolution backward weight (#524)

Optimizations

Improved the performance of the normalization kernel

Additions

New CMake flags:
- "DL_KERNELS"-* Must be set to "ON" in order to build the GEMM DL and batched_gemm_multi_d_dl instances
- "DTYPES" -- Can be set to any subset of "fp64;fp32;fp16;fp8;bf16;int8" to build an instance of the specified data types
- "INSTANCES_ONLY" -- Only builds CK library and instances without tests, examples, or profiler
New feature: if GPU_TARGETS is not set in the CMake command line, CK will be built for all targets supported by the compiler
Support for MI300A/MI300X
Support for AMD RDNA 3
New user tutorial (#563)
Additional instances for irregular GEMM sizes (#560)
New inter-wave consumer-producer programming model for GEMM kernels (#310)
GEMM with support multiple elementwise fusions (multi-D) (#534)
Multi-embeddings support (#542)
AMD RDNA 3 blockwise GEMM and real GEMM support (#541)
AMD RDNA grouped convolution backward weight support (#505)
MaxPool and AvgPool forward (#815); MaxPool backward (#750)

Changes

None

6.6 KiB Raw Blame History

Changelog for Composable Kernel

(Unreleased) Composable Kernel for ROCm

Added

Changed

Composable Kernel 1.1.0 for ROCm 7.1.0

Added

Upcoming changes

Composable Kernel 1.1.0 for ROCm 7.0.0

Added

Optimized

Changed

Composable Kernel 1.1.0 for ROCm 6.1.0

Additions

Optimizations

Fixes

Changes

Known issues

Composable Kernel 1.1.0 for ROCm 6.0.0

Fixes

Optimizations

Additions

Changes

Composable Kernel 0.2.0 for ROCm 5.7.0

Fixes

Optimizations

Additions

Changes

6.6 KiB

Raw Blame History