Amir Ghamarian 33b2015939 Add 16x16 MFMA tiny decode kernel (1 warp, kBlockM=16, kBlockQ=2) ...

Enable 16x16 MFMA for decode by making the softmax cross-warp reduction
conditional on the warp gemm M dimension: use permlane32_swap for 32x32
MFMA (2 lanes per row), fall back to block_tile_reduce_sync for 16x16
MFMA (4 lanes per row).

New tiny decode traits: 1 warp, sequence<1,1,1>, warp_gemm 16x16x32,
kBlockM=16, kBlockQ=2 for GQA-8. This matches Triton's BLOCK_M=16 /
BLOCK_Q=2 decode configuration exactly.

Also adds 4-tier dispatch: tiny (avg_q<=2) -> small (avg_q<=8) ->
medium (avg_q<=128) -> large (prefill).

Benchmark results (d64 GQA-8 via aiter, 363 shapes):
  Before: CK faster 135 (37.2%), Triton faster 228 (62.8%)
  After:  CK faster 247 (68.0%), Triton faster 116 (32.0%)

Key shapes:
  1-seq decode:   0.021ms (CK 0.75x, wins 25%)
  64-seq decode:  0.025ms vs Triton 0.029ms (CK wins 14%)
  512-seq decode: 0.018ms vs Triton 0.021ms (CK wins)
  Weighted end-to-end: CK/Triton = 0.999x (tied)

Verified correct on 10 shapes: bf16+fp16, d64 GQA-8 + d128 MHA,
batch 1-64, all 4 dispatch tiers.

Made-with: Cursor

2026-03-28 12:19:34 +00:00

..

add_rmsnorm2d_rdquant

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

batched_contraction

[CK Tile] batched contraction kernel generalizing (#3126)

2025-12-02 13:30:27 +01:00

batched_transpose

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

common

[CK_TILE] Stream-K Tree Reduction and Cache Skipping Integration (#3371)

2025-12-14 14:49:49 -07:00

elementwise

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

epilogue

fix mxfp8-gemm example failure (#3531)

2026-01-13 10:26:45 +08:00

flatmm

Fix error when building with -DCMAKE_BUILD_TYPE=Debug (#3541)

2026-01-15 09:35:24 -05:00

fmha

[CK_TILE][FMHA] Add new tile size for async (#3586)

2026-01-19 15:22:33 -08:00

fused_moe

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

gemm

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

gemm_quant

[CK TILE QUANT GEMM] use OverrideADataType in aquant pipeline (#3584)

2026-01-16 15:27:39 -08:00

grouped_convolution

[CK TILE] Fix grouped conv kernels splitk and double lds (#3527)

2026-01-08 07:59:38 +01:00

image_to_column

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

layernorm2d

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

norm_reduce

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

permute

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

pooling

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

reduce

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

rmsnorm2d

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

smoothquant

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

softmax/block

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

topk/block

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

topk_softmax

Shuffle fix for gfx950 (#3491)

2026-01-13 09:21:29 -08:00

unified_attention

Add 16x16 MFMA tiny decode kernel (1 warp, kBlockM=16, kBlockQ=2)

2026-03-28 12:19:34 +00:00

add_rmsnorm2d_rdquant.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

batched_contraction.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

batched_transpose.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

common.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

elementwise.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

epilogue.hpp

[CK_TILE] Epilogue chaining (Lwpck 3373) (#2773)

2025-12-18 10:02:02 +01:00

flatmm.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

fmha.hpp

[FMHA] Batch Prefill Support Improvements: Change KV Cache Layout & Large Page Size Support (#3442)

2026-01-05 18:41:47 +08:00

fused_moe.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

gemm_quant.hpp

[CK_TILE] add preshuffleB mode for ABQuant GEMM (#3495)

2026-01-06 12:35:01 -08:00

gemm.hpp

Joye/revise wp pipeline (#3493)

2026-01-05 13:49:26 -08:00

grouped_convolution.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

image_to_column.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

layernorm2d.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

moe_flatmm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

norm_reduce.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

permute.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

pooling.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

reduce.hpp

Dlejeune/ck tile 2d multiple reductions (#3147)

2026-01-09 11:16:37 +01:00

rmsnorm2d.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

smoothquant.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

softmax.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

topk_softmax.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

topk.hpp

feat(precommit-hooks): add check for correct copyright header (#3302)

2025-12-10 22:50:43 -08:00

unified_attention.hpp

refactor

2025-12-11 13:34:27 +00:00