ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-05-15 18:42:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f673873aa0aab7c61fbd661f4ecd938f074e8a91
composable_kernel/include/ck/tensor_operation/gpu/grid
History
Linjun-AMD f17537b7c2 [CK] add swiglustep_and_mul activation to gridwise_moe_gemm (#6873) 
Title：
feat(composablekernel): add swiglustep_and_mul activation to
gridwise_moe_gemm

  Description：
  ## Motivation

Step-3.5-Flash uses a clamped SwiGLU activation (`swiglu_limits[43]=7`,
  `swiglu_limits[44]=7`) for layers 43 and 44. Without this kernel path,
  those layers produce BOS token spam because unclamped gate/up values
  accumulate floating-point noise over 200+ decode steps, degrading
  output quality (cosine similarity drops from 0.999989 to ~0.998982).

  ## Changes

  Add `swiglustep_and_mul` as a new `Activation` enum branch in
  `gridwise_moe_gemm.hpp`, covering all 4 code paths:
  - Quantized (A×B scale) + IsInputGemm=true
  - Quantized (A×B scale) + IsInputGemm=false
  - Non-quantized + IsInputGemm=true
  - Non-quantized + IsInputGemm=false

  The activation computes:
  gate = silu(gate)
  gate = clamp(gate, max=7.0f)
  up   = clamp(up,   min=-7.0f, max=7.0f)
  output = gate * up

Also handles the `MulRoutedWeight` case (topk weight multiplication) and
  `pk_i4_t` weight scaling (×16 dequant factor).

  ## Verification

  - Tested on gfx950 (MI350X, 8×GPU)
- cosine similarity for layers 43/44: **0.999989** (vs 0.998982 before
fix)
  - End-to-end Step-3.5-Flash inference: no BOS spam, output coherent
  - BF16 tp=2/tp=4 and FP8 tp=2/tp=4 all verified PASS
- [x] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
2026-05-07 05:59:47 +00:00
..
batchnorm_multiblock
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gemm_layernorm
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
normalization
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
block_to_ctile_map.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
epilogue_cshuffle_v3_reduce_wmma.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
epilogue_cshuffle_v3_welford_wmma.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
epilogue_cshuffle_v3_wmma_base.hpp
Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)
2026-01-14 11:02:19 +01:00
epilogue_cshuffle_v3_wmma.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
epilogue_direct_store.hpp
Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)
2026-01-14 11:02:19 +01:00
gridwise_2d_multiple_reduction_multiblock.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_2d_multiple_reduction_threadwise.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_2d_reduction_multiblock.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_2d_reduction_threadwise_multi_d.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_2d_reduction_threadwise.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_ab_transfer_thread_tiles_preshuffle.hpp
Multi AB support for wave transfer (#3578)
2026-01-29 10:29:40 -08:00
gridwise_ab_transfer_thread_tiles.hpp
Multi AB support for wave transfer (#3578)
2026-01-29 10:29:40 -08:00
gridwise_ab_transfer_wave_tiles_interleave.hpp
Multi AB support for wave transfer (#3578)
2026-01-29 10:29:40 -08:00
gridwise_ab_transfer_wave_tiles.hpp
Multi AB support for wave transfer (#3578)
2026-01-29 10:29:40 -08:00
gridwise_batched_gemm_gemm_wmma_cshuffle_v3.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_batched_gemm_softmax_gemm_wmma_cshuffle.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_batchnorm_backward_blockwise_welford.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_batchnorm_forward_blockwise_welford.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_elementwise_1d_scale.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_elementwise_2d.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_elementwise_layernorm_welford_variance.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_fpAintB_gemm_wmma.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_gemm_dl_multiple_d.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_dl_v1r3.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_dpp.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_multiple_abd_xdl_cshuffle.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_gemm_multiple_d_wmma_cshuffle.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_multiple_d_xdl_cshuffle_lds_direct_load.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_multiple_d_xdl_cshuffle.hpp
CK: Remove 41 commented-out dead code blocks (~200 lines) (#6302)
2026-04-10 11:17:11 -04:00
gridwise_gemm_multiple_d_xdl_splitk_cshuffle.hpp
173 implement device grouped gemm fixed nk for rdna4 (#4299)
2026-02-19 09:13:05 +01:00
gridwise_gemm_pipeline_selector.hpp
Simplify includes for CK builder reflection (#3357)
2025-12-05 07:44:10 -08:00
gridwise_gemm_pipeline_v1.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_pipeline_v2.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_pipeline_v3.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_pipeline_v4_direct_load.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_reduce_xdl_cshuffle_v1.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_gemm_split_k_multiple_d_xdl_cshuffle_v2.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_waveletmodel.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_wmma_cshuffle_v3_ab_scale.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_wmma_cshuffle_v3_common.hpp
Implement device_grouped_gemm_fixed_nk_bias for RDNA4 (#4340)
2026-02-26 00:28:09 +00:00
gridwise_gemm_wmma_cshuffle_v3.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_wmma.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_gemm_xdl_cshuffle_common.hpp
projects/composablekernel: add SwigluStep support for MoE blockscale (#6118)
2026-04-21 07:24:48 +00:00
gridwise_gemm_xdl_cshuffle_conv_v3.hpp
[CK_TILE] Grouped Convolution Backward Data Direct Load (#6624)
2026-04-23 11:16:55 +02:00
gridwise_gemm_xdl_cshuffle_streamk_v3.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v1.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v2.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_b_preshuffle.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_b_scale.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_multi_abd.hpp
CK: Remove 41 commented-out dead code blocks (~200 lines) (#6302)
2026-04-10 11:17:11 -04:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_ab_scale.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_b_preshuffle.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_blockscale_b_preshuffle.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_multi_d.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_mx_bpreshuffle.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3_mx.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_cshuffle_v3.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_gemm_xdl_waveletmodel_cshuffle.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_xdlops_bwd_weight.hpp
[Compiler] Addressing new compiler warnings (#3640)
2026-02-02 09:39:48 -08:00
gridwise_gemm_xdlops_skip_b_lds_v1.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_xdlops_splitk_lds_direct_load.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdlops_streamk.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdlops_v2r3.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdlops_v2r4.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_xdlops_v2r4r2.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_gemm_xdlops_v3r1.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_xdlops_v3r2.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_gemm_xdlops_v3r3.hpp
[ck] add gridwise base class for in all xdl kernel (#186) (#3544)
2026-01-27 12:49:47 -08:00
gridwise_moe_gemm_blockscale.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_moe_gemm.hpp
[CK] add swiglustep_and_mul activation to gridwise_moe_gemm (#6873)
2026-05-07 05:59:47 +00:00
gridwise_moe_mx_gemm_bns.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_moe_mx_gemm_bpreshuffle.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_moe_mx_gemm.hpp
[CK] Fix/suppress clang lifetimebound warnings with staging compiler. (#6550)
2026-04-22 15:47:47 +00:00
gridwise_permute.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_put_element_1d.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_set_buffer_value.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_set_multiple_buffer_value.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00
gridwise_softmax.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_sparse_embeddings_forward_layernorm_builtins.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_sparse_embeddings_forward_layernorm.hpp
[CK] Replace nested static_for with static_ford to reduce device IR function emissions [1B] (#5031)
2026-03-18 08:45:22 -06:00
gridwise_tensor_rearrange.hpp
chore(copyright): update copyright header for include directory (#3293)
2025-11-26 11:00:05 -07:00