Enrico Degregori e79a263198 Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465) ...

- Add support for direct store in epilogue instead of cshuffle
 - Add padding support for wave transfer without transpose
 - Add wave transfer with interleaved layout to support direct store
 - Enable new functionalities on GEMMs
 - Add optional new functionality support for grouped convolution fwd
 - Add some fast instances for grouped convolution fwd with new functionalities (proper tuning needed)

2026-01-15 09:29:14 -05:00

..

batchnorm_multiblock

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gemm_layernorm

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

normalization

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

block_to_ctile_map.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_reduce_wmma.hpp

Wmma support for gemm_bias_add_reduce (#3316)

2026-01-15 09:29:12 -05:00

epilogue_cshuffle_v3_welford_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_wmma_base.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

epilogue_cshuffle_v3_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_direct_store.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

gridwise_2d_multiple_reduction_multiblock.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_multiple_reduction_threadwise.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_multiblock.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_threadwise_multi_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_threadwise.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_ab_transfer_thread_tiles_preshuffle.hpp

Wmma support for gemm_multiply_multiply_wp (#3278)

2025-12-03 07:38:23 -08:00

gridwise_ab_transfer_thread_tiles.hpp

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

gridwise_ab_transfer_wave_tiles_interleave.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

gridwise_ab_transfer_wave_tiles.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

gridwise_batched_gemm_gemm_wmma_cshuffle_v3.hpp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_softmax_gemm_wmma_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batchnorm_backward_blockwise_welford.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batchnorm_forward_blockwise_welford.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_1d_scale.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_2d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_layernorm_welford_variance.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_fpAintB_gemm_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp

Wmma support for gemm_bias_add_reduce (#3316)

2026-01-15 09:29:12 -05:00

gridwise_gemm_dl_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_dl_v1r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_dpp.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_abd_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_wmma_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_cshuffle_lds_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_splitk_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_selector.hpp

Simplify includes for CK builder reflection (#3357)

2025-12-05 07:44:10 -08:00

gridwise_gemm_pipeline_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v4_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_reduce_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_waveletmodel.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_wmma_cshuffle_v3_ab_scale.hpp

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

gridwise_gemm_wmma_cshuffle_v3_common.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

gridwise_gemm_wmma_cshuffle_v3.hpp

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-15 09:29:14 -05:00

gridwise_gemm_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_conv_v3.hpp

[CK] Allow tensors larger than 2GB in grouped conv bwd weight (#3169)

2026-01-15 09:29:12 -05:00

gridwise_gemm_xdl_cshuffle_streamk_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_b_scale.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_abd.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_ab_scale.hpp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_blockscale_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_mx_bpreshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_mx.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_waveletmodel_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_bwd_weight.hpp

[CK] Allow tensors larger than 2GB in grouped conv bwd weight (#3169)

2026-01-15 09:29:12 -05:00

gridwise_gemm_xdlops_skip_b_lds_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_splitk_lds_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_streamk.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r4.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r4r2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_gemm_blockscale.hpp

moe fp8 blockscale use nt (#3524)

2026-01-15 09:29:13 -05:00

gridwise_moe_gemm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm_bns.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm_bpreshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_permute.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_put_element_1d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_set_buffer_value.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_set_multiple_buffer_value.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_softmax.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_sparse_embeddings_forward_layernorm_builtins.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_sparse_embeddings_forward_layernorm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_tensor_rearrange.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00