yadaish c0ee71d735 Dev/a8w4 and a8w8splitk (#3447) ...

* Ck moe bs splitk pr (#3440)

* splitk kick-off. Compilation fail

* splitk hack pass

* fix scale offset calc.

* clang-format for a8w8_moe_blk_gemm1 splitk change

* fix testcase error

---------

Co-authored-by: oscar <huaiguxu@amd.com>
Co-authored-by: huaiguxu <145733371+huaiguxu@users.noreply.github.com>

* Zan/moe a8w4 (#3441)

* update

* update

* update ck moe a8w4

* update

* update

* update

* compile pass

* update

* update

* python3 op_tests/test_moe_2stage.py -t 16 -e 1 -k 1 -dim 256,256 ready

* support new a8w4 kernel

* update

* update ck_tile

* re format

* update

* update

* fix conflict

* fix build

* update ck_tile moe

* fix clang format

* fix the problem

* fix accruacy issue

* fix

---------

Co-authored-by: oscar <huaiguxu@amd.com>
Co-authored-by: huaiguxu <145733371+huaiguxu@users.noreply.github.com>
Co-authored-by: Zzz9990 <zanzhang@amd.com>
Co-authored-by: felix <felix.li@amd.com>

2025-12-19 09:26:52 +08:00

..

batchnorm_multiblock

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gemm_layernorm

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

normalization

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

block_to_ctile_map.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_reduce_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_welford_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_wmma_base.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

epilogue_cshuffle_v3_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_multiple_reduction_multiblock.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_multiple_reduction_threadwise.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_multiblock.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_threadwise_multi_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_2d_reduction_threadwise.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_ab_transfer_thread_tiles_preshuffle.hpp

Wmma support for gemm_multiply_multiply_wp (#3278)

2025-12-03 07:38:23 -08:00

gridwise_ab_transfer_thread_tiles.hpp

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

gridwise_ab_transfer_wave_tiles.hpp

Wmma support for gemm_multiply_multiply_wp (#3278)

2025-12-03 07:38:23 -08:00

gridwise_batched_gemm_gemm_wmma_cshuffle_v3.hpp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_softmax_gemm_wmma_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batchnorm_backward_blockwise_welford.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_batchnorm_forward_blockwise_welford.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_1d_scale.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_2d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_elementwise_layernorm_welford_variance.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_fpAintB_gemm_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_dl_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_dl_v1r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_dpp.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_abd_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_wmma_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_cshuffle_lds_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_multiple_d_xdl_splitk_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_selector.hpp

Simplify includes for CK builder reflection (#3357)

2025-12-05 07:44:10 -08:00

gridwise_gemm_pipeline_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_pipeline_v4_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_reduce_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_waveletmodel.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_wmma_cshuffle_v3_ab_scale.hpp

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

gridwise_gemm_wmma_cshuffle_v3_common.hpp

Grouped convolution forward device implementation and base flavors for RDNA3/4 (#2964)

2025-12-18 13:12:15 -07:00

gridwise_gemm_wmma_cshuffle_v3.hpp

Grouped convolution forward device implementation and base flavors for RDNA3/4 (#2964)

2025-12-18 13:12:15 -07:00

gridwise_gemm_wmma.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_conv_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_streamk_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_b_scale.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_abd.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_ab_scale.hpp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d_blockscale_b_preshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_multi_d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_mx_bpreshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3_mx.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_cshuffle_v3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdl_waveletmodel_cshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_bwd_weight.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_skip_b_lds_v1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_splitk_lds_direct_load.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_streamk.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r4.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v2r4r2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r1.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r2.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_gemm_xdlops_v3r3.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_gemm_blockscale.hpp

Dev/a8w4 and a8w8splitk (#3447)

2025-12-19 09:26:52 +08:00

gridwise_moe_gemm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm_bns.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm_bpreshuffle.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_moe_mx_gemm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_permute.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_put_element_1d.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_set_buffer_value.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_set_multiple_buffer_value.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_softmax.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_sparse_embeddings_forward_layernorm_builtins.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_sparse_embeddings_forward_layernorm.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

gridwise_tensor_rearrange.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00