Adam Osewski 8bb2bb4a05 Grouped Gemm + SplitK + simplified Kernel Args (#669) ...

* simplify karg in device/grid split-k op

* fix mk_kn_mn instances

* add more instances

* B2C with 3D grid for KSplit

* Remove unused code.

* Use default B2C (3D grid) in grid gemm v2r4r2.

* Device gemm splitk use B2C map.

* Device GroupedGemmXdlSplitKCShuffle

* Example for GroupedGemm Xdl SplitK

* Introduce Device GroupedGemmSplitK

* Fix updating kbatch size.

* Add instance mk-nk-mn

* Enable set kbatch in profiler.

* Add GGemmSplitK mk-kn-mn instances

* Add more instances & split into multiple files.

* minor fix

* tuning

* clean

* disabled failed instances

* use pipe v2

* Ignore arg on not supported arch.

* fix warning

---------

Co-authored-by: carlushuang <carlus.huang@amd.com>
Co-authored-by: Adam Osewski <aosewski@amd.com>
Co-authored-by: zjing14 <zhangjing14@gmail.com>
Co-authored-by: Jing Zhang <jizhan@amd.com>
Co-authored-by: root <root@ctr-ubbsmc15.amd.com>

2023-04-24 15:43:36 -05:00

..

batchnorm_multiblock

BatchNorm backward instance/external API/profiler/tests (#519)

2022-11-30 13:32:20 -06:00

gemm_layernorm

Gemm layernorm welford (#413)

2023-01-16 20:08:25 -06:00

block_to_ctile_map.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

gridwise_2d_multiple_reduction_multiblock.hpp

Batchnorm-forward and Batchnorm-infer Implemented using generic kernels (#320)

2022-08-15 10:11:02 -05:00

gridwise_2d_multiple_reduction_threadwise.hpp

Batchnorm-forward and Batchnorm-infer Implemented using generic kernels (#320)

2022-08-15 10:11:02 -05:00

gridwise_2d_reduction_multiblock.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_2d_reduction_threadwise.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp

batched_gemm + multiple_d + gemm + multiple_d (#394)

2022-09-14 17:54:18 -05:00

gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp

Remove the workaround for bf16 attention tests. (#586)

2023-02-14 18:06:24 -06:00

gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp

Generate output using Doxygen / Breathe (#598)

2023-03-06 11:39:16 -06:00

gridwise_batchnorm_backward_blockwise_welford.hpp

BatchNorm backward instance/external API/profiler/tests (#519)

2022-11-30 13:32:20 -06:00

gridwise_batchnorm_forward_blockwise_welford.hpp

BatchNorm backward implementation (#461)

2022-11-28 20:51:10 -06:00

gridwise_contraction_dlops_v1r2.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_elementwise_1d.hpp

Add 'Permute' device op & example (#408)

2022-09-19 21:30:25 -05:00

gridwise_elementwise_2d.hpp

Gridwise elementwise 2d (#466)

2022-12-12 09:18:10 -06:00

gridwise_elementwise_layernorm_welford_variance.hpp

Gemm layernorm welford (#413)

2023-01-16 20:08:25 -06:00

gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_dl_multiple_d.hpp

gemm/Conv xdlops + dlops quantization (#625)

2023-03-15 15:29:40 -05:00

gridwise_gemm_dl_v1r3.hpp

Add Grouped Conv Backward Weight on Navi21 for ResNet50. (#505)

2023-02-22 11:59:53 -06:00

gridwise_gemm_dlops_v1r2.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_gemm_dlops_v2.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_gemm_dlops_v3.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_multiple_d_wmma_cshuffle.hpp

Add memory index guard in wmma device ops (#667)

2023-04-11 15:42:47 -05:00

gridwise_gemm_multiple_d_xdl_cshuffle.hpp

add a marco to turn on/off denorm fix (off by default) (#673)

2023-04-11 07:44:43 -05:00

gridwise_gemm_pipeline_selector.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_pipeline_v1.hpp

Add a denorm test fix (#603)

2023-03-29 15:05:32 -05:00

gridwise_gemm_pipeline_v2.hpp

GEMM pipeline v2 (#317)

2022-07-08 15:55:14 -05:00

gridwise_gemm_reduce_xdl_cshuffle_v1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp

Example contraction splitk (#430)

2022-10-11 17:54:34 -05:00

gridwise_gemm_waveletmodel.hpp

Wavelet (inter-wave consumer-producer) GEMM (#310)

2023-01-18 12:00:02 -06:00

gridwise_gemm_wmma.hpp

Fix a typo (#676)

2023-04-15 21:57:34 -05:00

gridwise_gemm_xdl_cshuffle_v1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_xdl_waveletmodel_cshuffle.hpp

Wavelet (inter-wave consumer-producer) GEMM (#310)

2023-01-18 12:00:02 -06:00

gridwise_gemm_xdlops_bwd_weight.hpp

Add more macros to turn on/off denorm fix (#678)

2023-04-15 21:56:07 -05:00

gridwise_gemm_xdlops_skip_b_lds_v1.hpp

Fused GEMM+GEMM (#351)

2022-08-13 09:18:58 -05:00

gridwise_gemm_xdlops_v2r3.hpp

Add more macros to turn on/off denorm fix (#678)

2023-04-15 21:56:07 -05:00

gridwise_gemm_xdlops_v2r4.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

gridwise_gemm_xdlops_v2r4r2.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

gridwise_gemm_xdlops_v3r1.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_xdlops_v3r2.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_gemm_xdlops_v3r3.hpp

Add pipeline v1/v2 selector, add more instances (#381)

2022-11-02 16:50:48 -06:00

gridwise_normalization_naive_variance.hpp

Improve normalization (#580)

2023-02-15 11:59:35 -06:00

gridwise_normalization_selector.hpp

Improve normalization (#580)

2023-02-15 11:59:35 -06:00

gridwise_normalization_welford_variance.hpp

Improve normalization (#580)

2023-02-15 11:59:35 -06:00

gridwise_permute.hpp

Add 'Permute' device op & example (#408)

2022-09-19 21:30:25 -05:00

gridwise_set_buffer_value.hpp

Batchnorm-forward implemented using welford method to calculate variance (#403)

2022-10-27 18:52:54 -06:00

gridwise_set_multiple_buffer_value.hpp

Batchnorm-forward and Batchnorm-infer Implemented using generic kernels (#320)

2022-08-15 10:11:02 -05:00

gridwise_softmax.hpp

fix standalone softmax race condition around blockwise reduction (#323)

2022-07-14 22:52:45 -05:00

gridwise_sparse_embeddings_forward_layernorm.hpp

remove unused variable (#564)

2023-01-31 10:34:35 +08:00