Adam Osewski 8bb2bb4a05 Grouped Gemm + SplitK + simplified Kernel Args (#669) ...

* simplify karg in device/grid split-k op

* fix mk_kn_mn instances

* add more instances

* B2C with 3D grid for KSplit

* Remove unused code.

* Use default B2C (3D grid) in grid gemm v2r4r2.

* Device gemm splitk use B2C map.

* Device GroupedGemmXdlSplitKCShuffle

* Example for GroupedGemm Xdl SplitK

* Introduce Device GroupedGemmSplitK

* Fix updating kbatch size.

* Add instance mk-nk-mn

* Enable set kbatch in profiler.

* Add GGemmSplitK mk-kn-mn instances

* Add more instances & split into multiple files.

* minor fix

* tuning

* clean

* disabled failed instances

* use pipe v2

* Ignore arg on not supported arch.

* fix warning

---------

Co-authored-by: carlushuang <carlus.huang@amd.com>
Co-authored-by: Adam Osewski <aosewski@amd.com>
Co-authored-by: zjing14 <zhangjing14@gmail.com>
Co-authored-by: Jing Zhang <jizhan@amd.com>
Co-authored-by: root <root@ctr-ubbsmc15.amd.com>

2023-04-24 15:43:36 -05:00

..

device_batched_contraction_multiple_d_wmma_cshuffle.hpp

[gfx110x] support Navi3x architectures. (#628)

2023-03-09 07:56:40 -06:00

device_batched_contraction_multiple_d_xdl_cshuffle.hpp

Input/output permutation for fused attention (#460)

2022-10-27 14:58:20 -06:00

device_batched_gemm_e_permute_xdl.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_batched_gemm_gemm_xdl_cshuffle.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_batched_gemm_multi_d_xdl.hpp

fix GetTypeString

2022-11-29 14:18:10 +08:00

device_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle.hpp

Add layout check to IsSupportedArgument (#627)

2023-03-15 11:12:12 -05:00

device_batched_gemm_reduce_xdl_cshuffle.hpp

Add a flag to enable/disable debug output in many kernels. (#549)

2023-01-11 19:55:56 -06:00

device_batched_gemm_softmax_gemm_permute_xdl_cshuffle.hpp

fix a bug when building for gfx1030 target. (#591)

2023-02-16 13:54:08 -06:00

device_batched_gemm_softmax_gemm_xdl_cshuffle.hpp

Input/output permutation for fused attention (#460)

2022-10-27 14:58:20 -06:00

device_batched_gemm_xdl.hpp

Add a flag to enable/disable debug output in many kernels. (#549)

2023-01-11 19:55:56 -06:00

device_batchnorm_backward_impl.hpp

BatchNorm backward instance/external API/profiler/tests (#519)

2022-11-30 13:32:20 -06:00

device_batchnorm_forward_impl.hpp

BatchNorm forward instance/external api/profiler/tests/client example (#511)

2022-11-24 18:02:27 -06:00

device_cgemm_4gemm_xdl_cshuffle.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_contraction_multiple_d_xdl_cshuffle.hpp

disable tensor contraction f64 on MI100 (#602)

2023-02-23 16:59:37 -08:00

device_conv2d_backward_weight_xdl_c_shuffle_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv2d_bwd_data_xdl_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv2d_fwd_xdl_c_shuffle_bias_activation_add_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv2d_fwd_xdl_c_shuffle_bias_activation_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv2d_fwd_xdl_c_shuffle_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv2d_fwd_xdl_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_conv3d_fwd_naive_ndhwc_kzyxc_ndhwk.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_conv3d_fwd_xdl_ndhwc_kzyxc_ndhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_convnd_bwd_data_nwc_kxc_nwk_dl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_convnd_bwd_data_nwc_kxc_nwk_xdl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_elementwise_2d_impl.hpp

Batchnorm inference instances, external API, client examples and gtests (#531)

2023-01-25 17:09:04 -06:00

device_elementwise_impl.hpp

Batchnorm inference instances, external API, client examples and gtests (#531)

2023-01-25 17:09:04 -06:00

device_elementwise_normalization_impl.hpp

Add instance for elementwise normlization (#573)

2023-02-09 09:37:29 -08:00

device_gemm_bias_add_reduce_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_bias_e_permute_xdl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_dl.hpp

Add a flag to enable/disable debug output in many kernels. (#549)

2023-01-11 19:55:56 -06:00

device_gemm_multiple_d_dl.hpp

gemm/Conv xdlops + dlops quantization (#625)

2023-03-15 15:29:40 -05:00

device_gemm_multiple_d_layernorm_xdl_cshuffle.hpp

Gemm+layernorm instance, ckProfiler, client example (#568)

2023-02-09 15:02:55 -06:00

device_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_multiple_d_wmma_cshuffle.hpp

[Navi3x] Multiple issue fix (#612)

2023-03-10 17:04:28 -06:00

device_gemm_multiple_d_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_reduce_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_wmma.hpp

[Navi3x] Multiple issue fix (#612)

2023-03-10 17:04:28 -06:00

device_gemm_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_xdl_layernorm_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_gemm_xdl_skip_b_lds.hpp

Add a flag to enable/disable debug output in many kernels. (#549)

2023-01-11 19:55:56 -06:00

device_gemm_xdl_splitk_c_shuffle.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

device_gemm_xdl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_contraction_multiple_d_xdl_cshuffle.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_grouped_conv_bwd_data_multiple_d_xdl_cshuffle_v1.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_conv_bwd_weight_gnwc_gkxc_gnwk_dl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_conv_bwd_weight_gnwc_gkxc_gnwk_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_conv_fwd_multiple_d_multiple_r_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_conv_fwd_multiple_d_multiple_r.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_grouped_conv_fwd_multiple_d_wmma_cshuffle.hpp

Get rid of XDL parameters in WMMA kernel string. (#646)

2023-03-22 08:05:48 -07:00

device_grouped_conv_fwd_multiple_d_xdl_cshuffle.hpp

Add a denorm test fix (#603)

2023-03-29 15:05:32 -05:00

device_grouped_gemm_multiple_d_dl.hpp

Device Op GroupedGemmMultipleD + example fp16 (#633)

2023-03-15 11:22:59 -05:00

device_grouped_gemm_xdl_splitk_cshuffle.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

device_grouped_gemm_xdl.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_multiple_reduce_multiblock.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_multiple_reduce_threadwise.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_normalization_impl.hpp

Improve normalization (#580)

2023-02-15 11:59:35 -06:00

device_permute_impl.hpp

Add 'Permute' device op & example (#408)

2022-09-19 21:30:25 -05:00

device_pool2d_fwd_nhwc_nhwc.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_reduce_common.hpp

Refactor device op implementations into impl subdirectory. (#420)

2022-10-13 09:05:08 -05:00

device_reduce_multiblock.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_reduce_threadwise.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_softmax_impl.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_sparse_embeddings_forward_layernorm.hpp

add multi embeddings support (#542)

2023-01-18 11:32:12 -06:00