Adam Osewski 8bb2bb4a05 Grouped Gemm + SplitK + simplified Kernel Args (#669) ...

* simplify karg in device/grid split-k op

* fix mk_kn_mn instances

* add more instances

* B2C with 3D grid for KSplit

* Remove unused code.

* Use default B2C (3D grid) in grid gemm v2r4r2.

* Device gemm splitk use B2C map.

* Device GroupedGemmXdlSplitKCShuffle

* Example for GroupedGemm Xdl SplitK

* Introduce Device GroupedGemmSplitK

* Fix updating kbatch size.

* Add instance mk-nk-mn

* Enable set kbatch in profiler.

* Add GGemmSplitK mk-kn-mn instances

* Add more instances & split into multiple files.

* minor fix

* tuning

* clean

* disabled failed instances

* use pipe v2

* Ignore arg on not supported arch.

* fix warning

---------

Co-authored-by: carlushuang <carlus.huang@amd.com>
Co-authored-by: Adam Osewski <aosewski@amd.com>
Co-authored-by: zjing14 <zhangjing14@gmail.com>
Co-authored-by: Jing Zhang <jizhan@amd.com>
Co-authored-by: root <root@ctr-ubbsmc15.amd.com>

2023-04-24 15:43:36 -05:00

..

impl

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

convolution_backward_data_specialization.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

convolution_backward_weight_specialization.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

convolution_forward_specialization.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_base.hpp

Remove including of cmath (#551)

2023-01-11 19:52:47 -06:00

device_batched_contraction_multiple_d.hpp

Add batched/grouped_gemm contraction deviceOps (#349)

2022-08-10 12:20:29 -05:00

device_batched_gemm_e_permute.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_batched_gemm_gemm.hpp

Attention with output permutation (#370)

2022-08-23 14:52:56 -05:00

device_batched_gemm_multi_d.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_batched_gemm_multiple_d_gemm_multiple_d.hpp

batched_gemm + multiple_d + gemm + multiple_d (#394)

2022-09-14 17:54:18 -05:00

device_batched_gemm_softmax_gemm_permute.hpp

Add GemmAddSoftmaxGemm support for MSFT ORT (instances and client API) (#576)

2023-02-08 14:34:45 -06:00

device_batched_gemm_softmax_gemm.hpp

Input/output permutation for fused attention (#460)

2022-10-27 14:58:20 -06:00

device_batched_gemm.hpp

Gemm+Bilinear (#316)

2022-07-02 09:15:38 -05:00

device_batchnorm_backward.hpp

BatchNorm backward instance/external API/profiler/tests (#519)

2022-11-30 13:32:20 -06:00

device_batchnorm_forward.hpp

BatchNorm forward instance/external api/profiler/tests/client example (#511)

2022-11-24 18:02:27 -06:00

device_batchnorm_infer.hpp

BatchNorm forward instance/external api/profiler/tests/client example (#511)

2022-11-24 18:02:27 -06:00

device_cgemm.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

device_contraction_multiple_d.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_conv_bwd_data.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_conv_fwd_bias_activation_add.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

device_conv_fwd_bias_activation.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

device_conv_fwd.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_elementwise_normalization.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_elementwise.hpp

Batchnorm inference instances, external API, client examples and gtests (#531)

2023-01-25 17:09:04 -06:00

device_gemm_bias_e_permute.hpp

Clean up conv example, Instances, profiler and test (#324)

2022-07-29 18:19:25 -05:00

device_gemm_multiple_d_layernorm.hpp

Gemm layernorm welford (#413)

2023-01-16 20:08:25 -06:00

device_gemm_multiple_d_multiple_r.hpp

Refactor the design of DeviceGemmMultipleDMultipleR_Xdl_CShuffle (#378)

2022-08-24 10:12:54 -05:00

device_gemm_multiple_d.hpp

Refactor the design of DeviceGemmMultipleDMultipleR_Xdl_CShuffle (#378)

2022-08-24 10:12:54 -05:00

device_gemm_reduce.hpp

Improve external interface for GEMM and GEMM+add+add+fastgelu (#311)

2022-06-30 22:11:00 -05:00

device_gemm_splitk.hpp

Improve external interface for GEMM and GEMM+add+add+fastgelu (#311)

2022-06-30 22:11:00 -05:00

device_gemm_xdl_waveletmodel_cshuffle.hpp

Wavelet (inter-wave consumer-producer) GEMM (#310)

2023-01-18 12:00:02 -06:00

device_gemm.hpp

Refactor the design of DeviceGemmMultipleDMultipleR_Xdl_CShuffle (#378)

2022-08-24 10:12:54 -05:00

device_grouped_contraction_multiple_d.hpp

Add batched/grouped_gemm contraction deviceOps (#349)

2022-08-10 12:20:29 -05:00

device_grouped_conv_bwd_data_multiple_d.hpp

Remove interface 'DeviceGroupedConvBwdData' (#500)

2022-11-09 18:32:17 -06:00

device_grouped_conv_bwd_weight.hpp

Add client example of grouped conv2d backward weight (data type: fp16) (#498)

2022-11-09 18:50:03 -06:00

device_grouped_conv_fwd_dl_multiple_d_nhwc_kyxc_nhwk.hpp

gemm/Conv xdlops + dlops quantization (#625)

2023-03-15 15:29:40 -05:00

device_grouped_conv_fwd_dl_nhwc_kyxc_nhwk.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_conv_fwd_multiple_d.hpp

Add a denorm test fix (#603)

2023-03-29 15:05:32 -05:00

device_grouped_conv_fwd.hpp

Add client example of grouped conv2d forward (data type: fp16) (#488)

2022-11-09 19:01:58 -06:00

device_grouped_gemm_softmax_gemm_permute_xdl_cshuffle.hpp

Update GetTypeString function to generate unique kernel IDs. (#638)

2023-03-15 10:44:42 -05:00

device_grouped_gemm_softmax_gemm_permute.hpp

Input/output permutation for fused attention (#460)

2022-10-27 14:58:20 -06:00

device_grouped_gemm_splitk.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

device_grouped_gemm.hpp

Grouped Gemm + SplitK + simplified Kernel Args (#669)

2023-04-24 15:43:36 -05:00

device_multiple_reduce.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_normalization.hpp

Improve normalization (#580)

2023-02-15 11:59:35 -06:00

device_permute.hpp

Remove including of cmath (#551)

2023-01-11 19:52:47 -06:00

device_pool2d_fwd.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

device_reduce.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_softmax.hpp

Use double for all scaling values and float-point constant values at the Device Op API (#557)

2023-01-18 12:02:50 -06:00

device_splitk_contraction_multiple_d_xdl_cshuffle.hpp

Example contraction splitk (#430)

2022-10-11 17:54:34 -05:00

device_splitk_contraction_multiple_d.hpp

Example contraction splitk (#430)

2022-10-11 17:54:34 -05:00

gemm_specialization.hpp

Implement padding and sanity checks for fused GEMM+GEMM (#376)

2022-08-23 10:01:02 -05:00

masking_specialization.hpp

Input/output permutation for fused attention (#460)

2022-10-27 14:58:20 -06:00

matrix_padder.hpp

batched_gemm + multiple_d + gemm + multiple_d (#394)

2022-09-14 17:54:18 -05:00

reduction_operator_mapping.hpp

add license in file (#303)

2022-06-24 23:32:43 -05:00

tensor_layout.hpp

Conv bwd data multiple d (#404)

2022-09-19 11:25:28 -05:00

tensor_specialization.hpp

Add batched/grouped_gemm contraction deviceOps (#349)

2022-08-10 12:20:29 -05:00

welford_helper.hpp

Batchnorm-forward implemented using welford method to calculate variance (#403)

2022-10-27 18:52:54 -06:00