Zoltán Lakatos 301eb5cf08 Implement device grouped gemm fixed nk multi abd for rdna4 (#3619) ...

* device struct implementation

* added xdl grouped multi abd fixed nk testing

* wmma implementation fixed

* avoid unnecessary device mem allocation and code cleanups

* cleanup instances definitions

* wmma examples added

* code cleanups

* fix clang format

* typo and compilation fixes related to reference gemm

* fix compilation error due to std::remove_cvref_t

* added missing hip_check_error includes

* correction to example instances

* review commentes addressed

* removed split-k from testing

* code formatting

---------

Co-authored-by: Zoltán Lakatos <zoltan.lakatos@streamhpc.com>
Co-authored-by: illsilin_amdeng <Illia.Silin@amd.com>

2026-02-02 13:58:11 -08:00

..

avg_pool2d_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

avg_pool3d_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_add_relu_gemm_add

Implement batched gemm add relu gemm add for rdna4 (#3391)

2026-01-20 13:06:59 -08:00

batched_gemm_b_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_bias_permute

Implement batched gemm bias permute for RDNA4 (#3534)

2026-01-17 08:30:27 +01:00

batched_gemm_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_multi_d

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_reduce

WMMA support for batched_gemm_reduce (#3332)

2026-01-20 10:50:46 +01:00

batched_gemm_softmax_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_softmax_gemm_permute

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batchnorm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

column_to_image

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

contraction_bilinear

Add support to fp16 + compute fp16 and bf16 + compute bf16 contractions (#3598)

2026-01-20 09:39:57 -08:00

contraction_scale

Add support to fp16 + compute fp16 and bf16 + compute bf16 contractions (#3598)

2026-01-20 09:39:57 -08:00

conv1d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd_bias_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd_bias_relu_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv3d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

elementwise

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

elementwise_normalization

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_ab_scale

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gemm_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_add_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_multiply

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_relu_add_layernorm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_silu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_b_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_bias_add_reduce

Wmma support for gemm_bias_add_reduce (#3316)

2026-01-07 10:27:16 -08:00

gemm_bilinear

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_blockscale_wp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gemm_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multi_abd

Multi AB support for wave transfer (#3578)

2026-01-29 10:29:40 -08:00

gemm_multiply_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multiply_multiply

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multiply_multiply_wp

Wmma support for gemm_multiply_multiply_wp (#3278)

2025-12-03 07:38:23 -08:00

gemm_mx

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_splitk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_streamk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal

Padding support for wave transfer (#3537)

2026-01-26 12:57:09 -08:00

gemm_universal_batched

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal_preshuffle

Implement device_gemm_universal_preshuffle_instance for RDNA4 (#3429)

2026-01-15 07:19:31 -08:00

gemm_universal_reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal_streamk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv1d_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_conv1d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv2d_bwd_data

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv2d_bwd_weight

Grouped Conv Bwd Weight Direct Load (#3648)

2026-01-28 15:31:54 -06:00

grouped_conv2d_fwd

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-14 11:02:19 +01:00

grouped_conv2d_fwd_bias_bnorm_clamp

WMMA grouped conv fwd large tensor bias bnorm clamp (#3595)

2026-01-23 12:20:00 +01:00

grouped_conv2d_fwd_bias_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv2d_fwd_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv2d_fwd_dynamic_op

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_bwd_data

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_data_bilinear

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_data_scale

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_conv3d_bwd_weight_bilinear

Temporarily disable kernel instances that won't build on gfx1101 on Windows (#3499)

2025-12-31 13:12:45 -07:00

grouped_conv3d_bwd_weight_scale

Replace grouped conv bwd wei wmmaV3 bilin/scale bf16f32bf16 support with bf16bf16bf16 (#3470)

2025-12-29 12:58:29 +01:00

grouped_conv3d_fwd

Added large tensor support for grouped conv fwd wmma (#3437)

2025-12-18 21:55:50 -07:00

grouped_conv3d_fwd_bias_bnorm_clamp

WMMA grouped conv fwd large tensor bias bnorm clamp (#3595)

2026-01-23 12:20:00 +01:00

grouped_conv3d_fwd_bias_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv3d_fwd_bilinear

Adding support for scale and bilinear ops for WMMA grouped conv fwd (#3450)

2025-12-19 15:15:02 +01:00

grouped_conv3d_fwd_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv3d_fwd_convinvscale

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_fwd_convscale

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_fwd_convscale_add

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_fwd_convscale_relu

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_fwd_dynamic_op

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_conv3d_fwd_scale

Adding support for scale and bilinear ops for WMMA grouped conv fwd (#3450)

2025-12-19 15:15:02 +01:00

grouped_conv3d_fwd_scaleadd_ab

Post-merge cleanup for WMMA grouped conv fwd (#3468)

2025-12-22 15:57:45 +01:00

grouped_conv3d_fwd_scaleadd_scaleadd_relu

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00

grouped_convnd_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_gemm

Implement grouped gemm tile loop for RDNA4 (#3304)

2026-01-13 07:14:23 +01:00

grouped_gemm_bias

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_gemm_fastgelu

Implement grouped gemm fastgelu for RDNA4 (#3303)

2026-01-07 10:20:44 -08:00

grouped_gemm_fixed_nk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_gemm_fixed_nk_multi_abd

Implement device grouped gemm fixed nk multi abd for rdna4 (#3619)

2026-02-02 13:58:11 -08:00

grouped_gemm_tile_loop

Implement grouped gemm tile loop for RDNA4 (#3304)

2026-01-13 07:14:23 +01:00

image_to_column

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

max_pool_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

mha

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_bwd_gamma_beta

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

permute_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

pool2d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

pool3d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

quantization

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

softmax

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

transpose

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

CMakeLists.txt

Adding remaining conv, dynamic_op, and scaleadd_scaleadd_relu flavors for grouped conv fwd (#3529)

2026-01-30 17:02:14 +01:00