Enrico Degregori 2e49b6b2f7 Padding support for wave transfer (#3537) ...

* Add padding support with transpose

Also move check before writing storing is_src_valid during reading

* Add/modify instances to use wave transfer for gemm universal

Condition is changed so now the vectorsize of vmem reading and lds
writing must be equal to 8 in order to use the wave transfer

* Fix clang format

* Modify example

* Fix bwd data

* Add restriction for wave transfer with padding and transpose

Add test case which shows this limitation

* Fix validity checks 8 bit types

* Add validity check gemm_bias_add_reduce

* Add validity check grouped gemm tile loop

* Fix validity checks new flavours

* Minor fixes

* Fix clang format

2026-01-26 12:57:09 -08:00

..

avg_pool2d_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

avg_pool3d_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_add_relu_gemm_add

Implement batched gemm add relu gemm add for rdna4 (#3391)

2026-01-20 13:06:59 -08:00

batched_gemm_b_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_bias_permute

Implement batched gemm bias permute for RDNA4 (#3534)

2026-01-17 08:30:27 +01:00

batched_gemm_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_multi_d

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_reduce

WMMA support for batched_gemm_reduce (#3332)

2026-01-20 10:50:46 +01:00

batched_gemm_softmax_gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batched_gemm_softmax_gemm_permute

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

batchnorm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

column_to_image

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

contraction_bilinear

Add support to fp16 + compute fp16 and bf16 + compute bf16 contractions (#3598)

2026-01-20 09:39:57 -08:00

contraction_scale

Add support to fp16 + compute fp16 and bf16 + compute bf16 contractions (#3598)

2026-01-20 09:39:57 -08:00

conv1d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd_bias_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv2d_fwd_bias_relu_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

conv3d_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

elementwise

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

elementwise_normalization

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_ab_scale

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gemm_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_add_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_multiply

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_relu_add_layernorm

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_add_silu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_b_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_bias_add_reduce

Wmma support for gemm_bias_add_reduce (#3316)

2026-01-07 10:27:16 -08:00

gemm_bilinear

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_blockscale_wp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

gemm_fastgelu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multi_abd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multiply_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multiply_multiply

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_multiply_multiply_wp

Wmma support for gemm_multiply_multiply_wp (#3278)

2025-12-03 07:38:23 -08:00

gemm_mx

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_splitk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_streamk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal

Padding support for wave transfer (#3537)

2026-01-26 12:57:09 -08:00

gemm_universal_batched

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal_preshuffle

Implement device_gemm_universal_preshuffle_instance for RDNA4 (#3429)

2026-01-15 07:19:31 -08:00

gemm_universal_reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

gemm_universal_streamk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv1d_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_conv1d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv2d_bwd_data

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv2d_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_conv2d_fwd

Add support for direct store in epilogue and padding support for wave transfer without transpose (#3465)

2026-01-14 11:02:19 +01:00

grouped_conv2d_fwd_bias_bnorm_clamp

WMMA grouped conv fwd large tensor bias bnorm clamp (#3595)

2026-01-23 12:20:00 +01:00

grouped_conv2d_fwd_bias_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv2d_fwd_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv2d_fwd_dynamic_op

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_bwd_data

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_data_bilinear

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_data_scale

Grouped convolution backward data WMMA v3 implementation (#3460)

2025-12-30 16:25:08 +01:00

grouped_conv3d_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_conv3d_bwd_weight_bilinear

Temporarily disable kernel instances that won't build on gfx1101 on Windows (#3499)

2025-12-31 13:12:45 -07:00

grouped_conv3d_bwd_weight_scale

Replace grouped conv bwd wei wmmaV3 bilin/scale bf16f32bf16 support with bf16bf16bf16 (#3470)

2025-12-29 12:58:29 +01:00

grouped_conv3d_fwd

Added large tensor support for grouped conv fwd wmma (#3437)

2025-12-18 21:55:50 -07:00

grouped_conv3d_fwd_bias_bnorm_clamp

WMMA grouped conv fwd large tensor bias bnorm clamp (#3595)

2026-01-23 12:20:00 +01:00

grouped_conv3d_fwd_bias_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv3d_fwd_bilinear

Adding support for scale and bilinear ops for WMMA grouped conv fwd (#3450)

2025-12-19 15:15:02 +01:00

grouped_conv3d_fwd_clamp

WMMA grouped conv fwd large tensor extra flavors (#3582)

2026-01-23 12:19:51 +01:00

grouped_conv3d_fwd_convinvscale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_fwd_convscale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_fwd_convscale_add

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_fwd_convscale_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_fwd_dynamic_op

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_conv3d_fwd_scale

Adding support for scale and bilinear ops for WMMA grouped conv fwd (#3450)

2025-12-19 15:15:02 +01:00

grouped_conv3d_fwd_scaleadd_ab

Post-merge cleanup for WMMA grouped conv fwd (#3468)

2025-12-22 15:57:45 +01:00

grouped_conv3d_fwd_scaleadd_scaleadd_relu

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_convnd_bwd_weight

Wmma support for grouped convolution bwd weight (#2947)

2025-12-17 15:58:58 -08:00

grouped_gemm

Implement grouped gemm tile loop for RDNA4 (#3304)

2026-01-13 07:14:23 +01:00

grouped_gemm_bias

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_gemm_fastgelu

Implement grouped gemm fastgelu for RDNA4 (#3303)

2026-01-07 10:20:44 -08:00

grouped_gemm_fixed_nk

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_gemm_fixed_nk_multi_abd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

grouped_gemm_tile_loop

Implement grouped gemm tile loop for RDNA4 (#3304)

2026-01-13 07:14:23 +01:00

image_to_column

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

max_pool_bwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

mha

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_bwd_data

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_bwd_gamma_beta

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

normalization_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

permute_scale

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

pool2d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

pool3d_fwd

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

quantization

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

reduce

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

softmax

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

transpose

chore(copyright) update library wide CMakeLists.txt copyright header template (#3313)

2025-11-28 13:49:54 -08:00

CMakeLists.txt

Enable offload-compress for Windows if avaliable (#3521)

2026-01-07 07:05:03 -08:00