Enrico Degregori 2e49b6b2f7 Padding support for wave transfer (#3537) ...

* Add padding support with transpose

Also move check before writing storing is_src_valid during reading

* Add/modify instances to use wave transfer for gemm universal

Condition is changed so now the vectorsize of vmem reading and lds
writing must be equal to 8 in order to use the wave transfer

* Fix clang format

* Modify example

* Fix bwd data

* Add restriction for wave transfer with padding and transpose

Add test case which shows this limitation

* Fix validity checks 8 bit types

* Add validity check gemm_bias_add_reduce

* Add validity check grouped gemm tile loop

* Fix validity checks new flavours

* Minor fixes

* Fix clang format

2026-01-26 12:57:09 -08:00

..

impl

Padding support for wave transfer (#3537)

2026-01-26 12:57:09 -08:00

conv_tensor_rearrange_op.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

convolution_backward_data_specialization.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

convolution_backward_weight_specialization.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

convolution_forward_specialization.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_avgpool_bwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_base.hpp

Improve XDL to WMMA porting for grouped conv fwd (#3456)

2025-12-19 15:58:51 -07:00

device_batched_contraction_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_e_permute.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_gemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_multi_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_multiple_d_gemm_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_softmax_gemm_permute.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm_softmax_gemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batched_gemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batchnorm_backward.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batchnorm_forward.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_batchnorm_infer.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_cgemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_contraction_multiple_abd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_contraction_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_conv_bwd_data.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_conv_fwd_bias_activation_add.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_conv_fwd_bias_activation.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_conv_fwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_conv_tensor_rearrange.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_elementwise_normalization.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_elementwise_scale.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_elementwise.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_bias_e_permute.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_dequantB.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_multiple_abd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_multiple_d_ab_scale.hpp

Wmma support for gemm_ab_scale (#3314)

2025-12-11 09:06:20 +01:00

device_gemm_multiple_d_layernorm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_multiple_d_multiple_r.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_mx.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_reduce.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_splitk.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_streamk_v2.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_streamk.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm_v2.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_gemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_contraction_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_bwd_data_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_bwd_weight_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_bwd_weight.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_fwd_multiple_abd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_fwd_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_conv_fwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_fixed_nk.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_multi_abd_fixed_nk.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_multi_abd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_softmax_gemm_permute.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_splitk.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_grouped_gemm_tile_loop.hpp

Implement grouped gemm tile loop for RDNA4 (#3304)

2026-01-13 07:14:23 +01:00

device_grouped_gemm.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_max_pool_bwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_multiple_reduce.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_normalization_bwd_data.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_normalization_bwd_gamma_beta.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_normalization_fwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_permute.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_pool_fwd.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_put_element.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_reduce_multi_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_reduce.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_softmax.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

device_splitk_contraction_multiple_d.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

gemm_specialization.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

helper.hpp

chore(copyright): update copyright header for include directory (#3224)

2025-11-18 10:17:18 -08:00

masking_specialization.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

matrix_padder.hpp

Implement batched gemm add relu gemm add for rdna4 (#3391)

2026-01-20 13:06:59 -08:00

reduction_operator_mapping.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

tensor_layout.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

tensor_specialization.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00

welford_helper.hpp

chore(copyright): update copyright header for include directory (#3293)

2025-11-26 11:00:05 -07:00