Enrico Degregori 4ebc48a3cd WMMA gemm_add_relu_add_layernorm (#2989) ...

* Summary:

 - Refactor epilogue (with CShuffle) to support fused operations:
    - EpilogueCShuffleBase holds common parts
    - EpilogueCShuffle: runs CShuffle and write out
    - EpilogueWelfordCShuffle: holds Welford specific arguments, runs CShuffle, write out, Welford first part and Welford write out

 - Extend thread transfer v7r3:
    - Support for intermediate data type different from src and dst type
    - New functionality to write to dst buffer and keep data (to be able to use them for additional operations)

* Adress review comments

2025-10-31 11:19:26 -07:00

..

data_type_enum.hpp

Pool2d max/avg kernel in the BWD version (#1494)

2024-09-12 11:47:52 +02:00

profile_avg_pool2d_bwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_avg_pool3d_bwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batched_gemm_add_relu_gemm_add_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_batched_gemm_b_scale_impl.hpp

implement device batched gemm b scale for wmma (#2825)

2025-10-16 11:00:42 -07:00

profile_batched_gemm_bias_softmax_gemm_permute_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batched_gemm_gemm_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_batched_gemm_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batched_gemm_reduce_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_batched_gemm_softmax_gemm_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batched_gemm_softmax_gemm_permute_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batchnorm_backward_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batchnorm_forward_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_batchnorm_infer_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_contraction_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_contraction_utils.hpp

Extend support for contraction 6D (#1207)

2024-04-09 23:46:21 +02:00

profile_conv_bwd_data_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_conv_fwd_bias_relu_add_impl.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

profile_conv_fwd_bias_relu_impl.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

profile_conv_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_conv_tensor_rearrange_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_elementwise_layernorm_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_gemm_ab_scale_impl.hpp

Add KBatch support for gemm_ab_scale (#2740)

2025-10-09 08:33:16 +02:00

profile_gemm_add_add_fastgelu_impl.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

profile_gemm_add_fastgelu_impl.hpp

Wmma support for multiple Ds based GEMMs (#2613)

2025-09-05 16:31:08 +02:00

profile_gemm_add_impl.hpp

Add support for mixed-precision f16bf16_int8 gemm (#1127)

2024-02-07 15:54:13 +01:00

profile_gemm_add_multiply_impl.hpp

Wmma support for multiple Ds based GEMMs (#2613)

2025-09-05 16:31:08 +02:00

profile_gemm_add_relu_add_layernorm_impl.hpp

WMMA gemm_add_relu_add_layernorm (#2989)

2025-10-31 11:19:26 -07:00

profile_gemm_add_relu_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_add_silu_impl.hpp

Add support for mixed-precision f16bf16_int8 gemm (#1127)

2024-02-07 15:54:13 +01:00

profile_gemm_b_scale_impl.hpp

implement device batched gemm b scale for wmma (#2825)

2025-10-16 11:00:42 -07:00

profile_gemm_bias_add_reduce_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_bilinear_impl.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

profile_gemm_blockscale_wp_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_fastgelu_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_multi_abd_impl.hpp

Fix multi-abd tests bug (#3099)

2025-10-27 08:09:02 -07:00

profile_gemm_multiply_add_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_multiply_multiply_impl.hpp

[A8W8 GEMM] Optimized weight-preshuffled implementation & add quantization datatype for CK TILE rms_norm (#1862)

2025-02-20 14:00:27 -08:00

profile_gemm_multiply_multiply_wp_impl.hpp

[Block Scale GEMM] Optimized block scale gemm (#1950)

2025-03-11 10:11:21 -07:00

profile_gemm_mx_impl.hpp

MX GEMM - Add FP6 GEMM Test (#2488)

2025-07-11 15:32:12 -06:00

profile_gemm_quantization_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_reduce_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_gemm_splitk_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_gemm_streamk_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_universal_batched_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_universal_impl.hpp

implement device batched gemm b scale for wmma (#2825)

2025-10-16 11:00:42 -07:00

profile_gemm_universal_preshuffle_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_universal_reduce_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_gemm_universal_streamk_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_grouped_conv_bwd_data_impl.hpp

re-enable clang-format by default (#3030)

2025-10-15 07:43:11 -07:00

profile_grouped_conv_bwd_weight_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_grouped_conv_fwd_bias_bnorm_clamp_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_grouped_conv_fwd_bias_clamp_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_grouped_conv_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_grouped_conv_fwd_outelementop_impl.hpp

upgrade from clang-format-12 to clang-format-18 (#2568)

2025-07-28 11:34:07 -07:00

profile_grouped_gemm_fastgelu_impl.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

profile_grouped_gemm_fixed_nk_impl.hpp

Split env.hpp header from the ck.hpp header. (#2049)

2025-04-03 15:30:21 -07:00

profile_grouped_gemm_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_grouped_gemm_multiply_tile_loop_impl.hpp

Split env.hpp header from the ck.hpp header. (#2049)

2025-04-03 15:30:21 -07:00

profile_grouped_gemm_tile_loop_impl.hpp

Split env.hpp header from the ck.hpp header. (#2049)

2025-04-03 15:30:21 -07:00

profile_groupnorm_bwd_data_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_groupnorm_bwd_gamma_beta_impl.hpp

layernorm & groupnorm bwd gamma beta (#1133)

2024-01-25 19:53:15 +08:00

profile_groupnorm_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_layernorm_bwd_data_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_layernorm_bwd_gamma_beta_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_layernorm_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_max_pool2d_bwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_max_pool3d_bwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_permute_scale_impl.hpp

Verify HostTensorDescriptor when it is created (#2829)

2025-09-25 18:22:13 -07:00

profile_pool2d_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_pool3d_fwd_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_reduce_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_softmax_impl.hpp

[CK] Add command option instance_index and param_mask to run partial ck test (#2889)

2025-09-30 08:24:40 -07:00

profile_transpose_impl.hpp

Refactor elementwise kernels (#1222)

2024-04-19 13:31:17 +02:00