ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-04-19 22:39:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
e2d28a92af81139b9743e454570c1ada3146f87c
composable_kernel/include/ck/tensor_operation/gpu/grid
History
linqunAMD e2d28a92af Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722) 
Update Blockwise and Gridwise files to support both wave32 & wave64.

1. Calculate WaveSize from template parameter, instead of hard code it to 64, some "64" is also replace with WaveSize
2. Move BN0Shuffled and BK0Shuffled to device side. we can't get correct mfma inst info in host side.
3. Update b_thread_offset_n and b_thread_offset_k in gridwise_gemm_xdl_cshuffle_v3_b_scale.hpp for gfx11. in gfx11, input data is duplicated for each 16 threads, it is different with all of others.
4. Modify a1_threadwise_copy in gridwise_batched_*gemm*gemm for gfx11.  for gfx11, we need duplicate input and swizzle A if transposeC isn't enabled.
2025-09-04 08:33:40 +08:00
..
batchnorm_multiblock
Do not use warpSize as compile time constant as it is removed (#2320)
2025-06-17 11:54:30 -07:00
gemm_layernorm
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
normalization
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
block_to_ctile_map.hpp
Grouped conv bwd weight with grouped gemm (#2304)
2025-06-12 10:15:07 +02:00
gridwise_2d_multiple_reduction_multiblock.hpp
Fix for beta!=0 in reduce (#1440)
2024-08-06 09:10:39 -07:00
gridwise_2d_multiple_reduction_threadwise.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_2d_reduction_multiblock.hpp
Fix for beta!=0 in reduce (#1440)
2024-08-06 09:10:39 -07:00
gridwise_2d_reduction_threadwise_multi_d.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_2d_reduction_threadwise.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_batched_gemm_softmax_gemm_wmma_cshuffle.hpp
Merging the gfx12 code into public repo. (#1362)
2024-06-27 00:33:34 -07:00
gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_batchnorm_backward_blockwise_welford.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_batchnorm_forward_blockwise_welford.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_elementwise_1d_scale.hpp
Refactor elementwise kernels (#1222)
2024-04-19 13:31:17 +02:00
gridwise_elementwise_2d.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_elementwise_layernorm_welford_variance.hpp
enable compilation of INSTANCES_ONLY for Windows (#1082)
2023-12-20 14:34:53 -08:00
gridwise_fpAintB_gemm_wmma.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_dl_multiple_d.hpp
Add Grouped Conv Fwd Large Tensor kernel (#1432)
2024-08-06 10:06:10 +02:00
gridwise_gemm_dl_v1r3.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_dpp.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_multiple_abd_xdl_cshuffle.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_multiple_d_wmma_cshuffle.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_multiple_d_xdl_cshuffle_lds_direct_load.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_multiple_d_xdl_cshuffle.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_multiple_d_xdl_splitk_cshuffle.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_pipeline_selector.hpp
Rebase the PR #1520 to ROCm repo. (#1574)
2025-02-20 18:58:14 -08:00
gridwise_gemm_pipeline_v1.hpp
Navi3 rel (#1176)
2024-03-08 17:11:51 -08:00
gridwise_gemm_pipeline_v2.hpp
Fixed Gemmv2r3 kpad (#938)
2023-09-26 18:40:00 -05:00
gridwise_gemm_pipeline_v3.hpp
initial stream-k implementation with example (#699)
2023-07-26 14:18:15 -05:00
gridwise_gemm_pipeline_v4_direct_load.hpp
Add support for double buffering in direct load GEMM kernel (#1052)
2023-12-03 23:08:47 +01:00
gridwise_gemm_reduce_xdl_cshuffle_v1.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_split_k_multiple_d_xdl_cshuffle_v2.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_waveletmodel.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_gemm_wmma_cshuffle_v3_b_scale.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_wmma_cshuffle_v3_common.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_wmma_cshuffle_v3.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_gemm_wmma.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdl_cshuffle_conv_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_streamk_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v2.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdl_cshuffle_v3_b_preshuffle.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_b_scale.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_multi_abd.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_ab_scale.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_b_preshuffle.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_multi_d_blockscale_b_preshuffle.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_multi_d.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_mx_bpreshuffle.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3_mx.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_cshuffle_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdl_waveletmodel_cshuffle.hpp
Use new mfma instructions for FP8 on gfx950 (#2202)
2025-05-19 17:29:51 -07:00
gridwise_gemm_xdlops_bwd_weight.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_skip_b_lds_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_gemm_xdlops_splitk_lds_direct_load.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_streamk.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v2r3.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v2r4.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v2r4r2.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v3r1.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v3r2.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_gemm_xdlops_v3r3.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00
gridwise_moe_gemm_blockscale.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_moe_gemm.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_moe_mx_gemm_bns.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_moe_mx_gemm_bpreshuffle.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_moe_mx_gemm.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
gridwise_permute.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
gridwise_put_element_1d.hpp
Average pool backward deviceOP and example (#797)
2023-08-10 12:04:35 +08:00
gridwise_set_buffer_value.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_set_multiple_buffer_value.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_softmax.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
gridwise_sparse_embeddings_forward_layernorm_builtins.hpp
Replace buffer load/store intrinsics with builtins (#1876)
2025-03-05 14:33:28 -08:00
gridwise_sparse_embeddings_forward_layernorm.hpp
[HotFix] add config and version files to pass on build info (#856)
2023-08-23 11:36:17 -07:00
gridwise_tensor_rearrange.hpp
Remove !defined(__HIP_DEVICE_COMPILE__) in CK kernel (#2564)
2025-07-28 13:01:07 -07:00