ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-04-19 22:39:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
440358c16851de74575798c539feca1b0be0799f
composable_kernel/include/ck/tensor_operation/gpu/block
History
Enrico Degregori 440358c168 Wave Tile Transfer supporting global load with transpose (#3027) 
* Initial implementation:

 - add new thread group transfer supporting transpose instruction
 - refactor AB transfer to switch between thread and wave tiles methods

* Add some comments and remove explicit wave and lane calculations

* Remove compiler option for performance

* fp16 example: use tuned instance

* Missing cleanup

* Integrate wave transfer in existing gemm and batched gemm instances

* Add fast instances

* extend implementation for 8 bit datatypes

packed types not supported

* Address review comments

* Optimize pipeline v1 and re-introduce compiler option

* Disable wave tile approach for b scale gemm

* Fix for clang20

* Avoid code duplication of amd_global_load_transpose_to_vgpr function
2025-10-16 11:33:56 -07:00
..
blockwise_gemm_dl_v2r3.hpp
Implement DPP8 based GEMM for Navi21 (#826)
2023-08-14 15:46:27 -05:00
blockwise_gemm_dlops_v2r2.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
blockwise_gemm_dlops_v3.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
blockwise_gemm_dpp.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_mx_pipeline_xdlops_base.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_wmma_selector.hpp
Implement batched gemm gemm for RDNA (3 and 4) (#2612)
2025-09-04 14:10:24 -07:00
blockwise_gemm_pipeline_wmmaops_base.hpp
Implement batched gemm gemm for RDNA (3 and 4) (#2612)
2025-09-04 14:10:24 -07:00
blockwise_gemm_pipeline_wmmaops_v1.hpp
Wave Tile Transfer supporting global load with transpose (#3027)
2025-10-16 11:33:56 -07:00
blockwise_gemm_pipeline_wmmaops_v3.hpp
Implement batched gemm gemm for RDNA (3 and 4) (#2612)
2025-09-04 14:10:24 -07:00
blockwise_gemm_pipeline_wmmaops.hpp
DeviceGemm_Wmma_CShuffleV3 with BlockGemmPipelineVersion::v3 (#2096)
2025-04-28 10:14:21 +05:00
blockwise_gemm_pipeline_xdlops_ab_scale_selector.hpp
[A8W8 GEMM] Optimized weight-preshuffled implementation & add quantization datatype for CK TILE rms_norm (#1862)
2025-02-20 14:00:27 -08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_dequant_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_dequant_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_gufusion_dequant_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_gufusion_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_gufusion_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_mx_moe_gufusion_v3.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_mx_moe_selector.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_mx_moe_v3.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_selector.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_v2.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_preshuffle_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_b_scale_selector.hpp
[A8W8 GEMM] Optimized weight-preshuffled implementation & add quantization datatype for CK TILE rms_norm (#1862)
2025-02-20 14:00:27 -08:00
blockwise_gemm_pipeline_xdlops_base.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_blockscale_b_preshuffle_selector.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_blockscale_b_preshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_blockscale_b_preshuffle_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_moe_blockscale_b_preshuffle_gufusion_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_moe_blockscale_b_preshuffle_gufusion_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_moe_blockscale_b_preshuffle_selector.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_moe_blockscale_b_preshuffle_v1.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_moe_blockscale_b_preshuffle_v3.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_pipeline_xdlops_mx_bpreshuffle_selector.hpp
Optimized GEMMs for MX FP4/8 (#2294)
2025-06-05 13:54:15 -06:00
blockwise_gemm_pipeline_xdlops_mx_moe_gufusion_v3.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_nbs_gufusion_v3.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_nbs_selector.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_nbs_v1.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_nbs_v3.hpp
Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)
2025-06-12 09:25:59 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_selector.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_mx_moe_v3.hpp
[CK] Mxfp4 moe blockscale buf2lds version support (#2455)
2025-07-06 15:42:00 +08:00
blockwise_gemm_pipeline_xdlops_mx_selector.hpp
Optimized GEMMs for MX FP4/8 (#2294)
2025-06-05 13:54:15 -06:00
blockwise_gemm_pipeline_xdlops_selector.hpp
[A8W8 GEMM] Optimized weight-preshuffled implementation & add quantization datatype for CK TILE rms_norm (#1862)
2025-02-20 14:00:27 -08:00
blockwise_gemm_pipeline_xdlops_v1_ab_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v1_b_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v1_mx.hpp
Optimized GEMMs for MX FP4/8 (#2294)
2025-06-05 13:54:15 -06:00
blockwise_gemm_pipeline_xdlops_v1.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v2_ab_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v2_b_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v2.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v3_ab_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v3_b_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v3_mx_bpreshuffle.hpp
[CK] Fixed MPerBlock=32 build issue for MXFP4 GEMM decode (#2512)
2025-07-18 14:35:54 +08:00
blockwise_gemm_pipeline_xdlops_v3_mx.hpp
Optimized GEMMs for MX FP4/8 (#2294)
2025-06-05 13:54:15 -06:00
blockwise_gemm_pipeline_xdlops_v3.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v4_b_scale.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v4.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops_v5.hpp
Conv:TF32: add more instances - 1 (#2867)
2025-09-25 09:27:18 +08:00
blockwise_gemm_pipeline_xdlops.hpp
Extend XDL kernel to Support RDNA3/4 - Part 3 (#2723)
2025-09-09 11:22:36 +08:00
blockwise_gemm_smfmac_xdlops.hpp
Extend XDL kernel to Support RDNA3/4 - Part 3 (#2723)
2025-09-09 11:22:36 +08:00
blockwise_gemm_wmma.hpp
fix compilation errors for gfx12 with clang20 (#1606)
2024-10-28 19:02:48 -07:00
blockwise_gemm_xdlops_skip_b_lds.hpp
Extend XDL kernel to Support RDNA3/4 - Part 2 (#2722)
2025-09-04 08:33:40 +08:00
blockwise_gemm_xdlops.hpp
TF32 POC in Conv3d on MI30x platform #2763 (second attempt) (#2852)
2025-09-17 14:50:15 -07:00
blockwise_softmax.hpp
[HotFix] add config and version files to pass on build info (#856)
2023-08-23 11:36:17 -07:00
blockwise_tensor_slice_transfer_v5r1.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
blockwise_welford.hpp
Batchnorm splitk single kernel (#771)
2023-07-06 10:58:55 -05:00
reduction_functions_blockwise.hpp
Batchnorm splitk single kernel (#771)
2023-07-06 10:58:55 -05:00
thread_group_tensor_slice_transfer_direct_load.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
thread_group_tensor_slice_transfer_gather_direct_load.hpp
upgrade from clang-format-12 to clang-format-18 (#2568)
2025-07-28 11:34:07 -07:00
thread_group_tensor_slice_transfer_global.hpp
Wave Tile Transfer supporting global load with transpose (#3027)
2025-10-16 11:33:56 -07:00
thread_group_tensor_slice_transfer_v4r1_dequant.hpp
Navi3 rel (#1176)
2024-03-08 17:11:51 -08:00
thread_group_tensor_slice_transfer_v4r1_gather.hpp
Moe gemm activation (#2026)
2025-04-23 10:35:34 +08:00
thread_group_tensor_slice_transfer_v4r1.hpp
[A8W8 GEMM] Optimized weight-preshuffled implementation & add quantization datatype for CK TILE rms_norm (#1862)
2025-02-20 14:00:27 -08:00
thread_group_tensor_slice_transfer_v4r2.hpp
Add elementwise with dynamic vector dim (#1198)
2024-03-22 10:40:43 +01:00
thread_group_tensor_slice_transfer_v6r1.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
thread_group_tensor_slice_transfer_v6r1r2.hpp
initial stream-k implementation with example (#699)
2023-07-26 14:18:15 -05:00
thread_group_tensor_slice_transfer_v6r2.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
thread_group_tensor_slice_transfer_v6r3.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
thread_group_tensor_slice_transfer_v7.hpp
update copyright headers (#726)
2023-05-31 18:46:57 -05:00
thread_group_tensor_slice_transfer_v7r2.hpp
Codegen hipRTC compilation (#1579)
2025-01-31 09:48:39 -08:00
thread_group_tensor_slice_transfer_v7r3_scatter.hpp
Moe gemm activation (#2026)
2025-04-23 10:35:34 +08:00
thread_group_tensor_slice_transfer_v7r3.hpp
add f8 gemm multiD with both row/col wise scale (#1300)
2024-05-28 12:04:22 -05:00