Enrico Degregori 440358c168 Wave Tile Transfer supporting global load with transpose (#3027) ...

* Initial implementation:

 - add new thread group transfer supporting transpose instruction
 - refactor AB transfer to switch between thread and wave tiles methods

* Add some comments and remove explicit wave and lane calculations

* Remove compiler option for performance

* fp16 example: use tuned instance

* Missing cleanup

* Integrate wave transfer in existing gemm and batched gemm instances

* Add fast instances

* extend implementation for 8 bit datatypes

packed types not supported

* Address review comments

* Optimize pipeline v1 and re-introduce compiler option

* Disable wave tile approach for b scale gemm

* Fix for clang20

* Avoid code duplication of amd_global_load_transpose_to_vgpr function

2025-10-16 11:33:56 -07:00

..

amd_address_space.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

amd_buffer_addressing_builtins.hpp

Fix direct lds load for gfx950 and clang20 (#2346)

2025-06-15 15:22:34 -07:00

amd_buffer_addressing.hpp

Optimized GEMMs for MX FP4/8 (#2294)

2025-06-05 13:54:15 -06:00

amd_ck_fp8.hpp

Congma/ck tile/remove cpp 20 code (#2873)

2025-09-25 10:34:28 -07:00

amd_gemm_dpp.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

amd_inline_asm.hpp

Split up data_type header. (#1996)

2025-03-24 15:08:54 -07:00

amd_lds.hpp

Implement direct loads split-K GEMM kernel (#1137)

2024-02-07 01:08:34 +01:00

amd_smfmac.hpp

Added structural sparsity blockwise gemm (#1435)

2024-09-11 15:19:42 +02:00

amd_transpose_load.hpp

Wave Tile Transfer supporting global load with transpose (#3027)

2025-10-16 11:33:56 -07:00

amd_wave_read_first_lane.hpp

Rebase the PR #1520 to ROCm repo. (#1574)

2025-02-20 18:58:14 -08:00

amd_wmma.hpp

Fix a typo in intrin_wmma_bf16_16x16x16_bf16_w32 (#2727)

2025-09-03 08:07:09 +08:00

amd_xdlops.hpp

Conv:TF32: add more instances - 1 (#2867)

2025-09-25 09:27:18 +08:00

array_multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

array.hpp

Codegen hipRTC compilation (#1579)

2025-01-31 09:48:39 -08:00

blkgemmpipe_scheduler.hpp

Extend XDL kernel to Support RDNA3/4 - Part 1 (#2606)

2025-08-22 17:46:30 -04:00

c_style_pointer_cast.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

common_header.hpp

update the switch condition for buffer built-ins (#2602)

2025-08-01 14:30:07 -07:00

container_element_picker.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

container_helper.hpp

upgrade from clang-format-12 to clang-format-18 (#2568)

2025-07-28 11:34:07 -07:00

data_type.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

debug.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

dtype_fp64.hpp

Split up data_type header. (#1996)

2025-03-24 15:08:54 -07:00

dtype_vector.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

dynamic_buffer.hpp

Wave Tile Transfer supporting global load with transpose (#3027)

2025-10-16 11:33:56 -07:00

e8m0.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

enable_if.hpp

Rebase the PR #1520 to ROCm repo. (#1574)

2025-02-20 18:58:14 -08:00

env.hpp

Fix build errors on windows (#2456)

2025-07-16 07:58:23 -07:00

f8_utils.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

filter_tuple.hpp

Shard several of the most costly targets. (#2373)

2025-06-23 07:24:36 -07:00

flush_icache.hpp

Universal gemm flush cache (#1251)

2024-04-25 15:07:14 -05:00

functional2.hpp

Add MoE & FP8 Blockscale WP Kernels for GFX950 (#2297)

2025-06-12 09:25:59 +08:00

functional3.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional4.hpp

Codegen hipRTC compilation (#1579)

2025-01-31 09:48:39 -08:00

functional.hpp

Codegen hipRTC compilation (#1579)

2025-01-31 09:48:39 -08:00

generic_memory_space_atomic.hpp

Add declarations for atomic add for fp16 and unsigned short. (#2483)

2025-07-10 07:18:56 -07:00

get_id.hpp

Extend XDL kernel to Support RDNA3/4 - Part 5 (#2725)

2025-09-15 10:59:25 -07:00

get_shift.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

ignore.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

inner_product_dpp8.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

inner_product.hpp

Fix the DL kernel issues on Navi3x. (#998)

2023-10-19 09:34:39 -07:00

integral_constant.hpp

Optimized GEMMs for MX FP4/8 (#2294)

2025-06-05 13:54:15 -06:00

is_detected.hpp

upgrade from clang-format-12 to clang-format-18 (#2568)

2025-07-28 11:34:07 -07:00

is_known_at_compile_time.hpp

Add optimized copy to ck wrapper (#1126)

2024-01-19 11:29:00 +01:00

loop_scheduler.hpp

Rebase the PR #1520 to ROCm repo. (#1574)

2025-02-20 18:58:14 -08:00

magic_division.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

math_v2.hpp

Rebase the PR #1520 to ROCm repo. (#1574)

2025-02-20 18:58:14 -08:00

math.hpp

Extend available elementwise operations with conv examples (#995)

2023-10-19 17:23:19 +02:00

multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

mxf4_utils.hpp

MX GEMM - Expand MX MFMA Testing to BF8, FP6, and BF6 Data Types (#2199)

2025-05-19 16:52:51 -05:00

mxf6_utils.hpp

MX GEMM - Expand MX MFMA Testing to BF8, FP6, and BF6 Data Types (#2199)

2025-05-19 16:52:51 -05:00

mxf8_utils.hpp

Add accelerated stochastic rounding on gfx950 (#2355)

2025-06-23 09:31:46 -05:00

mxfp_utils.hpp

Fix return value bug that drops minus sign in some cases. (#2415)

2025-07-02 14:53:00 +08:00

number.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

numeric_limits.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

numeric_utils.hpp

Conv:TF32: add more instances - 2 (#2879)

2025-10-10 15:28:17 +08:00

random_gen.hpp

Upgrade to ROCm7.0.1 compiler. (#2909)

2025-09-24 10:00:53 -07:00

reduction_common.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

reduction_enums.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_functions_accumulate.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_operator.hpp

Remove unsupported (fp8) type from Add memory operation. (#1521)

2024-09-20 09:40:45 +02:00

scaled_type_convert.hpp

Update packed fp4 layout (#2523)

2025-07-21 16:58:59 -05:00

sequence_helper.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

sequence.hpp

upgrade from clang-format-12 to clang-format-18 (#2568)

2025-07-28 11:34:07 -07:00

span.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

static_buffer.hpp

Jing's contribution: prototype of mixed precision gemm FP16/BF16xint4 GEMM (#1762)

2025-01-02 11:48:06 +08:00

statically_indexed_array_multi_index.hpp

Codegen hipRTC compilation (#1579)

2025-01-31 09:48:39 -08:00

statically_indexed_array.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

synchronization.hpp

Wave Tile Transfer supporting global load with transpose (#3027)

2025-10-16 11:33:56 -07:00

thread_group.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

transpose_vectors.hpp

[GEMM] Gemm universal device operation (#1154)

2024-04-13 21:03:18 -05:00

tuple_helper.hpp

Moe gemm activation (#2026)

2025-04-23 10:35:34 +08:00

tuple.hpp

Codegen hipRTC compilation (#1579)

2025-01-31 09:48:39 -08:00

type_convert.hpp

[CK] Fix misc issues in CK examples (#2890)

2025-09-24 11:28:20 -07:00

type.hpp

Rebase the PR #1520 to ROCm repo. (#1574)

2025-02-20 18:58:14 -08:00

workgroup_barrier.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

workgroup_synchronization.hpp

Do not use warpSize as compile time constant as it is removed (#2320)

2025-06-17 11:54:30 -07:00