Adam Osewski 061ac0649c Polished Grouped GEMM APIs and new BF16 instances (#1600) ...

* Few small fixes.

* New GroupedGemm instances (BF16)

* Unify and refactor GroupedGEMM device API.

* Adapt changes to new API.

* Adapt grouped gemm profiler.

* Accept multiple kbatches for grouped gemm profiler.

- delete obsolete two stage as it is now covered by grouped gemm

* Update unit test for grouped gemm.

* Fix thresholds for BF16 and F8. Unblock tests.

* Fix few instances.

* Multiple small fixes.

* Adapt to new API, check dynamic casting.

* Uncomment few data types in grouped gemm profiler.

* Fix call to SetDeviceArgs.

* Fix profile grouped gemm multiply tile loop.

* Fix grouped gemm tile loop kernel args in client examples.

* Review comments.

2024-11-27 13:02:44 +01:00

..

amd_address_space.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

amd_buffer_addressing.hpp

[GEMM] gemm_universal related optimization (#1453)

2024-08-14 10:42:30 +08:00

amd_gemm_dpp.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

amd_inline_asm.hpp

Navi3 rel (#1176)

2024-03-08 17:11:51 -08:00

amd_lds.hpp

Implement direct loads split-K GEMM kernel (#1137)

2024-02-07 01:08:34 +01:00

amd_smfmac.hpp

Added structural sparsity blockwise gemm (#1435)

2024-09-11 15:19:42 +02:00

amd_wave_read_first_lane.hpp

Add read_first_lane function for int64 (#1347)

2024-06-18 15:05:30 -05:00

amd_wmma.hpp

Add bf16 and int8 wmma gemms for Navi3x and Navi4x. (#1671)

2024-11-18 14:07:04 -08:00

amd_xdlops.hpp

add int8 gemm multiply multiply a8w8 (#1591)

2024-10-26 16:39:34 +08:00

array_multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

array.hpp

CK Instance Gen (#1145)

2024-06-25 16:37:35 -05:00

blkgemmpipe_scheduler.hpp

[GEMM] Gemm universal device operation (#1154)

2024-04-13 21:03:18 -05:00

c_style_pointer_cast.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

common_header.hpp

FP8 enablement - add a pseudorandom number generator, add conversion methods (#708)

2023-06-19 11:20:35 -05:00

container_element_picker.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

container_helper.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

data_type.hpp

Generic threshold calculation (#1546)

2024-10-25 12:46:24 +02:00

debug.hpp

Grouped GEMM Multiple D tile loop. (#1247)

2024-04-25 15:12:53 -05:00

dynamic_buffer.hpp

[GEMM] gemm_universal related optimization (#1453)

2024-08-14 10:42:30 +08:00

enable_if.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

env.hpp

replace the ENV macro with CK_ENV (#1296)

2024-05-17 10:42:51 -07:00

f8_utils.hpp

Remove reinterpret_cast uses that result in undefined behaviour. (#1445)

2024-08-07 11:49:02 -07:00

flush_icache.hpp

Universal gemm flush cache (#1251)

2024-04-25 15:07:14 -05:00

functional2.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional3.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional4.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

generic_memory_space_atomic.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

get_id.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

get_shift.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

ignore.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

inner_product_dpp8.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

inner_product.hpp

Fix the DL kernel issues on Navi3x. (#998)

2023-10-19 09:34:39 -07:00

integral_constant.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

is_detected.hpp

add vector_type support into thread_copy_v3r1 (#969)

2023-10-13 15:11:43 -05:00

is_known_at_compile_time.hpp

Add optimized copy to ck wrapper (#1126)

2024-01-19 11:29:00 +01:00

loop_scheduler.hpp

Polished Grouped GEMM APIs and new BF16 instances (#1600)

2024-11-27 13:02:44 +01:00

magic_division.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

math_v2.hpp

Explicit cast values to half (#1593)

2024-10-22 11:17:32 -07:00

math.hpp

Extend available elementwise operations with conv examples (#995)

2023-10-19 17:23:19 +02:00

multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

number.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

random_gen.hpp

FP8 enablement - add a pseudorandom number generator, add conversion methods (#708)

2023-06-19 11:20:35 -05:00

reduction_common.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

reduction_enums.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_functions_accumulate.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_operator.hpp

Remove unsupported (fp8) type from Add memory operation. (#1521)

2024-09-20 09:40:45 +02:00

sequence_helper.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

sequence.hpp

Grouped GEMM Multiple D tile loop. (#1247)

2024-04-25 15:12:53 -05:00

span.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

static_buffer.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

statically_indexed_array_multi_index.hpp

Extend available elementwise operations with conv examples (#995)

2023-10-19 17:23:19 +02:00

statically_indexed_array.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

synchronization.hpp

Merging the gfx12 code into public repo. (#1362)

2024-06-27 00:33:34 -07:00

thread_group.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

transpose_vectors.hpp

[GEMM] Gemm universal device operation (#1154)

2024-04-13 21:03:18 -05:00

tuple_helper.hpp

Add tensor partition and generic copy for ck wrapper (#1108)

2024-01-03 01:10:57 +01:00

tuple.hpp

Add multiple A/B support (#906)

2023-09-26 21:16:23 -05:00

type_convert.hpp

Add Grouped Conv Fwd Large Tensor kernel (#1432)

2024-08-06 10:06:10 +02:00

type.hpp

bf16A_Int8B with fastgelu/bias (#1264)

2024-04-26 07:26:30 -05:00

workgroup_barrier.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

workgroup_synchronization.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00