Haocong WANG 3049b5467c [GEMM] gemm_universal related optimization (#1453) ...

* replace buffer_atomic with global_atomic

* fixed global_atomic_add

* added bf16 atomic_add

* format

* clang-format-12

* clean

* clean

* add guards

* Update gtest.cmake

* enabled splitk_gemm_multi_d

* format

* add ckProfiler

* format

* fixed naming

* format

* clean

* clean

* add guards

* fix clang format

* format

* add kbatch printout

* clean

* Add rocm6.2 related gemm optimization

* Limit bf16 atomic usage

* remove redundant RCR gemm_universal instance

* Add RRR fp8 gemm universal instance

* Bug fix

* Add GPU_TARGET guard to FP8/BF8 target

* bug fix

* update cmake

* remove all fp8/bf8 example if arch not support

* Enable fp8 RRR support in ckProfiler

* limit greedy-reverse flag to gemm_universal in ckProfiler

---------

Co-authored-by: Jing Zhang <jizhan@fb.com>
Co-authored-by: Jing Zhang <jizhan@meta.com>
Co-authored-by: zjing14 <zhangjing14@gmail.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>

2024-08-14 10:42:30 +08:00

..

amd_address_space.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

amd_buffer_addressing.hpp

[GEMM] gemm_universal related optimization (#1453)

2024-08-14 10:42:30 +08:00

amd_gemm_dpp.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

amd_inline_asm.hpp

Navi3 rel (#1176)

2024-03-08 17:11:51 -08:00

amd_lds.hpp

Implement direct loads split-K GEMM kernel (#1137)

2024-02-07 01:08:34 +01:00

amd_smfmac.hpp

Fix issue with multiple targets and remove smfmac tests from unsupported test targets (#1372)

2024-07-03 23:34:38 -07:00

amd_wave_read_first_lane.hpp

Add read_first_lane function for int64 (#1347)

2024-06-18 15:05:30 -05:00

amd_wmma.hpp

Merging the gfx12 code into public repo. (#1362)

2024-06-27 00:33:34 -07:00

amd_xdlops.hpp

Code clean-up (#1285)

2024-05-10 09:41:39 -07:00

array_multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

array.hpp

CK Instance Gen (#1145)

2024-06-25 16:37:35 -05:00

blkgemmpipe_scheduler.hpp

[GEMM] Gemm universal device operation (#1154)

2024-04-13 21:03:18 -05:00

c_style_pointer_cast.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

common_header.hpp

FP8 enablement - add a pseudorandom number generator, add conversion methods (#708)

2023-06-19 11:20:35 -05:00

container_element_picker.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

container_helper.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

data_type.hpp

Merging the gfx12 code into public repo. (#1362)

2024-06-27 00:33:34 -07:00

debug.hpp

Grouped GEMM Multiple D tile loop. (#1247)

2024-04-25 15:12:53 -05:00

dynamic_buffer.hpp

[GEMM] gemm_universal related optimization (#1453)

2024-08-14 10:42:30 +08:00

enable_if.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

env.hpp

replace the ENV macro with CK_ENV (#1296)

2024-05-17 10:42:51 -07:00

f8_utils.hpp

Remove reinterpret_cast uses that result in undefined behaviour. (#1445)

2024-08-07 11:49:02 -07:00

flush_icache.hpp

Universal gemm flush cache (#1251)

2024-04-25 15:07:14 -05:00

functional2.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional3.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional4.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

functional.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

generic_memory_space_atomic.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

get_id.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

get_shift.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

ignore.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

inner_product_dpp8.hpp

Redesign the DPP8 GEMM kernel to use warp-wise component (#863)

2023-09-06 11:44:09 -05:00

inner_product.hpp

Fix the DL kernel issues on Navi3x. (#998)

2023-10-19 09:34:39 -07:00

integral_constant.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

is_detected.hpp

add vector_type support into thread_copy_v3r1 (#969)

2023-10-13 15:11:43 -05:00

is_known_at_compile_time.hpp

Add optimized copy to ck wrapper (#1126)

2024-01-19 11:29:00 +01:00

loop_scheduler.hpp

Grouped GEMM Multiple D tile loop. (#1247)

2024-04-25 15:12:53 -05:00

magic_division.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

math_v2.hpp

Replace the using of __expf by __ocml_exp_f32 to work-around the test_softmax_rank4 failure (#1394)

2024-07-17 09:15:05 -07:00

math.hpp

Extend available elementwise operations with conv examples (#995)

2023-10-19 17:23:19 +02:00

multi_index.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

number.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

random_gen.hpp

FP8 enablement - add a pseudorandom number generator, add conversion methods (#708)

2023-06-19 11:20:35 -05:00

reduction_common.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

reduction_enums.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_functions_accumulate.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

reduction_operator.hpp

Add support for half_t and bfloat to reduction operations (#1395)

2024-07-24 12:12:37 -05:00

sequence_helper.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

sequence.hpp

Grouped GEMM Multiple D tile loop. (#1247)

2024-04-25 15:12:53 -05:00

span.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

static_buffer.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

statically_indexed_array_multi_index.hpp

Extend available elementwise operations with conv examples (#995)

2023-10-19 17:23:19 +02:00

statically_indexed_array.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

synchronization.hpp

Merging the gfx12 code into public repo. (#1362)

2024-06-27 00:33:34 -07:00

thread_group.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

transpose_vectors.hpp

[GEMM] Gemm universal device operation (#1154)

2024-04-13 21:03:18 -05:00

tuple_helper.hpp

Add tensor partition and generic copy for ck wrapper (#1108)

2024-01-03 01:10:57 +01:00

tuple.hpp

Add multiple A/B support (#906)

2023-09-26 21:16:23 -05:00

type_convert.hpp

Add Grouped Conv Fwd Large Tensor kernel (#1432)

2024-08-06 10:06:10 +02:00

type.hpp

bf16A_Int8B with fastgelu/bias (#1264)

2024-04-26 07:26:30 -05:00

workgroup_barrier.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

workgroup_synchronization.hpp

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00