carlushuang e7dca79d27 initial stream-k implementation with example (#699) ...

* initial stream-k implementation with example

* fix unexpected change in err

* improve a little bit performance by reorganize pipeline.

* improve perf a little bit by swizzle block idx

* add profiler

* update example

* fix spelling

* shrink karg for streamk

* support dynamic buffer using memory coherence glc_slc bit from template

* control memory coherence while construct dynamic buffer

* update reduction for streamk(not ready yet)

* Add template parameter to make_dynamic_buffer to support amd_buffer coherence setting

* fix build issue

* fix several bug

* now result is correct, everything works (but has scratch)

* remove scratch by manually reset coordinate

* update device code

* fix a bug in final reduce

* fix something in example

* update async memset

* fix enum as camel case

* modify coherence enum name

* clean code and use atomic streamk by default

* remove unused var

* throw exception if have empty pointer

* fix format

* fix CI warning

* fix type in init

* modify CI error

* filter out on gfx10+

* restore changed example code

---------

Co-authored-by: Qianfeng Zhang <Qianfeng.Zhang@amd.com>

2023-07-26 14:18:15 -05:00

..

batchnorm_multiblock

Batchnorm splitk single kernel (#771)

2023-07-06 10:58:55 -05:00

gemm_layernorm

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

normalization

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

block_to_ctile_map.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

gridwise_2d_multiple_reduction_multiblock.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_2d_multiple_reduction_threadwise.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_2d_reduction_multiblock.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_2d_reduction_threadwise.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_batched_gemm_gemm_xdl_cshuffle_v1.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_batched_gemm_multiple_d_gemm_multiple_d_xdl_cshuffle_v1.hpp

Add bias scalar vectorload = 1 for gemm bias gemm (#791)

2023-07-24 20:08:15 -05:00

gridwise_batched_gemm_multiple_d_softmax_gemm_xdl_cshuffle_v1.hpp

Fix flash attn mask bug (#733)

2023-06-12 08:35:31 -05:00

gridwise_batched_gemm_softmax_gemm_xdl_cshuffle_v1.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_batchnorm_backward_blockwise_welford.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_batchnorm_forward_blockwise_welford.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_elementwise_1d.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_elementwise_2d.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_elementwise_layernorm_welford_variance.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_bias_add_reduce_xdl_cshuffle_v1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_dl_multiple_d.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_dl_v1r3.hpp

Add Grouped Conv Backward Weight on Navi21 for ResNet50. (#505)

2023-02-22 11:59:53 -06:00

gridwise_gemm_multiple_d_multiple_r_xdl_cshuffle.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_multiple_d_wmma_cshuffle.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_multiple_d_xdl_cshuffle.hpp

Optimize bf16 conversion (#664)

2023-05-04 10:25:47 -05:00

gridwise_gemm_pipeline_selector.hpp

Split GEMM instance library & enable pipeline v2 optimization (#783)

2023-07-06 10:59:35 -05:00

gridwise_gemm_pipeline_v1.hpp

Add a denorm test fix (#603)

2023-03-29 15:05:32 -05:00

gridwise_gemm_pipeline_v2.hpp

Split GEMM instance library & enable pipeline v2 optimization (#783)

2023-07-06 10:59:35 -05:00

gridwise_gemm_pipeline_v3.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

gridwise_gemm_reduce_xdl_cshuffle_v1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_split_k_multiple_d_xdl_cshuffle.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_waveletmodel.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_wmma.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_xdl_cshuffle_v1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdl_layernorm_cshuffle_v1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdl_waveletmodel_cshuffle.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_gemm_xdlops_bwd_weight.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_skip_b_lds_v1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_streamk.hpp

initial stream-k implementation with example (#699)

2023-07-26 14:18:15 -05:00

gridwise_gemm_xdlops_v2r3.hpp

Split GEMM instance library & enable pipeline v2 optimization (#783)

2023-07-06 10:59:35 -05:00

gridwise_gemm_xdlops_v2r4.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_v2r4r2.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_v3r1.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_v3r2.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_gemm_xdlops_v3r3.hpp

Enable gfx941 and gfx942 architectures. (#752)

2023-06-15 08:20:59 -07:00

gridwise_permute.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_put_element_1d.hpp

Maxpool bwd (#750)

2023-06-19 09:44:22 -05:00

gridwise_set_buffer_value.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_set_multiple_buffer_value.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_softmax.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00

gridwise_sparse_embeddings_forward_layernorm.hpp

update copyright headers (#726)

2023-05-31 18:46:57 -05:00