Qianfeng 14c3cfb1c6 [CK_TILE] Improve headdim96 performance for fmha-bwd (#1573) ...

* Add kQKHeaddimForGemmN and kVHeaddimForGemmN in order to support headdim 96

* Remove the using of MakeKRegBlockDescriptor and MakeVRegBlockDescriptor

* Fix in bwd_piple_default_policy

* Remove kQKHeaddim and rename kQKHeaddimForGemmN to kQKHeaddim in the bwd kernel and pipelines

* Replace kVHeaddimForGemmN by kVHeaddim and kDoDvHeaddim

* Update to hd96 tile settings

* Add smoke test scripts for fmha-bwd hd96

* Revert "Add smoke test scripts for fmha-bwd hd96"

This reverts commit 7ca7e1a93d.

* Remove hd96 tile settings in fmha_bwd codegen to save compiling

* Fix lost code line in bwd_pipeline_default_policy

* Merge kDoDvHeaddim/kPadHeadDimDoDv to kVHeaddim/kPadHeadDimV and remove TileFmhaBwdTraits

* Rename KRegSliceBlockDescriptor/VRegSliceBlockDescriptor to KRegBlockDescriptor/VRegBlockDescriptor

* tiny adjustments

---------

Co-authored-by: Po Yen Chen <PoYen.Chen@amd.com>
Co-authored-by: danyao12 <Dan.Yao@amd.com>

2024-10-16 18:14:32 +08:00

..

block_fmha_bwd_convert_dq.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_bwd_dot_do_o.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_bwd_dq_dk_dv_pipeline_kr_ktr_vr_iglp.hpp

[CK_TILE] Improve headdim96 performance for fmha-bwd (#1573)

2024-10-16 18:14:32 +08:00

block_fmha_bwd_dq_dk_dv_pipeline_kr_ktr_vr.hpp

[CK_TILE] Improve headdim96 performance for fmha-bwd (#1573)

2024-10-16 18:14:32 +08:00

block_fmha_bwd_pipeline_default_policy.hpp

[CK_TILE] Improve headdim96 performance for fmha-bwd (#1573)

2024-10-16 18:14:32 +08:00

block_fmha_bwd_pipeline_enum.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_bwd_pipeline_problem.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_fwd_appendkv_pipeline_default_policy.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00

block_fmha_fwd_appendkv_pipeline.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00

block_fmha_fwd_splitkv_combine_pipeline_default_policy.hpp

[CK_TILE] Simplify the codes in splitkv_combine pipeline (#1549)

2024-10-08 10:44:34 +08:00

block_fmha_fwd_splitkv_combine_pipeline.hpp

[CK_TILE] Simplify the codes in splitkv_combine pipeline (#1549)

2024-10-08 10:44:34 +08:00

block_fmha_fwd_splitkv_pipeline_qr_ks_vs_default_policy.hpp

[CK_TILE] fmha forward split-kv + combine kernels (#1338)

2024-06-26 17:41:15 +08:00

block_fmha_fwd_splitkv_pipeline_qr_ks_vs.hpp

[CK_TILE] Change output accum tensor layout of fmha fwd split-kv & combine kernels (#1527)

2024-10-01 22:13:52 +08:00

block_fmha_pipeline_enum.hpp

[CK_TILE] support alibi (#1269)

2024-05-07 22:32:54 +08:00

block_fmha_pipeline_problem.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00

block_fmha_pipeline_qr_ks_vs_async_default_policy.hpp

introducing ck_tile! (#1216)

2024-04-15 19:27:12 -05:00

block_fmha_pipeline_qr_ks_vs_async.hpp

Early return if seqlen_k=0 on group mode (#1524)

2024-09-22 20:05:58 +08:00

block_fmha_pipeline_qr_ks_vs_default_policy.hpp

introducing ck_tile! (#1216)

2024-04-15 19:27:12 -05:00

block_fmha_pipeline_qr_ks_vs_fp8.hpp

CK Tile FA Training kernels (#1286)

2024-06-04 13:12:45 -05:00

block_fmha_pipeline_qr_ks_vs.hpp

Early return if seqlen_k=0 on group mode (#1524)

2024-09-22 20:05:58 +08:00

block_fmha_pipeline_qs_ks_vs_default_policy.hpp

introducing ck_tile! (#1216)

2024-04-15 19:27:12 -05:00

block_fmha_pipeline_qs_ks_vs.hpp

CK Tile FA Training kernels (#1286)

2024-06-04 13:12:45 -05:00

block_fmha_pipeline_qx_ks_vs_custom_policy.hpp

decouple the calling from gemm_pipeline (#1571)

2024-10-14 13:59:26 +08:00

tile_fmha_shape.hpp

CK Tile FA Training kernels (#1286)

2024-06-04 13:12:45 -05:00

tile_fmha_traits.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00