Po Yen Chen 7fbc9d6c97 [CK_TILE] FMHA FAv3 scheduling fine-tuning for performance (#2833) ...

* Re-mapping thread block indices for causal=True kernels

* Use more intuitive remap_opt value

* Fallback to origin remapping if seqlen_q >= 64K

* Use GenericAttentionMask to reduce mask computation

* Avoid unnecessary boundary check for IsMasking=false case

* Fix wrong kernel entry specifier

* Add s_nop to prevent delay wave0-3

* Refine scheduling

* Remove unnecessary sched_group_barrier()

* Move sched_group_barrier() call to scheduler

* Replace inline asm s_setprio with intrinsics

* Rephrase comments

* Expend some o_acc rescaling insts to avoid SIMD idle

* Fix block idx special mapping logic

* Tune block index mapping for causal=False cases

* Tune block index mapping for causal=True cases

* Fix wrong vmcnt()

* Remove parameter name

* Use boolean option for turn on/off causal mask

* Update benchmark_fwd_v3.sh option usages

* Add option if compiler support it

2025-09-16 11:32:38 +08:00

..

block_fmha_batch_prefill_pipeline_qr_ks_vs_async_default_policy.hpp

[CK_TILE] Add logits soft-capping & customization support to the FMHA forward kernel/pipelines (#2163)

2025-05-13 12:19:25 +08:00

block_fmha_batch_prefill_pipeline_qr_ks_vs_async.hpp

Revert "Eliminate warning caused by failed to meet occupancy requirement (#2389)" (#2514)

2025-07-17 10:09:01 +08:00

block_fmha_bwd_convert_dq.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_bwd_dot_do_o.hpp

[CK_TILE] FA bwd kernels optimization (#1397)

2024-08-16 13:40:10 -07:00

block_fmha_bwd_dq_dk_dv_pipeline_kr_ktr_vr_iglp.hpp

[CK_TILE] FMHA avoid unnecessary vmcnt0 (#2715)

2025-08-25 20:55:12 +08:00

block_fmha_bwd_dq_dk_dv_pipeline_kr_ktr_vr.hpp

[CK_TILE] FMHA avoid unnecessary vmcnt0 (#2715)

2025-08-25 20:55:12 +08:00

block_fmha_bwd_dq_dk_dv_pipeline_selector.hpp

[CK_TILE] FMHA BWD Decode Pipeline (#2643)

2025-08-12 17:02:52 +08:00

block_fmha_bwd_dq_dk_dv_pipeline_trload_kr_ktr_vr.hpp

Reapply "[CK_TILE] FMHA BWD Enable Tile 16x192 (#2741)" (#2757) (#2761)

2025-09-08 09:21:14 -07:00

block_fmha_bwd_dq_dk_dv_pipeline_trload_qr_qtr_dor.hpp

Reapply "[CK_TILE] FMHA BWD Enable Tile 16x192 (#2741)" (#2757) (#2761)

2025-09-08 09:21:14 -07:00

block_fmha_bwd_pipeline_default_policy.hpp

[CK_TILE] CK_TILE GEMM WMMA Support for GFX11/GFX12 (#2466)

2025-08-15 16:22:27 -07:00

block_fmha_bwd_pipeline_problem.hpp

[CK_TILE] FMHA BWD Optimization For GFX950 (#2628)

2025-08-12 11:11:55 +08:00

block_fmha_bwd_pipeline_trload_default_policy.hpp

Reapply "[CK_TILE] FMHA BWD Enable Tile 16x192 (#2741)" (#2757) (#2761)

2025-09-08 09:21:14 -07:00

block_fmha_fwd_appendkv_pipeline_default_policy.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00

block_fmha_fwd_appendkv_pipeline.hpp

[CK_TILE] Add PagedAttention kernels (#1387)

2024-08-28 20:50:43 +08:00

block_fmha_fwd_pagedkv_pipeline_qr_ks_vs_default_policy.hpp

ck tile pagedkv prefill (#2405)

2025-07-07 16:16:54 +08:00

block_fmha_fwd_pagedkv_pipeline_qr_ks_vs.hpp

Add separate mask checking for scope [aligned_physical_seqlen_k_start, physical_seqlen_k_end) (#2487)

2025-07-11 18:14:47 +08:00

block_fmha_fwd_splitkv_combine_pipeline_default_policy.hpp

[CK_TILE] Add fmha fwd N-Warp S-Shuffle pipeline (fmha fwd splitkv pipeline variant) (#1705)

2024-12-20 14:41:01 +08:00

block_fmha_fwd_splitkv_combine_pipeline.hpp

roll back splitkv

2025-01-08 10:09:54 -08:00

block_fmha_fwd_splitkv_pipeline_nwarp_sshuffle_qr_ks_vs_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_fwd_splitkv_pipeline_nwarp_sshuffle_qr_ks_vs.hpp

[CK_TILE] Add logits soft-capping & customization support to the FMHA forward kernel/pipelines (#2163)

2025-05-13 12:19:25 +08:00

block_fmha_fwd_splitkv_pipeline_qr_ks_vs_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_fwd_splitkv_pipeline_qr_ks_vs.hpp

ck tile pagedkv prefill (#2405)

2025-07-07 16:16:54 +08:00

block_fmha_fwd_v3_pipeline_default_policy.hpp

[CK_TILE] Add FAv3 fwd pipeline (#2731)

2025-09-01 09:16:45 +08:00

block_fmha_fwd_v3_pipeline.hpp

[CK_TILE] FMHA FAv3 scheduling fine-tuning for performance (#2833)

2025-09-16 11:32:38 +08:00

block_fmha_pipeline_enum.hpp

Re-enable optimization for gfx950 fmha fwd (#2671)

2025-08-13 14:57:43 +08:00

block_fmha_pipeline_problem.hpp

[CK_TILE] Add FAv3 fwd pipeline (#2731)

2025-09-01 09:16:45 +08:00

block_fmha_pipeline_qr_ks_vs_async_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_pipeline_qr_ks_vs_async_trload_policy.hpp

[CK_TILE] CK_TILE GEMM WMMA Support for GFX11/GFX12 (#2466)

2025-08-15 16:22:27 -07:00

block_fmha_pipeline_qr_ks_vs_async_trload.hpp

[CK_TILE] FMHA avoid unnecessary vmcnt0 (#2715)

2025-08-25 20:55:12 +08:00

block_fmha_pipeline_qr_ks_vs_async.hpp

Fix fmha fwd precision issue on MI3XX series (#2285)

2025-06-10 15:03:23 +08:00

block_fmha_pipeline_qr_ks_vs_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_pipeline_qr_ks_vs_fp8.hpp

[CK_TILE] Add fmha fwd headdim96 support (#1608)

2024-10-30 14:03:16 +08:00

block_fmha_pipeline_qr_ks_vs_whole_k_prefetch_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_pipeline_qr_ks_vs_whole_k_prefetch.hpp

Update for xformers (#2372)

2025-06-22 00:28:30 -07:00

block_fmha_pipeline_qr_ks_vs.hpp

use iglp to improve dim256 fmha fwd in qr_ks_vs pipeline (#2711)

2025-08-28 11:39:39 +08:00

block_fmha_pipeline_qs_ks_vs_default_policy.hpp

Ck tile/complete k prefetch (#1941)

2025-03-07 14:19:51 +08:00

block_fmha_pipeline_qs_ks_vs.hpp

[CK_TILE] Add logits soft-capping & customization support to the FMHA forward kernel/pipelines (#2163)

2025-05-13 12:19:25 +08:00

block_fmha_pipeline_qx_ks_vs_custom_policy.hpp

[CK_TILE][FMHA] Enable dwordx4 loading in async_load_tile_raw() (#2549)

2025-08-22 10:13:47 +08:00

tile_fmha_shape.hpp

[CK_TILE] FMHA BWD Decode Pipeline (#2643)

2025-08-12 17:02:52 +08:00

tile_fmha_traits.hpp

[CK_TILE] Add FAv3 fwd pipeline (#2731)

2025-09-01 09:16:45 +08:00