* uncomment all the headdim, use optdim to control
* change default back to -1
* uncomment splitkv instance
* Fix typo in receipt 4 for appendkv
* support optdim for bwd, splitkv and appendkv
* Fix 192 key error
---------
Co-authored-by: Max Podkorytov <4273004+tenpercent@users.noreply.github.com>
Co-authored-by: Andy Lugo <Andy.LugoReyes@amd.com>
* Wrap tile size mapping as class method
* Warp pipeline generating as class method
* Add constraint as kernel dispatching criteria
* Support mutltiple tile size for a (hdim, hdim_v) combination
* Use smaller tile size if CU utilization is low
* Use integar as the key of the tile size map
* Fix type error
* Simply override parent class method return value
* Add attribute to eliminate warnging
* Allow using environment variables to turn on/off custom factory
* Unify param naming style
* Add missing HIP runtime include directive
* Fix os.environ.get() usage
* add prefetching physical block id for pagedkv
* start add pagedkv prefill
* rename pipeline
* add kernel for pagedkv
* add an init version pagedkv prefill
* fix redefine issue
* add struct BlockFmhaFwdPagedKVPipelineProblem and fmha_fwd_pagedkv_args
* generate dispatch code
* add body generating code
* comipling pass
* remove dropout from pagedkv
* set lse to false in generating code
* start changing qr kernel to pagedkv
* init version of kernerl with pagedkv
* change names of file that are generated
* chang host validation for pagedkv prefill
* using iglp to change blockgemm
* add kernel files to op head file
* show parameters
* rewrite print parameter fun
* add fwd
* remove default parameter of GridSize
* format
* fix nhead issue and add seqlen_k_ptr to batch mode
* format code
* remove no-longer used code
* format
* fix some comments
---------
Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Po Yen Chen <PoYen.Chen@amd.com>
* Add constraint on traits/tile/pipeline
* Use kM0=128 if max_seqlen_q == 8192
* Re-format codegen script
* Remove redundant attr name postix
* Fix import error: default field in dataclass
* Use kK0=64 & kK1=64 to hide latency
* Use CU utilization to decide tile size
* return value with macro and revert the return value
* [CK-TILE] no-macro launch api solution (#1992)
* no-macro solution
* address -Wcomma
---------
Co-authored-by: Max Podkorytov <4273004+tenpercent@users.noreply.github.com>
* Update license year
* Add initial code to override decode problem
* Fix splitkv traits/args overriding error
* Reshape and transpose lse for decode
* Remove debug code
* Prettify example code
* Use better function name
* Add kMergeNumHeadGroupsSeqLenQ flag
Kernel user can use this switch to turn on/off optimization for
some problem sizes
* Add missing flag declarations
* Default turn off kMergeNumHeadGroupsSeqLenQ in codegen
* Group similar statements together
* Remove assumption of seqlen_q=1
* Remove kMergeNumHeadGroupsSeqLenQ from splitkv combine kernel
* Support kMergeNumHeadGroupsSeqLenQ=true in fmha splitkv kernel
* Run kMergeNumHeadGroupsSeqLenQ=true kernels when need
* Fix group mode block skip logics
* Undo changes of normal fwd kernel
* Update in GridSize() and using GridSize() for splitkv kernel (#1799)
---------
Co-authored-by: Qianfeng <qianfeng.zhang@amd.com>
* Remove using tile partitioner for fmha_fwd_kernel
* Remove using tile partitioner for fmha_fwd_splitkv and splitkv-combine kernels
* Remove using tile partitioner for fmha_fwd_appendkv kernel
* Unify the format of GetTileIndex
* Add check for zero values
* Add static assertions
* Remove invalid option '-e' in smoke_test.sh
* Use correct path of smoke_test.sh
* Avoid zero-sized shared memory array
* Add warning comment
* Replace expr by integer_divide_ceil() call
* Use more readable constant names
* Write down assumption as static assertion
* Add more diagnostic error messages
* Fix wrong BlockWarps when using default pipeline policy
* Add more static assertions for A LDS desc
* Allow using vector size < 8 for data type fp16/bf16
* Align vector size between DRAM dist & LDS desc
* Remove no-longer used func decl
* Fix wrong displayed piepline name
* Undo policy template changes for tile_example_gemm_basic
* Add missing space and make error message stands out
* Unify print precision
* Add missing include directive <iomanip>
* Replace constant 64 by get_warp_size() call
* Replace constant 128 by named variable: BankLength
* Add kAMBlock/kBNBlock attributes
* Allow usig different A/B warp dist for multiple blocks
* Add helper function to get warp dist encodings
* Add 4x64x4 fp16 warp gemm attribute impl
* Complete the A/B warp dist encoding logic
* Fix wrong thread mapping for C matrix
* Use smaller vector size for small tile
* Add static assert to block unsupported warp gemm impl
* Extract common code out as helper method
* Add 4x64x16 fp16 warp gemm type alias
* Add comment to warning developers
* Undo WarpGemmAtrributeMfma<> changes
* Use more clear static assertion error message
* Add trivial wrapper to get warp dstr encodings
* Only transpose warp gemm result if it's square
* Fix compilation error
* Support multi-block warp gemm (on N direction)
* Remove duplicated code
* Fix output encoding of warp gemm
* Fix wrong shape of WarpGemmAtrributeMfmaIterateK<>
* Remove unused code
* Fix wrong shape of WarpGemmAttributeMfmaImplF16F16F32M4N64K4
* Add type config for bf16_t
* Add 4x64x16 bf16 warp gemm
* Update WarpGemmAtrributeMfmaIterateKAndTransposedCDistribution
* Add 64x4x4 fp16/bf16 warp gemm impl
* Add 64x4x16 fp16/bf16 warp gemm
* Add static assertion for better error diagnostic
* Get Q dram dstr directly form block gemm
* Add missing header: fused_moe.hpp
* Allow specifying different warp-gemm for gemm0 & gemm1
* Store P matrix into LDS before gemm1
* Fix inconsistant kernel name
* Remove constraint on gemm0 & gemm1 block warps
* Remove unsupported vector size from checking list
* Allow using 4x64x16 warp gemm for gemm0
* Finish policy customization
* Finish pipeline modification
F#
* Use block warps in codegen
* Fix wrong rank of m_lds_window origin
* Use better distributed tensor
* Make P-store earlier
* Remove duplicated experssions
* Remove unnecessary tile window
* Create new files for new splitkv pipeline
* Separate old/new pipeline codegen logic
* Sync changes form develop
* Undo gemm kernel/pipeline changes
* Undo gemm example changes
* Remove blank lines
* Fix typo
* Use new warp gemm interface
* Fix link error
* Fix wrong pipeline tag
* Fix more link error
* Avoid unnecessary padding
* Always use vector load for K
* Padding on fastest dimension when necessary
* Force padding Q on hdim_q
* Set high dimension padding flag to false
* Re-format headers
* Use warps=<1, 4, 1> for both gemm0 & gemm1
* Fix complilation errors
* Remove m/l shuffle logics
* Ignore duplicate data when write lse_acc
* Use gemm0 block warps as lds tile width
* Remove hard-coded numbers
* Fix wrong distribution width
* Remove unnecessary code
* Add s_barrier before writing to LDS
* Store Q into LDS before gemm0
* Fix wrong Q tile size
* Use simple Q lds descriptor for debuging
* Use more realistic Q lds descriptor
* Add comment & use better variable name
* Make Q lds space not overlapped with others
* Remove unnecessary block_tile_reduce_sync() call
* Move Q load statements
* Move block_sync_lds() right before use
* Re-order instructions
* Remove necessary lambda expression
* Use 8 threads on kMaxSplits direction while doing reduction
* Tiny correction for using 8 threads on kMaxSplits direction for combine kernel
* Padding num_split direction of o_acc tile window to 4x
* Update splitkv combine pipeline design
* Add kN1 back to splitkv combine pipeline problem
* Fix compilation errors
* Add missing template parameter
* Fix wrong splitkv combine kernel name
* Fix wrong origin
* Fix wrong LDS descriptor shape
* Fix sync & reduction logics
* Remove unnecessary static assertions
* Extract tile size computation logics
* Make sure we can reuse padding flags in combine kernels
* Rename variables
* Use OaccDataType in BlockFmhaSplitKVCombinePipelineTileSizes<>
* Remove unnecessary static assertion
* Fix function name typo
* Add constraint on kN1 template parameter
* Hide K tile loading latency in earlier iteration
* Fix wrong splitkv kernel name
* Use s_shuffling to replace p_shuffling which removes the needs of cross-warp reduction
* Rename pipeline
* Fix wrong pipeline name attribute
* Add GetAlignmentQ() for NWarpSShuffle pipeline
* Separate Q tile into dram tile & register tile concepts
* Remove non-squre warp gemm transpose c type alias
* Fallback tile size changes for fmha fwd splitkv
* Remove redundant change
* Refine naming for the S tile
* Use better naming of the S tile dstr (read from lds)
* Share Q lds with K lds
* Tiny change
* Fix with using static_for for passing CI checking
---------
Co-authored-by: Qianfeng Zhang <Qianfeng.Zhang@amd.com>
* Change in fwd-splitkv kernel to support num_splits=1 case
* Update in codegen fwd-splitkv to make num_splits > 1 cases pass
* Specify instance traits in dispatch
* Fix link error for fp8 kernels
---------
Co-authored-by: Po Yen Chen <PoYen.Chen@amd.com>
* Generate group mode paged-attn kernel
* Enable paged-kvcache + group mode support
* Add missing header: fused_moe.hpp
* Add comment to explain kernel arg usage
* Make error message more clear
* Add comment for confusing data member names
* Add more comment for confusing variable names
* Fix typo in option description
* Add ceil_to_qualified_tile_length()
* Rename kK0BlockLength to kQKHeaddim
* Add kSubQKHeaddim concept to support headdim96
* Fix in math.hpp to avoid using __half interfaces
* Add LdsBufferSequence instance for headdim96
* Update in fmha_fwd/fmha_fwd_splitkv codegen to support hd96 testing
* Disable hd96 instance generation in codegen fmha_fwd and fmha_fwd_splitkv to save compiling time
* Reformat one file
* Fix text alignment in fmha_fwd_splitkv.py
---------
Co-authored-by: Po Yen Chen <PoYen.Chen@amd.com>
* Use pre-defined constants for readability
* Use vector write for o_acc tensor
* Remove no-longer used policy method
* Deprecate no-longer used policy/pipeline
* Specify gemm0/gemm1 block warps separately in codegen
* Fix wrong ps_idx creation logic
* Add single-warp block gemm
* Supoprt single-warp gemm0
* Make MakeCBlockTile() as static method
* Use MakeCBlockTile() to get underlying tile distribution
* Use kNumGemm1Warps to compute # threads for gemm1
* Put normal case in the if clause
* Refine fmha splitkv block mapping
* Refine & fix the lse_acc/o_acc layout
* Fix wrong LDS size for K tile
* Use kK0=64 for hdim=128,256 fmha splitkv kernels
* Use kK1=64 for hdim=32,64,128 fmha splitkv kernels
* Undo kK0/kK1 changes
* Use more reasonable GetAlignmentV() computation
* Using store_tile() in fmha splitkv kernel epilogue
