249 Commits

Author	SHA1	Message	Date
Qianfeng Zhang	fcd41a6f39	Re-arrange the codes section for using sched_group_barrier	2025-07-21 08:22:18 +00:00
Qianfeng Zhang	906ab84d6c	Fix in using sched_group_barrier()	2025-07-21 08:21:35 +00:00
Qianfeng Zhang	acb6cd89d9	Move store_tile() caled before the current iteration	2025-07-21 08:20:52 +00:00
Qianfeng Zhang	fed1474e4f	Revert "Disable support of hdim64 amnd hdim256 for quick compiling and testing" ... This reverts commit ed062f93e2.	2025-07-17 09:00:41 +00:00
Qianfeng Zhang	ed062f93e2	Disable support of hdim64 amnd hdim256 for quick compiling and testing	2025-07-17 09:00:15 +00:00
Qianfeng Zhang	f0c8dca9bf	Fix bug in generate_instances.py and re-generate the instances	2025-07-17 08:37:09 +00:00
Qianfeng Zhang	0306a1e6ab	Re-org the kernel parameters in HstuAttentionFwdBatchModeBaseKargs and HstuAttentionFwdJaggModeBaseKargs	2025-07-17 04:51:16 +00:00
Qianfeng Zhang	fdd9c117d4	Remove num_target from HstuBlockMask class member since it overlaps the meaning of max_uih_len	2025-07-15 16:21:40 +00:00
Qianfeng Zhang	c01695597e	Fix the calculation of number of instructions used by sched_group_barrier	2025-07-15 09:47:35 +00:00
Qianfeng Zhang	0206b3442a	[Performance] use iglp compiler instruction to tune the codes around gemm0 for window_size > 0 situation	2025-07-14 16:02:42 +00:00
Qianfeng Zhang	9171b35751	Add including of block_dropout.hpp in the hstu kernel to avoid potential compiling failure	2025-07-11 00:15:53 +00:00
Qianfeng Zhang	6825618dca	Moving code-lines in hstu pipeline	2025-07-07 09:50:11 +00:00
Qianfeng Zhang	8d30e46ba5	Remove using i_loop and num_loops since seqlen_k_curr and seqlen_k_end is enough	2025-07-06 15:46:56 +00:00
Qianfeng Zhang	5451912526	Let causal == 0 cases to do IsFullTileInsideMask() checking before calling IsTokenPairInsideMask()	2025-06-26 10:25:24 +00:00
Qianfeng Zhang	3c300d3069	Tiny movement in the code lines of the pipeline	2025-06-26 10:24:50 +00:00
Qianfeng Zhang	60d8ffb6cf	Use two work-groups per compute-unit for scheduling the kernel	2025-06-26 10:24:25 +00:00
Qianfeng Zhang	dc7e62a658	Simplify the codes in all host/device IsTokenPairInsideMask() trying to reduce branching	2025-06-23 14:23:53 +00:00
Qianfeng Zhang	63a47d7ec5	Fix masking for min_full_attn_seqlen > 0 situation	2025-06-22 16:27:27 +00:00
Qianfeng Zhang	c87a217475	Update to test_ck_hstu_mask.sh and test_pytorch_hstu_mask.py to align their testings	2025-06-22 16:26:53 +00:00
Qianfeng Zhang	463a19859a	Completely remove the dependency to include/ck_tile/ops/fmha/ops headers	2025-06-22 11:58:26 +00:00
Qianfeng Zhang	4fa6474254	Fix in using KV LdsBuffers to avoid un-expected over-writting that causes un-deterministic results	2025-06-21 13:52:32 +00:00
Qianfeng Zhang	a5f24d7470	Change while() do to do while() for the main loop to let the compiler to generate more elegant codes	2025-06-21 13:51:42 +00:00
Qianfeng Zhang	f9caae2d8b	Use batch dim as first grid dim by default and replace env ASSUME_LEAST_VARIED_SEQLEN by ASSUME_HIGHLY_VARIED_SEQLEN	2025-06-18 16:02:34 +00:00
Qianfeng Zhang	09ac14604c	Align the -seqlens=xxx in the mattn0_full0 and mattn256_full256 scripts with the required benchmarks	2025-06-18 16:02:04 +00:00
root	9e62359b59	Tiny fix in hstu attention IsFullTileInsideMask()	2025-06-18 16:01:40 +00:00
Qianfeng Zhang	08886e99d5	Enable BATCH_AS_FIRST_GRID_DIM grid-scheduling and use ASSUME_LEAST_VARIED_SEQLEN for building control	2025-06-10 15:44:29 +00:00
Qianfeng Zhang	4632d30cc0	Improve the VDramTileDistribution and VLds layout for better device loading and reduce bank-conflict	2025-06-08 13:25:57 +00:00
Qianfeng Zhang	84eb9adc71	Move GetKPackV() and GetAlignmentV() out of ck_tile fmha to hstu pipeline default policy for better visibility	2025-06-07 15:01:19 +00:00
Qianfeng Zhang	b2db644dcd	Add assert(contextual_seqlen >= 0) in example	2025-06-06 15:56:04 +00:00
Qianfeng Zhang	d7930cd541	Update IsFulleTileInsideMask() for kUseLocal is true situtation	2025-06-06 15:55:26 +00:00
Qianfeng Zhang	9e6a24010a	Move all test and bench scripts to folder scripts	2025-06-06 08:22:38 +00:00
Qianfeng Zhang	2bb59dff0a	Add two scripts	2025-06-06 08:15:25 +00:00
Qianfeng Zhang	9582ae2dff	Move dividing by max_seqlen to end of Gemm1 loop in the reference hstu-attention codes	2025-05-31 06:22:54 +00:00
Qianfeng Zhang	bec35abafe	Tune the settings for hdim-256	2025-05-30 08:51:49 +00:00
Qianfeng Zhang	832747c58d	Add example parameter alpha to ease the testing	2025-05-30 08:51:24 +00:00
Qianfeng Zhang	781cba355a	Convert P to fp16/bf16 before doing second gemm in reference hstu implementation	2025-05-29 01:07:04 +00:00
Qianfeng Zhang	36a0f2020c	not-critical updates in example and block_masking codes	2025-05-29 01:06:32 +00:00
Qianfeng Zhang	68a5ab8ff8	Add init_qkv and dump_output example parameters for easier debugging	2025-05-29 01:05:49 +00:00
Qianfeng Zhang	10c35125d2	Add example parameter max_seqlen and max_target	2025-05-27 14:43:43 +00:00
Qianfeng Zhang	c9e19351c7	Update to the method for calculating max_seqlen in the example	2025-05-27 10:38:52 +00:00
Qianfeng Zhang	dc0977faad	Use NRepetitions2DEpilogue for outputing o_acc tile	2025-05-26 14:43:30 +00:00
Qianfeng Zhang	81f7b139e0	Use LDS to in-directly load Q-tile to enable dwordx4 loading and avoid cachelines wasting	2025-05-22 09:49:11 +00:00
Qianfeng Zhang	a1346aaf3e	Update the reference hstu to not do fp32 to fp16/bf16 conversion before P@V gemm	2025-05-20 07:53:25 +00:00
Qianfeng Zhang	0a8ea6bd02	Adjust the threshold values for fp16/bf16 in the example	2025-05-20 07:52:37 +00:00
Qianfeng Zhang	29cf1610f1	Enable RTN fp32 to bf16 conversion by adding compiler option in CMakeLists.txt	2025-05-20 07:52:01 +00:00
Qianfeng Zhang	fac03abbda	Change do-while main-loop to while-do and remove early exiting check	2025-05-19 15:39:31 +00:00
Qianfeng Zhang	14ab6f154d	Adjust the codes before the main-loop	2025-05-19 13:59:57 +00:00
Qianfeng Zhang	f411d676f2	Move k_tile loading and v_tile loading earlier in the loop	2025-05-19 13:59:24 +00:00
Qianfeng Zhang	902b1c645c	Move k_tile loading in the loop earlier	2025-05-19 13:58:43 +00:00
Qianfeng Zhang	f582c21418	Replace s_acc and pcomp tile array by single tile object for simplification	2025-05-19 13:57:11 +00:00