644 Commits

Author	SHA1	Message	Date
Qianfeng Zhang	f27d8cefb7	Add attn_scale MakeKargs() parameter support and update in example, reference codes	2025-08-03 03:37:28 +00:00
Qianfeng Zhang	7c9032d2cf	Replace the integer max_seqlen by float scale_p as kernel/pipeline parameter	2025-08-03 03:36:23 +00:00
Qianfeng Zhang	de71d3359c	Use __builtin_amdgcn_sched_barrier(0x1) to prevent the compiler from unexpected codes arrangement	2025-08-01 06:57:21 +00:00
Qianfeng Zhang	3483af0516	Fix added case in test_hstu_attention.sh	2025-07-25 15:12:05 +00:00
Qianfeng Zhang	29d3dc9662	Update in GetTileRangeAlongX to consider for non-causal+local_size>0 situation and add test case to test_hstu_attention.sh	2025-07-25 14:56:13 +00:00
Qianfeng Zhang	43a97681b8	Add three scripts for verification of jagged causal cases	2025-07-25 11:20:46 +00:00
Qianfeng Zhang	01c123dedd	Fix in GetTileRangeAlongX() and IsFullTileInsideMask() of HstuBlockMaskWithLocal	2025-07-25 11:20:15 +00:00
Qianfeng Zhang	cf012c23fc	Adjust the codes related to calculate i_m0 in the kernel	2025-07-23 13:44:56 +00:00
Qianfeng Zhang	f49fe28ca2	[Performance] Use separate workgroups to handle seqlen scope [max_uih_len - minfull_attn_seqlen, seqlen]	2025-07-23 09:55:27 +00:00
Qianfeng Zhang	ce6a044440	Fix comments in test_pytorch_hstu_mask.py scripts	2025-07-22 13:22:49 +00:00
Qianfeng Zhang	203e22bf9c	Change the seqlen_q dim padding setting for o_dram and bias_dram	2025-07-22 13:22:12 +00:00
Qianfeng Zhang	ecf6a86d88	Correct some comments	2025-07-21 09:05:26 +00:00
Qianfeng Zhang	fcd41a6f39	Re-arrange the codes section for using sched_group_barrier	2025-07-21 08:22:18 +00:00
Qianfeng Zhang	906ab84d6c	Fix in using sched_group_barrier()	2025-07-21 08:21:35 +00:00
Qianfeng Zhang	acb6cd89d9	Move store_tile() caled before the current iteration	2025-07-21 08:20:52 +00:00
Qianfeng Zhang	fed1474e4f	Revert "Disable support of hdim64 amnd hdim256 for quick compiling and testing" ... This reverts commit ed062f93e2.	2025-07-17 09:00:41 +00:00
Qianfeng Zhang	ed062f93e2	Disable support of hdim64 amnd hdim256 for quick compiling and testing	2025-07-17 09:00:15 +00:00
Qianfeng Zhang	f0c8dca9bf	Fix bug in generate_instances.py and re-generate the instances	2025-07-17 08:37:09 +00:00
Qianfeng Zhang	0306a1e6ab	Re-org the kernel parameters in HstuAttentionFwdBatchModeBaseKargs and HstuAttentionFwdJaggModeBaseKargs	2025-07-17 04:51:16 +00:00
Qianfeng Zhang	fdd9c117d4	Remove num_target from HstuBlockMask class member since it overlaps the meaning of max_uih_len	2025-07-15 16:21:40 +00:00
Qianfeng Zhang	c01695597e	Fix the calculation of number of instructions used by sched_group_barrier	2025-07-15 09:47:35 +00:00
Qianfeng Zhang	0206b3442a	[Performance] use iglp compiler instruction to tune the codes around gemm0 for window_size > 0 situation	2025-07-14 16:02:42 +00:00
Qianfeng Zhang	9171b35751	Add including of block_dropout.hpp in the hstu kernel to avoid potential compiling failure	2025-07-11 00:15:53 +00:00
Qianfeng Zhang	6825618dca	Moving code-lines in hstu pipeline	2025-07-07 09:50:11 +00:00
Qianfeng Zhang	8d30e46ba5	Remove using i_loop and num_loops since seqlen_k_curr and seqlen_k_end is enough	2025-07-06 15:46:56 +00:00
Qianfeng Zhang	5451912526	Let causal == 0 cases to do IsFullTileInsideMask() checking before calling IsTokenPairInsideMask()	2025-06-26 10:25:24 +00:00
Qianfeng Zhang	3c300d3069	Tiny movement in the code lines of the pipeline	2025-06-26 10:24:50 +00:00
Qianfeng Zhang	60d8ffb6cf	Use two work-groups per compute-unit for scheduling the kernel	2025-06-26 10:24:25 +00:00
Qianfeng Zhang	dc7e62a658	Simplify the codes in all host/device IsTokenPairInsideMask() trying to reduce branching	2025-06-23 14:23:53 +00:00
Qianfeng Zhang	63a47d7ec5	Fix masking for min_full_attn_seqlen > 0 situation	2025-06-22 16:27:27 +00:00
Qianfeng Zhang	c87a217475	Update to test_ck_hstu_mask.sh and test_pytorch_hstu_mask.py to align their testings	2025-06-22 16:26:53 +00:00
Qianfeng Zhang	463a19859a	Completely remove the dependency to include/ck_tile/ops/fmha/ops headers	2025-06-22 11:58:26 +00:00
Qianfeng Zhang	4fa6474254	Fix in using KV LdsBuffers to avoid un-expected over-writting that causes un-deterministic results	2025-06-21 13:52:32 +00:00
Qianfeng Zhang	a5f24d7470	Change while() do to do while() for the main loop to let the compiler to generate more elegant codes	2025-06-21 13:51:42 +00:00
Qianfeng Zhang	f9caae2d8b	Use batch dim as first grid dim by default and replace env ASSUME_LEAST_VARIED_SEQLEN by ASSUME_HIGHLY_VARIED_SEQLEN	2025-06-18 16:02:34 +00:00
Qianfeng Zhang	09ac14604c	Align the -seqlens=xxx in the mattn0_full0 and mattn256_full256 scripts with the required benchmarks	2025-06-18 16:02:04 +00:00
root	9e62359b59	Tiny fix in hstu attention IsFullTileInsideMask()	2025-06-18 16:01:40 +00:00
Qianfeng Zhang	08886e99d5	Enable BATCH_AS_FIRST_GRID_DIM grid-scheduling and use ASSUME_LEAST_VARIED_SEQLEN for building control	2025-06-10 15:44:29 +00:00
Qianfeng Zhang	4632d30cc0	Improve the VDramTileDistribution and VLds layout for better device loading and reduce bank-conflict	2025-06-08 13:25:57 +00:00
Qianfeng Zhang	84eb9adc71	Move GetKPackV() and GetAlignmentV() out of ck_tile fmha to hstu pipeline default policy for better visibility	2025-06-07 15:01:19 +00:00
Qianfeng Zhang	b2db644dcd	Add assert(contextual_seqlen >= 0) in example	2025-06-06 15:56:04 +00:00
Qianfeng Zhang	d7930cd541	Update IsFulleTileInsideMask() for kUseLocal is true situtation	2025-06-06 15:55:26 +00:00
Qianfeng Zhang	9e6a24010a	Move all test and bench scripts to folder scripts	2025-06-06 08:22:38 +00:00
Qianfeng Zhang	2bb59dff0a	Add two scripts	2025-06-06 08:15:25 +00:00
Qianfeng Zhang	9582ae2dff	Move dividing by max_seqlen to end of Gemm1 loop in the reference hstu-attention codes	2025-05-31 06:22:54 +00:00
Qianfeng Zhang	bec35abafe	Tune the settings for hdim-256	2025-05-30 08:51:49 +00:00
Qianfeng Zhang	832747c58d	Add example parameter alpha to ease the testing	2025-05-30 08:51:24 +00:00
Qianfeng Zhang	781cba355a	Convert P to fp16/bf16 before doing second gemm in reference hstu implementation	2025-05-29 01:07:04 +00:00
Qianfeng Zhang	36a0f2020c	not-critical updates in example and block_masking codes	2025-05-29 01:06:32 +00:00
Qianfeng Zhang	68a5ab8ff8	Add init_qkv and dump_output example parameters for easier debugging	2025-05-29 01:05:49 +00:00