ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-05-21 21:39:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,909 Commits 753 Branches 38 Tags
d169ed21940ef26d06c0efd3a6c830804cd574fb
Commit Graph

1065 Commits

Author SHA1 Message Date
Qianfeng Zhang
a41371f734 Update in K-Lds laying-out to consider for both WarpGemm-32x32x16 and WarpGemm-16x16x16 2025-04-24 15:02:57 +00:00
rocking
02ce6d39ea Only generate specific hdim (#2120) 2025-04-24 18:52:58 +08:00
Qianfeng Zhang
cea919aefb Use 16x16x16 WarpGemm 2025-04-24 08:14:09 +00:00
Qianfeng Zhang
7848d15d39 Using __builtin_amdgcn_rcpf in siLU function 2025-04-24 06:28:16 +00:00
Qianfeng Zhang
aec19176d4 Combine minus with scale_s 2025-04-24 05:47:24 +00:00
Qianfeng Zhang
ce4665262b Move silu calculation to gemm1 iteration and try to interleave gemm_1 and silu 2025-04-24 04:49:58 +00:00
Qianfeng Zhang
2d2e1941a8 Update in using masking for the case where kMasking is false and kPadSeqLenK is true 2025-04-23 10:47:27 +00:00
Qianfeng Zhang
8dcde8d10f Fix in generate_instances.py and re-generated the instances 2025-04-23 10:30:40 +00:00
lalala-sh
39ba03f25d Moe gemm activation (#2026) 
* fix useless code and remove usless oob

* clang format

* fix coredump in e2e test

* fix2

* fix clang format

* fix output oob

* impl int64 but result not correct

* int64 index ok now

* input output all ok

* fix uint32

* revert v1 test

* use uint32

* mork to support 13w tokens

* moe sorting fix moebuf

* fix merge

* update moe api fix aiter build

* fix buid

* fuse silu

* silu ok

* acale ok

* add silu

* change code

* gemm2 ok

* gufusion compatible ok, fix warnings

* gu fusion for m32 m64 ok

* support bf16 cshuffle

* i4 gemm2 ok

* i4 gemm2 ok and i4 gemm1 build

* 16x16 run ok

* change flops; change cshuffle dtype

* fuse gelu silu act in moe gemm1

* fp8 with act ready

* int4 act ready

* remove useless changes

* remove useless code change

* fix clang format

* add the arch limit of int4 moe gemm

* fuse moe activation

* fix fp8 16x16

* fix no quant case

* fix bugs

* fix fp8 gufusion bug

* remove useless comments

* refine activation code & complete moe example

* fix int8 bugs

* merge tkw1

---------

Co-authored-by: coderfeli <coderfeli@163.com>
Co-authored-by: feli <felix.li@amd.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
Co-authored-by: root <root@hjbog-srdc-51.amd.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
 2025-04-23 10:35:34 +08:00
Gino Lu
504f563f78 [CK-Tile] warp-gemm support for using V_MFMA_F32_16x16x32_BF16 (#2073) 
* draft v_mfma_f32_16x16x32_bf16

* fix error config and add debug code.

* Solve the CShuffle Problem

* draft v_mfma_f32_16x16x32_bf16

* fix error config and add debug code.

* Solve the CShuffle Problem

* fix error while testing new command

* Finished the feature of new mfma 16*16*32

* Addressed the comment

---------

Co-authored-by: ThomasNing <thomas.ning@amd.com>
 2025-04-22 15:52:36 -07:00
Qianfeng Zhang
022ed3fd8a Back to use exp() instead of exp2() since exp() in ck_tile using fast __builtin_amdgcn_exp2f() 2025-04-22 14:47:24 +00:00
Qianfeng Zhang
26db7e0b7c Use kN0=64 to save vgprs 2025-04-22 14:45:27 +00:00
Qianfeng Zhang
65ddb1a863 Fix the script name 2025-04-22 13:46:48 +00:00
Qianfeng Zhang
58ab5533a6 Fix in GetTileRangeAlongX 2025-04-22 13:46:23 +00:00
Qianfeng Zhang
677fd60d10 Add script compare_with_triton_2.sh for measuring the jagged cases of seqlen 1024/2048/4096/8192/16384/32768 2025-04-22 13:45:48 +00:00
Qianfeng Zhang
2546e905ce Change gemm0 to iterate along kN0 so that BlockGemm can overlap with maksing and siLu 2025-04-20 13:23:15 +00:00
Qianfeng Zhang
ee259a8924 Fix the GetTileRangeAlongX() to align with the hstu masking definition when both causal=true and local=true 2025-04-18 15:37:49 +00:00
Qianfeng Zhang
efc786f6a3 Remove un-needed __builtin_amdgcn_sched_barrier(0) 2025-04-18 10:05:57 +00:00
Qianfeng Zhang
88e54a8989 Use shared ring Lds buffers for K/V to avoid over-lapping between first-K/last-V or last-K/first-V 2025-04-18 09:47:43 +00:00
Qianfeng Zhang
f12a47218f Tiny codes simplification in pipeline 2025-04-18 08:22:11 +00:00
lalala-sh
bcf5bb41be enable do top k weights in moe stage1 gemm (#2094) 
* add switch for mul topk weights

* fix bf16/f16 bugs

* complete
 2025-04-18 10:45:49 +08:00
Qianfeng Zhang
ca1ae84fc6 Remove one line of __builtin_amdgcn_sched_barrier(0) 2025-04-17 14:21:14 +00:00
Qianfeng Zhang
b0ae27046f Fix the integer overflow in total_flops calculation 2025-04-17 10:34:13 +00:00
Qianfeng Zhang
6086ead2f9 Add scripts for comparing with triton 2025-04-17 10:33:44 +00:00
Andriy Roshchenko
da54464cce MX GEMM - Add MX BF8 example (#2071) 
* Add MX GEMM example for MX BF8

* Verified MX FP8 with 16x16x128 scale builtin

* Verify MX BF8 GEMM with BF16 output
 2025-04-16 15:25:02 -06:00
BingYuan.Zhou
eaf1f0bf3b [flatmm] implement basic fp16 flatmm (#2089) 
* [flatmm] implement basic fp16 flatmm

* fix CI build fail

---------

Co-authored-by: root <root@hjbog-srdc-50.amd.com>
Co-authored-by: solin <bingzhou@amd.com>
 2025-04-16 16:51:17 +08:00
Qianfeng Zhang
1351d9cd1b Use exp2() to calculate exp() for better performance 2025-04-16 06:54:06 +00:00
Qianfeng Zhang
226a254723 Remove the comparing of row/col to max_uih_len in masking 2025-04-16 04:35:42 +00:00
felix
c5975529bb add preshuffle gemm fp16 (#2036) 
* add preshuffle gemm fp16

* clang format and test ok

* Update gemm_multiply_multiply_xdl_fp16_bpreshuffle.cpp

remove useless comments in example

* Update gemm_multiply_multiply_xdl_fp16_bpreshuffle.cpp

remove 2

---------

Co-authored-by: coderfeli <coderfeli@163.com>
 2025-04-16 10:53:21 +08:00
joyeamd
94d47b1680 fmha hdim256 vectorize improve (#2086) 
For hdim 256, will not have vectorized buffer load when seqlen % 256 != 0 and hdim % 256 = 0; this commit tries to solve this condition.
 2025-04-16 09:21:04 +08:00
Andriy Roshchenko
7106976a72 MX GEMM - New GEMM pipeline for MX data types (#2059) 
* Allow selection of mfma_scale instructions

* Read B tensor from LDS to VGPR in chunks of 16 in MFMA order

* Add constexpr and synchronize return type for `get_exponent_value`

* Pass scales by reference and add comments to `mfma_scale_f32_32x32x64`

* Add support for microscaling instructions in `XdlopsGemm`

* Fix `mfma_scale_f32_16x16x128f8f6f4` wrapper

* Remove software implementation of MX GEMM

* Make interface of `intrin_mfma_scale_f32_16x16x128f8f6f4<16, 16>` consistent with the other scale instruction

* Update README

* Updated CHANGELOG

* Remove unused static methods
 2025-04-15 17:17:07 -06:00
Qianfeng Zhang
d1749b3aae Use kM0=128 kN0=64 to completely remove the vgprs spilling 2025-04-15 15:08:46 +00:00
Qianfeng Zhang
3cd1b13e46 Split HstuBlockMasking into HstuBlockMaskWithLocal and HstuBlockMaskNoLocal to save vgprs for non-local situations 2025-04-15 14:40:55 +00:00
Qianfeng Zhang
cad1356170 Use packed cast_tile for fp16 2025-04-15 14:29:30 +00:00
Qianfeng Zhang
fff13b6c76 Update to partially reduce the register spilling 2025-04-15 07:44:33 +00:00
Mingtao Gu
56378f810f CK pk_i4_t test failures fix (SWDEV-518629) (#2075) 
* fix pk_i4_v3 tests failures in Unbuntu env.

* fix pk_i4_t tests failure on Unbuntu issues.

* some fixed.

---------

Co-authored-by: mtgu0705 <mtgu@amd.com>
 2025-04-14 16:58:57 +08:00
Qianfeng Zhang
c2e6ab8516 Add IsFirstVLdsBufferOverlapLastKLdsBuffer() check to reduce call of s_barrier() 2025-04-13 11:00:22 +00:00
Qianfeng Zhang
238e78d82e Update the in pipeline codes 2025-04-13 09:43:58 +00:00
Qianfeng Zhang
53e567977e Fix in calculation of total_flops and update benchmark scripts 2025-04-13 08:50:00 +00:00
jakpiase
6c61f4d237 [CK_TILE] Add 2:4 structured sparsity support for fp16 gemm (#1957) 
* add structured sparsity fp16 support for gemm

* added reviewer suggestions

* update changelog

* update changelog

* add reviewers suggestions

* Minor fix

* clang fix

* fix doxygen
 2025-04-11 12:18:26 +02:00
slippedJim
5f885d2b7a add fmha fwd splitkv receipt for aiter c++ api (#2068) 
* add s_randval for c++ api

* Fix bug of bias in splitkv

---------

Co-authored-by: rocking <ChunYu.Lai@amd.com>
 2025-04-10 23:21:13 +08:00
Illia Silin
3e6d21adeb enable gfx115x support (#2065) 2025-04-09 10:06:42 -07:00
Qianfeng Zhang
71697d9cb9 Add output of estimated TFLOPS 2025-04-09 14:50:18 +00:00
Qianfeng Zhang
1766e6d3be Update to the scripts and error thresholds 2025-04-09 10:34:37 +00:00
Qianfeng Zhang
dd2cd2cbcb Tune the input initialization to avoid over-flow in silu 2025-04-09 10:03:32 +00:00
Qianfeng Zhang
86c0e45987 Add benchmark_hstu_attention.sh 2025-04-09 08:28:05 +00:00
Qianfeng Zhang
9cb2dca958 Add several verification test cases 2025-04-08 16:38:35 +00:00
Qianfeng Zhang
561d490990 Fix in kernel and forward dispatch for jagged mode 2025-04-08 16:37:52 +00:00
Qianfeng Zhang
dc2f72a09f Fix in hstu-attention pipeline (which makes some testing cases passed) 2025-04-08 15:53:08 +00:00
Qianfeng Zhang
dbcf38aae9 Fixes and updates 2025-04-07 15:29:23 +00:00