ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-07-12 10:08:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,926 Commits 778 Branches 40 Tags
e2c8f98fef3bc08dc38ce5951cff15f1a7979bc0
Commit Graph

1926 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
aska-0096
e2c8f98fef generalize the pipeline scheduling. 2025-05-19 02:29:02 +00:00
aska-0096
3e8b07ef58 tempsave; modify the way we represent fp4 2025-05-19 02:28:23 +00:00
Ding, Yi
70e0d94932 Add f4 profiler examples 2025-05-16 07:49:55 +00:00
Ding, Yi
a663ca3395 Fix example compile error 2025-05-16 07:29:47 +00:00
Ding, Yi
dc30e7d025 Add f4 ckProfiler 2025-05-16 07:19:22 +00:00
Ding, Yi
b8fcc7e3b1 Add profiler filter 2025-05-16 07:16:15 +00:00
Ding, Yi
c04d44b5f6 Merge remote-tracking branch 'origin/develop' into wip-f4 2025-05-16 07:11:26 +00:00
Po Yen Chen
8cb0474b3d Use only qr_async pipeline for batch_prefill (#2195) 2025-05-15 11:47:29 -07:00
Khushbu Agarwal
3d8d6e75e4 Adding validation for tile sizes in Tile Engine (#2189) 
* Adding validation for tile sizes

* Add architecture in config, and shuffle lines of code in warp_gemm.hpp

* Enable MFMA for gfx950, and invalid tile handling
 2025-05-15 10:28:31 -07:00
Bartłomiej Kocot
7c0e29cc0f Extend 64x64 with 4 waves instances for grouped conv bwd wei (#2187) 
* Extend 64x64 with 4 waves instnaces for grouped conv bwd wei

* Fix

* fix

* fix
 2025-05-15 16:21:34 +02:00
Ding, Yi
9009d75c7a Pack e8m0 as int32_t 2025-05-15 09:12:17 +00:00
aska-0096
062e16d54a Improve the pipeline 2025-05-15 09:08:36 +00:00
Ding, Yi
4ba9fe186c Use random scale for init1 2025-05-14 05:42:39 +00:00
Ding, Yi
e7130d483c a/b thread_desc stride fix 2025-05-14 05:11:32 +00:00
BingYuan.Zhou
41c17d0a95 fix moe sorting build fail (#2190) 
* fix moe sorting build fail

* refile code

---------

Co-authored-by: solin <bingzhou@amd.com>
 2025-05-14 09:31:26 +08:00
Illia Silin
58f9e9ffbc Update the buffer load/store intrinsic names for clang>=20. (#2192) 
* fix the buffer load/store intrinsic names

* fix clang format
 2025-05-13 10:18:14 -07:00
Ding, Yi
521471c956 Fix fp8/bf8 B-row 2025-05-13 10:13:18 +00:00
Bartłomiej Kocot
c53b7bd22e Switch to v2 pipeline for grouped conv bwd data (#2181) 
* Change to old pipeline for grouped conv bwd data

* fix

* fix

* fix

* fix

* fix

* fix

* Fix
 2025-05-13 10:14:30 +02:00
Ding, Yi
178e361101 Fix fp8/bf8; remove duplicated code 2025-05-13 07:52:13 +00:00
Po Yen Chen
2920604786 [CK_TILE] Add logits soft-capping & customization support to the FMHA forward kernel/pipelines (#2163) 
* hack for cap logits

* fix bug

* Re-format files

* Allow specifying logits_soft_cap through APIs

* Support turn on/off logits_soft_cap in async pipeline

* Do not generate non-verified kernels

* Align receipt used in Aiter

* Sync logits soft-capping across pipelines

* Re-enable some hdim pipelines

* fix perf

* Add attention variant for logits_soft_cap

* Add newline at end-of-file

* Fix performance

* Add comment to explain logits_soft_cap pre-processing

* Unify code

* Unify floating-point literal style

* Use class data member to slience the compilation error

* [CK_TILE] Update attention customizaton interface: add LogitsMask() (#2133)

* Send 'mask' along with variant params to the LogitsMask()

* Send block indices to the variant

* Add indices parameters in variant interface

* Fix fmha bwd codegen error

* Allow switch logits_soft_cap impl

* Eliminate register spills

* Fix compilation errors

* Fix wrong LSE

* Fix LSE for splitkv kernel

* Sync splitkv pipeline changes

* Add batch_prefill kernel/pipeline

* Fix codegen error

* Undo changes in CMakeLists.txt

* Merge pipeline filtering check

* Use different code path if kHasLogitsSoftCap=false

* Remove [[maybe_unused]] attribute

* Use pre-existing compile-time flag to instantiate templates

* Sync pipeline changes

* Update CHANGELOG.md

---------

Co-authored-by: Bernard <bernaliu@amd.com>
Co-authored-by: coderfeli <coderfeli@163.com>
 2025-05-13 12:19:25 +08:00
Khushbu Agarwal
f05e45ba59 Disable SMFMA gfx90a (#2184) 
* sparsity fix for gfx90a

* reverting tile_engine changes
 2025-05-12 09:56:23 -07:00
aska-0096
79246e6cb8 function pass with inline asm hacky 2025-05-12 16:54:44 +00:00
Thomas Ning
b49f7de81f Improve the general performance of the Preshuffled GEMM V3 & delete the unnecessary instances (#2166) 
* make the work compiled

* Solved the example code, but still have the profiler error

* Finished the feature

* Clang format and update the CHANGELOG

* solve the preshuffle v1 & v2 problem

* Comment Addressed

* Comment Addressed
 2025-05-12 09:52:58 -07:00
Thomas Ning
9d1e44e56a Vectorized Transpose for Batched Transpose CK Tile Operator (#2131) 
* Shared Memory for single data point

* CKTile Transpose vectorize CP1

* CKTile Transpose vectorize CP2

* CKTile Transpose vectorize CP2.1

* fixed the compile error of the transpose tile 2d

* Have the correct result for the current test sample

* Changes to printing tensor

* fp8 support added

* Debugging for transpose

* solving the corner issue

* Changed padding flag

* Intermideate Debugging

* Intermidiate Debugging

* Intermediate Debugging

* Finished debugging of the transpose op

* Code Cleanup

* Adding edge case smoke tests

* Adding Transpose test to CI/CD

* Adding Transpose test to CI/CD

* Adding Transpose test to CI/CD

* Addressing Review Comment

* Addressing Comments

* Addressing Comments

* Measuring Perf Tests

* Code Cleanup

* Changlog

* Added the running iterations

* clang format

* Fix the changelog

* Fix the compilation error

* change the printing factor

---------

Co-authored-by: ThruptiRajLakshmanaGowda <tlakshma@amd.com>
 2025-05-12 00:41:45 -07:00
aska-0096
66f93b6e08 tempsave 2025-05-12 07:35:09 +00:00
Ding, Yi
4b19b934e8 fix fp8; fix even/odd 2025-05-12 07:31:28 +00:00
aska-0096
41ea1066ac implement shuffled scale mxfp4gemm, blocker: opsel not effect 2025-05-11 05:54:13 +00:00
Khushbu Agarwal
d8faf1c6a1 Support for swizzle and transpose for MFMA_16x16x32_F16/BF16 (#2172) 
* Changes for updating tile distribution for shuffle and transpose

* Fixed swizzle and transpose, removed comments

* clang formatted

* Adding support for bf16 type

* Addressing review comments
 2025-05-10 22:40:05 -07:00
aska-0096
6c761bf9b8 tempsave; buggy at passed 4 e8m0 to scaled mfma 2025-05-10 09:57:49 +00:00
Bartłomiej Kocot
6fddb5708c Add grouped conv fwd bias relu instances (#2179) 
* Add grouped conv fwd bias relu instances

* fixes

* fix
 2025-05-09 22:52:34 +02:00
aska-0096
087b20dc1d clang format 2025-05-09 16:15:10 +00:00
aska-0096
0987b0af44 remove unnecessary hacky 2025-05-09 16:07:22 +00:00
jefyang1
6b1a339b6f Fix grouped conv bwd data tests on gfx950 (#2173) 2025-05-09 09:01:06 -07:00
aska-0096
86bff9c46e Fix pipe v3 correctness issue 2025-05-09 15:54:43 +00:00
Mingtao Gu
a23390163d fix moe gemm2 for gfx950 (#2164) 
Co-authored-by: mtgu0705 <mtgu@amd.com>
 2025-05-09 08:25:31 -07:00
aska-0096
7bde4b8d34 Add pipeline v3. Have some runtime issue and register spill 2025-05-09 09:47:22 +00:00
aska-0096
bb043a3202 remove some unnecessary hacky; enable 256x256x256 tilesize 2025-05-09 07:54:28 +00:00
Khushbu Agarwal
ef72a4b9bc Disable SMFMA for gfx90a (#2182) 2025-05-09 00:18:07 -07:00
Illia Silin
3448e12609 Generate ckProfiler package for gfx942 only. (#2180) 
* build CI for gfx942 exclusively

* run the last stage in a docker with user jenkins

* update the image for the last stage

* ignore perf_log if not found

* archive and store all packages

* use ccache for building packages
 2025-05-08 13:29:14 -07:00
Andriy Roshchenko
cb27e7c77f Ensure MX GEMM Instances can be Cross-Compiled for Multiple Architectures (#2171) 
* Re-enable MX GEMM instances

* Fix compilation error when building MX GEMM for multiple architectures
 2025-05-08 13:26:03 -06:00
aska-0096
b2efb06315 Spilt the fp4 target. Fix the known bugs. 128x128x128 sanity checked; remove prints 2025-05-08 15:07:33 +00:00
aska-0096
32e6c8aad3 16x16x128 correct; 64x64x128 failed 2025-05-08 09:42:53 +00:00
Ding, Yi
6519ba3aa9 wip3 2025-05-08 07:44:22 +00:00
Thomas Ning
c757046d49 Revert "Disable the SMFMA instruction for gfx90a. (#2174)" (#2175) 
This reverts commit a32d907771.
 2025-05-08 00:07:03 -07:00
Khushbu Agarwal
a32d907771 Disable the SMFMA instruction for gfx90a. (#2174) 
* remove smfma for gfx90a

* clang formatted
 2025-05-07 23:09:22 -07:00
BingYuan.Zhou
6a3960c1e1 Flatmm merge (#2168) 
* sync with function interface of cshuffleepiloge,fix flatmm build fail

* move code from solin/flatmm which add mfma16*16*32fp8 and optimize flatmm

---------

Co-authored-by: solin <bingzhou@amd.com>
 2025-05-08 12:59:57 +08:00
Khushbu Agarwal
c7b8e86e34 [CK_Tile] Simplified Mem pipeline (#2159) 
* simplify code

* compiled the code

* Simplified example and codegen for mem pipeline

* Reveting config and universal gemm example

* clang formatted

* remove comments

* clang formatted

* Add memory operation changes for defualt pipeline

* fix config file

---------

Co-authored-by: ThomasNing <thomas.ning@amd.com>
 2025-05-07 18:37:31 -07:00
jakpiase
cb07ad84d5 fix for default epilogue (#2167) 2025-05-07 10:46:53 -07:00
Bartłomiej Kocot
397b9080a2 Move 16x16 grouped conv fwd instances from comp header (#2165) 
* Move 16x16 grouped conv fwd instances from comp header

* Improvements
 2025-05-07 08:04:31 -07:00
Ding, Yi
8d51a4ae96 wip2 2025-05-07 09:40:47 +00:00