ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-05-20 04:49:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,374 Commits 748 Branches 38 Tags
7b68b6eebbd0cf4db40484e4ef41ba4e1aa9c5fc
Commit Graph

2374 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andriy Roshchenko
79b0bfeb41 MX GEMM - Add FP8 GEMM Tests for Different Layouts (#2152) 
* Add gemm_mx_fp8_bf8 example with row-major B

* Add more overloads of MX MFMA instructions

* Add MK_KN (RRR) tests

* Add KM_NK (CCR) tests

* Add more problem sizes to Large tests

* Add test_gemm_mx to the list of regression tests
 2025-05-01 11:55:48 -06:00
Illia Silin
b9d17bdb11 add write permissions in workspace (#2154) 2025-05-01 07:04:57 -07:00
Aviral Goel
1d8ef40760 Add documentation for ck_tile::array<T,N> (#2078) 
* addded documentation for ck_tile::array<T,N>

* clang format fix

* spelling errros

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>

* spelling errros

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

* Revert "spelling errros"

This reverts commit 4179e7d193.

* Revert "spelling errros"

This reverts commit 3f90733dbe.

---------

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>
Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>
Co-authored-by: John Afaganis <john.afaganis@amd.com>
 2025-04-30 16:43:36 -07:00
spolifroni-amd
cfae863431 updated Doxyfile and added the class list (#2147) 
* updated Doxyfile and added the class list

* Update Doxyfile
 2025-04-30 14:58:40 -07:00
Illia Silin
9a9f59ae69 Revert "Add ck tile examples to package (#1880)" (#2150) 2025-04-30 10:20:16 -07:00
Bartłomiej Kocot
23de234dbe Add grouped conv fwd 16x16 mfma instruction instances (#2140) 
* Add grouped conv fwd 16x16 mfma instruction instances

* fix

* remove oddc

* fix

* fix
 2025-04-30 09:49:37 +02:00
Qianfeng Zhang
da89540ee0 Use kN0=32 2025-04-30 05:42:43 +00:00
Qianfeng Zhang
2972de4c88 Temporarily close the instance for hdim64 and hdim256 to save compiling time 2025-04-30 02:20:41 +00:00
Aviral Goel
1aea51d34e [Tile Engine] Improved README.md (#2134) 
* improved tile_engine readme

* changed ck tile explanation and json

* further improved readme

* fixed typo
 2025-04-29 17:37:07 -07:00
Max Podkorytov
6601931949 try building ck4inductor and testing it inside a virtual environment (#2142) 
use system virtualenv

use python-full ubuntu package in docker image

---------

Co-authored-by: illsilin <Illia.Silin@amd.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
 2025-04-29 17:22:38 -07:00
Illia Silin
8fcb4dff1a Run CI jobs as user jenkins (#2141) 
* run CI as jenkins

* remove user jenkins from docker image

* move inductor installation to a writeable path

* add a switch for inductor tests
 2025-04-29 07:35:10 -07:00
Aviral Goel
65f182d617 Add Matrix A and Matrix B Swizzle for LDS in Computev4 policy (#2136) 
* fixed computev4 policy bug for lds swizzle

* added swizzle for input matrix B

* Improved ComputeV4 policy and pipeline by swizzling A and B

* consolidated LDS descriptor functions in parent struct
 2025-04-28 18:20:47 -07:00
Khushbu Agarwal
d107f3c3a5 Support for MFMA_16x16x128 for fp8/bf8 (#2125) 
* Adding 16x16x128 support for gfx950

* Support for fp8 and bf8

* fix input arguments for MFMA scale instruction

* clang-formatted

* Fixes for lwpck-3145 (#2138)

* Fix lds tile & cmake dep & default epilogue

* Fallback BTypeToUse to ADataType in WOQ cases

* reverting instance json file

* reverting instance json file

---------

Co-authored-by: Yi DING <yi.ding@amd.com>
 2025-04-28 18:19:50 -07:00
Khushbu Agarwal
768c99eca9 [TileEngine] Support for sparsity in codegen (#2128) 
* Added sparsity flag in codegen

* remove comments

* clan formatted

* added sparsity as runtime argument

* updated README

* updated stream config variable

* fix typo for tail_num in hot loop
 2025-04-28 18:19:23 -07:00
Bartłomiej Kocot
4094ad158a Integrate universal gemm with conv bwd data and add SplitK (#1315) 
* Integrate universal gemm with conv bwd data

* Fix multi d kernel

* Add splitK support

* instances refactor

* instances refactor

* refactor

* fixeS

* fixes

* 16x16 instnaces

* Fixes

* Fix

* Fix

* Fix

* Fix

* Fix

* Fixes

* fix

* fix
 2025-04-28 23:54:49 +02:00
Daniel Su
d9786f3363 Check max-ilp-scheduling compiler option for moe_gemm examples (#2127) 2025-04-28 13:40:22 -07:00
jakpiase
434d19f696 Add ck tile examples to package (#1880) 
* add ck tile examples to package

* Update jenkinsfile

* fix for jenkinsfile

* fix for building ck tile code on non gfx9

* compile ck tile examples only for gfx94

* include ck tile examples in all target

* fix for basic gemm UseStructuredSparsity

* Update CMakeLists.txt

* Update gemm_pipeline_problem.hpp

* add targets to rocm install

---------

Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
 2025-04-28 09:53:19 -07:00
lalala-sh
83394e40d2 fix moe i4 example bug (#2139) 2025-04-28 09:49:31 -07:00
Qianfeng Zhang
d63dab90da Hack block_gemm_areg_bsmem_creg_v2 to let s_acc for gemm_0 not need be cleared first 2025-04-28 15:24:13 +00:00
Qianfeng Zhang
f1f4e249a6 Adjust the v_tile and k_tile loading location 2025-04-28 09:25:09 +00:00
Qianfeng Zhang
f53be61a74 Put two gemms call inside one n0loop unroll 2025-04-28 06:41:37 +00:00
Anton Gorenko
edd92fc546 DeviceGemm_Wmma_CShuffleV3 with BlockGemmPipelineVersion::v3 (#2096) 
* Prepare files for DeviceGemm_Wmma_CShuffleV3

* Implement main part of CShuffleV3 with block pipeline v3 for WMMA

* Remove unused functions and template params for A/B descriptors

* Support both gfx11 and gfx12

* Enable SplitK for gfx12 and disable for gfx11

* Added RowColRow layout for DeviceGemmV2 fp16

* Added more instances for Row, Col, Row data layout

* Added instances for DeviceGemm_Wmma_CShuffleV3, Col, Row, Row data layout

* Added instances for DeviceGemm_Wmma_CShuffleV3, Col, Col, Row data layout

* Added more instances for DeviceGemm_Wmma_CShuffleV3, Row, Row, Row data layout

* Fix formatting

* Add documentation

Based on e5ad48a784

* Enable gemm_universal profiling for gfx11/12

* Add WMMA intrinsics for F8/BF8

* Support F8/BF8 DeviceGemm_Wmma_CShuffleV3, add basic instances

* Add BF16 instances and tests

* Fix test_gemm_universal_wmma_fp8 by adding CK_USE_WMMA_FP8

---------

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
 2025-04-28 10:14:21 +05:00
Yi DING
8add2cf45d Fix fp8 convert & add option for basic example (#2129) 2025-04-27 16:26:05 -07:00
Qianfeng Zhang
1af27022ef Add IsFullTileInsideMask() to avoid pixel-by-pixel checking when kUseCausl=true but kUseLocal=false 2025-04-27 09:31:38 +00:00
Po Yen Chen
3d4d70d2fc Avoid using store_tile_raw() for fp32 tensors (#2072) 2025-04-26 23:07:41 -07:00
Qianfeng Zhang
054c397e05 Replace set_tile_if() by sweep_tile_span() to reduce branching 2025-04-27 05:00:09 +00:00
Qianfeng Zhang
95c93ba92e Update the GridSize() and GetTileIndex() in hstu kernel 2025-04-26 10:01:23 +00:00
Qianfeng Zhang
1b463e915d Add scripts for measuring jagged with/no causal cases 2025-04-25 15:59:51 +00:00
Qianfeng Zhang
9996270087 Tiny update in IsTokenPairInsideMask() 2025-04-25 15:36:58 +00:00
Qianfeng Zhang
27f7ab4f2c Use compiler builtin directly in f_silu for float type 2025-04-25 15:05:39 +00:00
Qianfeng Zhang
4ae9acd712 Revert "Update exp() in ck_tile/core/numeric/math.hpp to use __expf" 
This reverts commit 7316a44ff3.
 2025-04-25 14:42:33 +00:00
Qianfeng Zhang
80677eb6e0 Code re-arrangement in pipeline 2025-04-25 14:41:37 +00:00
Qianfeng Zhang
4a49119d98 Update the seqlen_k_curr inside the first gemm loop 2025-04-25 13:59:48 +00:00
joyeamd
41541aff7a SWDEV-52596 for hdim=256, when use splitkv pipeline, two new pipelines need to be added (#2126) 2025-04-25 16:31:09 +08:00
Qianfeng Zhang
7818cce1c3 Rename the performance measurement scripts 2025-04-25 06:09:17 +00:00
Qianfeng Zhang
05910ebe0b Add support for WarpGem-16x16x32 in QK-BlockGemm (which enables using ds_write/read_b128 for K 2025-04-25 06:06:50 +00:00
Khushbu Agarwal
a2ed34a112 MFMA_32x32x16 for gfx950 (#2121) 
* Enable MFMA_32x32x16 for fp16/BF16 for gfx950

* clang formatted
 2025-04-24 10:20:22 -07:00
Illia Silin
01cb8379cd make code compliant with std=c++20 (#2123) 2025-04-24 10:14:52 -07:00
alexxu-amd
ba97363acd Setup Doxygen API reference for Docs (#2115) 
* setup Doxygen settings

* add api_reference to requirements.txt

* add doxygen file header

* omit latex generation

* remove testing entry

* update Doxyfile
 2025-04-24 09:35:06 -06:00
Qianfeng Zhang
a41371f734 Update in K-Lds laying-out to consider for both WarpGemm-32x32x16 and WarpGemm-16x16x16 2025-04-24 15:02:57 +00:00
rocking
02ce6d39ea Only generate specific hdim (#2120) 2025-04-24 18:52:58 +08:00
Qianfeng Zhang
cea919aefb Use 16x16x16 WarpGemm 2025-04-24 08:14:09 +00:00
Qianfeng Zhang
7848d15d39 Using __builtin_amdgcn_rcpf in siLU function 2025-04-24 06:28:16 +00:00
Qianfeng Zhang
aec19176d4 Combine minus with scale_s 2025-04-24 05:47:24 +00:00
Qianfeng Zhang
ce4665262b Move silu calculation to gemm1 iteration and try to interleave gemm_1 and silu 2025-04-24 04:49:58 +00:00
carlushuang
5487289fc4 [CK_TILE] support gfx950 matrix core in 01_fmha fwd (#2110) 
* gfx950 01_fmha fwd

* fix comment

---------

Co-authored-by: Thomas Ning <Thomas.Ning@amd.com>
 2025-04-23 12:40:18 -07:00
John Afaganis
854159fd00 Update CODEOWNERS (#2119) 2025-04-23 10:25:41 -07:00
Qianfeng Zhang
2d2e1941a8 Update in using masking for the case where kMasking is false and kPadSeqLenK is true 2025-04-23 10:47:27 +00:00
Qianfeng Zhang
8dcde8d10f Fix in generate_instances.py and re-generated the instances 2025-04-23 10:30:40 +00:00
lalala-sh
39ba03f25d Moe gemm activation (#2026) 
* fix useless code and remove usless oob

* clang format

* fix coredump in e2e test

* fix2

* fix clang format

* fix output oob

* impl int64 but result not correct

* int64 index ok now

* input output all ok

* fix uint32

* revert v1 test

* use uint32

* mork to support 13w tokens

* moe sorting fix moebuf

* fix merge

* update moe api fix aiter build

* fix buid

* fuse silu

* silu ok

* acale ok

* add silu

* change code

* gemm2 ok

* gufusion compatible ok, fix warnings

* gu fusion for m32 m64 ok

* support bf16 cshuffle

* i4 gemm2 ok

* i4 gemm2 ok and i4 gemm1 build

* 16x16 run ok

* change flops; change cshuffle dtype

* fuse gelu silu act in moe gemm1

* fp8 with act ready

* int4 act ready

* remove useless changes

* remove useless code change

* fix clang format

* add the arch limit of int4 moe gemm

* fuse moe activation

* fix fp8 16x16

* fix no quant case

* fix bugs

* fix fp8 gufusion bug

* remove useless comments

* refine activation code & complete moe example

* fix int8 bugs

* merge tkw1

---------

Co-authored-by: coderfeli <coderfeli@163.com>
Co-authored-by: feli <felix.li@amd.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
Co-authored-by: root <root@hjbog-srdc-51.amd.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
 2025-04-23 10:35:34 +08:00