Commit Graph

1912 Commits

Author SHA1 Message Date
Anca Hamuraru
bbda71f096 Added padding instances for f16xf8xf16, f16xi4xf16 2025-05-16 08:31:32 +00:00
Anca Hamuraru
e97a7f246d Added padding instances for f8xf16xf16 2025-05-15 11:31:50 +00:00
Anca Hamuraru
537d19c459 Added packed instances for bf16xi4xbf16 2025-05-15 10:36:50 +00:00
Anca Hamuraru
a3d99c696d Merge branch '32-add-the-remaining-combination-of-data-types-mixed-precision-for-gemm_universal-pipeline3-for' into '33-wip'
Added mixed precision instances for gemm_universal

See merge request amd/ai/composable_kernel!11
2025-05-15 06:53:55 +00:00
Anca Hamuraru
0482b83fe6 Added padding instances for bf16xbf16, f8xf8 2025-05-14 14:36:10 +00:00
Anca Hamuraru
e2855633ee Fixes for padding instances for f16xf16xf16 2025-05-14 10:50:34 +00:00
Anca Hamuraru
ab28ac0b81 Merge branch '33-wip' of projects.streamhpc.com:amd/ai/composable_kernel into 33-wip 2025-05-14 09:43:36 +00:00
Anca Hamuraru
647024db6f Ammending changes for adding support for padding instances for f16xf16xf16 2025-05-14 09:41:19 +00:00
Anca Hamuraru
7107bccda3 Fixed cmake build errors related to test_fp8
(cherry picked from commit 12176616b6)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-14 08:46:34 +00:00
Anca Hamuraru
527c844a06 Merge branch 'cherry-pick-5bdc993d' into '33-wip'
Fixed cmake errors related to  gemm_bilinear. Previously, if the above flags...

See merge request amd/ai/composable_kernel!10
2025-05-14 08:46:15 +00:00
Anca Hamuraru
8b52033aae Fixed cmake errors related to gemm_bilinear. Previously, if the above flags are set, cmake build fails: GPU_TARGETS="gfx1100;gfx1201" -D DTYPES="fp16;bf16;fp8"
(cherry picked from commit 5bdc993dbf)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-14 08:45:34 +00:00
Anca Hamuraru
dd47f39a0a Added padding instances for f16xf16xf16 2025-05-14 07:54:46 +00:00
Thomas Ning
9d1e44e56a Vectorized Transpose for Batched Transpose CK Tile Operator (#2131)
* Shared Memory for single data point

* CKTile Transpose vectorize CP1

* CKTile Transpose vectorize CP2

* CKTile Transpose vectorize CP2.1

* fixed the compile error of the transpose tile 2d

* Have the correct result for the current test sample

* Changes to printing tensor

* fp8 support added

* Debugging for transpose

* solving the corner issue

* Changed padding flag

* Intermideate Debugging

* Intermidiate Debugging

* Intermediate Debugging

* Finished debugging of the transpose op

* Code Cleanup

* Adding edge case smoke tests

* Adding Transpose test to CI/CD

* Adding Transpose test to CI/CD

* Adding Transpose test to CI/CD

* Addressing Review Comment

* Addressing Comments

* Addressing Comments

* Measuring Perf Tests

* Code Cleanup

* Changlog

* Added the running iterations

* clang format

* Fix the changelog

* Fix the compilation error

* change the printing factor

---------

Co-authored-by: ThruptiRajLakshmanaGowda <tlakshma@amd.com>
2025-05-12 00:41:45 -07:00
Khushbu Agarwal
d8faf1c6a1 Support for swizzle and transpose for MFMA_16x16x32_F16/BF16 (#2172)
* Changes for updating tile distribution for shuffle and transpose

* Fixed swizzle and transpose, removed comments

* clang formatted

* Adding support for bf16 type

* Addressing review comments
2025-05-10 22:40:05 -07:00
Bartłomiej Kocot
6fddb5708c Add grouped conv fwd bias relu instances (#2179)
* Add grouped conv fwd bias relu instances

* fixes

* fix
2025-05-09 22:52:34 +02:00
jefyang1
6b1a339b6f Fix grouped conv bwd data tests on gfx950 (#2173) 2025-05-09 09:01:06 -07:00
Mingtao Gu
a23390163d fix moe gemm2 for gfx950 (#2164)
Co-authored-by: mtgu0705 <mtgu@amd.com>
2025-05-09 08:25:31 -07:00
Khushbu Agarwal
ef72a4b9bc Disable SMFMA for gfx90a (#2182) 2025-05-09 00:18:07 -07:00
Illia Silin
3448e12609 Generate ckProfiler package for gfx942 only. (#2180)
* build CI for gfx942 exclusively

* run the last stage in a docker with user jenkins

* update the image for the last stage

* ignore perf_log if not found

* archive and store all packages

* use ccache for building packages
2025-05-08 13:29:14 -07:00
Andriy Roshchenko
cb27e7c77f Ensure MX GEMM Instances can be Cross-Compiled for Multiple Architectures (#2171)
* Re-enable MX GEMM instances

* Fix compilation error when building MX GEMM for multiple architectures
2025-05-08 13:26:03 -06:00
Anca Hamuraru
744262dd88 Pre-commit hooks fixes 2025-05-08 13:48:18 +00:00
Anca Hamuraru
97e024918e Added missing test class for FP16_KM_NK 2025-05-08 10:50:49 +00:00
Anca Hamuraru
d892b5a0e9 Fixed IsSupportedArgument to check ComputeTypeA, ComputeTypeB instead of ADataType, BDataType 2025-05-08 10:49:47 +00:00
Thomas Ning
c757046d49 Revert "Disable the SMFMA instruction for gfx90a. (#2174)" (#2175)
This reverts commit a32d907771.
2025-05-08 00:07:03 -07:00
Khushbu Agarwal
a32d907771 Disable the SMFMA instruction for gfx90a. (#2174)
* remove smfma for gfx90a

* clang formatted
2025-05-07 23:09:22 -07:00
BingYuan.Zhou
6a3960c1e1 Flatmm merge (#2168)
* sync with function interface of cshuffleepiloge,fix flatmm build fail

* move code from solin/flatmm which add mfma16*16*32fp8 and optimize flatmm

---------

Co-authored-by: solin <bingzhou@amd.com>
2025-05-08 12:59:57 +08:00
Khushbu Agarwal
c7b8e86e34 [CK_Tile] Simplified Mem pipeline (#2159)
* simplify code

* compiled the code

* Simplified example and codegen for mem pipeline

* Reveting config and universal gemm example

* clang formatted

* remove comments

* clang formatted

* Add memory operation changes for defualt pipeline

* fix config file

---------

Co-authored-by: ThomasNing <thomas.ning@amd.com>
2025-05-07 18:37:31 -07:00
jakpiase
cb07ad84d5 fix for default epilogue (#2167) 2025-05-07 10:46:53 -07:00
Bartłomiej Kocot
397b9080a2 Move 16x16 grouped conv fwd instances from comp header (#2165)
* Move 16x16 grouped conv fwd instances from comp header

* Improvements
2025-05-07 08:04:31 -07:00
kylasa
956fe8f751 Simple copy kernel, which can be a tool to experiment with CK_Tile API with minimal code. (#2156)
* Test Copy kernel code for testing tile distribution logic

* Fix the error

* Solved the problem

* Updated comments and document formatting

* Removed unused tile distribution and code cleanup

* Added README.md and formatting for CI/CD.

---------

Co-authored-by: ThomasNing <thomas.ning@amd.com>
2025-05-07 00:02:59 -07:00
Aviral Goel
769336b640 [CK_TILE] Add type traits to detect tile window types at compile time (#2158)
* added WindowType enum to tile_window_structs and static assert checks in computev4 pipeline

* added type traits instead of enum to tile_window() and tile_window_linear() with debug comments

* removed comments, added documentation and clang format
2025-05-07 00:00:39 -07:00
Rostyslav Geyyer
8a0d659f92 Add FP4 MX MFMA tests (#2151)
* Add conversion tests

* Fix ctor

* Fix nan logic

* Fix conversion logic

* Permute packed f4_t values

* Fix conversion to float, repack vector elements

* Fix device tests

* Permute elements in a vector

* Add a repro test

* Add a conversion for a repro test

* Update test vectors

* Update conversion

* Fix the test

* Update test vector generator

* Fix vector sr conversion

* Permute conversion args

* Update conversion

* Test

* Fix packing

* Simplify conversion function

* Pack conversion in a loop

* Pack conversion in a loop

* Pack another conversion in a loop

* Pack one more conversion in a loop

* Pack the last conversion in a loop

* Clean up

* Add ops

* Add tests

* Add missing utils

* Update reference mx gemm

* Add f4x2 init mode

* Update host tensor utils

* Update chunk size for f4x2

* Add non scaled ops

* Add a type utility

* Update non scaled reference kernel

* Add non scaled tests

* Debug mfma arguments

* Add more debug info

* Update chunk size

* Update data layout

* Add more debugging

* Fix B stride

* Fix reference gemm

* Fix build

* One more reference fix

* Add more debug info

* Disable some tests

* Enable tests

* Add fp4 dimensions

* Update reference kernels

* Temp edits

* Remove leftovers

* Fix conflicts

* Clean up

* More clean up

* Revert "More clean up"

This reverts commit d8d35a0846.

* Add layouts to tests

---------

Co-authored-by: Andriy Roshchenko <107577548+andriy-ca@users.noreply.github.com>
2025-05-06 09:24:00 -05:00
carlushuang
4e9b76f88c [CK_TILE] optimize moe sorting kernel, boost large context case up to 20x (#2153)
* combine 2-3 as single stage

* support zeroing

* improve long tokens

* update specialization

* b16 ws

* 8bit topk optimize

* update 15 example
2025-05-06 17:32:07 +08:00
Muhammed Emin Ozturk
b8fa27bfef Fix failure in test_batched_gemm_softmax_gemm_permute for lower resource devices (#2117)
* Problematic test case are analyzed and turned off for lower resource GPUs

* update device info

* Update test_batched_gemm_softmax_gemm_permute_bf16_xdl.cpp

* Update test_batched_gemm_softmax_gemm_permute_fp16_xdl.cpp

* Update test/batched_gemm_softmax_gemm_permute/test_batched_gemm_device_utils.hpp

Co-authored-by: John Afaganis <john.afaganis@amd.com>
2025-05-05 13:12:22 -07:00
jakpiase
0bcb804ad0 [CK_TILE] Remove scratch usage from universal gemm (#2001)
* moves kbatch condition outside of kernel

* add reviewer comments

* fixes

* fix tests

* fixes after review

---------

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>
2025-05-05 18:46:44 +02:00
Andriy Roshchenko
79beaacdd1 Restrict MX GEMM instantiation to GFX950 arch (#2157) 2025-05-05 08:18:22 -07:00
dependabot[bot]
c4e4e592c1 Bump rocm-docs-core[api_reference] from 1.18.2 to 1.18.4 in /docs/sphinx (#2161)
Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.18.2 to 1.18.4.
- [Release notes](https://github.com/ROCm/rocm-docs-core/releases)
- [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.18.2...v1.18.4)

---
updated-dependencies:
- dependency-name: rocm-docs-core[api_reference]
  dependency-version: 1.18.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-05-05 07:29:07 -07:00
Anca Hamuraru
55f160278b Added support for F16xI4xF16 to gemm_wmma_universal 2025-05-05 10:36:59 +00:00
Anca Hamuraru
c5d99d2295 Added support for BF16xI4xBF16 to gemm_wmma_universal
(cherry picked from commit c6a4a69d2d43d59bae8bdabfae80d648646f217e)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-05 09:55:29 +00:00
Anca Hamuraru
636fbd5517 Added support for F16xF8xF16 to gemm_wmma_universal 2025-05-05 08:47:30 +00:00
Khushbu Agarwal
d58f2b8bd0 mfma_32x32x64_fp8/bf8 (#2148)
* support for mfma_32x32x64_fp8

* clang-formatted

* Fixing sparsity in codegen
2025-05-01 13:36:24 -07:00
Illia Silin
619fba3134 re-enable ck4inductor tests by default (#2155) 2025-05-01 12:37:27 -07:00
Andriy Roshchenko
79b0bfeb41 MX GEMM - Add FP8 GEMM Tests for Different Layouts (#2152)
* Add gemm_mx_fp8_bf8 example with row-major B

* Add more overloads of MX MFMA instructions

* Add MK_KN (RRR) tests

* Add KM_NK (CCR) tests

* Add more problem sizes to Large tests

* Add test_gemm_mx to the list of regression tests
2025-05-01 11:55:48 -06:00
Illia Silin
b9d17bdb11 add write permissions in workspace (#2154) 2025-05-01 07:04:57 -07:00
Anca Hamuraru
977d8a6649 Added support for F8xF16xF16 to gemm_wmma_universal
(cherry picked from commit 15c851de6daa513a12c2e3af299bab0176175fb5)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-01 12:53:15 +00:00
Anca Hamuraru
03c0446ed2 Adding support for RRR, F8xF16xF16 gemm_universal_wmma - wip
(cherry picked from commit f8c06322df0abcbd5945a56cdf5bffe56480f9f0)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-01 12:52:18 +00:00
Anca Hamuraru
df929f094a Updates to support mixed precision
(cherry picked from commit e65d71180393e7b66169c56565a6bac740427de6)

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-05-01 12:51:21 +00:00
Aviral Goel
1d8ef40760 Add documentation for ck_tile::array<T,N> (#2078)
* addded documentation for ck_tile::array<T,N>

* clang format fix

* spelling errros

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>

* spelling errros

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

* Revert "spelling errros"

This reverts commit 4179e7d193.

* Revert "spelling errros"

This reverts commit 3f90733dbe.

---------

Co-authored-by: Adam Osewski <19374865+aosewski@users.noreply.github.com>
Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>
Co-authored-by: John Afaganis <john.afaganis@amd.com>
2025-04-30 16:43:36 -07:00
spolifroni-amd
cfae863431 updated Doxyfile and added the class list (#2147)
* updated Doxyfile and added the class list

* Update Doxyfile
2025-04-30 14:58:40 -07:00
Illia Silin
9a9f59ae69 Revert "Add ck tile examples to package (#1880)" (#2150) 2025-04-30 10:20:16 -07:00