Commit Graph

648 Commits

Author SHA1 Message Date
feifei14119
32eefad0a4 fix buffer load for loop 2025-05-17 11:57:44 +08:00
mtgu0705
2151290dd4 minor updates 2025-05-15 03:49:20 -05:00
mtgu0705
b65bc1ba4a added mx moe block v3 support, function passed 2025-05-15 03:26:37 -05:00
mtgu0705
4e2ec31e4d rename moe block selector and pipeline 2025-05-15 02:12:50 -05:00
Your Name
e060999d29 v3 2025-05-15 13:19:07 +08:00
mtgu0705
dfba3c11e7 fix the bug, 128x128x256 tile function passed 2025-05-15 00:11:10 -05:00
mtgu0705
7cfd1db335 update debug 2025-05-14 21:41:28 -05:00
mtgu0705
102151ebcf temp save 2025-05-14 08:13:47 -05:00
mtgu0705
2700b217be 16x16x128 input size blockscale function passed 2025-05-14 03:20:59 -05:00
mtgu0705
1bbb50b212 mfma using asm, device result correct, host result need to check 2025-05-13 20:57:34 -05:00
mtgu0705
6dfe24c53e updated 2025-05-13 04:15:53 -05:00
mtgu0705
c11fef9197 added code for debug 2025-05-13 15:25:03 +08:00
mtgu0705
5ba86c210b updated and build passed 2025-05-13 14:49:37 +08:00
mtgu0705
cc43f88f08 add code for mxfp4 gemm, blockscale not supported yet 2025-05-12 20:56:50 +08:00
Your Name
ce51d2d099 Revert "use mem_op::set when topk=1"
This reverts commit def952a178.
2025-05-12 16:24:30 +08:00
Your Name
dff709c79f Revert "update"
This reverts commit 960b2bce1c.
2025-05-12 16:24:12 +08:00
Your Name
58f848cc07 Merge branch 'wjx/align_v3_pipeline' into fp4_gu_moe 2025-05-12 16:21:39 +08:00
aska-0096
22f8f5e5d8 fix bug for a lds read 2025-05-12 04:00:21 +00:00
mtgu0705
0bddd63d9c fix bugs, build passed 2025-05-11 17:49:43 +08:00
mtgu0705
d3f007c775 hotfix 2025-05-11 16:58:36 +08:00
mtgu0705
7d272f7f63 fixed some bugs 2025-05-10 22:01:20 +08:00
mtgu0705
70648240f9 added fp4_bpreshuffle example, build failures 2025-05-10 21:34:32 +08:00
Your Name
5421e71155 Merge branch 'wip-f4' into mt1 2025-05-10 14:56:21 +08:00
mtgu0705
bb49d2e623 fix the bug, functional test passed 2025-05-10 14:33:44 +08:00
mtgu0705
493a8639b3 fix a bug 2025-05-10 13:07:43 +08:00
aska-0096
087b20dc1d clang format 2025-05-09 16:15:10 +00:00
aska-0096
0987b0af44 remove unnecessary hacky 2025-05-09 16:07:22 +00:00
aska-0096
86bff9c46e Fix pipe v3 correctness issue 2025-05-09 15:54:43 +00:00
aska-0096
7bde4b8d34 Add pipeline v3. Have some runtime issue and register spill 2025-05-09 09:47:22 +00:00
mtgu0705
c0e010711c update for function debug 2025-05-09 08:37:04 +00:00
aska-0096
bb043a3202 remove some unnecessary hacky; enable 256x256x256 tilesize 2025-05-09 07:54:28 +00:00
mtgu0705
f2a474e2e9 fix update 2025-05-09 11:04:39 +08:00
mtgu0705
11f386108e some fixes 2025-05-08 23:38:27 +08:00
aska-0096
b2efb06315 Spilt the fp4 target. Fix the known bugs. 128x128x128 sanity checked; remove prints 2025-05-08 15:07:33 +00:00
mtgu0705
7c49f9dd31 add mx fp8 b_preshuffle support, function not yet tested. 2025-05-08 22:41:54 +08:00
lalala-sh
6c459e8c38 Merge branch 'develop' into wjx/align_v3_pipeline 2025-05-08 18:08:31 +08:00
lalala-sh
def952a178 use mem_op::set when topk=1 2025-05-08 09:49:16 +00:00
lalala-sh
960b2bce1c update 2025-05-08 09:48:23 +00:00
aska-0096
32e6c8aad3 16x16x128 correct; 64x64x128 failed 2025-05-08 09:42:53 +00:00
Ding, Yi
6519ba3aa9 wip3 2025-05-08 07:44:22 +00:00
Ding, Yi
8d51a4ae96 wip2 2025-05-07 09:40:47 +00:00
Ding, Yi
d12b750bb0 wip 2025-05-07 08:17:07 +00:00
Rostyslav Geyyer
8a0d659f92 Add FP4 MX MFMA tests (#2151)
* Add conversion tests

* Fix ctor

* Fix nan logic

* Fix conversion logic

* Permute packed f4_t values

* Fix conversion to float, repack vector elements

* Fix device tests

* Permute elements in a vector

* Add a repro test

* Add a conversion for a repro test

* Update test vectors

* Update conversion

* Fix the test

* Update test vector generator

* Fix vector sr conversion

* Permute conversion args

* Update conversion

* Test

* Fix packing

* Simplify conversion function

* Pack conversion in a loop

* Pack conversion in a loop

* Pack another conversion in a loop

* Pack one more conversion in a loop

* Pack the last conversion in a loop

* Clean up

* Add ops

* Add tests

* Add missing utils

* Update reference mx gemm

* Add f4x2 init mode

* Update host tensor utils

* Update chunk size for f4x2

* Add non scaled ops

* Add a type utility

* Update non scaled reference kernel

* Add non scaled tests

* Debug mfma arguments

* Add more debug info

* Update chunk size

* Update data layout

* Add more debugging

* Fix B stride

* Fix reference gemm

* Fix build

* One more reference fix

* Add more debug info

* Disable some tests

* Enable tests

* Add fp4 dimensions

* Update reference kernels

* Temp edits

* Remove leftovers

* Fix conflicts

* Clean up

* More clean up

* Revert "More clean up"

This reverts commit d8d35a0846.

* Add layouts to tests

---------

Co-authored-by: Andriy Roshchenko <107577548+andriy-ca@users.noreply.github.com>
2025-05-06 09:24:00 -05:00
lalala-sh
0ab978584d fix bugs 2025-05-06 07:36:59 +00:00
Andriy Roshchenko
79b0bfeb41 MX GEMM - Add FP8 GEMM Tests for Different Layouts (#2152)
* Add gemm_mx_fp8_bf8 example with row-major B

* Add more overloads of MX MFMA instructions

* Add MK_KN (RRR) tests

* Add KM_NK (CCR) tests

* Add more problem sizes to Large tests

* Add test_gemm_mx to the list of regression tests
2025-05-01 11:55:48 -06:00
lalala-sh
9c06c3817a [fix] align v3 gufusion pipeline 2025-04-30 02:27:39 +00:00
lalala-sh
b8427b812e align v3 gufusion pipeline 2025-04-30 01:35:49 +00:00
Bartłomiej Kocot
4094ad158a Integrate universal gemm with conv bwd data and add SplitK (#1315)
* Integrate universal gemm with conv bwd data

* Fix multi d kernel

* Add splitK support

* instances refactor

* instances refactor

* refactor

* fixeS

* fixes

* 16x16 instnaces

* Fixes

* Fix

* Fix

* Fix

* Fix

* Fix

* Fixes

* fix

* fix
2025-04-28 23:54:49 +02:00
Anton Gorenko
edd92fc546 DeviceGemm_Wmma_CShuffleV3 with BlockGemmPipelineVersion::v3 (#2096)
* Prepare files for DeviceGemm_Wmma_CShuffleV3

* Implement main part of CShuffleV3 with block pipeline v3 for WMMA

* Remove unused functions and template params for A/B descriptors

* Support both gfx11 and gfx12

* Enable SplitK for gfx12 and disable for gfx11

* Added RowColRow layout for DeviceGemmV2 fp16

* Added more instances for Row, Col, Row data layout

* Added instances for DeviceGemm_Wmma_CShuffleV3, Col, Row, Row data layout

* Added instances for DeviceGemm_Wmma_CShuffleV3, Col, Col, Row data layout

* Added more instances for DeviceGemm_Wmma_CShuffleV3, Row, Row, Row data layout

* Fix formatting

* Add documentation

Based on e5ad48a784

* Enable gemm_universal profiling for gfx11/12

* Add WMMA intrinsics for F8/BF8

* Support F8/BF8 DeviceGemm_Wmma_CShuffleV3, add basic instances

* Add BF16 instances and tests

* Fix test_gemm_universal_wmma_fp8 by adding CK_USE_WMMA_FP8

---------

Co-authored-by: Anca Hamuraru <anca@streamhpc.com>
2025-04-28 10:14:21 +05:00
aska-0096
bc9c819aa4 generalized bpreshuffle pipeline optimization 2025-04-27 11:50:30 +00:00