Commit Graph

127 Commits

Author SHA1 Message Date
Illia Silin
8c29e06f3c disable inductor codegen tests on legacy OS (#1816) 2025-01-15 12:11:54 -08:00
Max Podkorytov
c0b90f130f fix parsing instances for pt inductor (#1796)
add unit test for gen instances for gemms

add unit tests for conv and batched gemms

add unit test for preselected gemm instances

apply ruff lint

add license header for the unit test

add inductor pytest to CI

verbose pip install

switch the directory before installing python packages

move the inductor codegen test

try yet another workdir

Update Jenkinsfile

The directory looks right, fixing pip module not found by invoking pip directly

Update Jenkinsfile

invoke pytest directly since the module is not found

Update Dockerfile

Install setuptools

update package structure

bump setuptools

maybe fix data path for library sources

fix library search path for conv instances

fix path in pyproject definition

compare path used in gen_instances with one in pyproject.toml; fix the difference

Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
2025-01-13 13:51:08 -08:00
Thomas Ning
73a076eee1 Ck tile/gemm perf measure (#1750)
* Finished adding the performance benchmark for ck tile gemm

* Fix the executable rename problem

* fix the executable name error

* delete the unsupported layout combinations

* Update run_full_test.sh

* Update benchmark_mem_pipeline.sh

* Update benchmark_basic.sh

* change the executable of gemm_universal

* change ck_tile_gemm script permissions

* Addressed the comment

* Addressed the comment

* Fixed the comments

* Fixed Comment

* roll back the malfunctioned change

* Fix the Typo

* finalize the tile_gemm_fp16 performance monitoring

* fix the stash names for ck_tile gemm logs

* change the stashing logic

* change stashing syntax

---------

Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
2025-01-09 17:41:49 -08:00
Illia Silin
fdfe210230 upgrade sqlalchemy version (#1748)
* upgrade sqlalchemy version

* replace the connection with engine in to_sql call

* change the hipTes=nsor ctest syntax
2024-12-15 16:25:21 -08:00
Illia Silin
90d8410d56 Upgrade to Ubuntu22.04 as default OS. (#1738)
* upgrade to ubuntu 22.04

* try adding -u roof docker options for ubuntu 22
2024-12-10 08:48:51 -08:00
Illia Silin
23cf2026b4 build CI for gfx12 by default (#1734) 2024-12-09 14:11:20 -08:00
Illia Silin
2f088b8707 update CI timeout limits (#1733) 2024-12-09 09:32:14 -08:00
Illia Silin
355893cdd8 Refactor CI performance tests. (#1726)
* merge the build and performance tests CI stages together

* add gemm performance test on gfx11/gfx12

* add suffices to distinguish gemm performance logs from different archs

* use smaller gemm set in CI for gfx10/gfx11/gfx12

* disable performance tests on gfx1030

* fix the shashing logic

* fix finding python3 for mha instances
2024-12-06 13:04:25 -08:00
Illia Silin
86990558e3 Upgrade default compiler to ROCm6.3 (#1723)
* upgrade to rocm6.3 compiler

* Proposed solution to convnd test failures in ROCm 6.3

---------

Co-authored-by: Andriy Roshchenko <andriy.roshchenko@amd.com>
2024-12-05 17:29:12 -08:00
Illia Silin
aa6e2087f5 Reduce docker size and build time in CI. (#1699)
* refactor docker build in CI

* add Dockerfile.compiler

* add input args to Dockerfile.compiler

* rearrange the docker args
2024-11-28 10:42:19 -08:00
Illia Silin
cb8c7f42d6 update mainline compiler branch name (#1696) 2024-11-26 14:58:35 -08:00
Illia Silin
19d4b79039 add --squash flag when building dockers (#1686) 2024-11-22 17:16:08 -08:00
Illia Silin
54440cf562 remove gfx940;gfx941 from default target lists (#1640) 2024-11-05 13:56:20 -08:00
Paul Fultz II
10158b0ffd Build codegen as standalone (#1556)
* Build codegen as standalone

* Add exception for device tests

* Use local filesystem header

* add a codegen test CI stage and daily build

---------

Co-authored-by: illsilin <Illia.Silin@amd.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
2024-10-15 13:20:42 -07:00
Illia Silin
11444e4cf2 [CI] remove the --rm docker container flags (#1568) 2024-10-11 14:29:46 -07:00
Illia Silin
cfac9497e2 remove gfx12 targets from daily builds with rocm6.2 (#1560) 2024-10-09 10:18:05 -07:00
Illia Silin
7d8ea5f08b Fix build logic using GRU_ARCHS. (#1536)
* update build logic with GPU_ARCHS

* fix the GPU_ARCHS build for codegen

* unset GPU_TARGETS when GPU_ARCHS are set
2024-10-07 08:18:23 -07:00
Illia Silin
11b7a4db00 re-enable the FMHA performance monitoring (#1539) 2024-10-01 13:17:55 -07:00
Illia Silin
65f8d1440f make CK CI use different git credentials (#1529) 2024-09-25 09:05:48 -07:00
Illia Silin
f16ebf82d4 Add a daily CI build with legacy dockers. (#1525)
* add an option to build CK with legacy dockers

* change the custom docker settings

* add environment varianble for custom docker

* use a new variable for legacy docker name

* new way to pass docker names for legacy OS

* add legacy docker check in the Build_CK function

* change groovy syntax

* add a check for legacy docker in getDockerImage

* make sure the legacy docker name is not empty

* remove the dumb-init call

* disable the tests in legacy OS dockers

* disable tests in legacy dockers

* use a different way to disable tests in legacy dockers

* rearrange the CI stages for legacy OS

* use different way to disable tests in legacy dockers

* update LD_LIBRARY_PATH for legacy dockers and add cron job

* update LD_LIBRARY_PATH at docker launch

* change the sytax for setting LD_LIBRARY_PATH
2024-09-23 09:03:55 -07:00
Illia Silin
1658c0dc11 Add rocm6.3_rc1 docker image (#1518)
* add image for rocm6.3_rc1

* fix deb package url
2024-09-17 15:59:26 -07:00
Thomas Ning
844f5a1712 Ck tile GPU verification sample develop & Add the CK TILE GEMM to the CI/CD test (#1505)
* Finished the feature of gpu verification

* Add the ck_tile_gemm test in the CI CD

* add the include of tensor_layou in reference_gemm

* Comment Addressed

* split ck_tile fhma and gemm tests into separate stages

* restructure the reference gemm

* restructure a new reference_gemm api that could read the device mem

---------

Co-authored-by: carlushuang <carlus.huang@amd.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
2024-09-14 21:08:40 +08:00
Illia Silin
e07f1108c0 make sure to rebuild compilers if they changed (#1504) 2024-09-12 07:49:55 -07:00
Illia Silin
19d22e60c1 Enable daily ninja build traces. (#1487)
* add ninja trace to CI builds

* fix ninja trace logic

* update the ninja trace logic in jenkins file

* limit the number of threads to run ninja build

* use ninja for installation after build

* update the path to ninjatracing tool

* use ninja to run check when using build trace

* fix jenkins logic

* fix typos

* set proper setup_args for all stages

* fix ninja syntax

* replace ninja check with ninja test

* enable ninja tracing with mainline and staging compilers
2024-08-26 13:27:01 -07:00
Bartłomiej Kocot
2581727d2a Add performance and large tensor tests for grouped conv (#1456)
* Add performance and large tensor tests for grouped conv

* Resize tests

* Resize tests

* update the python script to parse the grouped_conv results

* Remove int8 tests

* change bwd wei layout

---------

Co-authored-by: illsilin <Illia.Silin@amd.com>
2024-08-16 07:48:30 -07:00
arai713
da214a5a58 Codegen build w/CK (#1428)
* initial push

* cleaned up compiler errors

* removed commented code

* build codegen folder only for gfx9 targets

* remove separate stage for codegen tests from CI

* removed commented code from CMake

---------

Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
2024-08-09 08:15:06 -07:00
Illia Silin
4a5ab67871 Enable CI on gfx12. (#1454)
* enable CI build and test on gfx1201

* skip DL kernels in CI for gfx12

* only run CI on gfx12 if rocm version >= 6.2

* remove the rocm version check for CI on gfx12

* add a switch for CI builds on gfx12
2024-08-08 16:29:15 -07:00
Illia Silin
5df10432d8 upgrade to rocm6.2 as new default compiler (#1448) 2024-08-07 09:38:43 -07:00
Illia Silin
12c1f68dd9 Run CK_TILE FMHA benchmarks and collect the performance data. (#1447)
* run ck_tile benchmarks after the smoke tests and store logs

* change the path of fmha benchmark logs

* change the way of stashig ck_tile fmha logs

* prevent the errors in stages where no logs are generated

* fix the ck_tile fmha log names and headers

* generate the fmha performance logs in the root folder

* change jenkins scrip arguments format

* use exact file names for stashing

* modify scripts to process FMHA performance results

* unstash FMHA logs before parsing them
2024-08-07 08:18:26 -07:00
Illia Silin
f31ba04afc [CI][Jenkins] delete CI docker container upon exit (#1437) 2024-08-05 08:13:56 -07:00
Illia Silin
d311c95396 Add compiler flags for ROCm versions 6.2+ (#1429)
* add compiler flags to fix compiler issues

* fix typo.

* disable test_smfmac_op on all devices except gfx942

* specify full path to compiler in CI
2024-08-01 08:27:52 -07:00
Illia Silin
98a01bbc72 Add CK_TILE tests to daily CI builds. (#1381)
* add ck_tile tests to CI

* build and run ck_tile tests on gfx90a and gfx942 in parallel

* fix groovy syntax

* turn ck_tile tests OFF by default

* skip creating the build folder

* build ck_tile examples with 64 threads

* build ck_tile examples with cmake-ck-dev.sh script

* add video group to docker on mi300

* do not retry to rebuild the early CI stages

* help prevent jenkins false failure

* restore cron trigger
2024-07-11 13:22:40 -07:00
Illia Silin
f914c228c6 [Jenkins] restore cron jobs (#1380)
* test the cron trigger

* fix the cron jobs

* restore the list of cron jobs
2024-07-11 10:28:11 -07:00
Jun Liu
959073842c Fix issue with multiple targets and remove smfmac tests from unsupported test targets (#1372) 2024-07-03 23:34:38 -07:00
Illia Silin
941d1f7ce0 Merging the gfx12 code into public repo. (#1362) 2024-06-27 00:33:34 -07:00
Illia Silin
3fa7e2a6c4 disable the hipTensor test by default, only run once daily (#1321) 2024-06-03 14:07:30 -07:00
Illia Silin
34f3dfdd61 Build CK library for all supported targets. (#1312)
* test library build for all supported targets

* increase the number of threads to build lib in CI to 64
2024-05-28 12:36:06 -07:00
Illia Silin
566b6480a2 Code clean-up (#1285)
* code clean-up

* remove the profiling output samples
2024-05-10 09:41:39 -07:00
Illia Silin
f0bf1e3125 [CI] Focus CI stages on MI200 nodes for resource optimization (#1273) 2024-05-01 10:07:14 -07:00
Illia Silin
f6b3f4715d [CI][Tests] Add a daily cron job to build CK instances for gfx9;gfx10;gfx11. (#1271)
* add a daily build for instances for gfx9;gfx10;gfx11

* fix jenkins logic for instances only build

* fix the path for instance_only build

* reduce the number of build threads to 32
2024-04-30 14:44:30 -07:00
Illia Silin
930f889c34 Make daily cron jobs use the rocm6.1 compiler. (#1253)
* add rocm6.1 docker and make it default for CI

* fix typo

* move the rocm6.1 image into public dockerhub repo

* upgrade daily cron jobs to use rocm6.1
2024-04-18 09:40:21 -07:00
Illia Silin
caae537d8e Upgrade to ROCm6.1 and turn on the -enable-post-misched=0 compiler flag. (#1250)
* add rocm6.1 docker and make it default for CI

* fix typo

* move the rocm6.1 image into public dockerhub repo
2024-04-18 11:10:23 -05:00
Haocong WANG
f83e9701e9 [GEMM] Gemm universal device operation (#1154)
* Optimize GEMM on MI200/300:
1. Add new blockwise gemm pipeline
2. Add irregular splitk intances

* clang format + typo fix

* Fix a bug

* initial commit

* Add more instances to irregular splitk

* blkgemm pipeline v1~4 prototype

* Sanity Checked. Known issue:
1. Poor performance of splitk
2. Register spill on blkgemmpipeline v3

* Sanity and Performance fix:
1. fix a bug related to sanity in grouped b2c mapping
2. fix a bug related to sanity and performance in splitk offset

* Sanity and API update:
1. Remove prefetch stage
2. Fix valid check bug
3, Add first gemm_universal instance into ckProfiler

* Add NN instances for gemm universal

* 1. Add NT instances for gemm_universal
2. Fix a bug about Kpadding in gemm_universal

* Fix a bug regarding padding Odd K number

* remove kernel print

* Fix KPadding bug...

* Update safety check

* another try to fix kpadding..

* Sanity checked

* new instances..

* clang format+typo fix

* remove clang format script's change

* Add non-hotloop compile option

* 1. Add fp16xfp8 example
2. pull packed convert f8 from pr1150

* Some miscs.. opt and fix

* Add pipeline description docs

* Split universal gemm instance library to cut profiler compiling time

* uncomment cmakefile

* Fix a bug caused by blockwise_gemm_pipe_v2

* reduce default splitk to 1

* Add 224x256x64 tile size

* update, including:
1. Experiment pipeline 5~7
2. Optimization for pipeline 4
3. Organized instance library

* temp save

* temp save

* Permuted lds layout, sanity and function checked

* clang format

* Move OOB check from RunRead to RunWrite, for better software pipeline.
TODO: agpr spill when NN layout

* clangformat

* A/B splitpipe scheduler for v3

* Fix two bugs

* bug fix

* fix a bug in oob check

* Example for mixed fp16_fp8 gemm

* Clean experimental code blocks

* Add mixed precision gemm into profiler

* tempsave

* optimize m/n major lds layout

* Add RRR GEMM  mixed precision instances

* Optimize f8 matrix transpose

* Add test_gemm_universal

* A/B spilt schedule for blkpip v5

* Take ds_read2 into iglp scheduling scheme

* format

* fixed cmake

* Add llvm-option into CI cmake flag

---------

Co-authored-by: Jing Zhang <jizhan@amd.com>
2024-04-13 21:03:18 -05:00
Illia Silin
cc1f733d0e allow the CI to pass even if can't connect to db (#1214) 2024-03-22 15:39:11 -07:00
Illia Silin
f52109531b Fix a couple of docker issues. (#1206)
* do not install sccache by default, only install rocm-llvm-dev for rocm6.1

* add sccache flag to docker build options
2024-03-19 08:38:52 -07:00
Illia Silin
bdcd037428 Re-enable the performance tracking in CI. (#1203)
* test CK with rocm6.1 RC2

* add docker credentials for pull

* update the performance db name

* use environment variable for db name

* add rocm-llvm-dev package to ck docker

* turn off verification for daily performance runs

* do not stash ckProfiler on MI300 node

* add processing of mixed gemms to qa, fix parsing of splitk gemm logs

* fix the splitk gemm log file name

* turn the timing on for splitk gemm performance
2024-03-18 09:48:29 -07:00
Paul Fultz II
8eff4d62b6 Add host lib (#1134)
* Format

* Format

* Format

* Remove const

* Use the right template

* Format

* Format

* add row/col instances

* Add missing file

* fixed

* Format

* Updates

* Format

* fixed rrr layout

* Format

* Update test and embed modules

* Restore older version

* Update year

* Set -fPIC

* Format

* Use double for isnan

* rename host folder to codegen + minor fix

* add codegen CI test

* add option to build components without building CK

* fix the groovy syntax

* fix typo

* use the correct function for the codegen stage

---------

Co-authored-by: Jing Zhang <jizha@amd.com>
Co-authored-by: Illia Silin <98187287+illsilin@users.noreply.github.com>
Co-authored-by: illsilin <Illia.Silin@amd.com>
2024-03-05 17:08:43 -08:00
Illia Silin
cf86621170 [CI] Add CI build and test stage on MI300. (#1185) 2024-03-05 10:42:16 -08:00
Illia Silin
a78be3f69e add docker credentials before pushing image (#1165) 2024-02-12 16:11:32 -08:00
Illia Silin
f0dd1da088 Delete any dangling images after building a new one. (#1155)
* delete dangling docker images

* fix groovy syntax

* fix groovy syntax again

* try a different way to delete dangling images
2024-02-05 10:34:47 -08:00