* update the performance test logic
* fix unstash perf logs logic
* untangle unstashing fmha logs for different archs
* run process stage after running fmha tests
* fix the processing of perf logs
* fix arguments for run_performance scripts
* merge the build and performance tests CI stages together
* add gemm performance test on gfx11/gfx12
* add suffices to distinguish gemm performance logs from different archs
* use smaller gemm set in CI for gfx10/gfx11/gfx12
* disable performance tests on gfx1030
* fix the shashing logic
* fix finding python3 for mha instances
* test CK with rocm6.1 RC2
* add docker credentials for pull
* update the performance db name
* use environment variable for db name
* add rocm-llvm-dev package to ck docker
* turn off verification for daily performance runs
* do not stash ckProfiler on MI300 node
* add processing of mixed gemms to qa, fix parsing of splitk gemm logs
* fix the splitk gemm log file name
* turn the timing on for splitk gemm performance
* build CK only once, use deb package in all subsequent stages
* update jenkins file
* change prefix for build_CK stage
* update writing deb metadata to control file
* update ubuntu source for docker, script syntax for deb package metadata
* try different way to create deb metadata
* clean up DEBIAN before creating one
* fix the CI folder names, fix splitK qa
* use correct docker in all stages, separate tests for splitK verification and performance
* clean old comments, change dir before packaging
* use different package syntax
* change packaging syntax
* package with cmake
* remove unnecessary build prefix
* get rid of unnecessary paths
* change paths during unpacking
* change script syntax while unpacking
* get rid of unneccesary steps
* get rid of comments in the scripts
* use double quotes for scripts
* add ccache during build, try dpkg -x
* pull and install each package separately
* use full package names
* try to use stashing for packages
* change stash/unstash syntax
* move unstash out of shell, run tests on any gpu node
* unpack each package separately
* try re-using existing workspace
* merge the build and test stages, only stash ckProfiler
* merge the build and test stages, only stash zipped ckProfiler
* fix syntax
* add GPU check before build and test, rename docker to usual name
* upgrade the OS and ROCM versions in CK docker
* add cxx flags to link code with rocm5.2 and ck-9110 compiler
* rename the docker image
* run ONNX gemms using init=1
* add processing for the onng_gemm and splitK_gemm
* add profile_onnx_gemm.sh
* add stderr to logfiles, add splitK and onnx gemm parsing
* enable splitK gemm wresults posting to db
* turn on full qa only on gfx90a, use int initialization
* change script syntax
* update script parsing clinfo, throw exception if 0 devices
* fix syntax
* try using toBoolean for the QA conditions
* run regular CI on MI100 only, use MI200 only for daily QA
* evaluate when conditions before agent
* launch QA on develop branch and update profile_reduce script
* update test script
* update script
* remove false dependency from dockerfile
* try removing rbuild completely
Co-authored-by: Chao Liu <chao.liu2@amd.com>
Co-authored-by: Chao Liu <lc.roy86@gmail.com>
* add verify flag and update scripts
* replace old check_error function with the new check_err
* fix syntax
* remove blank spaces
* remove empty line
* add check_err for tensors
* fix syntax
* replace tensors with vectors in check_err calls
* fix syntax
* remove blank spaces
* fix syntax
* add new line at end of file
* disable conv2d_bwd_weight test, add gpu check
* set check_gpu using export
* check GPU using runShell
* add definition of runShell
* fix script syntax
* reduce the number of threads, add full qa option
* run processing scripts in bash
* fix the branch and host names in performance scripts, add chronos
* replace parameterizedCron with cron
* archive the perf log files
* try to fix git call
* pass branch and host names as arguments into scripts
* fix script arguments
* fix script arguments
* process results on master
* fix pipeline
* add definition of gpu_arch
* run processing scripts in docker
* fix the brackets
* add agent master for the processing stage
* get rid of show_node_info call on master
* try using mici label instead of master, disable MI100 tests for now
* fix syntax
* simplify container for results processing
* remove node(master) from the process_results stage
* put all stages in original order
* change the agent label from master to mici for gfx908
* adding scripts for full perf test suite
* uncomment the sql queries
* fix typo and chmod a+x for scripts
* dos2unix for all new scripts
* disable verification in full performance test
* fix reduction scripts, add gfrouped_gemm hotfix
* fix the grouped_gemm hotfix and only run reduction for fp16
* change compiler flag syntax
* fix syntax
* add predefinition of dockerArgs
* avoid redefinitions of dockerArgs
* add blank space at the end of dockerArgs
* try to build with release compiler
* adding spaces inside if condition
* limit the number of threads for building 9110 compiler
* change the way HIP_CLANG_PATH is set
* remove the export command
* change the conditional ENV syntax
* set HIP_CLANG_PATH at docker run time
* update scripts for full qa
* enable the sql write query
* fix typo
* remove a comment from a script
