Copied from the llvm-project LLVM_PARALLEL_*_JOBS
Concurrent linking can break the build as well as having too many
compile jobs for the avaiable memory. These options allow the user
to fine tune the build to fit within their machines memory
constraints.
An example use on linux is
COMPILE_JOBS=`cat /proc/cpuinfo | grep -m 1 'cpu cores' | awk '{ print $4 }'`
if [ ${COMPILE_JOBS}x = x ]; then
COMPILE_JOBS=1
fi
BUILD_MEM=4
MEM_KB=0
MEM_KB=`cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'`
MEM_MB=`eval "expr ${MEM_KB} / 1024"`
MEM_GB=`eval "expr ${MEM_MB} / 1024"`
COMPILE_JOBS_MEM=`eval "expr 1 + ${MEM_GB} / ${BUILD_MEM}"`
if [ "$COMPILE_JOBS_MEM" -lt "$COMPILE_JOBS" ]; then
COMPILE_JOBS=$COMPILE_JOBS_MEM
fi
LINK_MEM=32
LINK_JOBS=`eval "expr 1 + ${MEM_GB} / ${LINK_MEM}"`
cmake -G Ninja -DCK_PARALLEL_LINK_JOBS=$LINK_JOBS
-DCK_PARALLEL_COMPILE_JOBS=$COMPILE_JOBS
Signed-off-by: Tom Rix <trix@redhat.com>
[ROCm/composable_kernel commit: efaf31061a]
* add daily build with mainline compiler
* fix the compiler paths for ci
* remove the -flto flag
* build with clang by default
[ROCm/composable_kernel commit: afe4622014]
* replace ccache with sccache, pin package versions
* put ccache back temporarily to avoid breaking other CI jobs
* add sccashe_wrapper.sh script
* fix the package version syntax
* fix the pymysql package issue
* run sccache_wrapper before build if ccache server found
* set the paths before calling the sccache_wrapper
* use /tmp instead of /usr/local for cache
* try using sccache --start-server instead of wrapper
* try using redis server with sccache
* define SCCACHE_REDIS
* add redis and ping packages, and redis port
* use the new sccache redis server
* do not use sccache with staging compiler
* fix the condition syntax
* add stunnel to redis
* add tunnel verification
* separate caches for different architectures
* fix syntax for the cache tag
* quse double brackets for conditions
* add bash line to the script
* add a switch for sccache and only use it in build stage
* run check_host function when enabling sccache
* fix the invocation tags for sccache
* fix groovy syntax
* set the invocation tag in groovy
* disable sccache in clang-format stage
* try another syntax for invocation tags
* use local sccache server if can't connect to redis
* fix script syntax
* update README
* refresh readme
* readme updates
* remove the timing and verification caveat from readme
---------
Co-authored-by: Lisa Delaney <lisa.delaney@amd.com>
[ROCm/composable_kernel commit: 4e44a9e8da]
* split the types in gemm_bilinear instances, add condition to cmake policy
* fix syntax
* split the data types in batchnorm examples
* fix the batchnorm_bwd test
* fix types in the batchnorm_bwd test
[ROCm/composable_kernel commit: 2ea75bd6d7]
* refactor cmake files for the tests
* refactor cmake files for examples
* fix cmake for gemm example
* fix the cmake file for all examples
* add splitting by data types in gemm_splitk instance header
* rename test to reflect only dl instances are used
* clean up CI workspace, update cmake for instances
* change the jenkinsfile syntax
* build all instances except DL on gfx11
* move workspace cleanup after stages
* clean up workspace after every stage
* isolate data types in grouped_conv_fwd header
* isolate dl instances for grouped_conv2d_fwd
* fix syntax
* fix cmake and batchnorm instances
* fix typo
* fix reduction instances
* fix grouped_conv headers
* fix syntax
* replace parsing logic for instances, replace bfp16 with bf16
* fix the client examples build
* clean up DTYPES from instances cmake files
* update the parsing logic in cmake files
* make an exception for reduction kernels
* update few remaining cmake files to handle DTYPES
* fix syntax
* fix cmake conflicts
* replace f8 with fp8 test name
* resolve conflicts for dpp instances
[ROCm/composable_kernel commit: bba085d2b5]
* experiment with config file
* experiment with version.h config
* add more info to version.h
* minor updates
* minor updates
* fix case where DTYPE is not used
* large amount of files but minor changes
* remove white space
* minor changes to add more MACROs
* fix cmakedefine01
* fix issue with CK internal conflict
* fix define and define value
* fix clang-format
* fix formatting issue
* experiment with cmake
* clang format v12 to be consistent with miopen
* avoid clang-format for config file
[ROCm/composable_kernel commit: c8a8385fdd]
* add fno-offload-uniform-block flag for rocm5.7 and up
* add a comment and compiler ticket number
* update the threshold rocm version
[ROCm/composable_kernel commit: cbbd172fd6]
* allow building CK for specific data types
* add CI build and test stage on Naiv3x without some int8 instances
* add missing gemm fp16 instances
* add the changes to the missed cmake file
* add empty lines at end of source files
* Do not build quantization client example on navi3 in CI
* disable batched_gemm_multi_d_int8 instances with DTYPES
* disable device_conv2d_bwd_data_instance with DTYPES
* fix ckprofiler for conv_bwd_data for int8
* properly isolate the conv_bwd_data int8 instances
* remove empty line
[ROCm/composable_kernel commit: 189ea3b9aa]
* reduce the number of default targets
* re-write the setting of target flags
* move all options to one place
* add new custom target instances for installing CK
[ROCm/composable_kernel commit: a5059f8f90]
* replace hipcc compiler with clang++
* build client app with hipcc
* build client app with clang
* add an option to build with hipcc ro clang
* fix the environment for client app
* fix setting up compiler in cmake_build
* change the way the compiler is set
[ROCm/composable_kernel commit: 1e5b59df22]
* Introduce int4 data type.
* Add unit-tests for int4
* Compile int4 UT only when int4 enabled.
* clang-format
Co-authored-by: Adam Osewski <aosewski@amd.com>
[ROCm/composable_kernel commit: e00149ac67]
* initial stub for gemm_gemm_xdl_cshuffle
* set up example code
* compiles
* prevent integer overflow
* harmonize interface between ref_gemm and ref_batched_gemm
* batched_gemm_gemm
* fix example
* host tensor gen: diagonal pattern in lowest two-dimensions only
* make c descriptors containing only integral constants
* clean up
* add BlockwiseGemmXdlops_v2 while exploring an unified approach
* implement proper interface
* tidy up example
* fix compilation warnings
* coarsely controlled 2nd gemm padding
* remove rocm-cmake's hard requirement for certain revision
* clang-format
* resolve merge conflict
* fix compilation error on gfx10
* adds acc0 elementwise op to interface
* attention host validation
* add blockwsie softmax v1
* iteratively update softmax+gemm
* transpose both gemm0 and gemm1 xdl output so as to avoid broadcasting softmax max/sum
* add init method for easier debugging
* do away with manual thread cluster calculation
* generalize blockwise softmax interface
* row-wise softmax sum & max
* format
* rename to DeviceBatchedGemmSoftmaxGemm
* add gemm_softmax_gemm instances and tests
* comment
Co-authored-by: ltqin <letao.qin@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
[ROCm/composable_kernel commit: cac014f173]
* validate examples in ctest runs
* format
* fix usage of check_err
* amend
* add example codes to custom target 'check'
Co-authored-by: Chao Liu <chao.liu2@amd.com>
[ROCm/composable_kernel commit: 9f71ff48e2]
* Use googletest for tests. Add conv2d_fwd UT.
* Add conv1D/3D to gtest UT.
* Fix: not duplicate test with CTest.
* Convert more tests to googltests.
* Fix: GIT_SHALLOW is not allowed for git commit hash.
* Clang-format
* use integer value for GEMM test
Co-authored-by: Adam Osewski <aosewski@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
Co-authored-by: Chao Liu <lc.roy86@gmail.com>
[ROCm/composable_kernel commit: 8eca05a633]
* add docker file and make default target buildable
* add Jenkinsfile
* remove empty env block
* fix package stage
* remove render group from docker run
* clean up Jenkins file
* add cppcheck as dev dependency
* update cmake file
* Add profiler build stage
* add hip_version config file for reduction operator
* correct jenkins var name
* Build release instead of debug
* Update test CMakeLists.txt
reorg test dir
add test stage
* reduce compile threads to prevent compiler crash
* add optional debug stage, update second test
* remove old test target
* fix tests to return proper results and self review
* Fix package name and make test run without args
* change Dockerfile to ues rocm4.3.1
* remove parallelism from build
* Lower paralellism
Co-authored-by: Chao Liu <chao.liu2@amd.com>
[ROCm/composable_kernel commit: 992f71e371]
* tweak conv for odd C
* update script
* clean up elementwise op
* fix build
* clean up
* added example for gemm+bias+relu+add
* added example for gemm+bias+relu
* add profiler for gemm_s_shuffle; re-org files
* add profiler
* fix build
* clean up
* clean up
* clean up
* fix build
[ROCm/composable_kernel commit: 823657ed12]
