ROCm/composable_kernel
1
0
Fork 0
mirror of https://github.com/ROCm/composable_kernel.git synced 2026-05-12 09:16:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
502 Commits 741 Branches 38 Tags
6dfb92bbef33b4caea55f6b4ed7c449927ae771c
Commit Graph

54 Commits

Author SHA1 Message Date
Jianfeng Yan
6dfb92bbef Conv3d new (#94) 
* conv3d compiles but has memory error

* conv3d works

* fix performance issue by using __builtin_amdgc_readfirstlane

* change MakeBlock2CTileMap to MakeDefaultBlock2CTileMap; change c_blockid_to* to cblockid_to*

* clang-format

* remove CK_EXPERIMENTAL_PASS_TENSOR_DECRIPTOR_BY_*; moved wrapper into DeviceConv3d

* format

* remove useless marc

* add comment

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2022-02-22 22:45:28 -06:00
JD
2778e99758 Initial Setup for CI (#86) 
* add docker file and make default target buildable

* add Jenkinsfile

* remove empty env block

* fix package stage

* remove render group from docker run

* clean up Jenkins file

* add cppcheck as dev dependency

* update cmake file

* Add profiler build stage

* add hip_version config file for reduction operator

* correct jenkins var name

* Build release instead of debug

* clean up

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2022-02-18 21:44:11 -06:00
ltqin
880fbee957 NHWC conv 2d: fwd bfp16/int8, Device level tuning and host API (#73) 
* add fwd bf16 conv

* change tunning parametor

* add int8 for conv fwd

* remove comments

* change tunning parametor for int8

* change init int8 example

* add test for conv2d fwd

* change device operation file pos because merge develop

* fwd int8 use reference

* test_conv_fwd use reference

* add braket for if statement

* rename fwd example name

* remove StaticBufferOfVectorTypeV2

* tweak example

Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2022-02-11 20:06:40 -06:00
Anthony Chang
904cbe2a8f fix build breaks (#81) 
- device_gemm_xdl_c_shuffle function signature matches split-k
- retire host_driver since it is no longer maintained
- linter error (unused variable)

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2022-02-10 23:52:19 -06:00
Chao Liu
823657ed12 GEMM+Bias+ReLU+Add (#76) 
* tweak conv for odd C

* update script

* clean up elementwise op

* fix build

* clean up

* added example for gemm+bias+relu+add

* added example for gemm+bias+relu

* add profiler for gemm_s_shuffle; re-org files

* add profiler

* fix build

* clean up

* clean up

* clean up

* fix build
 2022-02-06 22:32:47 -06:00
ltqin
690c75a7eb References for conv2d fwd bias relu and add (#75) 
* add reference

* clean up

* add reference for conv

* rename

Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2022-02-03 22:29:58 -06:00
zjing14
6d92959ad3 Replace llvm Intrinsics with clang buildins (#65) 
* test mfma builtins

* add fp16 buildins

* add int8 buildins

* add bfl16 buildins

* simplify host conv forward

* clean

* clean
 2022-02-02 23:13:09 -06:00
Chao Liu
acbd7bd7c5 Fusion Conv+Bias+ReLU(+Add) (#62) 
* fix relu

* clean up

* clean up

* adding 1x1 conv

* adding 1x1 conv

* added 1x1 conv

* refactor

* refactor

* refactor

* added profiler for conv+bias+relu+add

* clean up

* adding conv+bias+relu

* adding conv+bias+relu

* added conv+bias+relu

* Update README.md

* update cpu verification

* adding c shuffle

* update static_tensor for dealing with invalid element

* adding c shuffle

* debugging

* fix bug

* convert to fp16 before shuffle

* shuffle more than one M/NRepeat

* clean up

* remove coordinate step hack from GridwiseGemm_k0mk1_k0nk1_mn_xdlops_v3r1

* clean up

* remove coordinate step hack from all gridwise gemm xdl

* clean up coordinate step hack

* clean up coordinate step hack

* ThreadwiseTensorSliceTransfer_v3r2 support pointwise op on both src and dst

* adding output shuffle in conv+bias+relu+add

* update

* added conv+bias+relu+add with c shuffle

* added conv+bias+relu+add with c shuffle

* fix forward_sweep bugs in threadwise copy

* clean up

* refactor

* clean up

* clean up

* added conv_c_shuffle+bias_relu

* clean up

* added conv+bias+relu+atomic_add

* clean up

* clean up

* clean up

* clean up

* clean up

* clean up

* misc fixes; add 1x1 specialization

* clean up

* delete unused device op

* clean up

* add support for odd C value
 2021-12-26 07:43:42 -07:00
Chao Liu
41cdd3801a GEMM/Conv+BiasAdd+ReLU+Add (#55) 
* gemm+activation

* move C pointwise operation into threadwise copy

* add pointwise operation to A/B matrix

* update ckProfiler

* adding bias add

* adding bias add

* adding bias add

* added bias add; worked around compiler issues

* clean up

* clean up

* Update README.md

* Update README.md

* Update README.md

* clean up

* add conv_xdl example

* adding conv_xdl_bias_relu_add example

* add conv+bias+relu+add, but has register spill issue

* tweak

* tweak

* refactor

* Update README.md

update readme for example/2_gemm_xdl_bias_relu_add

* clean up

* Update README.md

update readme for example/3_conv_xdl

* Update README.md
 2021-12-02 20:07:37 -06:00
Jing Zhang
d798c9b8c6 fixed c_buffer alloc 2021-12-02 05:03:03 +00:00
Chao Liu
4041850f11 fix layout naming convention (#56) 2021-11-30 09:10:55 -06:00
Chao Liu
64350affc5 Use __builtin_memcpy to implement bit_cast and for accessing vector from pointer of scalars (#53) 
* reworking vector_type

* use __builtin_memcpy for bit_cast and vector access of scalar pointer

* clean up
 2021-11-18 09:11:15 -06:00
zjing14
970fa3e92e v5r1 fusion kernels for inference (#49) 
* init

* refactor for 1x1

* rename e0_e1

* add e1 with bugs

* debug

* fixed

* fixed e1

* add timer

* imprve threadwise gemm with dot2

* add e2

* tuning

* seperate c2

* add nhwc

* restore nchwc

* clean

* opt

* fixed; tuning

* add BGlobalMoveSliceWindowStepHacks{}

* tuning

* repeat running

* adjust

* merge v5r1 nchwc

* add adaptors

* split k0 k1 in c_thread_grid

* split h and w

* remove v5r1 nhwc

* clean for pr

* remove host_conv_add

* clean code

* clean

* add dynamic support

* static mode

* test static

* add conv+add fusion

* fixed validation

* naming fix

* use activ_enum

* make static

* refactor conv_add for InMem::add

* add bias

* add conv_out

* add configurable makeddesc

* add maxpool fusion

* add maxpool host for validation

* enable static desc

* conv-only use v5r1_add

* test

* test

* for binary dumps

* fixed incorrect results due to typo

* clean

* debugging maxpool

* workaround with offset trick

* clean code

* modularize ops of fusion

* add gridwise_gemm_v3

* create seperate fusion fun

* enable dynamic mode of conv and conv+resize_add

* add dynamic mode of maxpool

* add pass by point

* add activ_type as arguments

* merge develop

* clean

* reset config to old default

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2021-11-18 08:34:07 -06:00
zjing14
a651ea4f7a Fixed bfp16 host_conv_fwd (#52) 
* fixed bfloat16 issues

* refactor type_convert

* fixed host_convolution_forward for ushort

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2021-11-18 08:10:56 -06:00
zjing14
0a66c54e95 fixed multiple definition issue of bfp16/fp32 conversion function when building ckProfiler (#51) 
* fixed bfloat16 issues

* refactor type_convert

Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2021-11-16 15:44:17 -06:00
Jing Zhang
89e1ebd4d5 updated bfloat16_to_float 2021-11-16 18:01:25 +00:00
zjing14
3737bb039a Add bfp16/int8 support into XDL GEMM operator (#50) 
* init StaticBufferV2

* clean

* adopt old output stage for staticBufferV2

* clean

* remove hack

* clean

* clean

* add parameters

* clean code

* move c_buffer alloc into blockwise gemm

* add adaptors for m/n_thread_data_on_grid

* tweak gemm

* adjust blockwise_gemm_xdlops

* tweak

* update conv

* update script

* adding bwd 1x1

* update script

* adding 1x1 bwd

* debugging bwd 1x1 failure

* update script

* update script

* test

* test v100

* add bf16_1k

* clang-format

* clean

* add bfp16 for gfx908

* add verification

* clean up

* clean code

* restore bfl16

* clean

* add bfp16 support into gemm_driver

* apply new generator to other drivers

* add int8 support

* cleanb

* clean

* clean

* clean

Co-authored-by: Chao Liu <chao.liu2@amd.com>
Co-authored-by: Chao Liu <lc.roy86@gmail.com>
Co-authored-by: root <root@hayabusa6111.amd.com>
 2021-11-15 10:24:39 -06:00
Chao Liu
b491ebf384 FP16 data in-register transpose (#41) 
* start fixing 16bit data packing

* adding StaticTensor

* adding StaticTensor

* adding StaticTensor

* add missing constexpr

* adding static tensor

* adding static tensor

* adding transpose

* add inline asm for transpose 2x2 of half_t

* add general transpose_vectors(), but have unnecessary register initialization using v_mov

* fix unnecessary register initialization in transpose_vector by using more pass-by-reference

* add hardcoded logic for NHWC wrw

* improve asm for v_pack

* make ThreadwiseTensorSliceTransfer_v3r2 support any tensor

* tweak

* reorganize file
 2021-11-15 10:05:58 -06:00
Chao Liu
e823d518cb ckProfiler and device-level XDL GEMM operator (#48) 
* add DeviceGemmXdl

* update script

* fix naming issue

* fix comment

* output HostTensorDescriptor

* rename

* padded GEMM for fwd v4r4r4 nhwc

* refactor

* refactor

* refactor

* adding ckProfiler

* adding ckProfiler

* refactor

* fix tuning parameter bug

* add more gemm instances

* add more fp16 GEMM instances

* fix profiler driver

* fix bug in tuning parameter

* add fp32 gemm instances

* small fix

* refactor

* rename

* refactor gemm profiler; adding DeviceConv and conv profiler

* refactor

* fix

* add conv profiler

* refactor

* adding more GEMM and Conv instance

* Create README.md

Add build instruction for ckProfiler

* Create README.md

Add Readme for gemm_xdl example

* Update README.md

Remove build instruction from top most folder

* Update README.md

* clean up
 2021-11-14 11:28:32 -06:00
ltqin
6014185ac6 [Bug Fix] GridwiseGemm_bk0mk1_bk0nk1_mn_xdlops_v2r4 loop issue (#44) 
* change method computering kpad

* remove unusing variable: batchlen

* change KPerBlock to K0PerBlock

* fix bug for k0 == k0perblock

* fix bug for get k0 index

* use math::integer_divide_ceil

Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2021-10-27 09:39:18 -05:00
Chao Liu
d5297abae9 fix bug in gridwise gemm xdlops v2r3 (#45) 2021-10-21 16:42:24 -05:00
ltqin
fd49ff8080 add nchw atomic , nhwc and nhwc atomic method for backward weight (#30) 
* add add new algorithm from v4r4r2

* program once issue

* add split k functiion

* redefine code

* add a matrix unmerge

* add b matrix unmerge k0

* trans a and b to gridegemm

* nhwc init

* no hacks and vector load

* add hacks

* modify some parameter

* fix tuning prometer for fp32

* fix tuning prometer for fp16

* start change gridwise k split

* init ok

* revome a b matrix k0mk1 desc in grid

* carewrite lculate gridsize

* add kbatch to CalculateBottomIndex

* remove some unused funtion

* add clear data function before call kernel

* out hacks

* in hacks

* rename device convolution file and function name

* modify kBatch value

* fix some tuning code

* start from v4r4 nhwc

* nhwc atomic is able to run

* just for fp32

* enable nchw atomic

* tweak

* tweak

* re-arrange gridwise gemm hot loop for wrw

* add wrw v4r5

* v4r4r5 fp16

* v4r4r4 fp16

* v4r4r2 fp16

* V4R4R4XDLNHWC fp16

* V4R4R2XDLATOMICNCHW fp16

* adjust for fp16

* input gridsize

* change kbatch to gridsize

* testing wrw

* clean up

* k_batch to gridsize

* fix bug

* wrw v4r4r4 kbatch change to gride size

* wrw v4r4r2 kbatch change to gride size

* after merge , change gridwise gemm v2r4

* change MakeCBlockClusterAdaptor

* other method use new gridwise gemm

* clean up

* chapad method nge to make_right_pad_transform

* kbatch out from transform function

* clean up and fix bug

* fix bug

* using function type reduce template parameters

* using auto replace define fuction type

* clean up

Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
Co-authored-by: Jing Zhang <jizhan@amd.com>
 2021-10-19 18:42:34 -05:00
Chao Liu
b3e8d57d51 Tweak GEMM kernel (#38) 
* add parameters

* tweak gemm

* tweak

* update conv

* update script

* adding bwd 1x1

* update script

* adding 1x1 bwd

* debugging bwd 1x1 failure

* update script

* update script

* test

* test v100

* clean up
 2021-10-06 11:12:36 -05:00
Chao Liu
19613902b5 GEMM driver and kernel (#29) 
* add gemm driver

* tweak

* add gemm kernel: mk_kn_mn and km_kn_mn

* tweak

* add GEMM km_nk_mn

* fix comment
 2021-09-05 12:41:28 -05:00
ltqin
627d8ef35a Backward weight v4r4r2 with xdlops (#18) 
* start

* modify transformat

* modify device convolutiion

* modify host

* added host conv bwd and wrw

* remove bwd, seperate wrw

* clean

* hacall k to zero

* out log

* fixed

* fixed

* change to (out in wei)

* input hack

* hack to out

* format

* fix by comments

* change wei hacks(wei transform has not merge)

* fix program once issue

* fix review comment

* fix vector load issue

* tweak

Co-authored-by: ltqin <letaoqin@amd.com>
Co-authored-by: Jing Zhang <jizhan@amd.com>
Co-authored-by: Chao Liu <chao.liu2@amd.com>
 2021-08-30 22:49:17 -05:00
zjing14
9d3f634a3c Xdlops refactor fix (#22) 
* added constexpr ahead of adptor; clean unused driver; rename M/NPerWave to M/NPerXDL

* fixed bwd

* fixed comment
 2021-08-23 11:22:10 -05:00
zjing14
a2ad6d3531 refactor dynamic xdlops iGemm (#13) 
* xdlops refactor

* fixed commnt

* clean xdlops_gemm

* add make c into xldops-gemm

* change mfma_info

* refactor xdlops, hide c desc

* clean

* clean

* clean

* apply hacks changes to v4r4r4_nhwc

* rename hacks and use single stage adapter

* enable fp16 mfma
 2021-08-19 09:54:10 -05:00
zjing14
ba6f79a75e Added host_conv_wrw for verification (#15) 
* added host conv wrw
 2021-08-19 01:00:41 -05:00
Chao Liu
a91b68dfcb DynamicBuffer, StaticBuffer, amd_buffer_load support customized value for invalid element 2021-08-13 23:40:19 +00:00
Chao Liu
0834bc7635 compiler parameter use stream 2021-08-13 01:05:14 +00:00
Chao Liu
f2ac7832c6 make innner product compatiable on gfx900 2021-08-11 09:42:53 -05:00
Chao Liu
4e57b30a6a rename 2021-08-11 00:08:42 +00:00
Chao Liu
c03045ce2d rename 2021-08-10 23:45:36 +00:00
Chao Liu
2c48039d0e fix kernel filename 2021-08-10 22:15:23 +00:00
Chao Liu
ddd49ec9e7 fix clang warning suppression 2021-08-10 06:20:24 +00:00
Chao Liu
4f566c6221 vector/scalar pointer cast use c-style pointer cast instead of reinterpret_cast 2021-08-10 05:55:20 +00:00
Chao Liu
172036d728 add c-style pointer cast 2021-08-10 00:01:52 -05:00
Chao Liu
76f3131939 tidy 2021-08-09 18:49:59 -05:00
Chao Liu
d18428901e tidy 2021-08-09 18:20:02 -05:00
Chao Liu
f885c131d8 tidy 2021-08-09 22:13:47 +00:00
Chao Liu
c3efeb5e20 tidy 2021-08-09 19:32:07 +00:00
Chao Liu
56fc0842b3 tidy 2021-08-09 19:27:49 +00:00
Chao Liu
54fba515b3 tidy 2021-08-09 17:33:32 +00:00
Chao Liu
e62bae7a4a tidy 2021-08-09 15:11:35 +00:00
Chao Liu
61487e0a00 fix 2021-08-07 02:31:19 +00:00
Chao Liu
ae98b52ad8 remove online compilation from CK 2021-08-07 00:51:05 +00:00
Chao Liu
cb95421311 refactor 2021-08-06 22:17:51 +00:00
Chao Liu
7b1ec41e5b refactor 2021-08-06 20:50:01 +00:00
Chao Liu
49c33aaea7 refactor 2021-08-06 19:59:53 +00:00
Chao Liu
54b3e73d17 rename 2021-08-06 18:07:15 +00:00