diff --git a/example/01_gemm/README.md b/example/01_gemm/README.md index 3c757e6b46..7802c2e4ad 100644 --- a/example/01_gemm/README.md +++ b/example/01_gemm/README.md @@ -205,13 +205,6 @@ library/reference_tensor_operation/cpu/ - **DeviceGemmXdl** (in `device_gemm.hpp`): Main device API for launching GEMM kernels. - ```cpp - template - struct DeviceGemmXdl : public BaseOperator - ``` - **GridwiseGemmXdl** (in `gridwise_gemm_xdl.hpp`): Implements the tiled/blocking GEMM kernel for the GPU grid. - **BlockwiseGemmXdl** (in `blockwise_gemm_xdl.hpp`): diff --git a/example/02_gemm_bilinear/README.md b/example/02_gemm_bilinear/README.md index ae98ab73e8..6117affe72 100644 --- a/example/02_gemm_bilinear/README.md +++ b/example/02_gemm_bilinear/README.md @@ -101,13 +101,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceGemmMultipleD** (in `device_gemm_multiple_d.hpp`): Device API for GEMM with multiple auxiliary tensors and fused epilogues. - ```cpp - template - struct DeviceGemmMultipleD : public BaseOperator - ``` - **gridwise_gemm_multiple_d** (in `gridwise_gemm_multiple_d.hpp`): Implements the tiled/blocking GEMM kernel with multi-tensor epilogue. - **element_wise_operation** (in `element_wise_operation.hpp`): diff --git a/example/03_gemm_bias_relu/README.md b/example/03_gemm_bias_relu/README.md index ba32f5399d..c7bd6c597d 100644 --- a/example/03_gemm_bias_relu/README.md +++ b/example/03_gemm_bias_relu/README.md @@ -51,13 +51,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceGemmMultipleD** (in `device_gemm_multiple_d.hpp`): Device API for GEMM with auxiliary tensors and fused epilogues. - ```cpp - template - struct DeviceGemmMultipleD : public BaseOperator - ``` - **gridwise_gemm_xdl_cshuffle** (in `gridwise_gemm_xdl_cshuffle.hpp`): Implements the tiled/blocking GEMM kernel with fused epilogue. - **element_wise_operation** (in `element_wise_operation.hpp`): diff --git a/example/04_gemm_add_add_fastgelu/README.md b/example/04_gemm_add_add_fastgelu/README.md index 39b3c62740..f17556d796 100644 --- a/example/04_gemm_add_add_fastgelu/README.md +++ b/example/04_gemm_add_add_fastgelu/README.md @@ -62,13 +62,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceGemmMultipleD** (in `device_gemm_multiple_d.hpp`): Device API for GEMM with multiple auxiliary tensors and fused epilogues. - ```cpp - template - struct DeviceGemmMultipleD : public BaseOperator - ``` - **gridwise_gemm_multiple_d_xdl** (in `gridwise_gemm_multiple_d_xdl.hpp`): Implements the tiled/blocking GEMM kernel with multi-stage epilogue. - **element_wise_operation** (in `element_wise_operation.hpp`): diff --git a/example/09_convnd_fwd/README.md b/example/09_convnd_fwd/README.md index 8891149524..fb94f079ad 100644 --- a/example/09_convnd_fwd/README.md +++ b/example/09_convnd_fwd/README.md @@ -72,13 +72,6 @@ include/ck/tensor_operation/gpu/block/ - **DeviceConvNdFwd** (in `device_convnd_fwd.hpp`): Device API for N-dimensional convolution. - ```cpp - template - struct DeviceConvNdFwd : public BaseOperator - ``` - **gridwise_convnd_fwd_xdl** (in `gridwise_convnd_fwd_xdl.hpp`): Implements the tiled/blocking convolution kernel. - **blockwise_convnd_fwd_xdl** (in `blockwise_convnd_fwd_xdl.hpp`): diff --git a/example/10_convnd_fwd_multiple_d_multiple_reduce/README.md b/example/10_convnd_fwd_multiple_d_multiple_reduce/README.md index 92d3b25cd6..7b286a4703 100644 --- a/example/10_convnd_fwd_multiple_d_multiple_reduce/README.md +++ b/example/10_convnd_fwd_multiple_d_multiple_reduce/README.md @@ -51,14 +51,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceConvNdFwdMultipleDMultipleReduce** (in `device_convnd_fwd_multiple_d_multiple_reduce.hpp`): Device API for N-dimensional convolution with multiple outputs and reductions. - ```cpp - template - struct DeviceConvNdFwdMultipleDMultipleReduce : public BaseOperator - ``` - **gridwise_convnd_fwd_multiple_d_multiple_reduce** (in `gridwise_convnd_fwd_multiple_d_multiple_reduce.hpp`): Implements the tiled/blocking convolution kernel with multi-output/reduce epilogue. diff --git a/example/11_convnd_fwd_bias/README.md b/example/11_convnd_fwd_bias/README.md index faa11e1096..6fa892a939 100644 --- a/example/11_convnd_fwd_bias/README.md +++ b/example/11_convnd_fwd_bias/README.md @@ -51,14 +51,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceConvNdFwdBias** (in `device_convnd_fwd_bias.hpp`): Device API for N-dimensional convolution with bias. - ```cpp - template - struct DeviceConvNdFwdBias : public BaseOperator - ``` - **gridwise_convnd_fwd_bias** (in `gridwise_convnd_fwd_bias.hpp`): Implements the tiled/blocking convolution kernel with bias epilogue. diff --git a/example/12_reduce/README.md b/example/12_reduce/README.md index 49ba52a7e1..49042190dd 100644 --- a/example/12_reduce/README.md +++ b/example/12_reduce/README.md @@ -117,12 +117,6 @@ include/ck/tensor_operation/gpu/block/ - **DeviceReduce** (in `device_reduce.hpp`): Device API for reductions. - ```cpp - template - struct DeviceReduce : public BaseOperator - ``` - **gridwise_reduce** (in `gridwise_reduce.hpp`): Implements the tiled/blocking reduction kernel. - **blockwise_reduce** (in `blockwise_reduce.hpp`): diff --git a/example/13_pool2d_fwd/README.md b/example/13_pool2d_fwd/README.md index 212a91e66c..2fbe75fe14 100644 --- a/example/13_pool2d_fwd/README.md +++ b/example/13_pool2d_fwd/README.md @@ -98,12 +98,6 @@ include/ck/tensor_operation/gpu/block/ - **DevicePoolFwd** (in `device_pool_fwd.hpp`): Device API for pooling. - ```cpp - template - struct DevicePoolFwd : public BaseOperator - ``` - **gridwise_pool_fwd** (in `gridwise_pool_fwd.hpp`): Implements the tiled/blocking pooling kernel. - **blockwise_pool** (in `blockwise_pool.hpp`): diff --git a/example/14_gemm_quantization/README.md b/example/14_gemm_quantization/README.md index a6d7efd6ee..e2bee7f22f 100644 --- a/example/14_gemm_quantization/README.md +++ b/example/14_gemm_quantization/README.md @@ -52,14 +52,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceGemmQuantized** (in `device_gemm_quantized.hpp`): Device API for quantized GEMM. - ```cpp - template - struct DeviceGemmQuantized : public BaseOperator - ``` - **gridwise_gemm_quantized** (in `gridwise_gemm_quantized.hpp`): Implements the tiled/blocking quantized GEMM kernel. - **quantization_operations** (in `quantization_operations.hpp`): diff --git a/example/15_grouped_gemm/README.md b/example/15_grouped_gemm/README.md index 14429e213a..fbcc13fe84 100644 --- a/example/15_grouped_gemm/README.md +++ b/example/15_grouped_gemm/README.md @@ -58,13 +58,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceGroupedGemmXdl** (in `device_grouped_gemm_xdl.hpp`): Device API for grouped GEMM. - ```cpp - template - struct DeviceGroupedGemmXdl : public BaseOperator - ``` - **gridwise_grouped_gemm_xdl** (in `gridwise_grouped_gemm_xdl.hpp`): Implements the tiled/blocking grouped GEMM kernel. diff --git a/example/16_gemm_multi_d_multi_reduces/README.md b/example/16_gemm_multi_d_multi_reduces/README.md index 4d4a44b718..a24bc91751 100644 --- a/example/16_gemm_multi_d_multi_reduces/README.md +++ b/example/16_gemm_multi_d_multi_reduces/README.md @@ -50,14 +50,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceGemmMultiDMultiReduces** (in `device_gemm_multi_d_multi_reduces.hpp`): Device API for GEMM with multiple outputs and reductions. - ```cpp - template - struct DeviceGemmMultiDMultiReduces : public BaseOperator - ``` - **gridwise_gemm_multi_d_multi_reduces** (in `gridwise_gemm_multi_d_multi_reduces.hpp`): Implements the tiled/blocking GEMM kernel with multi-output/reduce epilogue. diff --git a/example/19_binary_elementwise/README.md b/example/19_binary_elementwise/README.md index 14910673fc..0245576c0a 100644 --- a/example/19_binary_elementwise/README.md +++ b/example/19_binary_elementwise/README.md @@ -30,23 +30,8 @@ The implementation for both cases relies on the efficient **grid-stride loop**, 1. **Grid Partitioning**: The problem is mapped to a 1D grid of threads based on the number of elements in the **output** tensor. 2. **Grid-Stride Loop**: Each thread iterates through a subset of the output elements. For each output index, it must calculate the corresponding indices into the input tensors A and B. - ``` - for (int out_idx = ...; out_idx < N_out; out_idx += ... ) - { - // Map 1D output index to multi-dimensional coordinates (i, j, k...) - Coord out_coord = get_coord(out_idx); - // Calculate input indices based on broadcasting rules - Coord a_coord = get_broadcast_coord(out_coord, shape_A); - Coord b_coord = get_broadcast_coord(out_coord, shape_B); - - // Map multi-dimensional input coords back to 1D indices - int a_idx = get_idx(a_coord, strides_A); - int b_idx = get_idx(b_coord, strides_B); - - C[out_idx] = A[a_idx] op B[b_idx]; - } - ``` +3. **Broadcasting Logic**: - The core of the broadcasting logic lies in the `get_broadcast_coord` function. If an input tensor's dimension is 1, the coordinate for that dimension is always set to 0, effectively reusing the same element across the broadcast dimension. If the dimension matches the output, the coordinate is passed through. - This strategy ensures that memory accesses to the larger tensor remain coalesced, while accesses to the smaller, broadcasted tensor will naturally involve re-reading the same values, which is efficiently handled by the GPU's cache hierarchy. diff --git a/example/21_gemm_layernorm/README.md b/example/21_gemm_layernorm/README.md index 197ae2a86e..b1f7db6e8d 100644 --- a/example/21_gemm_layernorm/README.md +++ b/example/21_gemm_layernorm/README.md @@ -51,13 +51,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceGemmLayerNorm** (in `device_gemm_layernorm.hpp`): Device API for GEMM fused with layer normalization. - ```cpp - template - struct DeviceGemmLayerNorm : public BaseOperator - ``` - **gridwise_gemm_layernorm** (in `gridwise_gemm_layernorm.hpp`): Implements the tiled/blocking GEMM kernel with layer normalization epilogue. diff --git a/example/23_softmax/README.md b/example/23_softmax/README.md index d50bb0789c..9703d048a7 100644 --- a/example/23_softmax/README.md +++ b/example/23_softmax/README.md @@ -67,12 +67,6 @@ include/ck/tensor_operation/gpu/block/ - **DeviceSoftmax** (in `device_softmax.hpp`): Device API for softmax. - ```cpp - template - struct DeviceSoftmax : public BaseOperator - ``` - **gridwise_softmax** (in `gridwise_softmax.hpp`): Implements the tiled/blocking softmax kernel. - **blockwise_softmax** (in `blockwise_softmax.hpp`): diff --git a/example/24_batched_gemm/README.md b/example/24_batched_gemm/README.md index fddfae8db8..de8b96d4d3 100644 --- a/example/24_batched_gemm/README.md +++ b/example/24_batched_gemm/README.md @@ -51,13 +51,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceBatchedGemmXdl** (in `device_batched_gemm_xdl.hpp`): Device API for batched GEMM. - ```cpp - template - struct DeviceBatchedGemmXdl : public BaseOperator - ``` - **gridwise_batched_gemm_xdl** (in `gridwise_batched_gemm_xdl.hpp`): Implements the tiled/blocking batched GEMM kernel. diff --git a/example/25_gemm_bias_e_permute/README.md b/example/25_gemm_bias_e_permute/README.md index 34849ddf6c..b3f85c7d50 100644 --- a/example/25_gemm_bias_e_permute/README.md +++ b/example/25_gemm_bias_e_permute/README.md @@ -50,12 +50,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceGemmBiasEPermute** (in `device_gemm_bias_e_permute.hpp`): Device API for GEMM fused with bias, elementwise, and permutation. - ```cpp - template - struct DeviceGemmBiasEPermute : public BaseOperator - ``` - **gridwise_gemm_bias_e_permute** (in `gridwise_gemm_bias_e_permute.hpp`): Implements the tiled/blocking GEMM kernel with fused epilogue and permutation. diff --git a/example/27_layernorm2d_fwd/README.md b/example/27_layernorm2d_fwd/README.md index d43e5b49fa..4420a3aa4d 100644 --- a/example/27_layernorm2d_fwd/README.md +++ b/example/27_layernorm2d_fwd/README.md @@ -53,12 +53,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceLayernormFwd** (in `device_layernorm_fwd.hpp`): Device API for layer normalization. - ```cpp - template - struct DeviceLayernormFwd : public BaseOperator - ``` - **gridwise_layernorm_fwd** (in `gridwise_layernorm_fwd.hpp`): Implements the tiled/blocking layernorm kernel. diff --git a/example/32_batched_gemm_scale_softmax_gemm/README.md b/example/32_batched_gemm_scale_softmax_gemm/README.md index 904e965ec9..0c22a5c92a 100644 --- a/example/32_batched_gemm_scale_softmax_gemm/README.md +++ b/example/32_batched_gemm_scale_softmax_gemm/README.md @@ -53,13 +53,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceBatchedGemmScaleSoftmaxGemm** (in `device_batched_gemm_scale_softmax_gemm.hpp`): Device API for fused attention. - ```cpp - template - struct DeviceBatchedGemmScaleSoftmaxGemm : public BaseOperator - ``` - **gridwise_batched_gemm_softmax** (in `gridwise_batched_gemm_softmax.hpp`): Implements the tiled/blocking fused attention kernel. - **gridwise_online_softmax** (in `gridwise_online_softmax.hpp`): diff --git a/example/34_batchnorm/README.md b/example/34_batchnorm/README.md index 1d0a800b3c..45e123ea44 100644 --- a/example/34_batchnorm/README.md +++ b/example/34_batchnorm/README.md @@ -133,12 +133,6 @@ include/ck/tensor_operation/gpu/grid/ - **DeviceBatchnormFwd** (in `device_batchnorm_fwd.hpp`): Device API for batch normalization. - ```cpp - template - struct DeviceBatchnormFwd : public BaseOperator - ``` - **gridwise_batchnorm_fwd** (in `gridwise_batchnorm_fwd.hpp`): Implements the tiled/blocking batchnorm kernel. diff --git a/example/39_permute/README.md b/example/39_permute/README.md index 9555b878c9..e408e94dd4 100644 --- a/example/39_permute/README.md +++ b/example/39_permute/README.md @@ -50,11 +50,6 @@ include/ck/tensor_operation/gpu/grid/ - **DevicePermute** (in `device_permute.hpp`): Device API for tensor permutation. - ```cpp - template - struct DevicePermute : public BaseOperator - ``` - **gridwise_permute** (in `gridwise_permute.hpp`): Implements the tiled/blocking permutation kernel. diff --git a/example/40_conv2d_fwd_quantization/README.md b/example/40_conv2d_fwd_quantization/README.md index 8467dfdc15..dca90502d2 100644 --- a/example/40_conv2d_fwd_quantization/README.md +++ b/example/40_conv2d_fwd_quantization/README.md @@ -53,14 +53,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceConv2dFwdQuantization** (in `device_conv2d_fwd_quantization.hpp`): Device API for quantized 2D convolution. - ```cpp - template - struct DeviceConv2dFwdQuantization : public BaseOperator - ``` - **gridwise_conv2d_fwd_quantization** (in `gridwise_conv2d_fwd_quantization.hpp`): Implements the tiled/blocking quantized conv2d kernel. - **quantization_operations** (in `quantization_operations.hpp`): diff --git a/example/44_elementwise_permute/README.md b/example/44_elementwise_permute/README.md index 1c2c68c1fc..ed49672508 100644 --- a/example/44_elementwise_permute/README.md +++ b/example/44_elementwise_permute/README.md @@ -51,11 +51,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceElementwisePermute** (in `device_elementwise_permute.hpp`): Device API for fused elementwise and permutation. - ```cpp - template - struct DeviceElementwisePermute : public BaseOperator - ``` - **gridwise_elementwise_permute** (in `gridwise_elementwise_permute.hpp`): Implements the tiled/blocking elementwise+permute kernel. - **element_wise_operation** (in `element_wise_operation.hpp`): diff --git a/example/46_gemm_add_multiply/README.md b/example/46_gemm_add_multiply/README.md index 42b34326f5..0c1992f6ff 100644 --- a/example/46_gemm_add_multiply/README.md +++ b/example/46_gemm_add_multiply/README.md @@ -60,13 +60,6 @@ include/ck/tensor_operation/gpu/element/ - **DeviceGemmMultipleD** (in `device_gemm_multiple_d.hpp`): Device API for GEMM with multiple auxiliary tensors and fused epilogues. - ```cpp - template - struct DeviceGemmMultipleD : public BaseOperator - ``` - **gridwise_gemm_multiple_d_xdl** (in `gridwise_gemm_multiple_d_xdl.hpp`): Implements the tiled/blocking GEMM kernel with multi-stage epilogue. - **element_wise_operation** (in `element_wise_operation.hpp`):