# Grouped Gemm Grouped General Matrix Multiplication (Grouped GEMM) is a technique used in GPU computing and high-performance computing to batch together multiple independent GEMM operations (matrix multiplications) into a single kernel launch in order to improve performance and efficiency. This folder contains Grouped GEMM examples that use the ck_tile tile-programming implementation. ## Quick Tour for New Users The `Grouped GEMM` operators are versions of GEMM that run multiple GEMM operations within a single kernel call. Each GEMM operation performs a matrix multiplication. Unlike regular batched GEMM operations where both matrices must be of the same size and have the same configuration, Grouped GEMM operations can take matrices with different sizes and configurations, making them more flexible for diverse workloads. Let's now break the example into the following parts: parsing arguments, preparing host and device buffers, preparing data, invoking GEMM, and building the example, while explaining each function. ### Parsing Arguments The example takes three arguments: `group_count`, `repeat`, and `warmup`: - `group_count`: the number of GEMM operations in the group, - `repeat`: the number of times to repeat the kernel for benchmarking - `warmup`: the number of iterations before the actual kernel run time measure. ```cpp // Example const int group_count = arg_parser.get_int("group_count"); const int repeat = arg_parser.get_int("repeat"); const int warmup = arg_parser.get_int("warmup"); ``` In the next step, the input parameters `Ms`, `Ns`, `Ks`, as well as the corresponding `stride_As`, `stride_Bs`, and `stride_Cs` are either provided from the comand line or generated by default. Since one or more input data sets are expected for `A` and `B`, each parameter is stored in a `std::vector`. The size of the `vector` is defined by `group_count`. ```cpp // Example std::vector Ms = arg_parser.get_int_vec("Ms"); std::vector Ns = arg_parser.get_int_vec("Ns"); std::vector Ks = arg_parser.get_int_vec("Ks"); std::vector stride_As = arg_parser.get_int_vec("stride_As"); std::vector stride_Bs = arg_parser.get_int_vec("stride_Bs"); std::vector stride_Cs = arg_parser.get_int_vec("stride_Cs"); ``` Where: - `Ms` is the M dimension of each GEMM. - `Ns` is the N dimension of each GEMM. - `Ks` is the K dimension of each GEMM. - `stride_As` is the stride values for matrix A. - `stride_Bs` is the stride values for matrix B. - `stride_Cs` is the stride values for matrix C. ### HostTensor and Device Memory Buffers (for CPU and GPU) Each parameter `Ms`, `Ns`, `Ks`, `stride_As`, `stride_Bs` and `stride_Cs` contains values for more than one matrix, meaning different matrix sizes and strides can be used for different grouped GEMM computations. The next step is to properly load the input values. For each input matrix, `A` and `B`, and for each output matrix, `C`, you need to create both `HostTensor` and `DeviceMemory`, where: - `HostTensor` represents the matrix data on the host (CPU). It stores the data before they are transferred to the device for computation. - `DeviceMemory` represents the matrix data on the device (GPU). This will store the data on the GPU for computation during the Grouped GEMM operation. #### HostTensor Buffers (for CPU) In the first step, create `HostTensor` for `A`, `B`, `C`. `HostTensor` allocates memory on the host (CPU) to store the matrices, initializing the memory with the appropriate dimensions and values to store the data. Below is an example code showing how to create HostTensors for those tensors: ```cpp // Example std::vector> a_m_k_tensors; std::vector> b_k_n_tensors; std::vector> c_m_n_tensors; ``` Where: - `a_m_k_tensors` is the vector of `HostTensor` objects for matrix `A` (with dimensions `M × K`). Each tensor stores the data for single GEMM operation. - `b_k_n_tensors` is the vector of `HostTensor` objects for matrix `B` (with dimensions `K × N`). - `c_m_n_tensors` is the vector of `HostTensor` objects for matrix `C` (the output matrix with dimensions `M × N`). The `std::vector` container is used for this purpose throughout. As mentioned above, the number of HostTensors is equal to `group_count`. #### Device Memory Buffers (for GPU) Now it's time to allocate memory on the device (GPU) and transfer the data from `HostTensor` to `DeviceMemory` for actual computation.. ```cpp // Example std::vector> a_m_k_dev_buf; std::vector> b_k_n_dev_buf; std::vector> c_m_n_dev_buf; ``` Where: - `a_m_k_dev_buf` is the buffer used for storing matrix A on the GPU. - `b_k_n_dev_buf` is the buffer used for storing matrix B on the GPU. - `c_m_n_dev_buf` is the buffer used for storing the result matrix C on the GPU. ## Prepare data In the next step, the input tensors are populated. A pseudorandom number generator, an existing distribution (e.g., `FillUniformDistribution`), or user data can be used to populate the tensors. Descriptors also need to be create for each input tensor. Use `get_default_stride` to get the strides for A, B, and C. `get_default_stride` is a template function that calculates the default stride for a 2D array based on whether it is row-major or column-major. Template parameter determines whether the storage order is row-major (true) or column-major (false). The function takes four params `row`, `col`, `stride` and `bool_constant`. If the stride is explicitly provided (`stride != 0`), the stride is returned as-is. If the stride is not provided (`stride == 0`), the function computes the default stride. For the Row-major order (`is_row_major == true`), the stride is set to the number of columns (col). For the column-major order (`is_row_major == false`), the stride is set to the number of rows (row). This function is useful when working with dynamically allocated 2D arrays, where the user may not specify the stride explicitly. It ensures a natural default stride based on the chosen storage order. ```cpp // Example, API template auto get_default_stride(std::size_t row, std::size_t col, std::size_t stride, bool_constant) { // code } ``` Where: - `is_row_major` is a bool template parameter that determines whether the storage order is row-major (true) or column-major (false). - `row` is the number of rows in the matrix. - `col` is the number of columns in the matrix. - `stride` is the current stride (the distance between consecutive elements in memory). - `bool_constant` is a tag type that helps in differentiating behavior at compile-time. Next host descriptors for each of the input tensors, A, B, and C are created. Use the `f_host_tensor_descriptor` function defined below. This function takes four parameters, row, col, stride, and layout, and returns a HostTensorDescriptor based on the specified layout. ```cpp // Example for tensor A ck_tile::HostTensor(f_host_tensor_descriptor(M, K, stride_As[i], a_layout))) ``` After creating the host_tensors, create `deviceMem` for each tensor `A`, `B`, and `C`, and then transfer the data to the device. The `get_element_space_size_in_bytes()` function is used to get the buffer size in bytes. Use `ToDevice()` to transfer data from the host to the device. The data that was previously generated (`a_m_k_tensors[i].data()`) is passed as a parameter to `ToDevice()`. The final step before running the GEMM operation is to retrieve the pointers to the buffers of `A`, `B`, and `C` stored on the device using `->GetDeviceBuffer()` and pack them into a shared container. For example: `gemm_descs.push_back({p_a, p_b, p_c, M, N, K, stride_As[i], stride_Bs[i], stride_Cs[i]})`, where `gemm_descs` is `std::vector gemm_descs` ([Code](https://github.com/ROCm/composable_kernel/blob/develop/example/ck_tile/17_grouped_gemm/run_grouped_gemm_example.inc#L221)). The container should include values such as: ```cpp struct GroupedGemmHostArgs { const void* a_ptr; const void* b_ptr; void* c_ptr; index_t M; index_t N; index_t K; index_t stride_A; index_t stride_B; index_t stride_C; }; ``` The data prepared in this way can be passed to the `invoke_gemm` function. This is a templated function that also takes three template parameters: `ALayout`, `BLayout`, and `CLayout`: ```cpp // Example, API template float invoke_gemm(int n_warmup, int n_repeat, int group_count, const std::vector& args) ``` `invoke_gemm` returns the run time in milliseconds. The workspace memory required for computation is allocated. Workspace memory on the GPU refers to temporary memory buffers allocated when some operations are run. This extra space is needed to hold GEMM descriptions. The following structure can be used to allocate workspace: ```cpp // Example ck_tile::DeviceMem gemm_workspace; gemm_workspace.Realloc(GetWorkspaceSize(args)); ``` Finally the arguments are passed to group_gemm and the kernel is launched. ```cpp // API template float grouped_gemm(const std::vector& gemm_descs, const ck_tile::stream_config& s, void* kargs_ptr) ``` All the necessary parameters are set, the tiling is computed, the GEMM pipeline and epilogue are prepared, and the GroupedGemmKernel is launched. ## Build ``` # in the root of ck_tile mkdir build && cd build # you can replace with the appropriate architecture (for example gfx90a or gfx942) or leave it blank ../script/cmake-ck-dev.sh ../ # The basic pipeline method on the gemm calculation make tile_example_grouped_gemm -j ``` This will result in an executable `build/bin/tile_example_grouped_gemm` ## example ``` args: -Ms M dimensions - empty by default. (default:) -Ns N dimensions - empty by default. (default:) -Ks K dimensions - empty by default. (default:) -stride_As Tensor A strides - it is empty by default. (default:) -stride_Bs Tensor B strides - it is empty by default. (default:) -stride_Cs Tensor C strides - it is empty by default. (default:) -a_layout A tensor data layout - Row by default. (default:R) -b_layout B tensor data layout - Row by default. (default:C) -c_layout C tensor data layout - Row by default. (default:R) -validate 0. No validation, 1. Validation on CPU. (default:1) -warmup number of iterations before benchmark the kernel. (default:10) -repeat number of iterations to benchmark the kernel. (default:100) -group_count group count. (default:8) -kbatch kbatch for SplitK (default:1) -json 0: No Json, 1: Dump Results in Json format (default:0) -jsonfile json file name to dump results (default:grouped_gemm.json) ```