mirror of
https://github.com/ROCm/composable_kernel.git
synced 2026-05-16 10:59:55 +00:00
81 lines
2.8 KiB
Markdown
81 lines
2.8 KiB
Markdown
# Client Example: Group Normalization (Forward and Backward)
|
|
|
|
## Theory
|
|
|
|
This client example demonstrates **group normalization** in both forward and backward modes, including fusion with Swish activation. Group normalization normalizes activations across groups of channels, improving training stability for small batch sizes or non-i.i.d. data.
|
|
|
|
**Mathematical Formulation:**
|
|
Given input $X[N, C, ...]$ divided into $G$ groups:
|
|
- For each group $g$:
|
|
- Mean: $\mu_g = \frac{1}{|g|} \sum_{i \in g} X_i$
|
|
- Variance: $\sigma^2_g = \frac{1}{|g|} \sum_{i \in g} (X_i - \mu_g)^2$
|
|
- Normalized: $\hat{X}_i = \frac{X_i - \mu_g}{\sqrt{\sigma^2_g + \epsilon}}$
|
|
- Output: $Y_i = \gamma \hat{X}_i + \beta$
|
|
|
|
$\gamma$, $\beta$ are learnable scale and shift parameters.
|
|
|
|
- Swish activation: $\text{Swish}(x) = x \cdot \sigma(x)$, where $\sigma$ is the sigmoid function.
|
|
|
|
**Algorithmic Background:**
|
|
- Forward pass computes mean, variance, normalization, and affine transformation per group.
|
|
- Backward pass computes gradients with respect to input, gamma, and beta.
|
|
- Swish activation can be fused with normalization for efficiency.
|
|
|
|
## How to Run
|
|
|
|
### Prerequisites
|
|
|
|
Please follow the instructions in the main [Build Guide](../../README.md#building-ck) section as a prerequisite to building and running this example.
|
|
|
|
### Build and run
|
|
```bash
|
|
cd composable_kernel/client_example/18_groupnorm
|
|
mkdir build && cd build
|
|
cmake -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc ..
|
|
make -j
|
|
|
|
# Example run (forward with Swish)
|
|
./groupnorm_swish_fwd
|
|
|
|
# Example run (backward, data)
|
|
./groupnorm_bwd_data
|
|
|
|
# Example run (backward, gamma/beta)
|
|
./groupnorm_bwd_gamma_beta
|
|
```
|
|
|
|
## Source Code Structure
|
|
|
|
### Directory Layout
|
|
```
|
|
client_example/18_groupnorm/
|
|
├── groupnorm_swish_fwd.cpp # Groupnorm forward with Swish activation
|
|
├── groupnorm_bwd_data.cpp # Groupnorm backward (data)
|
|
├── groupnorm_bwd_gamma_beta.cpp # Groupnorm backward (gamma/beta)
|
|
├── CMakeLists.txt # Build configuration for the example
|
|
```
|
|
|
|
### Key Functions
|
|
|
|
- **main()** (in each `.cpp`):
|
|
Sets up input tensors, configures groupnorm parameters, launches the forward or backward kernel, and verifies the result.
|
|
- **GroupNorm kernel invocation**:
|
|
Uses the Composable Kernel device API to launch group normalization for different modes.
|
|
|
|
---
|
|
|
|
## Additional Details
|
|
|
|
- Supports fusion with Swish activation for efficiency.
|
|
- Example parameters can be adjusted in the source for different workloads.
|
|
|
|
---
|
|
|
|
## Related Examples
|
|
|
|
- [42_groupnorm_fwd](../../example/42_groupnorm_fwd/README.md): Group normalization in the main example directory
|
|
- [54_groupnorm_bwd](../../example/54_groupnorm_bwd/README.md): Group normalization backward in the main example directory
|
|
|
|
---
|
|
[Back to Client Examples](../README.md)
|