* Add SM103 grouped block-scaled GEMM kernel
Implements sm103_grouped_blockscaled_gemm.py combining SM103 kernel
internals (7-warp layout, MXF4/NVF4 ops, K=768, dedicated TMA SF warp)
with SM100 grouped scheduling (StaticPersistentGroupTileScheduler,
per-group tensormap updates via SMEM, 5 tensormaps).
* [CuTeDSL] Port SM103 grouped block-scaled GEMM to v4.5 + fix scale correctness
Rebases the SM103 grouped block-scaled GEMM onto the current tree, which
reorganized the CuTeDSL examples (#3202) and changed the TMEM-storage API.
Port:
- Move kernel to cute/blackwell/kernel/blockscaled_grouped_gemm/
sm103_grouped_blockscaled_gemm.py (beside the SM100 grouped kernel);
update the test import to the new package path.
- Adopt the v4.5 TMEM storage API: struct field tmem_dealloc_mbar_ptr ->
tmem_dealloc_mbar; pointers via storage.<field>.ptr; local
tmem_holding_buf -> tmem_holding_buf_ptr.
- Rename 3xFP4 -> FP4 Ultra terminology to match the v4.5 dense kernel.
Correctness fix at scale:
The AB/SF TMA producer wait tokens (try_acquire peek) were initialized
once before the persistent tile loop. Each tile's final stage skips the
next try_acquire(), so a stale token was carried into the next work tile,
letting acquire_and_advance(True) overwrite a pipeline buffer stage before
MMA released it -> wrong results once total output tiles exceeded ~1-2K
(launch failure at the largest sizes), while small-shape tests passed.
Refresh ab_producer / sf_producer at each work-tile boundary inside the
loop, matching the SM103 dense and SM100 grouped kernels. Add a large
persistent regression test (8 x 2048^3).
Verified on NVIDIA GB300 (sm_103, CUTLASS DSL 4.5.2): pytest
test/examples/CuTeDSL/sm_103/ --runtime-sm 103 => 23 passed;
compute-sanitizer memcheck clean on 8 x 4096^3.
Select the Mamba2 SSD register split from the target CUDA backend. CUDA 12.9 uses 48 uniform and 88 epilogue registers, while CUDA 13.1 and newer retain the 32/104 split.
Use the same backend predicate for grouped GEMM accumulation and compiler options. Older backends keep the register-friendly per-element update, with explicit O3 for contiguous GEMM and default optimization for masked GEMM. Newer backends retain vector accumulation and O2.
For example, the tested (2, 4, 2, 40, 32, 64, 128, 128) SSD shape now selects 48/88 with CUDA 12.9 and 32/104 with CUDA 13.3.
Checks: Python byte compilation; Ruff format and targeted lint; B200 correctness for SSD none/scalar/vector and contiguous/masked grouped GEMM on CUDA 12.9 and 13.3; contiguous PTX and cubin identity for both backend selections.
* v4.6 dev update.
* Remove CUTLASS_HOST_DEVICE from CudaHostAdapater::memsetDevice (#3286)
* [SM120] Add ptr-array TMA collective for tensor/token-scaled FP8 grouped GEMM (#3280)
* gemm: add SM120 array TMA collective for tensor/token-scaled FP8 grouped GEMM
Adds CollectiveMma and CollectiveBuilder specializations for
MainloopSm120ArrayTmaWarpSpecialized, enabling ptr-array grouped GEMM
(MoE expert dispatch) with tensor- and token-level FP8 scaling on
SM_120/SM_121 consumer Blackwell (RTX 5090/5080/5070, DGX Spark GB10).
New files:
- include/cutlass/gemm/collective/sm120_mma_array_tma.hpp
CollectiveMma specialization for MainloopSm120ArrayTmaWarpSpecialized.
Handles both Cooperative (4x2 atom layout) and Pingpong (2x2) schedules.
Grouped GEMM via pointer-array indirection through params.ptr_A / ptr_B.
Supports F8F6F4 MMA with TMA loads for both A and B operands.
- include/cutlass/gemm/collective/builders/sm120_array_mma_builder.inl
CollectiveBuilder specialization for KernelPtrArrayTmaWarpSpecialized
Cooperative/PingpongSm120<N> schedule tags. Computes tile/stage counts
from smem capacity, routes to MainloopSm120ArrayTmaWarpSpecialized
dispatch policy, produces correctly-typed CollectiveOp.
Modified files:
- collective_mma.hpp: include sm120_mma_array_tma.hpp
- collective_builder.hpp: include sm120_array_mma_builder.inl
- sm120_mma_builder.inl: remove ptr-array schedules from enable_if
(they now route to sm120_array_mma_builder.inl) and drop the
IsPtrArrayKernel static_assert that enforced the restriction
Validated on real SM_121 hardware (DGX Spark, 128 GB LPDDR5X) running
vLLM with RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic (Gemma 4 MoE, 26B
total / 4B active). Previously fell back to a non-CUTLASS Triton path;
with this patch, the SM120 CUTLASS grouped GEMM collective activates and
produces correct outputs. Short-sequence throughput improved ~7% vs the
fallback baseline (76.3 → 81.9 tok/s).
Closes#3263
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tyler Merritt <tgmerritt@gmail.com>
* test: add SM120 ptr-array grouped GEMM unit tests
Adds 6 device-level tests for the CollectiveMma/CollectiveBuilder
specializations introduced for MainloopSm120ArrayTmaWarpSpecialized,
covering both KernelPtrArrayTmaWarpSpecializedPingpongSm120<2> and
KernelPtrArrayTmaWarpSpecializedCooperativeSm120<2> schedule tags across
e4m3×e4m3 (symmetric), e4m3×e5m2 (mixed), float and bfloat16 outputs,
and two tile shapes.
Tests land in test/unit/gemm/device/sm120_tensorop_gemm/ under the new
cutlass_test_unit_sm120_grouped_gemm_device_tensorop CMake target, per
reviewer request in PR #3280.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Signed-off-by: Tyler Merritt <tgmerritt@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Tyler Merritt <tgmerritt@gmail.com>
Co-authored-by: Alex Georgiev <89279829+alexngUNC@users.noreply.github.com>
Co-authored-by: Tyler <tgmerritt@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
The internal DSL package refactored atomic_max_float32 to atomic_fmax,
which properly handles negative floats via sign-bit-aware integer
atomics. Update the example to use the new API so it works with
current DSL wheels.
Co-authored-by: Questa Wang <questaw@computelab-frontend-7.nvidia.com>
