## Summary
Adds a unified low-latency (LL) expert-parallel benchmark for comparing
mscclpp EP against NVIDIA NCCL-EP on equal footing, and fixes an LL
combine
performance regression in the feature/ep kernels.
## Commits
- **ep/ll: scale combine grid to numCombinedTokens (fix combine
regression).**
The LL combine host launched a fixed grid of
`ceil(numExperts/kNumWarpGroups)`
(= 43 SMs for 128 experts), but `combineRecv`'s per-token weighted
reduction
strides `tokenIdx` by the grid size, so it only used 43 blocks for 128
tokens.
Scale the grid to `numCombinedTokens` (capped by the device SM count).
Extra
blocks are recv-only (send is guarded by `responsibleExpertIdx <
numExperts`)
and every block still hits `cg::this_grid().sync()`. Measured (e128 t128
d7168
k8, 1-node): combine avg 56 → 43 us (-24%), min 50 → 38 us. Bit-exact
(`test_low_latency_multirank.py`).
- **ep(bench): unified driver + three backends** — mscclpp (Python
MoECommunicator), mscclpp-cpp (pure C++ MoERuntime), nccl-ep
(`ep_bench`).
- **ep(bench): pure-C++ LL benchmark** (`mscclpp_ep_bench.cu`) calling
`MoERuntime::dispatch/combine` directly, with CUPTI kernel timing, built
via CMake.
## Build
No impact on the core library build: the benchmark's
`test/python/ep/CMakeLists.txt` is standalone (no `add_subdirectory`
from any
parent CMake) and this PR does not touch the top-level `CMakeLists.txt`
/
`pyproject.toml` / `setup.py`. The only library change is the combine
grid size.
The Python driver (`run_ep_bench.py`) and the mscclpp Python backend
(`ep_bench_ll.py`) need **no build**. Only the **mscclpp-cpp** backend
needs a
one-time standalone build (it recompiles the two LL translation units +
links the installed `libmscclpp.so`):
```bash
cmake -S test/python/ep -B build \
-DMSCCLPP_EP_NUM_MAX_NVL_PEERS=4 -DCMAKE_CUDA_ARCHITECTURES=100 # GB200 sm_100
cmake --build build -j
# -> build/mscclpp_ep_bench
```
Requires nvcc/CUDA, MPI, and CUPTI (all `find_package REQUIRED`).
## Usage
Common workload: `-e 128 -t 128 -d 7168 -k 8 -w 10 -i 100`. The driver
prints a
unified summary with **host-observed** and **kernel-only** (CUPTI) rows.
```bash
# mscclpp (Python MoECommunicator) — no build needed
python test/python/ep/run_ep_bench.py --ep-lib mscclpp -e 128 --cupti-inproc
# mscclpp-cpp (pure C++ MoERuntime) — after the Build step above
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100
# 2 / 4-node (one NVLink domain): list peer IPs; the driver builds the hostfile + mpirun
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nodes "10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4"
# nccl-ep (NVIDIA reference ep_bench)
python test/python/ep/run_ep_bench.py --ep-lib nccl-ep -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nccl-lib-path /path/to/nccl/build/lib
# all backends side-by-side
python test/python/ep/run_ep_bench.py --ep-lib all -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nccl-lib-path /path/to/nccl/build/lib
# add --kernel-only for just the CUPTI rows, --dry-run to print the commands
```
Useful flags: `--nodes "<ip ...>"`, `--nproc-per-node 4`,
`--kernel-only`,
`--dry-run`, `--cupti-inproc`, `--mscclpp-cpp-bench <path>`,
`--nccl-ep-bench <path>`,
`--nccl-lib-path <dir>`, `--hpcx <dir>`, `--iface <nic>`.
## Validation
Combine fix bit-exact and benchmarked 1/2/4-node on GB200 NVL72
(sm_100);
dispatch/combine host + kernel times on par with NCCL-EP at all scales.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
