Use Bootstrap node and NVML fabric-domain discovery for optimized LL, collectively gate cross-host fabric payload and semaphore support, remove obsolete topology env defaults, and refresh the EP documentation and performance guidance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: efbacae6-f679-430b-bc16-b45ae162fc76
Resolve the C++ benchmark conflict by combining NCCL-EP-style random top-k routing and masked selections with the current BF16/FP8 MoERuntime API and format-aware byte accounting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: efbacae6-f679-430b-bc16-b45ae162fc76
Refresh the Python and C++ benchmark paths for BF16 and FP8 dispatch, current MoERuntime signatures, active kernel sources, portable CUPTI discovery, realistic routing, and safe unified reporting. Remove the merged change to the inactive legacy implementation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: efbacae6-f679-430b-bc16-b45ae162fc76
## Summary
Replaces the deterministic `(t*K+j) % E` expert assignment in
`mscclpp_ep_bench.cu`
with the same randomized top-k routing scheme used by the NCCL-EP
benchmark, so the
C++ LL benchmark exercises a realistic, non-uniform expert distribution.
## Changes
- Per-token scores drawn from `|N(0,1)| + 1` with a rank-seeded
`std::mt19937(1+rank)`.
- `std::partial_sort` selects the top-K experts per token (descending).
- Randomly masks 10 (token, slot) positions to `-1` to exercise the LL
kernels'
invalid-index handling (guarded at `low_latency.cu:278` and `:726`).
- Byte accounting now counts only valid (`>= 0`) selections.
- Added `<random>` / `<utility>` includes; clang-format-14 clean.
---------
Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
## Summary
Adds a unified low-latency (LL) expert-parallel benchmark for comparing
mscclpp EP against NVIDIA NCCL-EP on equal footing, and fixes an LL
combine
performance regression in the feature/ep kernels.
## Commits
- **ep/ll: scale combine grid to numCombinedTokens (fix combine
regression).**
The LL combine host launched a fixed grid of
`ceil(numExperts/kNumWarpGroups)`
(= 43 SMs for 128 experts), but `combineRecv`'s per-token weighted
reduction
strides `tokenIdx` by the grid size, so it only used 43 blocks for 128
tokens.
Scale the grid to `numCombinedTokens` (capped by the device SM count).
Extra
blocks are recv-only (send is guarded by `responsibleExpertIdx <
numExperts`)
and every block still hits `cg::this_grid().sync()`. Measured (e128 t128
d7168
k8, 1-node): combine avg 56 → 43 us (-24%), min 50 → 38 us. Bit-exact
(`test_low_latency_multirank.py`).
- **ep(bench): unified driver + three backends** — mscclpp (Python
MoECommunicator), mscclpp-cpp (pure C++ MoERuntime), nccl-ep
(`ep_bench`).
- **ep(bench): pure-C++ LL benchmark** (`mscclpp_ep_bench.cu`) calling
`MoERuntime::dispatch/combine` directly, with CUPTI kernel timing, built
via CMake.
## Build
No impact on the core library build: the benchmark's
`test/python/ep/CMakeLists.txt` is standalone (no `add_subdirectory`
from any
parent CMake) and this PR does not touch the top-level `CMakeLists.txt`
/
`pyproject.toml` / `setup.py`. The only library change is the combine
grid size.
The Python driver (`run_ep_bench.py`) and the mscclpp Python backend
(`ep_bench_ll.py`) need **no build**. Only the **mscclpp-cpp** backend
needs a
one-time standalone build (it recompiles the two LL translation units +
links the installed `libmscclpp.so`):
```bash
cmake -S test/python/ep -B build \
-DMSCCLPP_EP_NUM_MAX_NVL_PEERS=4 -DCMAKE_CUDA_ARCHITECTURES=100 # GB200 sm_100
cmake --build build -j
# -> build/mscclpp_ep_bench
```
Requires nvcc/CUDA, MPI, and CUPTI (all `find_package REQUIRED`).
## Usage
Common workload: `-e 128 -t 128 -d 7168 -k 8 -w 10 -i 100`. The driver
prints a
unified summary with **host-observed** and **kernel-only** (CUPTI) rows.
```bash
# mscclpp (Python MoECommunicator) — no build needed
python test/python/ep/run_ep_bench.py --ep-lib mscclpp -e 128 --cupti-inproc
# mscclpp-cpp (pure C++ MoERuntime) — after the Build step above
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100
# 2 / 4-node (one NVLink domain): list peer IPs; the driver builds the hostfile + mpirun
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nodes "10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4"
# nccl-ep (NVIDIA reference ep_bench)
python test/python/ep/run_ep_bench.py --ep-lib nccl-ep -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nccl-lib-path /path/to/nccl/build/lib
# all backends side-by-side
python test/python/ep/run_ep_bench.py --ep-lib all -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
--nccl-lib-path /path/to/nccl/build/lib
# add --kernel-only for just the CUPTI rows, --dry-run to print the commands
```
Useful flags: `--nodes "<ip ...>"`, `--nproc-per-node 4`,
`--kernel-only`,
`--dry-run`, `--cupti-inproc`, `--mscclpp-cpp-bench <path>`,
`--nccl-ep-bench <path>`,
`--nccl-lib-path <dir>`, `--hpcx <dir>`, `--iface <nic>`.
## Validation
Combine fix bit-exact and benchmarked 1/2/4-node on GB200 NVL72
(sm_100);
dispatch/combine host + kernel times on par with NCCL-EP at all scales.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This pull request makes significant improvements to the MoE (Mixture of
Experts) Python API and documentation, focusing on clarifying and
expanding the Expert Parallel (EP) interface, especially around
quantization, dispatch/combine handles, and overlap configuration. The
changes introduce new data structures, update function signatures, and
improve documentation to better reflect the current and planned
capabilities of the system. Additionally, the base development container
is updated to CUDA 13.0, and minor corrections are made to extension
naming.
