ep: unified low-latency EP benchmark + LL combine SM-scaling fix (#831)

## Summary
Adds a unified low-latency (LL) expert-parallel benchmark for comparing
mscclpp EP against NVIDIA NCCL-EP on equal footing, and fixes an LL
combine
performance regression in the feature/ep kernels.

## Commits
- **ep/ll: scale combine grid to numCombinedTokens (fix combine
regression).**
The LL combine host launched a fixed grid of
`ceil(numExperts/kNumWarpGroups)`
(= 43 SMs for 128 experts), but `combineRecv`'s per-token weighted
reduction
strides `tokenIdx` by the grid size, so it only used 43 blocks for 128
tokens.
Scale the grid to `numCombinedTokens` (capped by the device SM count).
Extra
blocks are recv-only (send is guarded by `responsibleExpertIdx <
numExperts`)
and every block still hits `cg::this_grid().sync()`. Measured (e128 t128
d7168
  k8, 1-node): combine avg 56 → 43 us (-24%), min 50 → 38 us. Bit-exact
  (`test_low_latency_multirank.py`).
- **ep(bench): unified driver + three backends** — mscclpp (Python
MoECommunicator), mscclpp-cpp (pure C++ MoERuntime), nccl-ep
(`ep_bench`).
- **ep(bench): pure-C++ LL benchmark** (`mscclpp_ep_bench.cu`) calling
`MoERuntime::dispatch/combine` directly, with CUPTI kernel timing, built
via CMake.

## Build
No impact on the core library build: the benchmark's
`test/python/ep/CMakeLists.txt` is standalone (no `add_subdirectory`
from any
parent CMake) and this PR does not touch the top-level `CMakeLists.txt`
/
`pyproject.toml` / `setup.py`. The only library change is the combine
grid size.

The Python driver (`run_ep_bench.py`) and the mscclpp Python backend
(`ep_bench_ll.py`) need **no build**. Only the **mscclpp-cpp** backend
needs a
one-time standalone build (it recompiles the two LL translation units +
links the installed `libmscclpp.so`):
```bash
cmake -S test/python/ep -B build \
  -DMSCCLPP_EP_NUM_MAX_NVL_PEERS=4 -DCMAKE_CUDA_ARCHITECTURES=100   # GB200 sm_100
cmake --build build -j
# -> build/mscclpp_ep_bench
```
Requires nvcc/CUDA, MPI, and CUPTI (all `find_package REQUIRED`).

## Usage

Common workload: `-e 128 -t 128 -d 7168 -k 8 -w 10 -i 100`. The driver
prints a
unified summary with **host-observed** and **kernel-only** (CUPTI) rows.

```bash
# mscclpp (Python MoECommunicator) — no build needed
python test/python/ep/run_ep_bench.py --ep-lib mscclpp -e 128 --cupti-inproc

# mscclpp-cpp (pure C++ MoERuntime) — after the Build step above
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100

# 2 / 4-node (one NVLink domain): list peer IPs; the driver builds the hostfile + mpirun
python test/python/ep/run_ep_bench.py --ep-lib mscclpp-cpp -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
    --nodes "10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4"

# nccl-ep (NVIDIA reference ep_bench)
python test/python/ep/run_ep_bench.py --ep-lib nccl-ep -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
    --nccl-lib-path /path/to/nccl/build/lib

# all backends side-by-side
python test/python/ep/run_ep_bench.py --ep-lib all -e 128 -t 128 -d 7168 -k 8 -w 10 -i 100 \
    --nccl-lib-path /path/to/nccl/build/lib
# add --kernel-only for just the CUPTI rows, --dry-run to print the commands
```

Useful flags: `--nodes "<ip ...>"`, `--nproc-per-node 4`,
`--kernel-only`,
`--dry-run`, `--cupti-inproc`, `--mscclpp-cpp-bench <path>`,
`--nccl-ep-bench <path>`,
`--nccl-lib-path <dir>`, `--hpcx <dir>`, `--iface <nic>`.

## Validation
Combine fix bit-exact and benchmarked 1/2/4-node on GB200 NVL72
(sm_100);
dispatch/combine host + kernel times on par with NCCL-EP at all scales.

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

test/python/ep/cupti_kernel_timer.cpp

@@ -0,0 +1,122 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+// In-process CUPTI kernel timer for the mscclpp LL benchmark.
+// name, and exposes an extern "C" ABI so it can be driven from Python via ctypes
+// (no cuda-python CUPTI bindings exist in this env, but libcupti.so is loadable).
+//
+// This is near-zero host perturbation (out-of-band buffer callbacks), unlike
+// torch.profiler's in-process tracing which serialized the LL dispatch recv-spin
+// and inflated one rank's device time into the millisecond range. It matches
+// ep_bench's methodology exactly: start() after warmup, stop() after the timed
+// loop, get_avg_us("dispatch"/"combine") buckets by mangled-name substring.
+//
+// COOPERATIVE-LAUNCH NOTE (GB200 / CUDA 13): the mscclpp LL dispatch/combine
+// kernels are launched with cudaLaunchCooperativeKernel. Those are NOT reported
+// by CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL on this driver, but they ARE reported
+// by CUPTI_ACTIVITY_KIND_KERNEL (the serialized-kernel activity), which is what
+// we subscribe to below. KIND_KERNEL only serializes *inter*-kernel concurrency;
+// in this dispatch->sync->combine->sync paired loop the kernels already run one
+// at a time, so the measured per-kernel GPU duration is unaffected. The activity
+// record carries the RAW MANGLED name (e.g. ...low_latency8dispatch...), so the
+// caller matches the substring "dispatch"/"combine" (present in the mangled form)
+// rather than the demangled "low_latency::dispatch".
+//
+// Build (host-only C++, links libcupti):
+//   g++ -O2 -fPIC -shared cupti_kernel_timer.cpp -o libcupti_kernel_timer.so \
+//       -I<cuda>/targets/sbsa-linux/include -L<cuda>/targets/sbsa-linux/lib -lcupti
+#include <cupti.h>
+
+#include <cstdint>
+#include <cstdlib>
+#include <cstring>
+#include <map>
+#include <mutex>
+#include <string>
+
+namespace {
+
+struct KernelStat {
+  uint64_t total_ns = 0;
+  uint64_t count = 0;
+};
+
+std::map<std::string, KernelStat> g_stats;
+std::mutex g_mutex;
+
+constexpr size_t kBufSize = 8 * 1024 * 1024;  // 8 MB, matches ep_bench
+
+void CUPTIAPI bufferRequested(uint8_t** buffer, size_t* size, size_t* maxNumRecords) {
+  // 8-byte aligned; aligned_alloc requires size to be a multiple of alignment.
+  *buffer = static_cast<uint8_t*>(aligned_alloc(8, kBufSize));
+  *size = kBufSize;
+  *maxNumRecords = 0;
+}
+
+void CUPTIAPI bufferCompleted(CUcontext, uint32_t, uint8_t* buffer, size_t, size_t validSize) {
+  CUpti_Activity* record = nullptr;
+  std::lock_guard<std::mutex> lock(g_mutex);
+  while (cuptiActivityGetNextRecord(buffer, validSize, &record) == CUPTI_SUCCESS) {
+    if (record->kind == CUPTI_ACTIVITY_KIND_KERNEL) {
+      // CUpti_ActivityKernel10 is the record layout for CUDA 13 CUPTI. start/end
+      // (GPU HW timestamps, ns) and name have been stable across versions.
+      auto* k = reinterpret_cast<CUpti_ActivityKernel10*>(record);
+      if (k->name) {
+        auto& s = g_stats[k->name];
+        s.total_ns += (k->end - k->start);
+        s.count += 1;
+      }
+    }
+  }
+  free(buffer);
+}
+
+}  // namespace
+
+extern "C" {
+
+// Clear stats, register the buffer callbacks, and enable concurrent-kernel
+// activity recording. Call AFTER warmup (like ep_bench's KernelTimer::start()).
+int kt_start() {
+  {
+    std::lock_guard<std::mutex> lock(g_mutex);
+    g_stats.clear();
+  }
+  CUptiResult r = cuptiActivityRegisterCallbacks(bufferRequested, bufferCompleted);
+  if (r != CUPTI_SUCCESS) return static_cast<int>(r);
+  r = cuptiActivityEnable(CUPTI_ACTIVITY_KIND_KERNEL);
+  return static_cast<int>(r);
+}
+
+// Flush pending buffers and disable recording. Returns CUPTI result code (0=ok).
+int kt_stop() {
+  cuptiActivityFlushAll(0);
+  CUptiResult r = cuptiActivityDisable(CUPTI_ACTIVITY_KIND_KERNEL);
+  return static_cast<int>(r);
+}
+
+// Average GPU execution time (microseconds) over every recorded kernel whose
+// mangled name contains `substr`. Returns 0 if none matched.
+double kt_get_avg_us(const char* substr) {
+  std::lock_guard<std::mutex> lock(g_mutex);
+  uint64_t total_ns = 0, count = 0;
+  for (const auto& kv : g_stats) {
+    if (kv.first.find(substr) != std::string::npos) {
+      total_ns += kv.second.total_ns;
+      count += kv.second.count;
+    }
+  }
+  return count ? static_cast<double>(total_ns) / static_cast<double>(count) / 1000.0 : 0.0;
+}
+
+// Number of recorded kernel instances whose name contains `substr`.
+long kt_get_count(const char* substr) {
+  std::lock_guard<std::mutex> lock(g_mutex);
+  uint64_t count = 0;
+  for (const auto& kv : g_stats) {
+    if (kv.first.find(substr) != std::string::npos) count += kv.second.count;
+  }
+  return static_cast<long>(count);
+}
+
+}  // extern "C"