[GH-ISSUE #2607] [RFC]: Comprehensive MoE / EP Kernel Benchmark Under Incast Traffic #683

New issue

Open

opened 2026-10-29 21:18:05 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented 2026-10-29 21:18:05 +00:00

Owner

Copy link

Originally created by @Srinivasoo7 on GitHub (Jun 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2607

Changes proposed

Summary

This RFC proposes adding a dedicated benchmark suite for Mooncake EP dispatch/combine kernels under both uniform and incast traffic patterns from Issue #1883 Milestone 15

Mooncake EP already has correctness coverage through existing tests that exercise dispatch/combine behavior, expert routing, data types, zero-copy paths, async finish behavior, and receive hooks. However, the project does not yet have a standardized performance benchmark that measures throughput and tail latency under realistic expert-parallel MoE serving traffic.

The main target workload for this RFC is incast traffic, where many ranks simultaneously route tokens into a small set of hot experts. This traffic pattern stresses the all-to-all dispatch path much more aggressively than uniform random routing and is important for detecting regressions in EP kernel and transport behavior.

The proposed benchmark will live separately from correctness tests, likely under:

benchmarks/ep_benchmark/

The benchmark should be CLI-driven, support JSON output, and allow repeatable comparison across CUDA and other supported backends when hardware is available.

Motivation

Mooncake EP is responsible for high-performance expert-parallel communication patterns used by MoE workloads. Correctness tests are necessary but not sufficient for kernel and transport development because they do not answer questions such as:

• How does dispatch latency change as routing skew increases?
• What is the p99 or p999 latency under expert hot-spotting?
• How does zero-copy compare against non-zero-copy execution?
• How much imbalance appears across ranks and experts?
• Does a backend change regress throughput under realistic incast load?
• Are CUDA and non-CUDA backends behaving similarly under equivalent benchmark configurations?
  Current correctness tests are valuable because they validate functional behavior, but they are not designed to provide repeatable performance statistics over warmup and measured iterations.

A dedicated EP benchmark would make M15 optimization work easier to evaluate and would give maintainers and contributors a common tool for validating EP performance changes.

Goals

1. Add a standalone EP benchmark harness.
2. Reuse the same public EP setup path used by existing tests:
   • mooncake.mooncake_ep_buffer.Buffer
   • mooncake.pg
   • dispatch/combine APIs
3. Support multiple routing distributions:
   • uniform routing baseline
   • K-hot-expert incast
   • Zipfian/skewed routing
4. Report throughput and latency statistics:
   • dispatch latency
   • combine latency
   • end-to-end dispatch+combine latency
   • tokens/sec
   • p50 / p90 / p99 / p999 latency
   • per-rank load imbalance
   • per-expert hot-spot concentration
5. Support configurable benchmark knobs:
   • number of experts
   • hidden size
   • top-k
   • batch size / number of tokens
   • dtype, including bf16 and fp8 where supported
   • zero-copy mode
   • async finish
   • return receive hook
   • number of warmup iterations
   • number of measured iterations
6. Produce machine-readable JSON output.
7. Keep performance gating out of mandatory CI initially unless suitable multi-GPU runners are available.

Non-Goals

1. This is not an end-to-end model-serving benchmark.
2. This does not introduce new EP kernel functionality.
3. This does not replace existing correctness tests.
4. This does not require performance results to be a hard CI gate in the first version.
5. This does not attempt to benchmark every possible MoE architecture or model framework.

Current State

Mooncake EP currently has correctness-oriented tests that exercise dispatch and combine behavior. These tests are useful for checking that output tensors match expected values and that different API modes behave correctly.

However, correctness tests are not ideal for repeated performance measurement because they usually:

• do not run enough iterations for stable latency statistics,
• do not provide structured benchmark output,
• do not model incast traffic explicitly,
• do not separate warmup from measured iterations,
• do not expose benchmark sweeps as a stable CLI contract.
  This RFC proposes adding a benchmark harness rather than extending the correctness test directly.

Proposed Design

Directory Layout

Proposed initial structure:

benchmarks/
  ep_benchmark/
    README.md
    run_ep_benchmark.py
    routing.py
    metrics.py
    configs/
      cuda_uniform.json
      cuda_incast_k_hot.json
      cuda_zipfian.json
    results/
      .gitignore

Possible future structure if the benchmark grows:

benchmarks/
  ep_benchmark/
    mooncake_ep_benchmark/
      __init__.py
      runner.py
      routing.py
      metrics.py
      config.py
      report.py
    run_ep_benchmark.py
    README.md

CLI Interface

Example command:

python benchmarks/ep_benchmark/run_ep_benchmark.py \
  --backend cuda \
  --num-ranks 8 \
  --num-experts 64 \
  --hidden-size 7168 \
  --top-k 8 \
  --num-tokens 4096 \
  --dtype bf16 \
  --routing-mode k_hot \
  --hot-experts 4 \
  --warmup-iters 20 \
  --iters 100 \
  --zero-copy \
  --async-finish \
  --return-recv-hook \
  --json-output results/ep_benchmark_cuda_khot.json

Routing Modes

1. Uniform Routing

Uniform routing should approximate the current correctness-test routing pattern and serve as a baseline.

Example behavior:

Each token chooses top-k experts from the full expert set with roughly uniform probability.

2. K-Hot-Expert Incast

K-hot routing models many ranks sending tokens to a small hot expert set.

Example behavior:

Given num_experts = 64 and hot_experts = 4, most or all ranks route a configurable percentage of tokens into experts [0, 1, 2, 3].

Suggested knobs:

--routing-mode k_hot
--hot-experts 4
--hot-fraction 0.90

This allows both extreme incast and mixed traffic.

3. Zipfian Routing

Zipfian routing models skewed real-world load where a small number of experts receive a disproportionate share of tokens.

Suggested knobs:

--routing-mode zipf
--zipf-alpha 1.2

Metrics

The benchmark should report:

{
  "benchmark": "mooncake_ep",
  "backend": "cuda",
  "world_size": 8,
  "num_experts": 64,
  "hidden_size": 7168,
  "top_k": 8,
  "num_tokens": 4096,
  "dtype": "bf16",
  "routing_mode": "k_hot",
  "hot_experts": 4,
  "hot_fraction": 0.9,
  "zero_copy": true,
  "async_finish": true,
  "return_recv_hook": true,
  "warmup_iters": 20,
  "iters": 100,
  "metrics": {
    "dispatch_latency_ms": {
      "p50": 0.0,
      "p90": 0.0,
      "p99": 0.0,
      "p999": 0.0,
      "mean": 0.0
    },
    "combine_latency_ms": {
      "p50": 0.0,
      "p90": 0.0,
      "p99": 0.0,
      "p999": 0.0,
      "mean": 0.0
    },
    "end_to_end_latency_ms": {
      "p50": 0.0,
      "p90": 0.0,
      "p99": 0.0,
      "p999": 0.0,
      "mean": 0.0
    },
    "tokens_per_second": 0.0,
    "expert_load": {
      "max_tokens_per_expert": 0,
      "min_tokens_per_expert": 0,
      "mean_tokens_per_expert": 0.0,
      "imbalance_ratio": 0.0
    }
  }
}

Actual metric values should be populated by the benchmark.

Timing Method

The benchmark should separate:

1. setup,
2. warmup iterations,
3. measured iterations,
4. result aggregation.
   For CUDA, timing should use appropriate synchronization around measured regions so that async GPU work is accounted for correctly. If backend-specific timing APIs differ, the benchmark should isolate backend timing behavior behind helper functions.

Backend Support

Initial implementation should target CUDA first because it is the most common validation environment.

MUSA or other backend support can be added after confirming:

• build support,
• hardware availability,
• expected backend flags,
• whether the same benchmark code path can be reused.
  The benchmark should not assume all contributors have access to every backend.

CI Strategy

The benchmark should not become a hard performance gate initially.

Suggested CI behavior for the first version:

1. Run lightweight import/config parsing checks.
2. Run unit tests for routing distribution generation.
3. Optionally run a tiny single-process or mock smoke test if feasible.
4. Do not require multi-GPU performance results in standard CI.
   Future CI options:

• self-hosted multi-GPU runner,
• nightly benchmark job,
• manual benchmark workflow,
• threshold-based regression checks after the benchmark is stable.

Implementation Plan

Phase 1 : Minimal Benchmark Harness

• Add benchmarks/ep_benchmark/.
• Add CLI parser.
• Add uniform routing mode.
• Reuse existing EP setup pattern.
• Measure dispatch/combine latency over warmup and measured iterations.
• Emit JSON output.

Phase 2 : Incast Routing

• Add K-hot-expert routing mode.
• Add hot expert fraction.
• Add per-expert load metrics.
• Add README examples.

Phase 3 : Zipfian Routing and Sweep Support

• Add Zipfian routing mode.
• Add simple config-file support.
• Add benchmark sweep examples.

Phase 4 : Backend Parity

• Add backend selection where supported.
• Validate CUDA first.
• Add MUSA or other backend support only where maintainers confirm hardware/toolchain availability.

Phase 5 : Optional CI / Nightly Integration

• Add smoke checks to standard CI.
• Consider a manual or nightly benchmark job if suitable hardware exists.

Validation Plan

The benchmark should be validated in three ways:

1. Correctness sanity check: for small inputs, verify benchmark dispatch/combine outputs remain consistent with existing test behavior.
2. Routing validation: ensure uniform, K-hot, and Zipfian modes produce expected expert-load distributions.
3. Performance sanity check: run repeated benchmark executions on the same hardware and confirm variance is within an acceptable range.

Alternatives Considered

Extend Existing Correctness Test

Rejected for initial implementation.

Correctness tests and performance benchmarks have different goals. Mixing repeated-trial statistics, warmup, JSON output, and sweep logic into a correctness test would make both harder to maintain.

Rely Only on External MoE Benchmarks

External benchmarks may be useful for comparison, but Mooncake EP should have a benchmark that exercises its own public API and backend paths directly.

Add CI Perf Gate Immediately

Rejected for the first version.

Kernel performance benchmarking requires stable hardware, stable drivers, and controlled environments. A hard CI gate should only be added after the benchmark is stable and the project has appropriate runners.

Risks

1. Benchmark noise: GPU performance can vary across hosts and runs.
2. Hardware availability: contributors may not have access to multi-GPU or non-CUDA environments.
3. Backend divergence: CUDA and other backends may require slightly different timing or setup paths.
4. Scope creep: benchmark should remain focused on EP dispatch/combine, not full model serving.

Open Questions

1. What hardware can maintainers use for official EP benchmark runs?
2. Should the first benchmark target only single-node multi-GPU, or should cross-node EP/RDMA behavior be included later?
3. Which backend combinations should be considered required for the first version?
4. Should benchmark results be stored in-tree as examples, or only generated locally?
5. What metric thresholds would be meaningful enough for future regression detection?

Expected Outcome

This RFC should produce a repeatable Mooncake EP benchmark that helps contributors measure dispatch/combine performance under both normal and incast traffic. The first implementation should be useful as a manual benchmark tool, with CI integration kept lightweight until stable multi-GPU runners are available.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Srinivasoo7 on GitHub (Jun 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2607 ### Changes proposed ## Summary This RFC proposes adding a dedicated benchmark suite for Mooncake EP dispatch/combine kernels under both uniform and incast traffic patterns from Issue #1883 **Milestone 15** Mooncake EP already has correctness coverage through existing tests that exercise dispatch/combine behavior, expert routing, data types, zero-copy paths, async finish behavior, and receive hooks. However, the project does not yet have a standardized performance benchmark that measures throughput and tail latency under realistic expert-parallel MoE serving traffic. The main target workload for this RFC is **incast traffic**, where many ranks simultaneously route tokens into a small set of hot experts. This traffic pattern stresses the all-to-all dispatch path much more aggressively than uniform random routing and is important for detecting regressions in EP kernel and transport behavior. The proposed benchmark will live separately from correctness tests, likely under: ```text benchmarks/ep_benchmark/ ``` The benchmark should be CLI-driven, support JSON output, and allow repeatable comparison across CUDA and other supported backends when hardware is available. ## Motivation Mooncake EP is responsible for high-performance expert-parallel communication patterns used by MoE workloads. Correctness tests are necessary but not sufficient for kernel and transport development because they do not answer questions such as: * How does dispatch latency change as routing skew increases? * What is the p99 or p999 latency under expert hot-spotting? * How does zero-copy compare against non-zero-copy execution? * How much imbalance appears across ranks and experts? * Does a backend change regress throughput under realistic incast load? * Are CUDA and non-CUDA backends behaving similarly under equivalent benchmark configurations? Current correctness tests are valuable because they validate functional behavior, but they are not designed to provide repeatable performance statistics over warmup and measured iterations. A dedicated EP benchmark would make M15 optimization work easier to evaluate and would give maintainers and contributors a common tool for validating EP performance changes. ## Goals 1. Add a standalone EP benchmark harness. 2. Reuse the same public EP setup path used by existing tests: * `mooncake.mooncake_ep_buffer.Buffer` * `mooncake.pg` * dispatch/combine APIs 3. Support multiple routing distributions: * uniform routing baseline * K-hot-expert incast * Zipfian/skewed routing 4. Report throughput and latency statistics: * dispatch latency * combine latency * end-to-end dispatch+combine latency * tokens/sec * p50 / p90 / p99 / p999 latency * per-rank load imbalance * per-expert hot-spot concentration 5. Support configurable benchmark knobs: * number of experts * hidden size * top-k * batch size / number of tokens * dtype, including bf16 and fp8 where supported * zero-copy mode * async finish * return receive hook * number of warmup iterations * number of measured iterations 6. Produce machine-readable JSON output. 7. Keep performance gating out of mandatory CI initially unless suitable multi-GPU runners are available. ## Non-Goals 1. This is not an end-to-end model-serving benchmark. 2. This does not introduce new EP kernel functionality. 3. This does not replace existing correctness tests. 4. This does not require performance results to be a hard CI gate in the first version. 5. This does not attempt to benchmark every possible MoE architecture or model framework. ## Current State Mooncake EP currently has correctness-oriented tests that exercise dispatch and combine behavior. These tests are useful for checking that output tensors match expected values and that different API modes behave correctly. However, correctness tests are not ideal for repeated performance measurement because they usually: * do not run enough iterations for stable latency statistics, * do not provide structured benchmark output, * do not model incast traffic explicitly, * do not separate warmup from measured iterations, * do not expose benchmark sweeps as a stable CLI contract. This RFC proposes adding a benchmark harness rather than extending the correctness test directly. ## Proposed Design ### Directory Layout Proposed initial structure: ```text benchmarks/ ep_benchmark/ README.md run_ep_benchmark.py routing.py metrics.py configs/ cuda_uniform.json cuda_incast_k_hot.json cuda_zipfian.json results/ .gitignore ``` Possible future structure if the benchmark grows: ```text benchmarks/ ep_benchmark/ mooncake_ep_benchmark/ __init__.py runner.py routing.py metrics.py config.py report.py run_ep_benchmark.py README.md ``` ### CLI Interface Example command: ```bash python benchmarks/ep_benchmark/run_ep_benchmark.py \ --backend cuda \ --num-ranks 8 \ --num-experts 64 \ --hidden-size 7168 \ --top-k 8 \ --num-tokens 4096 \ --dtype bf16 \ --routing-mode k_hot \ --hot-experts 4 \ --warmup-iters 20 \ --iters 100 \ --zero-copy \ --async-finish \ --return-recv-hook \ --json-output results/ep_benchmark_cuda_khot.json ``` ### Routing Modes #### 1. Uniform Routing Uniform routing should approximate the current correctness-test routing pattern and serve as a baseline. Example behavior: ```text Each token chooses top-k experts from the full expert set with roughly uniform probability. ``` #### 2. K-Hot-Expert Incast K-hot routing models many ranks sending tokens to a small hot expert set. Example behavior: ```text Given num_experts = 64 and hot_experts = 4, most or all ranks route a configurable percentage of tokens into experts [0, 1, 2, 3]. ``` Suggested knobs: ```text --routing-mode k_hot --hot-experts 4 --hot-fraction 0.90 ``` This allows both extreme incast and mixed traffic. #### 3. Zipfian Routing Zipfian routing models skewed real-world load where a small number of experts receive a disproportionate share of tokens. Suggested knobs: ```text --routing-mode zipf --zipf-alpha 1.2 ``` ### Metrics The benchmark should report: ```json { "benchmark": "mooncake_ep", "backend": "cuda", "world_size": 8, "num_experts": 64, "hidden_size": 7168, "top_k": 8, "num_tokens": 4096, "dtype": "bf16", "routing_mode": "k_hot", "hot_experts": 4, "hot_fraction": 0.9, "zero_copy": true, "async_finish": true, "return_recv_hook": true, "warmup_iters": 20, "iters": 100, "metrics": { "dispatch_latency_ms": { "p50": 0.0, "p90": 0.0, "p99": 0.0, "p999": 0.0, "mean": 0.0 }, "combine_latency_ms": { "p50": 0.0, "p90": 0.0, "p99": 0.0, "p999": 0.0, "mean": 0.0 }, "end_to_end_latency_ms": { "p50": 0.0, "p90": 0.0, "p99": 0.0, "p999": 0.0, "mean": 0.0 }, "tokens_per_second": 0.0, "expert_load": { "max_tokens_per_expert": 0, "min_tokens_per_expert": 0, "mean_tokens_per_expert": 0.0, "imbalance_ratio": 0.0 } } } ``` Actual metric values should be populated by the benchmark. ### Timing Method The benchmark should separate: 1. setup, 2. warmup iterations, 3. measured iterations, 4. result aggregation. For CUDA, timing should use appropriate synchronization around measured regions so that async GPU work is accounted for correctly. If backend-specific timing APIs differ, the benchmark should isolate backend timing behavior behind helper functions. ### Backend Support Initial implementation should target CUDA first because it is the most common validation environment. MUSA or other backend support can be added after confirming: * build support, * hardware availability, * expected backend flags, * whether the same benchmark code path can be reused. The benchmark should not assume all contributors have access to every backend. ## CI Strategy The benchmark should not become a hard performance gate initially. Suggested CI behavior for the first version: 1. Run lightweight import/config parsing checks. 2. Run unit tests for routing distribution generation. 3. Optionally run a tiny single-process or mock smoke test if feasible. 4. Do not require multi-GPU performance results in standard CI. Future CI options: * self-hosted multi-GPU runner, * nightly benchmark job, * manual benchmark workflow, * threshold-based regression checks after the benchmark is stable. ## Implementation Plan ### Phase 1 : Minimal Benchmark Harness * Add `benchmarks/ep_benchmark/`. * Add CLI parser. * Add uniform routing mode. * Reuse existing EP setup pattern. * Measure dispatch/combine latency over warmup and measured iterations. * Emit JSON output. ### Phase 2 : Incast Routing * Add K-hot-expert routing mode. * Add hot expert fraction. * Add per-expert load metrics. * Add README examples. ### Phase 3 : Zipfian Routing and Sweep Support * Add Zipfian routing mode. * Add simple config-file support. * Add benchmark sweep examples. ### Phase 4 : Backend Parity * Add backend selection where supported. * Validate CUDA first. * Add MUSA or other backend support only where maintainers confirm hardware/toolchain availability. ### Phase 5 : Optional CI / Nightly Integration * Add smoke checks to standard CI. * Consider a manual or nightly benchmark job if suitable hardware exists. ## Validation Plan The benchmark should be validated in three ways: 1. **Correctness sanity check:** for small inputs, verify benchmark dispatch/combine outputs remain consistent with existing test behavior. 2. **Routing validation:** ensure uniform, K-hot, and Zipfian modes produce expected expert-load distributions. 3. **Performance sanity check:** run repeated benchmark executions on the same hardware and confirm variance is within an acceptable range. ## Alternatives Considered ### Extend Existing Correctness Test Rejected for initial implementation. Correctness tests and performance benchmarks have different goals. Mixing repeated-trial statistics, warmup, JSON output, and sweep logic into a correctness test would make both harder to maintain. ### Rely Only on External MoE Benchmarks External benchmarks may be useful for comparison, but Mooncake EP should have a benchmark that exercises its own public API and backend paths directly. ### Add CI Perf Gate Immediately Rejected for the first version. Kernel performance benchmarking requires stable hardware, stable drivers, and controlled environments. A hard CI gate should only be added after the benchmark is stable and the project has appropriate runners. ## Risks 1. **Benchmark noise:** GPU performance can vary across hosts and runs. 2. **Hardware availability:** contributors may not have access to multi-GPU or non-CUDA environments. 3. **Backend divergence:** CUDA and other backends may require slightly different timing or setup paths. 4. **Scope creep:** benchmark should remain focused on EP dispatch/combine, not full model serving. ## Open Questions 1. What hardware can maintainers use for official EP benchmark runs? 2. Should the first benchmark target only single-node multi-GPU, or should cross-node EP/RDMA behavior be included later? 3. Which backend combinations should be considered required for the first version? 4. Should benchmark results be stored in-tree as examples, or only generated locally? 5. What metric thresholds would be meaningful enough for future regression detection? ## Expected Outcome This RFC should produce a repeatable Mooncake EP benchmark that helps contributors measure dispatch/combine performance under both normal and incast traffic. The first implementation should be useful as a manual benchmark tool, with CI integration kept lightweight until stable multi-GPU runners are available. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi commented 2026-10-29 21:18:06 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 24, 2026):

Thanks for opening this issue, @Srinivasoo7!

Field	Value
Issue	#2607
GitHub user ID	194645829
Reporter	@Srinivasoo7

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4793351626 --> @github-actions[bot] commented on GitHub (Jun 24, 2026): Thanks for opening this issue, @Srinivasoo7! | Field | Value | |-------|-------| | **Issue** | #2607 | | **GitHub user ID** | `194645829` | | **Reporter** | @Srinivasoo7 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:18:06 +00:00

Author

Owner

Copy link

@PACTHEMAN123 commented on GitHub (Jul 30, 2026):

Hi, I'd like to take this up. I have access to a single node with 8× NVIDIA H20 GPUs and can implement and validate the CUDA incast benchmark. Is anyone already working on it? @Srinivasoo7

<!-- gh-comment-id:5130873037 --> @PACTHEMAN123 commented on GitHub (Jul 30, 2026): Hi, I'd like to take this up. I have access to a single node with 8× NVIDIA H20 GPUs and can implement and validate the CUDA incast benchmark. Is anyone already working on it? @Srinivasoo7

bsakaguchi commented 2026-10-29 21:18:06 +00:00

Author

Owner

Copy link

@Srinivasoo7 commented on GitHub (Jul 30, 2026):

Hi, I'd like to take this up. I have access to a single node with 8× NVIDIA H20 GPUs and can implement and validate the CUDA incast benchmark. Is anyone already working on it? @Srinivasoo7

Hi @PACTHEMAN123
Thanks for stepping up. Let's collaborate and work on this!

<!-- gh-comment-id:5130942160 --> @Srinivasoo7 commented on GitHub (Jul 30, 2026): > Hi, I'd like to take this up. I have access to a single node with 8× NVIDIA H20 GPUs and can implement and validate the CUDA incast benchmark. Is anyone already working on it? [@Srinivasoo7](https://github.com/Srinivasoo7) Hi @PACTHEMAN123 Thanks for stepping up. Let's collaborate and work on this!

bsakaguchi commented 2026-10-29 21:18:07 +00:00

Author

Owner

Copy link

@PACTHEMAN123 commented on GitHub (Jul 31, 2026):

Opened #3213 for Phases 1–3 (uniform, k-hot, zipfian routing + metrics). Phases 4–5 to follow. @Srinivasoo7 would appreciate a review when you have time.

<!-- gh-comment-id:5139415671 --> @PACTHEMAN123 commented on GitHub (Jul 31, 2026): Opened #3213 for Phases 1–3 (uniform, k-hot, zipfian routing + metrics). Phases 4–5 to follow. @Srinivasoo7 would appreciate a review when you have time.

bsakaguchi commented 2026-10-29 21:18:07 +00:00

Author

Owner

Copy link

@UNIDY2002 commented on GitHub (Aug 4, 2026):

Thanks for putting together this RFC, and for driving the initial implementation in #3213. A standalone benchmark for routing skew and incast traffic would be a useful addition to Mooncake EP.

One organizational question: since this benchmark directly exercises the Mooncake EP runtime, I think mooncake-ep/benchmarks/ would be a better home for the initial implementation. It would sit alongside the existing elastic_buffer_perf.py benchmark and follow the repository's component-local benchmark layout.

A top-level benchmarks/ location would become more natural if this grows into a cross-implementation benchmark that compares Mooncake EP with external backends or serving frameworks. Would you be open to moving the initial suite under mooncake-ep/benchmarks/ as part of #3213?

<!-- gh-comment-id:5181682374 --> @UNIDY2002 commented on GitHub (Aug 4, 2026): Thanks for putting together this RFC, and for driving the initial implementation in #3213. A standalone benchmark for routing skew and incast traffic would be a useful addition to Mooncake EP. One organizational question: since this benchmark directly exercises the Mooncake EP runtime, I think `mooncake-ep/benchmarks/` would be a better home for the initial implementation. It would sit alongside the existing `elastic_buffer_perf.py` benchmark and follow the repository's component-local benchmark layout. A top-level `benchmarks/` location would become more natural if this grows into a cross-implementation benchmark that compares Mooncake EP with external backends or serving frameworks. Would you be open to moving the initial suite under `mooncake-ep/benchmarks/` as part of #3213?

bsakaguchi commented 2026-10-29 21:18:08 +00:00

Author

Owner

Copy link

@PACTHEMAN123 commented on GitHub (Aug 5, 2026):

Thanks for putting together this RFC, and for driving the initial implementation in #3213. A standalone benchmark for routing skew and incast traffic would be a useful addition to Mooncake EP.

One organizational question: since this benchmark directly exercises the Mooncake EP runtime, I think mooncake-ep/benchmarks/ would be a better home for the initial implementation. It would sit alongside the existing elastic_buffer_perf.py benchmark and follow the repository's component-local benchmark layout.

A top-level benchmarks/ location would become more natural if this grows into a cross-implementation benchmark that compares Mooncake EP with external backends or serving frameworks. Would you be open to moving the initial suite under mooncake-ep/benchmarks/ as part of #3213?

Thanks for your advise, I have moved the suite to mooncake-ep/benchmarks/ep_benchmark/ in the new commit 6478ba2e, alongside elastic_buffer_perf.py. We can revisit a top-level location if it later grows into a cross-backend comparison.

<!-- gh-comment-id:5189939282 --> @PACTHEMAN123 commented on GitHub (Aug 5, 2026): > Thanks for putting together this RFC, and for driving the initial implementation in [#3213](https://github.com/kvcache-ai/Mooncake/pull/3213). A standalone benchmark for routing skew and incast traffic would be a useful addition to Mooncake EP. > > One organizational question: since this benchmark directly exercises the Mooncake EP runtime, I think `mooncake-ep/benchmarks/` would be a better home for the initial implementation. It would sit alongside the existing `elastic_buffer_perf.py` benchmark and follow the repository's component-local benchmark layout. > > A top-level `benchmarks/` location would become more natural if this grows into a cross-implementation benchmark that compares Mooncake EP with external backends or serving frameworks. Would you be open to moving the initial suite under `mooncake-ep/benchmarks/` as part of [#3213](https://github.com/kvcache-ai/Mooncake/pull/3213)? Thanks for your advise, I have moved the suite to `mooncake-ep/benchmarks/ep_benchmark/` in the new commit 6478ba2e, alongside `elastic_buffer_perf.py`. We can revisit a top-level location if it later grows into a cross-backend comparison.

bsakaguchi commented 2026-10-29 21:18:08 +00:00

Author

Owner

Copy link

@Srinivasoo7 commented on GitHub (Aug 23, 2026):

Thanks @PACTHEMAN123 for taking over the implementation, @UNIDY2002 for guiding us on the same!

<!-- gh-comment-id:5388888883 --> @Srinivasoo7 commented on GitHub (Aug 23, 2026): Thanks @PACTHEMAN123 for taking over the implementation, @UNIDY2002 for guiding us on the same!

bsakaguchi referenced this issue 2026-10-29 21:29:57 +00:00

[PR #683] [MERGED] bugfix(nvlink): Add explicit P2P access enablement and error handling for NvlinkTransport #1740

bsakaguchi referenced this issue 2026-10-29 21:31:02 +00:00

[PR #869] [MERGED] Fix nvlink_transport bug: revert #683 #1875

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#683

No description provided.