kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1421] [Help Wanted] High-performance send/recv implementation for Mooncake PG #456

New issue

Closed

opened 2026-10-29 21:14:52 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:14:52 +00:00

Owner

Originally created by @UNIDY2002 on GitHub (Jan 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1421

Originally assigned to: @Bruce-x-1997 on GitHub.

Motivation

Mooncake PG is a torch.distributed backend built on top of Mooncake TE, providing a full set of collective communication primitives. While most collectives are performance-oriented, the current implementation of point-to-point primitives (torch.send / torch.recv) is significantly under-optimized and has become a performance bottleneck in practice.

We are looking for help to design and implement a high-performance send/recv path for Mooncake PG.

Improving send/recv performance directly unlocks better scalability and responsiveness for elastic MoE and LLM serving systems, where dynamic data movement is on the critical path.

Use Case

A representative use case is Expert Parallel Load Balancing (EPLB) in LLM serving. In this workflow, expert parameters are dynamically exchanged across ranks using torch.distributed.batch_isend_irecv, which relies heavily on efficient send/recv under the hood.

Typical characteristics:

Frequent point-to-point transfers
Large payloads (can be up to 100 MB per message)
Sensitivity to bandwidth

Example: https://github.com/sgl-project/sglang/pull/12068 relies on high performance send/recv of Mooncake.

Constraints / Assumptions

Tensors are CUDA tensors
The transfer engine already supports GPUDirect RDMA
Send/recv is largely independent from other collective implementations
- Only the transfer-engine initialization logic is acturally shared

Notes for Contributors

Please do not limit designs to the current implementation, which is known to be inefficient
Alternative designs, data paths, or abstractions are all welcome

Where to Start

If you’re interested in contributing, the following entry points may be helpful:

MooncakeBackend::send, MooncakeBackend::recv implementations in mooncake_backend.cpp
PR #1236, which initially introduces send/recv.

Originally created by @UNIDY2002 on GitHub (Jan 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1421 Originally assigned to: @Bruce-x-1997 on GitHub. ## Motivation Mooncake PG is a `torch.distributed` backend built on top of Mooncake TE, providing a full set of collective communication primitives. While most collectives are performance-oriented, the current implementation of point-to-point primitives (`torch.send` / `torch.recv`) is **significantly under-optimized** and has become a performance bottleneck in practice. We are looking for help to design and implement a **high-performance send/recv path** for Mooncake PG. Improving send/recv performance directly unlocks **better scalability and responsiveness for elastic MoE and LLM serving systems**, where dynamic data movement is on the critical path. ## Use Case A representative use case is **Expert Parallel Load Balancing (EPLB)** in LLM serving. In this workflow, expert parameters are dynamically exchanged across ranks using `torch.distributed.batch_isend_irecv`, which relies heavily on efficient send/recv under the hood. Typical characteristics: - Frequent point-to-point transfers - Large payloads (can be up to 100 MB per message) - Sensitivity to bandwidth Example: https://github.com/sgl-project/sglang/pull/12068 relies on high performance send/recv of Mooncake. ## Constraints / Assumptions - Tensors are **CUDA tensors** - The transfer engine already supports **GPUDirect RDMA** - Send/recv is largely independent from other collective implementations - Only the transfer-engine initialization logic is acturally shared ## Notes for Contributors - Please do not limit designs to the current implementation, which is known to be inefficient - Alternative designs, data paths, or abstractions are all welcome ## Where to Start If you’re interested in contributing, the following entry points may be helpful: - `MooncakeBackend::send`, `MooncakeBackend::recv` implementations in `mooncake_backend.cpp` - PR #1236, which initially introduces send/recv.

bsakaguchi

2026-10-29 21:14:52 +00:00

closed this issue
added the
help wanted
label

bsakaguchi commented

2026-10-29 21:14:53 +00:00

Author

Owner

@staryxchen commented on GitHub (Jan 23, 2026):

Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness.

@staryxchen commented on GitHub (Jan 23, 2026): Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness.

bsakaguchi commented

2026-10-29 21:14:53 +00:00

Author

Owner

@yuechen-sys commented on GitHub (Jan 23, 2026):

Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness.

We could refer to nccl-test to implement our benchmark. It provides a comprehensive characterization of collective communication operations.

@yuechen-sys commented on GitHub (Jan 23, 2026): > Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness. We could refer to [nccl-test](https://github.com/NVIDIA/nccl-tests) to implement our benchmark. It provides a comprehensive characterization of collective communication operations.

bsakaguchi commented

2026-10-29 21:14:53 +00:00

Author

Owner

@staryxchen commented on GitHub (Jan 23, 2026):

Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness.

We could refer to nccl-test to implement our benchmark. It provides a comprehensive characterization of collective communication operations.

nccl-test is indeed a good benchmark. I believe we only need to implement a small portion of it.

@staryxchen commented on GitHub (Jan 23, 2026): > > Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness. > > We could refer to [nccl-test](https://github.com/NVIDIA/nccl-tests) to implement our benchmark. It provides a comprehensive characterization of collective communication operations. nccl-test is indeed a good benchmark. I believe we only need to implement a small portion of it.

bsakaguchi commented

2026-10-29 21:14:54 +00:00

Author

Owner

@yuechen-sys commented on GitHub (Jan 23, 2026):

Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness.

We could refer to nccl-test to implement our benchmark. It provides a comprehensive characterization of collective communication operations.

nccl-test is indeed a good benchmark. I believe we only need to implement a small portion of it.

Agreed. Currently, we only need standard metrics like bandwidth and latency to guide further development. We can align our benchmark's test cases with NCCL to understand the performance gap between NCCL and Mooncake PG. But the implementation could be simple.

@yuechen-sys commented on GitHub (Jan 23, 2026): > > > Can we develop a benchmark to measure current performance? Using this benchmark, we can quantify any optimization implemented, thereby evaluating their effectiveness. > > > > > > We could refer to [nccl-test](https://github.com/NVIDIA/nccl-tests) to implement our benchmark. It provides a comprehensive characterization of collective communication operations. > > nccl-test is indeed a good benchmark. I believe we only need to implement a small portion of it. Agreed. Currently, we only need standard metrics like [bandwidth](https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md) and latency to guide further development. We can align our benchmark's test cases with NCCL to understand the performance gap between NCCL and Mooncake PG. But the implementation could be simple.

bsakaguchi commented

2026-10-29 21:14:54 +00:00

Author

Owner

@Bruce-x-1997 commented on GitHub (Jan 26, 2026):

maybe I could try this one

@Bruce-x-1997 commented on GitHub (Jan 26, 2026): maybe I could try this one

bsakaguchi commented

2026-10-29 21:14:54 +00:00

Author

Owner

@ykwd commented on GitHub (Jan 28, 2026):

maybe I could try this one

@Bruce-x-1997 Many thanks. Do you have any thoughts or ideas so far on performance optimizations for PG?

@ykwd commented on GitHub (Jan 28, 2026): > maybe I could try this one @Bruce-x-1997 Many thanks. Do you have any thoughts or ideas so far on performance optimizations for PG?