[PR #805] [MERGED] [Misc] Mooncake EP & Mooncake Backend #1834

New issue

Closed

opened 2026-10-29 21:30:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:30:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/805
Author: @UNIDY2002
Created: 9/4/2025
Status: ✅ Merged
Merged: 9/26/2025
Merged by: @alogfans

Base: main ← Head: sunxun/mooncake-backend-dev

📝 Commits (10+)

34d7deb Initialize a mooncake backend
fa80700 Add pybind
a401f32 Fix incorrect backend registration
b1e5dfc Fix wheel building of mooncake_ep
b535bbb Add a fake allreduce implementation
779b447 Introduce transfer_engine to mooncake_backend
fb8918e Add a basic CPU proxy execution framework
cf75d22 Implement a seemingly working allgather
1427db9 Remove mooncake_ep's dependency on etcd
fab5451 Implement _allgather_base

📊 Changes

46 files changed (+2532 additions, -992 deletions)

View changed files

📝 .github/workflows/ci.yml (+92 -0)
📝 .github/workflows/release.yaml (+97 -1)
📝 .typos.toml (+3 -4)
📝 CMakeLists.txt (+7 -0)
📝 doc/en/build.md (+5 -0)
➕ doc/en/ep-backend.md (+70 -0)
➕ mooncake-ep/CMakeLists.txt (+42 -0)
➖ mooncake-ep/csrc/CMakeLists.txt (+0 -25)
➖ mooncake-ep/csrc/mxa_ep.cpp (+0 -594)
➕ mooncake-ep/example/CMakeLists.txt (+0 -0)
➕ mooncake-ep/include/CMakeLists.txt (+0 -0)
➕ mooncake-ep/include/mooncake_backend.h (+86 -0)
📝 mooncake-ep/include/mooncake_ep_api.cuh (+4 -9)
➕ mooncake-ep/include/mooncake_ep_buffer.h (+168 -0)
📝 mooncake-ep/include/mooncake_ep_configs.cuh (+0 -0)
📝 mooncake-ep/include/mooncake_ep_event.h (+11 -8)
📝 mooncake-ep/include/mooncake_ep_exception.cuh (+0 -2)
📝 mooncake-ep/include/mooncake_ep_launch.cuh (+1 -1)
📝 mooncake-ep/include/mooncake_ep_utils.cuh (+3 -3)
📝 mooncake-ep/include/mooncake_ibgda/compiler.h (+0 -0)

...and 26 more files

📄 Description

In this PR, we propose Mooncake EP and the Mooncake Backend.

Mooncake EP is an adaptation of DeepEP that supports fault tolerance for large-scale MoE inference. It remains API-compatible with DeepEP, with an extra broken_ranks tensor to track failed ranks.

Mooncake Backend is a PyTorch distributed backend, designed as a fault-tolerant replacement for NCCL and Gloo. it can continue to perform collective communication under rank failures and reports them to upper layers for graceful handling.

Tests

Since the C++ APIs are not intended for direct use, no C++ unit tests are provided. Instead, three Python unit tests are included under mooncake-wheel/tests/:

test_mooncake_ep.py: Adapted from DeepEP’s test_low_latency.py. Verifies the correctness of the EP APIs and includes a basic performance test.
test_mooncake_backend.py: Validates the correctness of the Mooncake Backend.
test_mooncake_backend_perf.py: Compares the performance of the Mooncake Backend against NCCL and Gloo.

Performance

Tested on a 8 * H100 node.

Mooncake EP (pure RDMA)

Impl	Dispatch bandwidth	Dispatch latency	Combine bandwidth	Combine latency
Mooncake	41 GB/s	184 us	38 GB/s	387 us
DeepEP	46 GB/s	163 us	46 GB/s	318 us

Mooncake Backend

Here is the preliminary performance result of the Mooncake Backend. Further optimizations will be done in the future.

All data are in microseconds.

Mooncake v.s. Gloo

Allgather

Data Size	Mooncake	Gloo
1K	94	681
4K	125	834
16K	288	1121
64K	928	6253
256K	3715	8163
1M	7929	37067
4M	31239	142334

Allreduce

Data Size	Mooncake	Gloo
1K	87	1334
4K	163	1358
16K	476	1482
64K	1623	1606
256K	6382	2202
1M	23194	5324
4M	92664	15734

Broadcast

Data Size	Mooncake	Gloo
1K	61	101
4K	87	129
16K	142	177
64K	389	449
256K	1389	1130
1M	1662	2759
4M	7876	11559

Mooncake v.s. NCCL

Allgather

Data Size	Mooncake	NCCL
1K	67	93
4K	69	88
16K	78	93
64K	122	84
256K	293	81
1M	1038	178
4M	4158	521

Allreduce

Data Size	Mooncake	NCCL
1K	57	34
4K	60	30
16K	77	31
64K	122	30
256K	300	31
1M	1112	53
4M	14421	119

Broadcast

Data Size	Mooncake	NCCL
1K	50	28
4K	38	26
16K	47	27
64K	100	28
256K	246	34
1M	834	28
4M	3196	68

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/805 **Author:** [@UNIDY2002](https://github.com/UNIDY2002) **Created:** 9/4/2025 **Status:** ✅ Merged **Merged:** 9/26/2025 **Merged by:** [@alogfans](https://github.com/alogfans) **Base:** `main` ← **Head:** `sunxun/mooncake-backend-dev` --- ### 📝 Commits (10+) - [`34d7deb`](https://github.com/kvcache-ai/Mooncake/commit/34d7debe4fe6ca8a555214a00e8eaffa3580cf03) Initialize a mooncake backend - [`fa80700`](https://github.com/kvcache-ai/Mooncake/commit/fa807003947650e84caa09148f8f1e803a1caf69) Add pybind - [`a401f32`](https://github.com/kvcache-ai/Mooncake/commit/a401f320f7faa5e93b70d69e08305aeab82e013b) Fix incorrect backend registration - [`b1e5dfc`](https://github.com/kvcache-ai/Mooncake/commit/b1e5dfcbe819a87c35b726e0c8ff9d9071b51f7d) Fix wheel building of mooncake_ep - [`b535bbb`](https://github.com/kvcache-ai/Mooncake/commit/b535bbb495ac4bea79e7e3296b7a15e70aac26e2) Add a fake allreduce implementation - [`779b447`](https://github.com/kvcache-ai/Mooncake/commit/779b447bf92b82a90bdba3a8c3596eb9ec3533e2) Introduce transfer_engine to mooncake_backend - [`fb8918e`](https://github.com/kvcache-ai/Mooncake/commit/fb8918e4aae571aec23aab1b94560410f3856f51) Add a basic CPU proxy execution framework - [`cf75d22`](https://github.com/kvcache-ai/Mooncake/commit/cf75d220f6e7359ee18ece0d7f650f8ec16f0734) Implement a seemingly working allgather - [`1427db9`](https://github.com/kvcache-ai/Mooncake/commit/1427db920df937548e55984a67ec9e3961adaec8) Remove mooncake_ep's dependency on etcd - [`fab5451`](https://github.com/kvcache-ai/Mooncake/commit/fab5451e4509421cd9227bfcbf7b670849e6f4b1) Implement `_allgather_base` ### 📊 Changes **46 files changed** (+2532 additions, -992 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+92 -0) 📝 `.github/workflows/release.yaml` (+97 -1) 📝 `.typos.toml` (+3 -4) 📝 `CMakeLists.txt` (+7 -0) 📝 `doc/en/build.md` (+5 -0) ➕ `doc/en/ep-backend.md` (+70 -0) ➕ `mooncake-ep/CMakeLists.txt` (+42 -0) ➖ `mooncake-ep/csrc/CMakeLists.txt` (+0 -25) ➖ `mooncake-ep/csrc/mxa_ep.cpp` (+0 -594) ➕ `mooncake-ep/example/CMakeLists.txt` (+0 -0) ➕ `mooncake-ep/include/CMakeLists.txt` (+0 -0) ➕ `mooncake-ep/include/mooncake_backend.h` (+86 -0) 📝 `mooncake-ep/include/mooncake_ep_api.cuh` (+4 -9) ➕ `mooncake-ep/include/mooncake_ep_buffer.h` (+168 -0) 📝 `mooncake-ep/include/mooncake_ep_configs.cuh` (+0 -0) 📝 `mooncake-ep/include/mooncake_ep_event.h` (+11 -8) 📝 `mooncake-ep/include/mooncake_ep_exception.cuh` (+0 -2) 📝 `mooncake-ep/include/mooncake_ep_launch.cuh` (+1 -1) 📝 `mooncake-ep/include/mooncake_ep_utils.cuh` (+3 -3) 📝 `mooncake-ep/include/mooncake_ibgda/compiler.h` (+0 -0) _...and 26 more files_ </details> ### 📄 Description In this PR, we propose Mooncake EP and the Mooncake Backend. Mooncake EP is an adaptation of [DeepEP](https://github.com/deepseek-ai/DeepEP) that supports **fault tolerance** for large-scale MoE inference. It remains API-compatible with DeepEP, with an extra `broken_ranks` tensor to track failed ranks. Mooncake Backend is a PyTorch distributed backend, designed as **a fault-tolerant replacement for NCCL and Gloo**. it can continue to perform collective communication under rank failures and reports them to upper layers for graceful handling. Read more at `doc/en/ep-backend.md`. --- ## Tests Since the C++ APIs are not intended for direct use, no C++ unit tests are provided. Instead, three Python unit tests are included under `mooncake-wheel/tests/`: - **test_mooncake_ep.py**: Adapted from [DeepEP’s test_low_latency.py](https://github.com/deepseek-ai/DeepEP/blob/df4debe30c257b2ed186ec209ea76ea70b39cf75/tests/test_low_latency.py). Verifies the correctness of the EP APIs and includes a basic performance test. - **test_mooncake_backend.py**: Validates the correctness of the Mooncake Backend. - **test_mooncake_backend_perf.py**: Compares the performance of the Mooncake Backend against NCCL and Gloo. --- ## Performance Tested on a 8 * H100 node. ### Mooncake EP (pure RDMA) | Impl | Dispatch bandwidth | Dispatch latency | Combine bandwidth | Combine latency | |:--------:|:--------:|:--------:|:--------:|:--------:| | Mooncake | 41 GB/s | 184 us | 38 GB/s | 387 us | | DeepEP | 46 GB/s | 163 us | 46 GB/s | 318 us | ### Mooncake Backend Here is the preliminary performance result of the Mooncake Backend. Further optimizations will be done in the future. All data are in microseconds. #### Mooncake v.s. Gloo **Allgather** | Data Size | Mooncake | Gloo | |:---------:|:--------:|:------:| | 1K | 94 | 681 | | 4K | 125 | 834 | | 16K | 288 | 1121 | | 64K | 928 | 6253 | | 256K | 3715 | 8163 | | 1M | 7929 | 37067 | | 4M | 31239 | 142334 | **Allreduce** | Data Size | Mooncake | Gloo | |:---------:|:--------:|:------:| | 1K | 87 | 1334 | | 4K | 163 | 1358 | | 16K | 476 | 1482 | | 64K | 1623 | 1606 | | 256K | 6382 | 2202 | | 1M | 23194 | 5324 | | 4M | 92664 | 15734 | **Broadcast** | Data Size | Mooncake | Gloo | |:---------:|:--------:|:------:| | 1K | 61 | 101 | | 4K | 87 | 129 | | 16K | 142 | 177 | | 64K | 389 | 449 | | 256K | 1389 | 1130 | | 1M | 1662 | 2759 | | 4M | 7876 | 11559 | #### Mooncake v.s. NCCL **Allgather** | Data Size | Mooncake | NCCL | |:---------:|:--------:|:------:| | 1K | 67 | 93 | | 4K | 69 | 88 | | 16K | 78 | 93 | | 64K | 122 | 84 | | 256K | 293 | 81 | | 1M | 1038 | 178 | | 4M | 4158 | 521 | **Allreduce** | Data Size | Mooncake | NCCL | |:---------:|:--------:|:------:| | 1K | 57 | 34 | | 4K | 60 | 30 | | 16K | 77 | 31 | | 64K | 122 | 30 | | 256K | 300 | 31 | | 1M | 1112 | 53 | | 4M | 14421 | 119 | **Broadcast** | Data Size | Mooncake | NCCL | |:---------:|:--------:|:------:| | 1K | 50 | 28 | | 4K | 38 | 26 | | 16K | 47 | 27 | | 64K | 100 | 28 | | 256K | 246 | 34 | | 1M | 834 | 28 | | 4M | 3196 | 68 | --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:30:43 +00:00