deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[PR #605] [MERGED] [Public release 26/04] Introducing EPv2: faster EP, and Engram/PP/CP supports #658

New issue
Closed
opened 2026-10-27 21:04:57 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-27 21:04:57 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/605
Author: @LyricZhao
Created: 4/23/2026
Status: ✅ Merged
Merged: 4/29/2026
Merged by: @LyricZhao

Base: main ← Head: epv2-release

📝 Commits (7)

📊 Changes

106 files changed (+14948 additions, -3932 deletions)

View changed files

➕ .editorconfig (+58 -0)
📝 .gitignore (+24 -4)
➕ .gitmodules (+3 -0)
➕ CMakeLists.txt (+52 -0)
📝 README.md (+324 -244)
➕ build.sh (+12 -0)
📝 csrc/CMakeLists.txt (+0 -35)
➖ csrc/deep_ep.cpp (+0 -1893)
➖ csrc/deep_ep.hpp (+0 -300)
➕ csrc/elastic/buffer.hpp (+1307 -0)
➕ csrc/elastic/utils.hpp (+28 -0)
➕ csrc/indexing/main.cu (+21 -0)
➕ csrc/jit/api.hpp (+20 -0)
➕ csrc/jit/cache.hpp (+34 -0)
➕ csrc/jit/compiler.hpp (+269 -0)
➕ csrc/jit/device_runtime.hpp (+67 -0)
➕ csrc/jit/handle.hpp (+155 -0)
➕ csrc/jit/include_parser.hpp (+80 -0)
➕ csrc/jit/kernel_runtime.hpp (+87 -0)
➕ csrc/jit/launch_runtime.hpp (+74 -0)

...and 80 more files

📄 Description

With the evolution of hardware, networking, and model architectures, the previous DeepEP V1 had accumulated too much legacy baggage and performance issues. Today, we are excited to introduce DeepEP V2, which includes a complete refactoring of Expert Parallelism — achieving extreme performance with several times fewer SM resources compared to V1, while supporting significantly larger scale-up and scale-out domains — as well as experimental 0 SM Engram, 0 SM Pipeline Parallelism, and 0 SM Context Parallelism all-gather.

We are also happy to announce that we have switched from the NVSHMEM backend to the more lightweight NCCL Gin backend.

New Features

  • Fully JIT (Just-In-Time compilation)
  • NCCL Gin backend
    • Header-only & lightweight
    • Able to reuse existing NCCL communicators
  • EPv2
    • CUDA graph compatible, or faster CPU sync in saving memory mode
    • High-throughput and low-latency APIs unified into a single interface, with a new GEMM layout
    • Larger scale-up & scale-out domain support (up to EP2048)
    • Analytical SM & QP count calculation — no more auto-tuning needed
    • Both hybrid & direct modes remain supported
    • For V3-like legacy training, SM usage reduced from 24 to 4 - 6 while maintaining equivalent or better performance
  • 0 SM Engram (with RDMA)
  • 0 SM PP (with RDMA)
  • 0 SM CP (with Copy Engine)

Notes

  • Buffer size consumption is larger than V1
  • 0 SM RDMA low-latency EP is no longer supported
  • Engram, PP, and CP are experimental features

Still On-going Features

  • Elastic GPU & CPU buffers: A contiguous virtual address space that maps to a hybrid of GPU and CPU physical memory under the hood, enabling fully automatic and transparent Engram or imbalanced EP
  • Reducing intermediate buffer sizes by leveraging EP replay to handle load imbalance
  • All-gather updates and reduce-scatter implementations for DP & TP

Performance

Following V3's configuration, we tested with 8K tokens per batch, 7168 hidden dimensions, top 8 experts, FP8 dispatching, and BF16 combining, and obtained the following results:

Arch NIC type Topo Dispatch Bottleneck Bandwidth Combine Bottleneck Bandwidth #SMs
SM90 CX7 EP 8 x 2 90 GB/s (RDMA) 81 GB/s (RDMA) 12
SM90 CX7 EP 8 x 4 61 GB/s (RDMA) 61 GB/s (RDMA) 6
SM100 CX7 EP 8 x 2 90 GB/s (RDMA) 91 GB/s (RDMA) 12
SM100 N/A EP 8 726 GB/s (NVLink) 740 GB/s (NVLink) 64 (Max perf)
SM100 N/A EP 8 643 GB/s (NVLink) 675 GB/s (NVLink) 24 (Min #SM)

Notes, the results are logical bandwidth. For example, under the EP 8 x 2 case, 90 GB/s actually contains local rank traffic.

Comparing with V1, V2 achieves up to 1.3x peak performance, while saving up to 4x SM count.

We omit results for larger EP configurations for the time being, but encourage interested users to benchmark them directly. Based on our internal experience, we expect the kernel to continue saturating hardware bandwidth at scale.

Contributors

  • Overall design & impls: @LyricZhao @sphish
  • Networking backends & elastic buffers: @sphish @tianr22
  • Optimizations: @interestingLSY @zheanxu @bucket-xv
  • Engram: @tianr22 @sphish @LyricZhao
  • PP & CP: @skip2004 @LyricZhao
  • Testing & benchmarking: @sphish @yukuai26
  • Acknowledgement: @sjeaugey @pakmarkthub @sb17v @xiaofanl-nvidia, and the NCCL team

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/605 **Author:** [@LyricZhao](https://github.com/LyricZhao) **Created:** 4/23/2026 **Status:** ✅ Merged **Merged:** 4/29/2026 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `epv2-release` --- ### 📝 Commits (7) - [`4151890`](https://github.com/deepseek-ai/DeepEP/commit/41518903e065390a0532be9ce6f8b922bdf53061) Merge with private repo - [`a6c8525`](https://github.com/deepseek-ai/DeepEP/commit/a6c8525998ca32fa9e75b8235498907dce8f272e) Fix compilation - [`6305c46`](https://github.com/deepseek-ai/DeepEP/commit/6305c4694d10df2c8d292dfcd229a771ecd68662) Add readme - [`3e490fb`](https://github.com/deepseek-ai/DeepEP/commit/3e490fb6f9e3949c07a3a2a4db70cb2162833a34) Update docs - [`bae7664`](https://github.com/deepseek-ai/DeepEP/commit/bae7664ba05930a49a69df83af66adf92476c891) Fix submodule - [`76f1427`](https://github.com/deepseek-ai/DeepEP/commit/76f14279f110278fec8e9b1a78b7a76c226c1845) Fix cu128 compilation - [`018f21e`](https://github.com/deepseek-ai/DeepEP/commit/018f21ea62c6c2888d9b46806aad6d90b399b23d) enhance: add env:EP_NIC_NAME to config nic name (#610) ### 📊 Changes **106 files changed** (+14948 additions, -3932 deletions) <details> <summary>View changed files</summary> ➕ `.editorconfig` (+58 -0) 📝 `.gitignore` (+24 -4) ➕ `.gitmodules` (+3 -0) ➕ `CMakeLists.txt` (+52 -0) 📝 `README.md` (+324 -244) ➕ `build.sh` (+12 -0) 📝 `csrc/CMakeLists.txt` (+0 -35) ➖ `csrc/deep_ep.cpp` (+0 -1893) ➖ `csrc/deep_ep.hpp` (+0 -300) ➕ `csrc/elastic/buffer.hpp` (+1307 -0) ➕ `csrc/elastic/utils.hpp` (+28 -0) ➕ `csrc/indexing/main.cu` (+21 -0) ➕ `csrc/jit/api.hpp` (+20 -0) ➕ `csrc/jit/cache.hpp` (+34 -0) ➕ `csrc/jit/compiler.hpp` (+269 -0) ➕ `csrc/jit/device_runtime.hpp` (+67 -0) ➕ `csrc/jit/handle.hpp` (+155 -0) ➕ `csrc/jit/include_parser.hpp` (+80 -0) ➕ `csrc/jit/kernel_runtime.hpp` (+87 -0) ➕ `csrc/jit/launch_runtime.hpp` (+74 -0) _...and 80 more files_ </details> ### 📄 Description With the evolution of hardware, networking, and model architectures, the previous DeepEP V1 had accumulated too much legacy baggage and performance issues. Today, we are excited to introduce **DeepEP V2**, which includes a complete refactoring of Expert Parallelism — achieving extreme performance with several times fewer SM resources compared to V1, while supporting significantly larger scale-up and scale-out domains — as well as experimental 0 SM Engram, 0 SM Pipeline Parallelism, and 0 SM Context Parallelism all-gather. We are also happy to announce that we have switched from the NVSHMEM backend to the more lightweight **NCCL Gin backend**. --- ### New Features - Fully JIT (Just-In-Time compilation) - NCCL Gin backend - Header-only & lightweight - Able to reuse existing NCCL communicators - EPv2 - CUDA graph compatible, or faster CPU sync in saving memory mode - High-throughput and low-latency APIs unified into a single interface, with a new GEMM layout - Larger scale-up & scale-out domain support (up to EP2048) - Analytical SM & QP count calculation — no more auto-tuning needed - Both hybrid & direct modes remain supported - For V3-like legacy training, SM usage reduced from 24 to 4 - 6 while maintaining equivalent or better performance - 0 SM Engram (with RDMA) - 0 SM PP (with RDMA) - 0 SM CP (with Copy Engine) --- ### Notes - Buffer size consumption is larger than V1 - 0 SM RDMA low-latency EP is no longer supported - Engram, PP, and CP are experimental features --- ### Still On-going Features - **Elastic GPU & CPU buffers**: A contiguous virtual address space that maps to a hybrid of GPU and CPU physical memory under the hood, enabling fully automatic and transparent Engram or imbalanced EP - Reducing intermediate buffer sizes by leveraging EP replay to handle load imbalance - All-gather updates and reduce-scatter implementations for DP & TP --- ### Performance Following V3's configuration, we tested with 8K tokens per batch, 7168 hidden dimensions, top 8 experts, FP8 dispatching, and BF16 combining, and obtained the following results: | Arch | NIC type | Topo | Dispatch Bottleneck Bandwidth | Combine Bottleneck Bandwidth | #SMs | |--|--|--|--|--|--| | SM90 | CX7 | EP 8 x 2 | 90 GB/s (RDMA) | 81 GB/s (RDMA) | 12 | | SM90 | CX7 | EP 8 x 4 | 61 GB/s (RDMA) | 61 GB/s (RDMA) | 6 | | SM100 | CX7 | EP 8 x 2 | 90 GB/s (RDMA) | 91 GB/s (RDMA) | 12 | | SM100 | N/A | EP 8 | 726 GB/s (NVLink) | 740 GB/s (NVLink) | 64 (Max perf) | | SM100 | N/A | EP 8 | 643 GB/s (NVLink) | 675 GB/s (NVLink) | 24 (Min #SM) | Notes, the results are logical bandwidth. For example, under the `EP 8 x 2` case, 90 GB/s actually contains local rank traffic. Comparing with V1, V2 achieves up to 1.3x peak performance, while saving up to 4x SM count. We omit results for larger EP configurations for the time being, but encourage interested users to benchmark them directly. Based on our internal experience, we expect the kernel to continue saturating hardware bandwidth at scale. --- ### Contributors - Overall design & impls: @LyricZhao @sphish - Networking backends & elastic buffers: @sphish @tianr22 - Optimizations: @interestingLSY @zheanxu @bucket-xv - Engram: @tianr22 @sphish @LyricZhao - PP & CP: @skip2004 @LyricZhao - Testing & benchmarking: @sphish @yukuai26 - Acknowledgement: @sjeaugey @pakmarkthub @sb17v @xiaofanl-nvidia, and the NCCL team --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-27 21:04:57 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#658
No description provided.