deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #6] how to limit the alltoall op in 4 nodes #4

New issue

Closed

opened 2026-10-26 09:52:16 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:52:16 +00:00

Owner

Originally created by @rubbberrabbit on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/6

To effectively leverage the different bandwidths of IB and NVLink, we limit each
token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its
routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node
index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is
instantaneously forwarded via NVLink to specific GPUs that host their target experts, without
being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink
are fully overlapped, and each token can efficiently select an average of 3.2 experts per node
without incurring additional overhead from NVLink.
这一段的实现是和EP的实现代码有关系吗？

Originally created by @rubbberrabbit on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/6 To effectively leverage the different bandwidths of IB and NVLink, we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts, without being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink. 这一段的实现是和EP的实现代码有关系吗？

bsakaguchi referenced this issue

2026-10-26 09:52:16 +00:00

[GH-ISSUE #4] deepseek #5

bsakaguchi closed this issue

2026-10-26 09:52:16 +00:00

bsakaguchi commented

2026-10-26 09:52:17 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 25, 2025):

这一段的实现是和EP的实现代码有关系吗？

Yes.

You just give the correct topk_idx, and the all-to-all kernels will automatically do it. For example, if topk_idx[i] of token i indicate goes to Y experts in X nodes, the kernel will only send X RDMA messages and broadcast to Y experts via NVLink.

@LyricZhao commented on GitHub (Feb 25, 2025): > 这一段的实现是和EP的实现代码有关系吗？ Yes. You just give the correct `topk_idx`, and the all-to-all kernels will automatically do it. For example, if `topk_idx[i]` of token `i` indicate goes to $Y$ experts in $X$ nodes, the kernel will only send $X$ RDMA messages and broadcast to $Y$ experts via NVLink.

bsakaguchi commented

2026-10-26 09:52:17 +00:00

Author

Owner

@rubbberrabbit commented on GitHub (Feb 25, 2025):

thx for explain, it is really helpful. It is grateful to see you guys' work continue the best!

@rubbberrabbit commented on GitHub (Feb 25, 2025): thx for explain, it is really helpful. It is grateful to see you guys' work continue the best!

bsakaguchi commented

2026-10-26 09:52:17 +00:00

Author

Owner

@yanweian commented on GitHub (Feb 26, 2025):

这一段的实现是和EP的实现代码有关系吗？

Yes.

You just give the correct topk_idx, and the all-to-all kernels will automatically do it. For example, if topk_idx[i] of token i indicate goes to Y experts in X nodes, the kernel will only send X RDMA messages and broadcast to Y experts via NVLink.

Hi, the impl of the broadcast via NVLink is use multiple p2ps? or one broadcast? If use one broadcast, why 3.2 experts in "In this way, communications via IB and NVLink
are fully overlapped, and each token can efficiently select an average of 3.2 experts per node
without incurring additional overhead from NVLink."
Thanks.

@yanweian commented on GitHub (Feb 26, 2025): > > 这一段的实现是和EP的实现代码有关系吗？ > > Yes. > > You just give the correct `topk_idx`, and the all-to-all kernels will automatically do it. For example, if `topk_idx[i]` of token `i` indicate goes to Y experts in X nodes, the kernel will only send X RDMA messages and broadcast to Y experts via NVLink. Hi, the impl of the broadcast via NVLink is use multiple p2ps? or one broadcast? If use one broadcast, why 3.2 experts in "In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink." Thanks.

bsakaguchi commented

2026-10-26 09:52:17 +00:00

Author

Owner

@LyricZhao commented on GitHub (Feb 26, 2025):

We use CUDA IPC (memory semantics) for NVLink, so you just use normal ld instrs to read RDMA buffer (only once), and write into multiple NVL memory remote address using st instrs.
That's what we called "broadcast", nothing special.

@LyricZhao commented on GitHub (Feb 26, 2025): We use CUDA IPC (memory semantics) for NVLink, so you just use normal `ld` instrs to read RDMA buffer (only **once**), and write into **multiple** NVL memory remote address using `st` instrs. That's what we called "broadcast", nothing special.

bsakaguchi referenced this issue

2026-10-26 09:53:39 +00:00

[GH-ISSUE #74] test_internode.py hangs at internode.cu:dispatch(...) #68

bsakaguchi referenced this issue

2026-10-26 09:54:16 +00:00

[GH-ISSUE #106] how can I run the test across multi nodes？ #97

bsakaguchi referenced this issue

2026-10-26 09:56:29 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200

bsakaguchi referenced this issue

2026-10-26 09:56:40 +00:00

[GH-ISSUE #290] Running the test case on a 2-node H20 3E machine failed with internode.cu:1029 'unspecified launch failure' #208