deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #493] [Question] Unexpectedly low bandwidth when scaling from 2 nodes to 4 nodes #319

New issue
Open
opened 2026-10-26 09:58:32 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:58:32 +00:00
Owner
Copy link

Originally created by @tkandi on GitHub (Nov 13, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/493

Hi,

I’m testing DeepEP across multiple nodes and found that bandwidth looks normal on 2 nodes, but drops sharply when scaling to 4 nodes.

Could you please suggest possible causes or configurations that might affect this? Any tips for debugging or tuning would be appreciated.

Setup:

GPUs: NVIDIA H200

Network: CX7 400 Gb/s InfiniBand

DeepEP version: a84a248(commit ID)

Test result(only list best):

# 2 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 20: 44.37 GB/s (RDMA), 145.16 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 16: 40.89 GB/s (RDMA), 133.78 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 24: 38.19 GB/s (RDMA), 124.92 GB/s (NVL)

# 4 nodes
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 8: 12.53 GB/s (RDMA), 25.05 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 12, RDMA chunk 4: 12.27 GB/s (RDMA), 24.51 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 4, RDMA chunk 4: 13.15 GB/s (RDMA), 26.27 GB/s (NVL)

Thanks!

Originally created by @tkandi on GitHub (Nov 13, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/493 Hi, I’m testing DeepEP across multiple nodes and found that bandwidth looks normal on 2 nodes, but drops sharply when scaling to 4 nodes. Could you please suggest possible causes or configurations that might affect this? Any tips for debugging or tuning would be appreciated. Setup: GPUs: NVIDIA H200 Network: CX7 400 Gb/s InfiniBand DeepEP version: a84a248(commit ID) Test result(only list best): ``` # 2 nodes [tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 20: 44.37 GB/s (RDMA), 145.16 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 28, RDMA chunk 16: 40.89 GB/s (RDMA), 133.78 GB/s (NVL) [tuning] Best dispatch (FP8): SMs 24, NVL chunk 28, RDMA chunk 24: 38.19 GB/s (RDMA), 124.92 GB/s (NVL) # 4 nodes [tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 8: 12.53 GB/s (RDMA), 25.05 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 12, RDMA chunk 4: 12.27 GB/s (RDMA), 24.51 GB/s (NVL) [tuning] Best dispatch (FP8): SMs 24, NVL chunk 4, RDMA chunk 4: 13.15 GB/s (RDMA), 26.27 GB/s (NVL) ``` Thanks!
bsakaguchi commented 2026-10-26 09:58:34 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Nov 14, 2025):

This is most likely related to the network. You can try to check the InfiniBand congestion statistics.

<!-- gh-comment-id:3530564093 --> @sphish commented on GitHub (Nov 14, 2025): This is most likely related to the network. You can try to check the InfiniBand congestion statistics.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#319
No description provided.