[GH-ISSUE #361] H20's internode RDMA & NVL bandwidth drops to 50% after pr 245 #249

New issue

Open

opened 2026-10-26 09:57:21 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:57:21 +00:00

Owner

Originally created by @TheRainstorm on GitHub (Aug 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/361

Environment

Hardware: 2 nodes × 8× H20 GPUs,
Network: RoCE, CX6 InfiniBand NICs

Problem

Internode RDMA bandwidth dropped by ~50% after commit bc118b ( pr #245 ) , confirmed via bisect.

the commit before bc118b (9eb2f)

[tuning] Best dispatch (FP8): SMs 24, NVL chunk 24, RDMA chunk 4: 44.00 GB/s (RDMA), 144.10 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 8: 45.35 GB/s (RDMA), 148.50 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 43.93 GB/s (RDMA), 143.85 GB/s (NVL)

after commit bc118b

[tuning] Best dispatch (FP8): SMs 24, NVL chunk 8, RDMA chunk 12: 23.18 GB/s (RDMA), 75.91 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 12: 23.72 GB/s (RDMA), 77.68 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 23.41 GB/s (RDMA), 76.65 GB/s (NVL)

I have reviewed the modifications in the submission, but I find them too challenging to grasp. Could the developers reproduce and resolve this issue?

Originally created by @TheRainstorm on GitHub (Aug 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/361 ### Environment - Hardware: 2 nodes × 8× H20 GPUs, - Network: RoCE, CX6 InfiniBand NICs ### Problem Internode RDMA bandwidth dropped by ~50% after commit `bc118b` ( pr #245 ) , confirmed via bisect. the commit before `bc118b` (`9eb2f`) ```shell [tuning] Best dispatch (FP8): SMs 24, NVL chunk 24, RDMA chunk 4: 44.00 GB/s (RDMA), 144.10 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 8: 45.35 GB/s (RDMA), 148.50 GB/s (NVL) [tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 43.93 GB/s (RDMA), 143.85 GB/s (NVL) ``` after commit `bc118b` ```shell [tuning] Best dispatch (FP8): SMs 24, NVL chunk 8, RDMA chunk 12: 23.18 GB/s (RDMA), 75.91 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 4, RDMA chunk 12: 23.72 GB/s (RDMA), 77.68 GB/s (NVL) [tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 8: 23.41 GB/s (RDMA), 76.65 GB/s (NVL) ``` I have reviewed the modifications in the submission, but I find them too challenging to grasp. Could the developers reproduce and resolve this issue?

bsakaguchi commented

2026-10-26 09:57:23 +00:00

Author

Owner

@sphish commented on GitHub (Aug 18, 2025):

This change has resulted in a performance improvement in our environment, and we are unable to reproduce the reported issue.

Before this modification, RDMASenders prepare data in a one-by-one, sequential manner. After the change, RDMASenders can prepare data out of order within the send window. In theory, this increases the concurrency of RDMASenders, although it does slightly change the network transmission behavior.

We don‘t have H20 or CX6, so it might be helpful for others who have H20 or CX6 to share their testing results to aid in diagnosing the problem.

@sphish commented on GitHub (Aug 18, 2025): This change has resulted in a performance improvement in our environment, and we are unable to reproduce the reported issue. Before this modification, `RDMASenders` prepare data in a one-by-one, sequential manner. After the change, `RDMASenders` can prepare data out of order within the send window. In theory, this increases the concurrency of `RDMASenders`, although it does slightly change the network transmission behavior. We don‘t have H20 or CX6, so it might be helpful for others who have H20 or CX6 to share their testing results to aid in diagnosing the problem.

bsakaguchi commented

2026-10-26 09:57:23 +00:00

Author

Owner

@TheRainstorm commented on GitHub (Aug 18, 2025):

Before this modification, RDMASenders prepare data in a one-by-one, sequential manner. After the change, RDMASenders can prepare data out of order within the send window.

Though RDMASenders can prepare data in parallel, could the RDMASenderCoordinator still experience blocking? Specifically, if the processing of the N-th token becomes significantly delayed while tokens N+1 to N+10 complete rapidly, the tail pointer might remain stuck at position N, preventing pipeline advancement.

Could the redesigned code potentially introduce a stop-and-go transmission pattern:

Network idling for extended periods
Followed by sudden bursts of data traffic?

Conversely, would the legacy serial implementation maintain steady micro-bursts of data transfer, thereby preserving continuous network pipeline flow and potentially achieving higher bandwidth?

@TheRainstorm commented on GitHub (Aug 18, 2025): > Before this modification, `RDMASenders` prepare data in a one-by-one, sequential manner. After the change, `RDMASenders` can prepare data out of order within the send window. Though RDMASenders can prepare data in parallel, could the RDMASenderCoordinator still experience blocking? Specifically, if the processing of the N-th token becomes significantly delayed while tokens N+1 to N+10 complete rapidly, the tail pointer might remain stuck at position N, preventing pipeline advancement. Could the redesigned code potentially introduce a stop-and-go transmission pattern: - Network idling for extended periods - Followed by sudden bursts of data traffic? Conversely, would the legacy serial implementation maintain steady micro-bursts of data transfer, thereby preserving continuous network pipeline flow and potentially achieving higher bandwidth?

bsakaguchi referenced this issue

2026-10-27 21:02:59 +00:00

[PR #249] Computation communication overlap #485