mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #254] When running the two-machine test_low_latency.py (EP16), there is a significant difference in the test results between two machine #191
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#191
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @nannaer on GitHub (Jun 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/254
When running the two-machine test_low_latency.py (EP16), there is a significant difference in the test results of dispatch and combine on the two machines. My version number is
9fe9021, and I haven't modified any code. As follows. I'm not sure if it's my machine's network that's malfunctioning. @sphishMachine 1 8 GPU
Machine 2 8 GPU
Version
@sphish commented on GitHub (Jun 25, 2025):
This seems very similar to #183. We haven’t encountered this problem, but my guess is that it might be caused by misaligned kernel launch times. You can try generating a timeline using the torch profiler to take a closer look.
@liuhe-spec commented on GitHub (Jun 25, 2025):
We have also encountered this performance issue and identified that it seems from behavioral differences between IB and RoCE.
After applying the following configurations, performance normalized.
@sphish commented on GitHub (Jun 25, 2025):
I discussed this issue with @liuhe-spec on WeChat, and we strongly suspect it is likely related to RoCE network congestion control.
If possible, you can ask your cluster's network administrators to try the following actions:
@nannaer commented on GitHub (Jun 25, 2025):
Thanks! I will try it!
@polarstormx commented on GitHub (Jul 2, 2025):
@nannaer I'm also experiencing this issue. Are there any updates ? Thanks!
@sunmac commented on GitHub (Jul 3, 2025):
We encountered the same issue, but in the opposite direction: the dispatch bandwidth on the master node is low, while the combine bandwidth is high. The problem persists even after swapping the master and worker roles.
The issue was resolved after adding dist.barrier().
Can you determine whether the issue stems from network problems or the inference code itself?
@rubbberrabbit commented on GitHub (Jul 3, 2025):
We encounter the same issue, but in our Roce network their is no ecn which means DCQCN does not work. We disable the adaptive routing in enviroment. Can you explain more details about the reason about the AR actions ? Thanks !
@Enjia commented on GitHub (Jul 3, 2025):
Does that mean you indeed encounter congestion while running dispatch and combine than get that situation resolved through enabling AR?
@elevenxiang commented on GitHub (Jul 3, 2025):
I have profiled and found below truth:
finally, dispatch and combine will have huge performance gap !
@elevenxiang commented on GitHub (Jul 3, 2025):
Hi Manner,
Did you get any progress ?
Thanks
@sphish commented on GitHub (Jul 3, 2025):
You can try adding a dist.barrier before each alltoall operation
@Enjia commented on GitHub (Jul 3, 2025):
Thanks! That works in pure deepEP dispatch+combine test, and I find the same send/recv time mismatching phenomenon in SGLang when it executing cross node inference because SGLang guarantees no barrier when calling dispatch or combine
@Kevin-XiongC commented on GitHub (Jul 30, 2025):
@sphish Hi, I am facing the same issue (master node has low combine bw but high dispatch bw and vice versa for the other node) on an IB-based cluster. Do these actions apply to IB as well (IMO, IB does not have DCQCN and congestion control)?
I've added
group.barrier(), but it doesn't help. It is still far behind.@viralbhadeshiya commented on GitHub (Oct 1, 2025):
This works. I just tried it with my DeepEP setup and it stabilize bandwidth number in each rank. For anyone who is still having this problem :)
@kzlxd commented on GitHub (Nov 2, 2025):
@viralbhadeshiya
def bench_kineto(fn, kernel_names: Union[str, tuple], num_tests: int = 30, suppress_kineto_output: bool = False,
trace_path: Optional[str] = None, barrier_comm_profiling: bool = False,
num_kernels_per_period: int = 1):
# Profile
suppress = suppress_stdout_stderr if suppress_kineto_output else empty_suppress
with suppress():
schedule = torch.profiler.schedule(wait=1, warmup=0, active=1, repeat=1)
with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA], schedule=schedule) as prof:
for i in range(2):
# NOTES: use a large kernel and a barrier to eliminate the unbalanced CPU launch overhead
if barrier_comm_profiling:
lhs = torch.randn((8192, 8192), dtype=torch.float, device='cuda')
rhs = torch.randn((8192, 8192), dtype=torch.float, device='cuda')
lhs @ rhs
dist.all_reduce(torch.ones(1, dtype=torch.float, device='cuda'))
for _ in range(num_tests):
fn()
torch.cuda.synchronize()
prof.step()
I'm encountering the same situation here, but upon reviewing the code, I see that synchronisation has already been implemented prior to executing this function. Why then is synchronisation being added again before
test_func?@viralbhadeshiya commented on GitHub (Nov 4, 2025):
@kzlxd The synchronization inside bench_kineto() ensures that GPU work launched within the profiled window has completed before advancing the profiler step it only aligns CUDA kernel completion with the profiler timeline.
However, it does not guarantee inter-rank phase alignment. In test_low_latency.py, each rank can enter the dispatch or combine phase at slightly different times due to async collectives and kernel scheduling. This causes one rank’s dispatch window to overlap with the other’s combine phase, leading to asymmetric per-phase bandwidths even though total throughput is correct.
@kzlxd commented on GitHub (Nov 11, 2025):
@viralbhadeshiya thanks for your explaining