mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #54] python tests/test_low_latency.py Hang on a single node #51
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#51
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Huixxi on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/54
On single node with 8 H800, the program Hang after NCCL INFO [6] NCCL INFO Connected all rings
@sphish commented on GitHub (Mar 9, 2025):
Is the program hanging inside NCCL?
@Huixxi commented on GitHub (Mar 9, 2025):
Yes, it's absolutely right. test_internode also have the same problem. torch.dist can't connection through RoCE RDMA after I install the gdrcopy v2.4.4. I don't know why.
@sphish commented on GitHub (Mar 10, 2025):
I haven't encountered this issue before. Maybe you can submit an issue to gdrcopy repo to ask about it.
@Huixxi commented on GitHub (Mar 11, 2025):
I've check the code and find it hang at deep_ep/buffer.py
self.runtime.sync(device_ids, ipc_handles, root_unique_id), any suggestions?@sphish commented on GitHub (Mar 11, 2025):
@Huixxi commented on GitHub (Mar 11, 2025):
Ok, thanks. I've checked that all ranks could print the log text above this line codes. And after I added this flag, the final logs looks like this then hang:
@sphish commented on GitHub (Mar 11, 2025):
It seems that there is no useful information from the logs. Perhaps you could use gdb to attach and check the stack trace.
@Huixxi commented on GitHub (Mar 11, 2025):
there seems a cuda error: cudaErrorInvalidValue 1 cudaErrorInvalidSymbol 13 cudaErrorInvalidMemcpyDirection 21 cudaErrorNoKernelImageForDevice 209 does it matter?
@sphish commented on GitHub (Mar 11, 2025):
This is a normal log. Doesn't matter.
@Huixxi commented on GitHub (Mar 11, 2025):
Some gdb info:
@sphish commented on GitHub (Mar 11, 2025):
Do all ranks reach this barrier?
@Huixxi commented on GitHub (Mar 13, 2025):
Thx. We've checked this problem and initially determined that it is associated with virtual network possibly. Maybe need to close the sriov's vf and use hostnetwork directly.
@zhouheyang-alibaba commented on GitHub (Mar 13, 2025):
I've met the same problem on sriov vf network. Is there any way to solve this problem on a virtualized network?
@DeepTecher commented on GitHub (Dec 3, 2025):
+1
@alpha-baby commented on GitHub (Dec 19, 2025):
maybe you can try off PCIe ACS or open ACS Upstream Forwarding.
detail: https://developer.aliyun.com/article/1071405
@Ken-Liu commented on GitHub (Dec 19, 2025):
Looks like the NIC has VFs enabled, but the PCIe switch chip on the target server doesn't support routing VF BAR space addresses. This causes the GPU's PCIe posted writes to be dropped when it rings the doorbell.
This issue typically occurs only in PCIe downstream ports.
To work around this issue, we can redirect such TLP packets to the Root Complex (RC) for indirect forwarding via the upstream port when they do not match any routing rules. This can be achieved by enabling the ACS (Access Control Services) Upstream Forwarding feature.
intel iommu=OFF;
NIC ATS=OFF
PCIE ACS=0010 (Upstream Forwarding Enable)
Only PCIe posted requests (like doorbell rings) take the forwarding path shown in the next picture, so they don’t really affect overall performance.