mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #74] test_internode.py hangs at internode.cu:dispatch(...) #68
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#68
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @yanminjia on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/74
When I run test_internode.py on 2 nodes with 16 GPUs, it hang at the kernel function internode.cu:dispatch(...).
And when reached the timeout (NUM_TIMEOUT_CYCLES), it crashed because trap() was called with following error messages:
It would be highly appreciated if any suggestion.
Many thanks.
@LyricZhao commented on GitHub (Mar 14, 2025):
Which commit are you using? Can you try the latest one? Normal kernels have similar problems with adaptive routing (AR) enabled, and now AR is fully supported.
@yanminjia commented on GitHub (Mar 14, 2025):
Many thanks for your response. We are using this commit.
We are testing DeepEP over RoCE network. I'm not sure if the latest code fixed some kind of problem relevant to the RoCE. Anyway, will try the latest code.
@yanminjia commented on GitHub (Mar 14, 2025):
Still cannot work with the latest code. It looks hanging at the same code. By the way, IBRC is used in nvshmem layer in my test case.
@LyricZhao commented on GitHub (Mar 17, 2025):
@sphish Do you have some idea about RoCE?
@sphish commented on GitHub (Mar 17, 2025):
Is adaptive routing enabled in your environment? Also, Are SR-IOV and IOMMU enabled on the host machine?
@zhoumo02 commented on GitHub (Mar 18, 2025):
I have the same problem, I run test_internode.py on 4 nodes with 32 GPUs, the timeout occurred in combine in addition to dispatch. We also used the latest version with adaptive routing enabled, iommu=pt
Hang in dispatch:
DeepEP dispatch NVL receiver timeout, channel: 10, ROMA: 1, nvl: 1, src RDMA: 3, src nul: 8, start: 8, end: 8
DeepEP dispatch NVL receiver timeout, channel: 10, RDMA: 1, nvl: 5, src ROMA: 3, src nvl: 0, start: 8, end: 0
DeepEP dispatch NVL receiver timeout, channel: 6, RDMA: 1, nvl: 6, sre ROMA: 3, src nvl: 0, start: 0, end: 0
Hang in combine RDMA receiver:
DeepEP combine RDMA receiver timeout, channel: 8, RDMA: 3, nvl: 5, src RDMA: 3, tail: 0, waiting: 902, expect: 18
DeepEP combine RDMA receiver timeout, channel: 8, RDMA: 3, nvl: 5, src RDMA: 3, tail: 0, waiting: 898, expect: 14
DeepEP combine RDMA receiver timeout, channel: 8, RDMA: 3, nvl: 5, src RDMA: 3, tail: 0, waiting: 886, expect: 6
DeepEP combine RDMA receiver timeout, channel: 8, RDMA: 3, nvl: 5, src RDMA: 3, tail: 0, waiting: 903, expect: 19
DeepEP combine RDMA receiver timeout, channel: 8, RDMA: 3, nvl: 5, src RDMA: 3, tail: 0, waiting: 882, expect: 2
Hang in combine RDMA forwarder:
DeepEP combine formarder (WL check) timeout, chanel: 11, ROMA: 1, nvl: 7, src WL: 0, dst RDMA: 0, tail: 0, waiting: 1, total: 68, sub: 1, large: 4, expected: 0
DeepEP combine forwarder (NWL check) timeout, channel: 7, RDMA: 1, nvl: 7, src NVL: 8, dst RDMA: 0, tail: 8, waiting: 2, total: 78, sub: 2, large: 4, expected: 0
DepEP combine forwarder (WL check) timeout, channel: 4, RDMA: 1, nvl: 7, src NWL: 0, dst ROMA: 8, tail: 8, waiting: 1, total: 69, sub: 1, large: 4, expected: 1
DeepEP combine forwarder (WL check) timeout, channel: 4, ROMA: 1, nvl: 7, src WL: 0, dst RDMA: 0, tail: 0, waiting: 0, total: 69, sub: 0, large: 4 expected: 0
Any suggestions can offer?
Many thanks！
@sphish commented on GitHub (Mar 19, 2025):
This appears to be related to some ordering issues. Could you try disabling adaptive routing to see if that resolves the problem? @zhoumo02
@zhoumo02 commented on GitHub (Mar 19, 2025):
Thanks！I will try @sphish
@yanminjia commented on GitHub (Mar 19, 2025):
No. Adaptive routing is not available in our RoCE enironment. Additionally, SR-IOV and IOMMU are closed on the side of GPU server.
@sphish commented on GitHub (Mar 19, 2025):
What GPUs and NICs are you using?
@yanminjia commented on GitHub (Mar 19, 2025):
GPU: H100
RNIC: dual-port ConnectX-7
As you know, Only single-port RNIC is supported by IBRC in NVSHMEM code. We modified the NVSHMEM code to support dual-port RNIC in IBRC. Basically, RDMA RPs are built on the 2 ports of the dual-port RNIC. And the traffic is distributed to the RPs on the 2 ports of the dual-port RNIC. It looks this hanging problem is relavant to the AMO operation associated with the dual-port RNIC. We have a walk around solution - AMO WR is only applied to the QP when the WRs on the other port of dual-port RNIC are served by calling check_poll_avail(ep, WAIT_ALL) in ibrc.cc:nvshmemt_ibrc_amo(...). But we guess this extra latency would have some kind of impact on performance.
@sphish commented on GitHub (Mar 19, 2025):
I don't have experience with dual-port NICs. What is RP you mentioned? Also, would traffic of a single QP be distributed across both ports? If that's the case, this could indeed cause the bug you're experiencing. @yanminjia
@sphish commented on GitHub (Mar 19, 2025):
I think one solution would be to pin each QP's traffic to a specific port, which should prevent this ordering issues.
@yanminjia commented on GitHub (Mar 19, 2025):
Sorry for this typo. It's RDMA QP.
The traffic of a specific QP has no way to be distributed to both ports. We build 2 QPs on the 2 ports of the dual-port RNIC. And the traffic issued from the GPU corresponding to this dual-port RNIC is distributed to these 2 QPs to fully utilize the bandwidth provided by this dual-port RNIC.
@yanminjia commented on GitHub (Mar 19, 2025):
So, we need to created a seperate QP for AMO operation, right? @sphish
@sphish commented on GitHub (Mar 19, 2025):
In the normal kernels, atomic operations are used to move buffer pointers, and you need to guarantee ordering between the atomic operation and the preceding put data operations. Therefore, isolating atomic operations to a separate QP isn't feasible.
I think a viable solution would be to partition our channels: assign all RDMA operations from half of the channels to one QP (one port), and all operations from the other half of the channels to another QP. This way, we maintain the necessary ordering while utilizing both ports.
@yanminjia commented on GitHub (Mar 19, 2025):
So the data and the associated amo operation from the same channel should go into the same QP. That could ensure the necessary ordering of RDMA and AMO operation, right?
@sphish commented on GitHub (Mar 19, 2025):
Yes
@yanminjia commented on GitHub (Mar 19, 2025):
Many thanks for your kindly help! @sphish