mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #137] [test_internode.py] failed on multi-QP: dispatch timeout on ROCE network with testing 2*H20 nodes #119
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#119
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jeffye-dev on GitHub (Apr 27, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/137
When I run the across-node test with
MASTER_ADDR=<ip> MASTER_PORT=30001 WORLD_SIZE=2 RANK=0 python test_internode.pyon 2*H20 nodes, I got the following timeout log:This issue only happens after the Multi-QP patch:
github.com/deepseek-ai/DeepEP@5ab80c28f3is merged.It's probably related with multi-QP.
@sphish commented on GitHub (Apr 27, 2025):
Is adaptive routing enabled in your NIC and switch configuration?
@sphish commented on GitHub (Apr 27, 2025):
This change has been successfully tested in our own IB environment and in RoCE environments of several cloud service providers. However, I have indeed discovered that it fails in some RoCE environments where adaptive routing is enabled.
@jeffye-dev commented on GitHub (Apr 27, 2025):
The AR is OFF in my environment. Is it caused by multi-QP. I tried earlier version and find it's working.
@alpha-baby commented on GitHub (Apr 28, 2025):
@jeffye-dev try set this var to
Truegithub.com/deepseek-ai/DeepEP@5ab80c28f3 (diff-c77f4e0d77)@alpha-baby commented on GitHub (Apr 28, 2025):
github.com/deepseek-ai/DeepEP@007fcfcf97 (diff-c77f4e0d77)in the latest version, not allow set
test_ll_compatibility=False? @sphishBecause test will be failed when set
test_ll_compatibility=False. test will be success when set test_ll_compatibility=True`.failed log:
@sphish commented on GitHub (Apr 28, 2025):
@alpha-baby Is this issue occurring intermittently, or does it happen every time you run the test? I haven't encountered this problem in our own testing environment. Have you recompiled the C code?
@alpha-baby commented on GitHub (Apr 28, 2025):
I use this commit code:
github.com/deepseek-ai/DeepEP@007fcfcf97 (diff-c77f4e0d77)I just modified the variable test_ll_compatibility=False in the
test_internode.pyfile, and didn't recompile the C code, which can always be reproduced in my environment. My test environment uses ROCE.@sphish
@sphish commented on GitHub (Apr 28, 2025):
@alpha-baby What I meant is, after switching to this commit, did you recompile the C code? If you haven't done so, you should recompile the C code.
@alpha-baby commented on GitHub (Apr 28, 2025):
yes, i recompiled the C code. I found that using the new commit really improved the performance.
@sphish commented on GitHub (Apr 28, 2025):
@alpha-baby It's quite strange. I can't reproduce this issue. Theoretically, setting
test_ll_compatibility=Falsenow only changes the way NVSHMEM init group, and shouldn't affect correctness.@Qizhi697 commented on GitHub (Apr 29, 2025):
I encountered this issue on 2*H800, and it occurs whether test_ll_compatibility is set to True or False. It may happen occasionally, so you can try multiple times to reproduce this issue.
@whybeyoung commented on GitHub (Apr 30, 2025):
the same error after upgrade to multi-qp relation code
@whybeyoung commented on GitHub (Apr 30, 2025):
the same
@whybeyoung commented on GitHub (May 1, 2025):
log
@alpha-baby commented on GitHub (May 6, 2025):
my env:
two node
@sphish In my test environment, I analyzed this problem. When I configure these two environment variables(
NVSHMEM_ENABLE_NIC_PE_MAPPING=1 NVSHMEM_HCA_PE_MAPPING="mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"), the program will time out and reappear 100%.If I don't configure these two environment variables, the program won't time out, but the bandwidth is only over 20 GB/s.
config:
test_ll_compatibility=True
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_PE_MAPPING="mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
test result:
config:
test_ll_compatibility=True
export NVSHMEM_HCA_LIST="mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1"
test result:
@whybeyoung
Have you also configured these two environment variables?
NVSHMEM_ENABLE_NIC_PE_MAPPING=1 NVSHMEM_HCA_PE_MAPPING="mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
Can you show me your machine environment?
command:
nvidia-smi topo -mibv_devinfo@sphish commented on GitHub (May 6, 2025):
Hi there, could you please try this branch https://github.com/deepseek-ai/DeepEP/tree/try_fix_roce_mqp and see if it resolves the issue?
@alpha-baby commented on GitHub (May 6, 2025):
Thank you for your help.
The submission still does not solve the problem, or the same error is reported. I tried many times and it was 100% reappearance.
@whybeyoung commented on GitHub (May 6, 2025):
@alpha-baby commented on GitHub (May 8, 2025):
I found a way to fix the problem.
config env:
hack in nvshmem :
@sphish commented on GitHub (May 9, 2025):
@alpha-baby Hi, I'm not very familiar with bonded NICs. Would it be possible to add me on WeChat: Sphizzz? I have a few questions I'd like to ask.
@whybeyoung commented on GitHub (May 11, 2025):
your solution not works for me
@whybeyoung commented on GitHub (May 11, 2025):
yes this resolve my test timeout problem
@polarstormx commented on GitHub (May 13, 2025):
@sphish May I ask if this change will be incorporated into the main branch?
@sphish commented on GitHub (May 14, 2025):
@polarstormx Have you tested this modification and confirmed its effectiveness? Since I haven't been able to reproduce this issue internally, I need to collect some feedback.
@cscyuge commented on GitHub (May 14, 2025):
not work on H20, tested with latest DeepEP:
bb393e7760@alpha-baby commented on GitHub (May 14, 2025):
@cscyuge Can you show me your machine environment? command:
nvidia-smi topo -mibv_devinfo@cscyuge commented on GitHub (May 14, 2025):
nvidia-smi topo -m：
ibv_devinfo:
@alpha-baby commented on GitHub (May 14, 2025):
Your machine topo environment is different from mine, mine only has four network cards.
My patch should not apply to you, so you don't need to configure
NVSHMEM_ENABLE_NIC_PE_MAPPINGconfiguration.you should config:
NVSHMEM_ENABLE_NIC_PE_MAPPING=0@cscyuge@cscyuge commented on GitHub (May 14, 2025):
I have applied the change of
nvshmemi_setup_connectionsand tried 4 8*H20 nodes with:I got the same timeout error running with the script above.
And I tried 2 8*H20 nodes, got another error message:
If I remove
NVSHMEM_ENABLE_NIC_PE_MAPPINGandNVSHMEM_HCA_PE_MAPPING, 2 8*H20 nodes can pass test_internode.py (can also pass without the change ofnvshmemi_setup_connections),but 4 nodes still timeout.
@alpha-baby commented on GitHub (May 15, 2025):
you should config:
@alpha-baby commented on GitHub (May 15, 2025):
@cscyuge look this: https://github.com/deepseek-ai/DeepEP/issues/155#issuecomment-2879733393
@cscyuge commented on GitHub (May 16, 2025):
I tried
with
NVSHMEM_ENABLE_NIC_PE_MAPPING=1andNVSHMEM_ENABLE_NIC_PE_MAPPING=0,also tried with hacked
nvshmemi_setup_connectionsand without hackednvshmemi_setup_connections,still time out with 4 8*H20 nodes.
But i can pass test_internode.py on 2 8*H20 nodes without the configurations above.
@guyueh1 commented on GitHub (May 20, 2025):
I am seeing the same kind of timeout in internode unit test that always reproduces. I am using the code commit
github.com/deepseek-ai/DeepEP@a84a24808fso it's before multi-QP was merged.And my machine topology is
Any suggestions on how to fix it?
@alpha-baby commented on GitHub (May 21, 2025):
You have configured this environment variable: NVSHMEM_HCA_PE_MAPPING. If you want this environment variable to take effect, you need to configure this environment variable: nvshmem _ enable _ NIC _ PE _ mapping = 1. Please understand the explanation of this environment variable in the official document.
@alpha-baby commented on GitHub (May 21, 2025):
GPU0 -> use -> NIC4
GPU1 -> use -> NIC7
...
@cong543 commented on GitHub (Jun 4, 2025):
@alpha-baby ，I used 2xH20 node and tried the methods mentioned above, but both failed. The commands I executed were like this:
both two node:
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2"test_internode.py
Rank0:
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_4 NCCL_SOCKET_IFNAME=ens19f0 MASTER_ADDR=172.16.1.134 WORLD_SIZE=2 RANK=0 python DeepEP/tests/test_internode.pyRank1:
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_4 NCCL_SOCKET_IFNAME=ens19f0 MASTER_ADDR=172.16.1.134 WORLD_SIZE=2 RANK=1 python DeepEP/tests/test_internode.pyError:
test_low_latency
Rank0:
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_4 NCCL_SOCKET_IFNAME=ens19f0 MASTER_ADDR=172.16.1.134 WORLD_SIZE=2 RANK=0 python DeepEP/tests/test_low_latencyRank1:
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_4 NCCL_SOCKET_IFNAME=ens19f0 MASTER_ADDR=172.16.1.134 WORLD_SIZE=2 RANK=1 python DeepEP/tests/test_low_latencyError:
Environment：
@cong543 commented on GitHub (Jun 4, 2025):
@whybeyoung ，I also encountered a problem similar to yours.
"Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x5577a8d44)"."DeepEP timeout check failed: 0".Could you share your DeepEP environment? I built it using this container method:
github.com/sgl-project/sglang@180ff5eecc/docker/Dockerfile.deepep@alpha-baby commented on GitHub (Jun 10, 2025):
@cong543 you can use this patch to fix segmentation fault when config
NVSHMEM_HCA_PE_MAPPINGpatch for nvshmem
@alokprasad commented on GitHub (Nov 4, 2025):
where to set test_ll_compatibility=True , i dont see that in latest code