mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #38] Test test_low_latency.py failed on H100 with ROCE #36
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#36
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ImbaPlayer on GitHub (Mar 3, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/38
Issue Description
I'm working on an H100 GPU cluster with RoCE drivers properly installed on the network interface cards.
While the
test_intranode.pyscript runs successfully and produces expected results, thetest_low_latencyconsistently fails with errors.Technical Details:
I would greatly appreciate any assistance or insights to resolve this. Below are the specific error messages for reference:
Actual Result
Env
Ubuntu2204
nvidia-smi
ibv_devinfo
@sphish commented on GitHub (Mar 4, 2025):
This is a bug in NVSHMEM 3.1.7 and can be resolved by using NVSHMEM 3.2.5.
https://github.com/deepseek-ai/DeepEP/issues/17#issuecomment-2684327121
@Baibaifan commented on GitHub (Mar 4, 2025):
How to deal with conflicts in deepep patch packages?
@sphish commented on GitHub (Mar 4, 2025):
For the current patch, the conflict is caused by a commit modifying the CMake file. You can skip this commit. I will later upload a new patch compatible with NVSHMEM 3.2.5.
@Baibaifan commented on GitHub (Mar 4, 2025):
I tried to modify the
CMakeLists.txtfile, butCMakeLists.txthas been refactored, and I don't know how to modify the original patch 140 and 165. I hope you can give me some suggestions. I'll wait for you to update the patch. @sphish@sphish commented on GitHub (Mar 5, 2025):
I have created a new branch that updates NVSHMEM to version 3.2.5. However, I don't have a RoCE environment for verification. Can you test this out? @Baibaifan
Branch: https://github.com/deepseek-ai/DeepEP/tree/roce-support
@Baibaifan commented on GitHub (Mar 5, 2025):
@sphish commented on GitHub (Mar 5, 2025):
Nope.
@kunfupanda-hw commented on GitHub (Mar 5, 2025):
I have the same problem on one 8*H100 cluster node for
test_low_latency.py, and update NVSHMEM to version 3.2.5 with the newest patch. However, it doesn't work, it lacks for, with the same logs, i.e.,/work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 /work/nvshmem_src_3.1.7-1/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:417: non-zero status: 110 ibv_modify_qp failed ibv_modify_qp failed. However, I have tried theib_write_bwcmd successfully. Any kind suggestion for this problem?@sphish commented on GitHub (Mar 6, 2025):
Could you provide the logs from your NVSHMEM 3.2.5 run? It appears you're still using NVSHMEM 3.1.7.
@Baibaifan commented on GitHub (Mar 6, 2025):
Hi @sphish, The process works, but the performance does not seem to meet expectations.
env:
rank0
NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=0 python tests/test_internode.pyrank1
NCCL_DEBUG=INFO MASTER_ADDR=xxx WORLD_SIZE=2 RANK=1 python tests/test_internode.pyThe difference between the above speed and the official data is greater than the difference between the network cards. How can I test the performance with
perftest? Is there any reference?@kunfupanda-hw commented on GitHub (Mar 6, 2025):
sorry for that old logs. The newest logs are as below, with some my comment logs.
By following the logs, we found the calltrace was
test_loop() --> buffer = deep_ep.Buffer() --> __init__ --> self.runtime.sync(device_ids, ipc_handles, root_unique_id)in DeepEP and thennvshmemt_ibrc_connect_endpoints --> nvshmemt_ibrc_ep_connect --> ep_connect --> status = ftable.modify_qp(ep->qp, &attr, flags);in nvshmem_src_3.2.5-1. The status110is usuallyETIMEDOUT。@Baibaifan commented on GitHub (Mar 6, 2025):
I also encountered this problem, but
tests/test_internode.pycan be run, but the performance is worse. Is it related to the fact that I have 4 NICs per machine? @sphish@kunfupanda-hw commented on GitHub (Mar 6, 2025):
Hi, @Baibaifan, are MASTER_ADDRs of rank0/1 same? Are they IP of CX7 cards or host IP？
@Baibaifan commented on GitHub (Mar 6, 2025):
MASTER_ADDRs are the same, both are host ip. It is to build Bootstrap use.
@sphish commented on GitHub (Mar 6, 2025):
Could you add the environment variable NVSHMEM_DEBUG=TRACE, run it again, and provide the log？
@sphish commented on GitHub (Mar 6, 2025):
We haven't tested on a machine with 4 nics, but I don't think the performance should be this low. Could you try running alltoall_perf from NCCL_tests?
@Baibaifan commented on GitHub (Mar 7, 2025):
After modifying the wrong
OOBconfiguration, the current speed of the 4 network NICs is:Is this reasonable, since I have 4 network NICs? @sphish
@kunfupanda-hw commented on GitHub (Mar 7, 2025):
Hi, @sphish,
In my setting, I run
test_internode.pyon two nodes andtest_intranode.pyon one node successfully, buttest_low_latency.pystill cannot pass withibv_modfiy_qpfailed. Then I runperftestcmd and get some observations:RTRstatus, e.g.,mlx5_0tomlx5_1, while succeed for same indexed CX cards, e.g.,mlx5_0tomlx5_0.perftestcmd pass.And I also set environment variable NVSHMEM_DEBUG=TRACE, and run it. Please see the part of logs as below. The logs show that the nvshmem driver tries to build EP connection with mutiple times, and finally failed.
@sphish commented on GitHub (Mar 9, 2025):
@Baibaifan What is
OOBconfiguration? Regarding the performance issue, I agree. It appears that the bandwidth is limited by the NICs.@Baibaifan commented on GitHub (Mar 10, 2025):
Could you please send me the command to run
perftestfor reference? @kunfupanda-hw@Baibaifan commented on GitHub (Mar 10, 2025):
@MinhuiWan commented on GitHub (Mar 10, 2025):
Hi @Baibaifan what's configuration have you changed, i have 4 IB per node, there is also the large difference between self-testing data （4.47GB/s RDMA） and offical data.
@kunfupanda-hw commented on GitHub (Mar 10, 2025):
server: ib_write_bw -d your_card_name -a --report_gbits -n 1000 -q 16 --CPU-freq -p 1
client: ib_write_bw -d your_card_name -a --report_gbits -n 1000 -q 16 your_ip --CPU-freq -p 1
@Baibaifan commented on GitHub (Mar 10, 2025):
I mean the tests in
perftest/perftest_install, in thenvshmem_srcdirectory. @kunfupanda-hw@Baibaifan commented on GitHub (Mar 10, 2025):
Please check the nccl network configuration. For example:
NCCL_IB_HCAandNCCL_SOCKET_IFNAME. @MinhuiWan@sphish commented on GitHub (Mar 10, 2025):
@Baibaifan If you are looking for some performance references for nvshmem perftest, there are some official P2P benchmark results in this link. As for collective communication, we haven't tested it either.
@sphish commented on GitHub (Mar 10, 2025):
@kunfupanda-hw I'm sorry, I don't have any ideas. Are you using a RoCE network? You can try running the shmem_put_bw in nvshmem. If that doesn't work, it might be an issue with your IP configuration.
@Baibaifan commented on GitHub (Mar 10, 2025):
@MinhuiWan commented on GitHub (Mar 10, 2025):
thanks for your reply @Baibaifan. I will set NCCL_IB_HCA and NCCL_SOCKET_IFNAME to test. But i have a question: DeepEP does not rely on NCCL, so why can NCCL environment variables be set to control it?
@Baibaifan commented on GitHub (Mar 10, 2025):
The unit test used nccl to create a communication group, and I found that there was a problem with my previous settings. @MinhuiWan
@MinhuiWan commented on GitHub (Mar 10, 2025):
@Baibaifan I want use IB to test. My environment is 2 nodes，8 H100 GPUs， 4 IB per node， 1 ethernet per node。Do you test with roce by ethernet? Now I set env:
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_LIST=mlx5_1:1,mlx5_7:1,mlx5_9:1,mlx5_13:1
to specify the list of HCAs。
@kunfupanda-hw commented on GitHub (Mar 11, 2025):
Hi, @Baibaifan, what's the problem with your previous settings? Did you pass the
test_lowlatency.py? In my setting, there are still QP modification errors. And I run./nvshmrun -n 2 /work/nvshmem_src_3.2.5-1/nvshmem_src/perftest/perftest_install/device/pt-to-pt/shmem_put_bw, then reported same errors asibv_modify_qp failed.@gpzlx1 commented on GitHub (Mar 15, 2025):
Hi everyone,
I’m also encountering an issue with
test_low_latency.py, but my error log is slightly different. Here’s the output:What's more, I’m able to successfully pass the tests for
nvshmem/pt-to-pt/shmem_put_bw,test_internode.py, andtest_intranode.py.@sphish commented on GitHub (Mar 17, 2025):
@gpzlx1 It appears you've encountered a failure during the bootstrap phase of NVSHMEM. Since you were able to bootstrap successfully in normal mode, you can check the low_latency_mode NVSHMEM environment
github.com/deepseek-ai/DeepEP@82dcf48fd3/deep_ep/buffer.py (L72-L79)variables and try to identify which specific variable is causing the bootstrap failure.@gpzlx1 commented on GitHub (Mar 17, 2025):
@sphish Thank you for your prompt response. However, it appears that the issue is more complex than anticipated, as adjusting those settings did not resolve my problem. We will perform an in-depth analysis to pinpoint the bug.
@yyyyyt123 commented on GitHub (Mar 17, 2025):
Same issue and even identical error log when testing
test_low_latency.pyon eight H800 with RoCE after switching to NVSHMEM V3.2.5. Does anyone know know to fix this error?@yyyyyt123 commented on GitHub (Mar 17, 2025):
We can pass the
test_intranode.pyandtest_internode.py. But face the above error intest_low_latency.py.@Alexhaoge commented on GitHub (Oct 25, 2025):
Any updates on this issue(
110 ibv_modify_qp failed)? Same error still occurs for nvshmem 3.3.9.