mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #55] test_low_latency failed #52
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#52
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @hyesung84 on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/55
I am experiencing an issue with NVSHMEM failing to initialize due to transport errors. The error message indicates that NVSHMEM is unable to detect the system topology and cannot initialize any transport layers. However, test_intranode.py passed successfully...
I would like to know how to resolve this problem.
System Information
GPU Model: H100 (8 GPUs, single node)
OS: Ubuntu 22.04
CUDA Version: 12.5
NVSHMEM Version: 3.2.5
Error Log
@sphish commented on GitHub (Mar 9, 2025):
What is your network hardware configuration? Could you please run
nvidia-smi topo -mpandibv_devinfoand share the results?@BigValen commented on GitHub (Mar 24, 2025):
I'm seeing a similar issue:
@sphish commented on GitHub (Mar 25, 2025):
@BigValen It appears that nvshmem cannot initialize ibrc transport, which is typically related to network configuration issues. However, the
ibv_devinfoandnvidia-smioutputs you provided look normal. Could you try runningib_write_bwand nvshmem'sshmem_put_bwto see if they work properly? This will help us determine if the issue is specific to nvshmem or if there might be a more general RDMA connectivity problem.@liusy58 commented on GitHub (Mar 31, 2025):
@sphish Same issue. Any help?
@sphish commented on GitHub (Apr 1, 2025):
@liusy58 Can you run the NVSHMEM's
shmem_put_bwtest, and will you encounter the same issue?@liusy58 commented on GitHub (Apr 1, 2025):
@sphish emmm, some features are not supported on my machine, I will try to fix it. Thank you a lot~~
@liusy58 commented on GitHub (Apr 2, 2025):
@sphish Hi, output of
shmem_put_bwis shown below. I cannot resolve this, could you please give me some guidance?@sphish commented on GitHub (Apr 2, 2025):
@liusy58 You need load nvidia_peermem kernel module.
@liusy58 commented on GitHub (Apr 3, 2025):
Thank you~
@Cydia2018 commented on GitHub (Apr 24, 2025):
After running the command
shmem_put_bw, I encountered the following error. Could you give me further guidance? Thanks a lot.@sphish commented on GitHub (Apr 25, 2025):
I suspect this is related to the CUDA driver version.
@koanho commented on GitHub (Apr 27, 2025):
@sphish
Hi, I got a similar issue. When testing
./shmem_put_bw, got an error below.@sphish commented on GitHub (Apr 27, 2025):
@koanho Can you check if the nvidia-peermem module is correctly installed and loaded?
@koanho commented on GitHub (Apr 27, 2025):
Thank you for reply @sphish.
I think nvidia-peermem is correctly installed and loaded.
@sphish commented on GitHub (Apr 27, 2025):
@koanho Have you modified drvier config? https://github.com/deepseek-ai/DeepEP/tree/main/third-party#4-configure-nvidia-driver
@koanho commented on GitHub (Apr 27, 2025):
Thank you @sphish.
I couldn't modify the driver configuration because I don't have root permissions on my training cluster 😞
It seems the error may have occurred because IBGDA is not properly enabled.
Is IBGDA necessary to use DeepEP, right?
@sphish commented on GitHub (Apr 27, 2025):
@koanho If you want to use low latency mode, Yes. If you only want to use the normal mode for training, you can use old version DeepEP, which use IBRC transport.
@vinjn commented on GitHub (May 8, 2025):
@sphish which commit uses IBRC transport? Thanks
@ch-tiger1 commented on GitHub (May 9, 2025):
I have the same error, the error is as follows
modinfo nvidia-peermem
lsmod | grep nvidia_peermem
I've been working on this for a long time, is there any good solution?
@vinjn commented on GitHub (May 9, 2025):
I've found a "good old version" that works with "IBGDA disabled" machines, which is
github.com/deepseek-ai/DeepEP@a84a24808f@ch-tiger1 commented on GitHub (May 9, 2025):
Thank you. I just looked at the modified patch and found that it is the same as what I am using now. I don’t know what went wrong?
In fact, I saw the sglang issue mentioning the dockerfile for creating this environment, but the above problems occurred.
@koanho commented on GitHub (May 9, 2025):
@vinjn
I've tested commit a84a248, but encountered a runtime error during DeepEP setup.
It seems that DeepEP cannot be set up properly when NVSHMEM is installed with IBGDA disabled.
Could you help me resolve this issue?
@ch-tiger1 commented on GitHub (May 9, 2025):
I found that when I executed
shmem_put_bw, aSegmentation faulterror would be reported, but when I loaded and executedmodprobe nvidia_peermem, this command did not work either.@ch-tiger1 commented on GitHub (May 9, 2025):
@sphish I used the above method to configure the NVIDIA driver, but still got Segmentation fault. Is there any other solution?
@Kevin-XiongC commented on GitHub (May 15, 2025):
Have you resolved this? I've been stuck here for days. @ch-tiger1
@wwj-2017-1117 commented on GitHub (May 21, 2025):
me too
@kwu130 commented on GitHub (Jun 7, 2025):
@sphish Hi
Excuse me, how can I determine whether my IB NIC or Driver supports the IBGDA feature? I used ConnectX-7, and OFED-23.10-2.1.3
According to the NVSHMEM code, if the GPU supports dmabuf, then nvidia-peermem is not required — is that correct?
Does using DeepEP have any requirements on the Linux kernel version?
Is GDRcopy necessary for IBGDA? What is its purpose?
Thank you! I look forward to your response.
@kwu130 commented on GitHub (Jun 7, 2025):
you need run this example use exactly two PE，so you can use nvshmrun to run it
eg: nvshmrun -n 2 ./shmem_put_bw
btw, nvshmrun is a progress launcher, you can install it follow install_hydra.sh in nvshmem project, good luck!
@sphish commented on GitHub (Jun 9, 2025):
@kwu130 For information about which devices and driver versions support IBGDA, it’s best to consult NVIDIA directly. We are not entirely sure about the exact compatibility details either. You can refer to our environment setup here: https://github.com/deepseek-ai/DeepEP/issues/36#issuecomment-2892652482.
I think that gdrcopy isn't required for IBGDA. However, the last time I tried to build without gdrcopy, I encountered some errors. I haven’t looked into this issue in detail yet.
@kwu130 commented on GitHub (Jun 12, 2025):
copy that, thx very much.
@sphish commented on GitHub (Jun 12, 2025):
@kwu130 By the way, after https://github.com/deepseek-ai/DeepEP/pull/201, DeepEP can be built without depending on gdrcopy.