mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #36] Is setting IBGDA necessary for test_internode.py? #34
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#34
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zhangml on GitHub (Mar 2, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/36
I noticed the following steps in the guide:
Due to some environment permission issues, I can't do this step for now. Is it possible to run test_internode.py without doing this step?
@LyricZhao commented on GitHub (Mar 3, 2025):
Yes, you can. The normal kernels use IBRC instead of IBGDA. But we plan to support AR later, which always requires IBGDA.
@zhangml commented on GitHub (Mar 3, 2025):
I ran 2 H20 node test_internode.py without IBGDA and encountered the following error, I'm not sure if it's related to IBGDA. @LyricZhao
@haswelliris commented on GitHub (Mar 3, 2025):
@yanminjia commented on GitHub (Mar 3, 2025):
I would like to run test_internode.py with IBGDA enabled because it looks dual-port RNIC is supported by going with IBGDA over RoCE network. Could you please give me a detailed configuration to enable IBGDA?
Many thanks.
@Baibaifan commented on GitHub (Mar 3, 2025):
I used megatron-lm to test two H100s and 16 cards in RoCE network with ep=16. I also encountered the above
bootstrap_net_recv:99: Message truncated: received 40 bytes instead of 8. I set IBGDA, but it prompts:WARN: init failed for remote transport: ibrc.@sphish commented on GitHub (Mar 3, 2025):
You can set envrionments
to enable IBGDA.
@sphish commented on GitHub (Mar 3, 2025):
This appears to be an error during NVSHMEM bootstrap. Please verify your network configuration is correct. It's recommended to run the tests in NVSHMEM perftest first to validate your network setup.
Note that even when IBGDA is enabled, NVSHMEM will still create IBRC connections, so seeing this warning message makes sense. For more details, please refer to the NVSHMEM documentation.
@Baibaifan commented on GitHub (Mar 3, 2025):
My script:
RANK=0 result:
RANK=1 result:
@haswelliris commented on GitHub (Mar 3, 2025):
@Baibaifan The message
neither nv_peer_mem nor nvidia_peermem detectedindicates that your system environment does not currently support GPU Direct RDMA. To resolve this, please try loading the GDR kernel module by running one of the following commands:This should enable GPU Direct RDMA functionality.
@Baibaifan commented on GitHub (Mar 3, 2025):
@haswelliris After I successfully set
modprobe nv_peer_mem, repeating the above command appears:rank0:
There is no error output for rank0.
rank1:
@yanminjia commented on GitHub (Mar 3, 2025):
Many thanks, will try it.
@ghghliu commented on GitHub (Mar 3, 2025):
Is there any way to determine if IBGDA is correctly enabled, because the performance seems no difference weather I set NVSHMEM_IB_ENABLE_IBGDA=1 or 0, and the result looks ok. And is it necessary to compile libgdsync(https://github.com/gpudirect/libgdsync) before compiling nvshmem?
@yanminjia commented on GitHub (Mar 3, 2025):
Many thanks for your kindly response. It looks not work because ibrc.cxx:progress_send(...) is called for transferring data by checking a log message added to this function (ibrc.cxx:progress_send(...)). Maybe, any other configuration missed to enable IBGDA?
@yanminjia commented on GitHub (Mar 3, 2025):
By debugging this issue, ibgda.cc:nvshmemt_init(...) is failed with the error message as following:
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
This problem is caused by mlx5dv_devx_umem_reg(...) failure.
Any suggestion would be appreciated.
Thanks.
@NDk9856 commented on GitHub (May 19, 2025):
We encountered the same problem as above mlx5dv_devx_umem_reg function failed using dma-buf or nv_peer_mem. Can someone give the version of the kernel that IBGDA can work on and the ofed driver version number of the Mellanox nic? Great debt of gratitude.
@sphish commented on GitHub (May 20, 2025):
@NDk9856 We use linux kernel version 5.15 and OFED version 5.8.1.0.1 with nvidia_peermem.
@NDk9856 commented on GitHub (May 21, 2025):
Thank you so much @sphish. We fixed the mlx5dv_devx_umem_reg failed problem by disabling nv_peer_mem module then enable nvidia_peermem module manually.We work on Ubuntu 22.04 LTS with kernel version 5.15.0-140, MOFED 5.8-6. Hope helpful for someone else.