mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #13] Test fail use 2node on tests/test_internode.py #15
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#15
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xesdiny on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/13
script:
generate log:
This error is in
csrc/deep_ep.cppL765It is caught and thrown after timeout. How to analyze the specific problem?
@Desperadoze commented on GitHub (Feb 25, 2025):
hello，can I ask you how you compile nvshmem,my gdrcopy_copybw had data performence,but when compile nvshmem,I can't find the path of gdrcopy.it's different with gdrcopy_copybw?
@xesdiny commented on GitHub (Feb 25, 2025):
1.install gdrcopy
when you has installed,
check your compile gdrcopy version.
1.2 Soft link full path
1.3 install deb package
2.install nvshmem_src
2.1Enable IBGDA by modifying &Update kernel configuration:
2.2 build
cd nvshmem_src
@HanHan009527 commented on GitHub (Feb 25, 2025):
I think the script should be
MASTER_ADDR=WL0 WORLD_SIZE=2 RANK=0 python tests/test_internode.py
MASTER_ADDR=WL0 WORLD_SIZE=2 RANK=1 python tests/test_internode.py
@xesdiny commented on GitHub (Feb 25, 2025):
Yeah, there's a little glitch in my paste run script.
@haswelliris commented on GitHub (Feb 25, 2025):
It appears that you are experiencing network connectivity issues. Please provide additional information, including:
Additionally, include hardware details, such as:
Furthermore, please share the results of the following reports, if available:
gdrcopy_copybwreport:nvshmem perftestreport:@xesdiny commented on GitHub (Mar 3, 2025):
@haswelliris
Operating system: ubuntu 20.04
GPU and NIC driver version:
A100-SXM 525.105.17, NVSHMEM v3.1.7
CUDA Version: 12.1
Network card:
GPU topology
gdrcopy_copybw
nvshmem perftest report:
@haswelliris commented on GitHub (Mar 3, 2025):
@xesdiny
ucx_perftestlocally (self-to-self) and is utilizing CPU host memory.To properly test NVSHMEM performance, you can find the
perftestdirectory in your NVSHMEM installation path. After building it with CMake, you can run thedevice/pt-to-pt/shmem_put_bwtest for accurate GPU-to-GPU performance measurements.NVSHMEM_IB_GID_INDEXNVSHMEM_IB_TRAFFIC_CLASS@renwuli commented on GitHub (Mar 7, 2025):
hi @haswelliris I have 8 Hopper GPUs and 4 50GB/s Mellanox CX7 NICs per node, NCCL 2 node allreduce busbw can achieve 193GB/, I following the deepep guide and run test_intranode.py, but the performance is extramely low (while intranode is normal), do you have any ideas?
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 12, RDMA chunk 12: 5.36 GB/s (RDMA), 17.49 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 16, RDMA chunk 8: 5.51 GB/s (RDMA), 17.99 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 1, RDMA chunk 20: 6.45 GB/s (RDMA), 21.05 GB/s (NVL)
@zhangml commented on GitHub (Mar 19, 2025):
I encountered the same issue. Could you please explain how you resolved it and what the cause was? @xesdiny
@juinshell commented on GitHub (Mar 2, 2026):
same issue, did you fix it? @zhangml
@fanfanaaaa commented on GitHub (Aug 12, 2026):
same issue, did you fix it? @juinshell