deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #316] H20-3e internode-test error IPC failure #222

New issue
Open
opened 2026-10-26 09:56:53 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:56:53 +00:00
Owner
Copy link

Originally created by @wpf19911118 on GitHub (Jul 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/316

IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:775: Fatal IPC Failure

使用sglang 0.4.9 镜像，pip 安装nvshmem，使用了目前最新的master分支 DeepEP

env：
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 12.575.51
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvshmem-cu12 3.3.9
nvidia-nvtx-cu12 12.6.77
pynvml 12.0.0
python-dotenv 1.1.0

启动命令
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=0 python3 tests/test_internode.py
MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=1 python3 tests/test_internode.py

Originally created by @wpf19911118 on GitHub (Jul 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/316 IPC failure: Sending data over socket failed: No such file or directory /dvs/p4/build/sw/rel/gpgpu/toolkit/r12.8/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:775: Fatal IPC Failure 使用sglang 0.4.9 镜像，pip 安装nvshmem，使用了目前最新的master分支 DeepEP env： nvidia-cublas-cu12 12.6.4.1 nvidia-cuda-cupti-cu12 12.6.80 nvidia-cuda-nvrtc-cu12 12.6.77 nvidia-cuda-runtime-cu12 12.6.77 nvidia-cudnn-cu12 9.5.1.17 nvidia-cufft-cu12 11.3.0.4 nvidia-cufile-cu12 1.11.1.6 nvidia-curand-cu12 10.3.7.77 nvidia-cusolver-cu12 11.7.1.2 nvidia-cusparse-cu12 12.5.4.2 nvidia-cusparselt-cu12 0.6.3 nvidia-ml-py 12.575.51 nvidia-nccl-cu12 2.26.2 nvidia-nvjitlink-cu12 12.6.85 nvidia-nvshmem-cu12 3.3.9 nvidia-nvtx-cu12 12.6.77 pynvml 12.0.0 python-dotenv 1.1.0 启动命令 MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=0 python3 tests/test_internode.py MASTER_ADDR=192.168.10.20 MASTER_PORT=10086 WORLD_SIZE=2 RANK=1 python3 tests/test_internode.py
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 22, 2025):

看起来是 nvshmem 初始化的时候用 socket 通信出了问题，你可以去看看 nvshmem 代码

<!-- gh-comment-id:3100241601 --> @sphish commented on GitHub (Jul 22, 2025): 看起来是 nvshmem 初始化的时候用 socket 通信出了问题，你可以去看看 nvshmem 代码
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@Crystal-lcx commented on GitHub (Jul 31, 2025):

Is there any update here? I also met the same question.

<!-- gh-comment-id:3139176054 --> @Crystal-lcx commented on GitHub (Jul 31, 2025): Is there any update here? I also met the same question.
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@junchenglai-mthreads commented on GitHub (Sep 19, 2025):

我也遇到了一样的问题，我也是用的sglang镜像，请问您解决了吗

<!-- gh-comment-id:3310923919 --> @junchenglai-mthreads commented on GitHub (Sep 19, 2025): 我也遇到了一样的问题，我也是用的sglang镜像，请问您解决了吗
bsakaguchi commented 2026-10-26 09:56:55 +00:00
Author
Owner
Copy link

@junchenglai-mthreads commented on GitHub (Sep 19, 2025):

我解决了，两台主机的hostname不能一样

<!-- gh-comment-id:3311028937 --> @junchenglai-mthreads commented on GitHub (Sep 19, 2025): 我解决了，两台主机的hostname不能一样
bsakaguchi commented 2026-10-26 09:56:55 +00:00
Author
Owner
Copy link

@ZhenguoYao1 commented on GitHub (Sep 19, 2025):

我解决了，两台主机的hostname不能一样

解决的真及时，我昨天也遇到同样的问题。折腾了半天也没搞好，原来是这个问题

<!-- gh-comment-id:3311466006 --> @ZhenguoYao1 commented on GitHub (Sep 19, 2025): > 我解决了，两台主机的hostname不能一样 解决的真及时，我昨天也遇到同样的问题。折腾了半天也没搞好，原来是这个问题
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#222
No description provided.