[GH-ISSUE #86] socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3) #79

New issue

Open

opened 2026-10-26 09:53:56 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:53:56 +00:00

Owner

Originally created by @zhoutianzi666 on GitHub (Mar 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/86

:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: /optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3)
:
:/op tnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed 
:
:/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed 
:
:/optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed 
:connect EPS failed 
:
:
:/optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 nvshmem setup connections failed 
:nvshmem setup connections failed

你好，请问我跑python tests/test_low_latency.py的时候，2机16卡可以，但是换到4机32卡报出这个错误？请问有什么办法吗？
多谢了。

Originally created by @zhoutianzi666 on GitHub (Mar 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/86 ```cpp :/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: /optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketStartConnect:488: socketStartConnect: exceeded timeouts (3)socketStartConnect: exceeded timeouts (3) : :/op tnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6 :/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_send:421: non-zero status: -6 :/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed : :/optnvshmem_src_3.1.7-1/src/modules/bootstrap/uid/bootstrap_uid.cpp:540: non-zero status: -6 /optnvshmem_src_3.1.7-1/src/modules/transport/ibrc/ibrc.cpp:1493: non-zero status: -6 allgather of ep handles failed : :/optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed :connect EPS failed : : :/optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 /optnvshmem_src_3.1.7-1/src/host/init/init.cu:1002: non-zero status: 7 nvshmem setup connections failed :nvshmem setup connections failed ``` 你好，请问我跑`python tests/test_low_latency.py`的时候，2机16卡可以，但是换到4机32卡报出这个错误？请问有什么办法吗？多谢了。

bsakaguchi commented

2026-10-26 09:53:57 +00:00

Author

Owner

@sphish commented on GitHub (Mar 24, 2025):

It appears there might be an issue with the Ethernet configuration. You could try running tests between different pairs from the four machines to see which combinations cannot work properly.

@sphish commented on GitHub (Mar 24, 2025): It appears there might be an issue with the Ethernet configuration. You could try running tests between different pairs from the four machines to see which combinations cannot work properly.

bsakaguchi referenced this issue

2026-10-26 10:00:40 +00:00

[PR #79] [MERGED] Support zero-copy for low-latency combine #430