deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #150] Caught signal 11 on 2 * 8 H20 internode test #132

New issue
Closed
opened 2026-10-26 09:54:57 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-26 09:54:57 +00:00
Owner
Copy link

Originally created by @cscyuge on GitHub (May 8, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/150

I encounted an error when running test_internode.py on 2 * 8 H20 nodes.

script:

NCCL_SOCKET_IFNAME=eth0 NCCL_IB_GID_INDEX=3 MASTER_ADDR=my_master_ip WORLD_SIZE=2 RANK=0 python test_internode.py

NCCL_SOCKET_IFNAME=eth0 NCCL_IB_GID_INDEX=3 MASTER_ADDR=my_master_ip WORLD_SIZE=2 RANK=1 python test_internode.py

error log:

[VM-12-3-centos:132060:0:132730] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078)
[VM-12-3-centos:132061:0:132732] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078)
[VM-12-3-centos:132062:0:132731] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000050)
[VM-12-3-centos:132064:0:132727] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000060)
[VM-12-3-centos:132063:0:132733] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078)
[VM-12-3-centos:132066:0:132729] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000070)
[VM-12-3-centos:132067:0:132734] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078)
[VM-12-3-centos:132065:0:132728] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000068)
==== backtrace (tid: 132732) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
==== backtrace (tid: 132734) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
==== backtrace (tid: 132730) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
==== backtrace (tid: 132728) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
==== backtrace (tid: 132729) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
==== backtrace (tid: 132727) ====
==== backtrace (tid: 132733) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 0 0x0000000000042520 __sigaction()  ???:0
 2 0x00000000000112d5 progress_recv()  :0
 1 0x0000000000010a03 process_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 2 0x00000000000112d5 progress_recv()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 4 0x000000000021d71c progress_transports()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
=================================
 7 0x0000000000126850 __xmknodat()  ???:0
==== backtrace (tid: 132731) ====
=================================
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000010a03 process_recv()  :0
 2 0x00000000000112d5 progress_recv()  :0
 3 0x00000000000113cc nvshmemt_ibrc_progress()  :0
 4 0x000000000021d71c progress_transports()  ???:0
 5 0x000000000021de02 nvshmemi_proxy_progress()  ???:0
 6 0x0000000000094ac3 pthread_condattr_setpshared()  ???:0
 7 0x0000000000126850 __xmknodat()  ???:0
=================================
W0508 12:53:24.783000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132060 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132061 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132062 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132063 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132065 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132066 via signal SIGTERM
W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132067 via signal SIGTERM
Traceback (most recent call last):
  File "/sgl-workspace/DeepEP/tests/test_internode.py", line 247, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 196, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with signal SIGSEGV

I have already modified kNumWarpsPerGroup and kNumWarpGroups in DeepEP/csrc/kernels/internode_ll.cu to 8 and 4 , respectively, (referring to https://github.com/deepseek-ai/DeepEP/issues/15#issuecomment-2709715782). Any solution?

Originally created by @cscyuge on GitHub (May 8, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/150 I encounted an error when running `test_internode.py` on 2 * 8 H20 nodes. script: ``` NCCL_SOCKET_IFNAME=eth0 NCCL_IB_GID_INDEX=3 MASTER_ADDR=my_master_ip WORLD_SIZE=2 RANK=0 python test_internode.py NCCL_SOCKET_IFNAME=eth0 NCCL_IB_GID_INDEX=3 MASTER_ADDR=my_master_ip WORLD_SIZE=2 RANK=1 python test_internode.py ``` error log: ``` [VM-12-3-centos:132060:0:132730] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078) [VM-12-3-centos:132061:0:132732] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078) [VM-12-3-centos:132062:0:132731] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000050) [VM-12-3-centos:132064:0:132727] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000060) [VM-12-3-centos:132063:0:132733] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078) [VM-12-3-centos:132066:0:132729] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000070) [VM-12-3-centos:132067:0:132734] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000078) [VM-12-3-centos:132065:0:132728] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x11000068) ==== backtrace (tid: 132732) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= ==== backtrace (tid: 132734) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= ==== backtrace (tid: 132730) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= ==== backtrace (tid: 132728) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= ==== backtrace (tid: 132729) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= ==== backtrace (tid: 132727) ==== ==== backtrace (tid: 132733) ==== 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 0 0x0000000000042520 __sigaction() ???:0 2 0x00000000000112d5 progress_recv() :0 1 0x0000000000010a03 process_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 2 0x00000000000112d5 progress_recv() :0 4 0x000000000021d71c progress_transports() ???:0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 4 0x000000000021d71c progress_transports() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 7 0x0000000000126850 __xmknodat() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 ================================= 7 0x0000000000126850 __xmknodat() ???:0 ==== backtrace (tid: 132731) ==== ================================= 0 0x0000000000042520 __sigaction() ???:0 1 0x0000000000010a03 process_recv() :0 2 0x00000000000112d5 progress_recv() :0 3 0x00000000000113cc nvshmemt_ibrc_progress() :0 4 0x000000000021d71c progress_transports() ???:0 5 0x000000000021de02 nvshmemi_proxy_progress() ???:0 6 0x0000000000094ac3 pthread_condattr_setpshared() ???:0 7 0x0000000000126850 __xmknodat() ???:0 ================================= W0508 12:53:24.783000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132060 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132061 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132062 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132063 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132065 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132066 via signal SIGTERM W0508 12:53:24.784000 131995 torch/multiprocessing/spawn.py:169] Terminating process 132067 via signal SIGTERM Traceback (most recent call last): File "/sgl-workspace/DeepEP/tests/test_internode.py", line 247, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 196, in join raise ProcessExitedException( torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with signal SIGSEGV ``` I have already modified `kNumWarpsPerGroup` and `kNumWarpGroups` in DeepEP/csrc/kernels/internode_ll.cu to `8` and `4` , respectively, (referring to https://github.com/deepseek-ai/DeepEP/issues/15#issuecomment-2709715782). Any solution?
bsakaguchi commented 2026-10-26 09:54:58 +00:00
Author
Owner
Copy link

@cscyuge commented on GitHub (May 8, 2025):

Network card:

Infiniband device 'mlx5_bond_0' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:212a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_1' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:3566
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_2' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:4256
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet
                                                                                                                                                                                                        [0/1989]
Infiniband device 'mlx5_bond_3' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:214a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_4' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:284a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_5' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:213a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_6' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:23ba
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_7' port 1 status:
        default gid:     fe80:0000:0000:0000:5e25:73ff:fe94:288a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      Ethernet

GPU topology:

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     PHB     NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    PHB     PIX     NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    96-191,288-383  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    96-191,288-383  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PHB     96-191,288-383  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    PHB     PIX     96-191,288-383  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    PIX     PHB     NODE    SYS     SYS     SYS     SYS     NODE     X      PHB     NODE    SYS     SYS     SYS     SYS
NIC2    NODE    PHB     PIX     NODE    SYS     SYS     SYS     SYS     NODE    PHB      X      NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PHB     SYS     SYS     SYS     SYS     NODE    NODE     X      PHB
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    PHB     PIX     SYS     SYS     SYS     SYS     NODE    NODE    PHB      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7
<!-- gh-comment-id:2862974478 --> @cscyuge commented on GitHub (May 8, 2025): Network card: ``` Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:212a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_1' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:3566 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_2' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:4256 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet [0/1989] Infiniband device 'mlx5_bond_3' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:214a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_4' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:284a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_5' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:213a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_6' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:23ba base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet Infiniband device 'mlx5_bond_7' port 1 status: default gid: fe80:0000:0000:0000:5e25:73ff:fe94:288a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: Ethernet ``` GPU topology: ``` GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX PHB NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE PHB PIX NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYS SYS SYS SYS 0-95,192-287 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE NODE NODE 96-191,288-383 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX NODE NODE 96-191,288-383 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE NODE PIX PHB 96-191,288-383 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE PHB PIX 96-191,288-383 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX PHB NODE SYS SYS SYS SYS NODE X PHB NODE SYS SYS SYS SYS NIC2 NODE PHB PIX NODE SYS SYS SYS SYS NODE PHB X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX PHB SYS SYS SYS SYS NODE NODE X PHB NIC7 SYS SYS SYS SYS NODE NODE PHB PIX SYS SYS SYS SYS NODE NODE PHB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7 ```
bsakaguchi commented 2026-10-26 09:54:59 +00:00
Author
Owner
Copy link

@cscyuge commented on GitHub (May 8, 2025):

Turns out I didn’t have the right NVSHMEM installed on the physical machine.
Sorry for the trouble.

<!-- gh-comment-id:2863095455 --> @cscyuge commented on GitHub (May 8, 2025): Turns out I didn’t have the right NVSHMEM installed on the physical machine. Sorry for the trouble.
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@ayrnb commented on GitHub (May 23, 2025):

On 2 * 8 H20 it works, but on 4 * 8 H20, I meet same errors. How to check whether the right NVSHMEM is installed?

<!-- gh-comment-id:2903178018 --> @ayrnb commented on GitHub (May 23, 2025): On 2 * 8 H20 it works, but on 4 * 8 H20, I meet same errors. How to check whether the right NVSHMEM is installed?
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@cscyuge commented on GitHub (Jun 4, 2025):

On 2 * 8 H20 it works, but on 4 * 8 H20, I meet same errors. How to check whether the right NVSHMEM is installed?

check bynvshmem-info -a
or try to reinstall by following the steps in https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile.deepep

<!-- gh-comment-id:2939629918 --> @cscyuge commented on GitHub (Jun 4, 2025): > On 2 * 8 H20 it works, but on 4 * 8 H20, I meet same errors. How to check whether the right NVSHMEM is installed? check by`nvshmem-info -a` or try to reinstall by following the steps in https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile.deepep
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@XiaobinZhao commented on GitHub (Jun 10, 2025):

@cscyuge i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test

node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0  python tests/test_internode.py
node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1  python tests/test_internode.py

the output is:

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting...
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

bootstrap_loader_init returned error for mode UID
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed
/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7
nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed
nvshmem_internal_init_thread failed


/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread

what should i do?

<!-- gh-comment-id:2958083171 --> @XiaobinZhao commented on GitHub (Jun 10, 2025): @cscyuge i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test ``` node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0 python tests/test_internode.py node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1 python tests/test_internode.py ``` the output is: ``` /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID bootstrap_loader_init returned error for mode UID /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed nvshmem_internal_init_thread failed /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread aborting due to error in nvshmemi_init_thread /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread ``` what should i do?
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@cscyuge commented on GitHub (Jun 12, 2025):

@cscyuge i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test

node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0  python tests/test_internode.py
node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1  python tests/test_internode.py

the output is:

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting...
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

bootstrap_loader_init returned error for mode UID
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed
/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7
nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed
nvshmem_internal_init_thread failed


/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread

what should i do?

the image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep) is invaild now, sorry i didn't encounter these errors before, maybe try opening a new issue about your problem

<!-- gh-comment-id:2964973185 --> @cscyuge commented on GitHub (Jun 12, 2025): > [@cscyuge](https://github.com/cscyuge) i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test > > ``` > node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0 python tests/test_internode.py > node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1 python tests/test_internode.py > ``` > > the output is: > > ``` > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... > aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > bootstrap_loader_init returned error for mode UID > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > bootstrap_init failed > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 > nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > nvshmem_internal_init_thread failed > > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > ``` > > what should i do? the image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep) is invaild now, sorry i didn't encounter these errors before, maybe try opening a new issue about your problem
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@XiaobinZhao commented on GitHub (Jun 12, 2025):

@cscyuge i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test

node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0  python tests/test_internode.py
node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1  python tests/test_internode.py

the output is:

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting...
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

bootstrap_loader_init returned error for mode UID
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed
/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7
nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed
nvshmem_internal_init_thread failed


/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread

what should i do?

the image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep) is invaild now, sorry i didn't encounter these errors before, maybe try opening a new issue about your problem

@cscyuge lmsysorg/sglang:dev-deepep is AVAILABLE in the docker hub; sorry, i show you the url is wrong.

<!-- gh-comment-id:2965007478 --> @XiaobinZhao commented on GitHub (Jun 12, 2025): > > [@cscyuge](https://github.com/cscyuge) i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test > > ``` > > node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0 python tests/test_internode.py > > node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1 python tests/test_internode.py > > ``` > > > > > > > > > > > > > > > > > > > > > > > > the output is: > > ``` > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... > > aborting due to error in nvshmemi_init_thread > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > > `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > > > Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > > /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > > > bootstrap_loader_init returned error for mode UID > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > > bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > > bootstrap_init failed > > > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 > > nvshmem_bootstrap failed > > > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > nvshmem_internal_init_thread failed > > > > > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > > aborting due to error in nvshmemi_init_thread > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > > ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > > ``` > > > > > > > > > > > > > > > > > > > > > > > > what should i do? > > the image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep) is invaild now, sorry i didn't encounter these errors before, maybe try opening a new issue about your problem @cscyuge lmsysorg/sglang:dev-deepep is AVAILABLE in the docker hub; sorry, i show you the url is wrong.
bsakaguchi commented 2026-10-26 09:55:00 +00:00
Author
Owner
Copy link

@ztxdcyy commented on GitHub (Jul 2, 2025):

@cscyuge i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test

node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0  python tests/test_internode.py
node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1  python tests/test_internode.py

the output is:

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting...
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

bootstrap_loader_init returned error for mode UID
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed
/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7
bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed
/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7
nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed
nvshmem_internal_init_thread failed


/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread
aborting due to error in nvshmemi_init_thread
/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init'
        ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init

/sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID

/sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed

/sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed

/sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed

/sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread

what should i do?

Hi, we have met the same problem, have your found out a solution for that? Many thanks

<!-- gh-comment-id:3027344782 --> @ztxdcyy commented on GitHub (Jul 2, 2025): > [@cscyuge](https://github.com/cscyuge) i just use the latest(2025-0606) dev-deepep image(https://hub.docker.com/layers/lmsysorg/sglang/dev-deepep); and i run the test > > ``` > node1: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA='mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=0 python tests/test_internode.py > node2: NCCL_SOCKET_IFNAME=eno1 NCCL_IB_HCA="mlx5_0,mlx5_1,mlx5_2,mlx5_5,mlx5_6,mlx5_7,mlx5_8,mlx5_9" NCCL_DEBUG=INFO MASTER_ADDR=10.10.251.105 WORLD_SIZE=2 RANK=1 python tests/test_internode.py > ``` > > the output is: > > ``` > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... > aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > `▒▒▒▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > /usr/bin/python: undefined symbol: nvshmemi_bootstrap_plugin_init > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > bootstrap_loader_init returned error for mode UID > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > bootstrap_init failed > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 > bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 > nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > nvshmem_internal_init_thread failed > > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > aborting due to error in nvshmemi_init_thread > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap_loader.cpp:100: NULL value Bootstrap failed to get symbol 'nvshmemi_bootstrap_plugin_init' > ▒;▒$▒U: undefined symbol: nvshmemi_bootstrap_plugin_init > > /sgl-workspace/nvshmem/src/host/bootstrap/bootstrap.cpp:238: non-zero status: 7 bootstrap_loader_init returned error for mode UID > > /sgl-workspace/nvshmem/src/host/init/init.cu:327: non-zero status: 7 bootstrap_init failed > > /sgl-workspace/nvshmem/src/host/init/init.cu:1149: non-zero status: 7 nvshmem_bootstrap failed > > /sgl-workspace/nvshmem/src/device/init/init_device.cu:145: non-zero status: 7 nvshmem_internal_init_thread failed > > /sgl-workspace/nvshmem/install/include/host/nvshmemx_api.h:63: non-zero status: 7: Connection timed out, exiting... aborting due to error in nvshmemi_init_thread > ``` > > what should i do? Hi, we have met the same problem, have your found out a solution for that? Many thanks
bsakaguchi commented 2026-10-26 09:55:01 +00:00
Author
Owner
Copy link

@XiaobinZhao commented on GitHub (Jul 3, 2025):

@ztxdcyy you can try to install the latest sglang code and reinstall the deepep

1. 更新 sglang代码
    1. git pull
    2. uv pip install -e 'python[all]' --system
2. 重装deepep
    1. cd DeepEP
    2. RUN NVSHMEM_DIR=/sgl-workspace/nvshmem/install pip install --break-system-packages .
3. 启动物理机上的IBGDA
<!-- gh-comment-id:3030069988 --> @XiaobinZhao commented on GitHub (Jul 3, 2025): @ztxdcyy you can try to install the latest sglang code and reinstall the deepep ``` 1. 更新 sglang代码 1. git pull 2. uv pip install -e 'python[all]' --system 2. 重装deepep 1. cd DeepEP 2. RUN NVSHMEM_DIR=/sgl-workspace/nvshmem/install pip install --break-system-packages . 3. 启动物理机上的IBGDA ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#132
No description provided.