deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #355] test_internode.py runs failed due to "nvshmem detect topo failed" #246

New issue
Closed
opened 2026-10-26 09:57:16 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-26 09:57:16 +00:00
Owner
Copy link

Originally created by @new-TonyWang on GitHub (Aug 8, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/355

I'm using 2 nodes with 16 H200 GPU to run this test in docker container. And I can run "test_intranode.py" successfuly ，but failed run "test_internode.py".
these are my card status

Image ibv_devinfo can show ibv devices 
[root:libnvshmem-linux-x86_64-3.3.9_cuda12-archive]$ ibv_devinfo 
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      e09d:7303:000d:1fc0
        sys_image_guid:                 e09d:7303:000d:1fc0
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      e09d:7303:0009:9738
        sys_image_guid:                 e09d:7303:0009:9738
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_2
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      b8e9:2403:002a:5fee
        sys_image_guid:                 b8e9:2403:002a:5fee
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_10
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:00b3:2274
        sys_image_guid:                 58a2:e103:00b3:2274
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000001045
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_11
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:00b3:2275
        sys_image_guid:                 58a2:e103:00b3:2274
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000001045
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_3
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      e09d:7303:0009:89e0
        sys_image_guid:                 e09d:7303:0009:89e0
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_4
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      b8e9:2403:002a:5f86
        sys_image_guid:                 b8e9:2403:002a:5f86
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_5
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      b8e9:2403:002a:538e
        sys_image_guid:                 b8e9:2403:002a:538e
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_6
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      e09d:7303:0009:6370
        sys_image_guid:                 e09d:7303:0009:6370
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_7
        transport:                      InfiniBand (0)
        fw_ver:                         28.42.1000
        node_guid:                      b8e9:2403:002d:f8a6
        sys_image_guid:                 b8e9:2403:002d:f8a6
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_bond_0
        transport:                      InfiniBand (0)
        fw_ver:                         26.43.2026
        node_guid:                      7c8c:0903:0004:2b52
        sys_image_guid:                 7c8c:0903:0004:2b52
        vendor_id:                      0x02c9
        vendor_part_id:                 4127
        hw_ver:                         0x0
        board_id:                       MT_0000000531
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

here are the script to run test_internode.py

export NCCL_DEBUG=INFO
export NVSHMEM_DEBUG=TRACE
export NVSHMEM_DEBUG_SUBSYS=ALL
export NVSHMEM_DEBUG_FILE=nvdebug_$RANK

export MASTER_ADDR=${MASTER_ADDR:-localhost}
export RANK=${PET_NODE_RANK:-0}

python3 tests/test_internode.py 2>&1 | tee test_result_internodes_"$RANK".log

test_result_internodes_0.log

test_result_internodes_1.log

running shmem_put_bw on two node with the following scripts:

export NVSHMEM_DEBUG=TRACE
export NVSHMEM_DEBUG_SUBSYS=ALL
export NVSHMEM_DEBUG_FILE=nvdebug
nvshmrun -n 16 -f libnvshmem-linux-x86_64-3.3.20_cuda12-archive/bin/perftest/device/pt-to-pt/shmem_put_bw 2>&1 | tee put_bw_twonodes.log

get these log:

put_bw_twonodes.log

nvdebug.txt

Originally created by @new-TonyWang on GitHub (Aug 8, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/355 I'm using 2 nodes with 16 H200 GPU to run this test in docker container. And I can run "test_intranode.py" successfuly ，but failed run "test_internode.py". these are my card status <img width="3134" height="1524" alt="Image" src="https://github.com/user-attachments/assets/989bf033-72d5-4ea1-9043-ed002f603f41" /> ibv_devinfo can show ibv devices ```bash [root:libnvshmem-linux-x86_64-3.3.9_cuda12-archive]$ ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: e09d:7303:000d:1fc0 sys_image_guid: e09d:7303:000d:1fc0 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: e09d:7303:0009:9738 sys_image_guid: e09d:7303:0009:9738 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_2 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: b8e9:2403:002a:5fee sys_image_guid: b8e9:2403:002a:5fee vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_10 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:00b3:2274 sys_image_guid: 58a2:e103:00b3:2274 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000001045 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_11 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:00b3:2275 sys_image_guid: 58a2:e103:00b3:2274 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000001045 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_3 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: e09d:7303:0009:89e0 sys_image_guid: e09d:7303:0009:89e0 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_4 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: b8e9:2403:002a:5f86 sys_image_guid: b8e9:2403:002a:5f86 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_5 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: b8e9:2403:002a:538e sys_image_guid: b8e9:2403:002a:538e vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_6 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: e09d:7303:0009:6370 sys_image_guid: e09d:7303:0009:6370 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_7 transport: InfiniBand (0) fw_ver: 28.42.1000 node_guid: b8e9:2403:002d:f8a6 sys_image_guid: b8e9:2403:002d:f8a6 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 26.43.2026 node_guid: 7c8c:0903:0004:2b52 sys_image_guid: 7c8c:0903:0004:2b52 vendor_id: 0x02c9 vendor_part_id: 4127 hw_ver: 0x0 board_id: MT_0000000531 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet ``` here are the script to run test_internode.py ```bash export NCCL_DEBUG=INFO export NVSHMEM_DEBUG=TRACE export NVSHMEM_DEBUG_SUBSYS=ALL export NVSHMEM_DEBUG_FILE=nvdebug_$RANK export MASTER_ADDR=${MASTER_ADDR:-localhost} export RANK=${PET_NODE_RANK:-0} python3 tests/test_internode.py 2>&1 | tee test_result_internodes_"$RANK".log ``` [test_result_internodes_0.log](https://github.com/user-attachments/files/21678653/test_result_internodes_0.log) [test_result_internodes_1.log](https://github.com/user-attachments/files/21678719/test_result_internodes_1.log) running shmem_put_bw on two node with the following scripts: ```shell export NVSHMEM_DEBUG=TRACE export NVSHMEM_DEBUG_SUBSYS=ALL export NVSHMEM_DEBUG_FILE=nvdebug nvshmrun -n 16 -f libnvshmem-linux-x86_64-3.3.20_cuda12-archive/bin/perftest/device/pt-to-pt/shmem_put_bw 2>&1 | tee put_bw_twonodes.log ``` get these log: [put_bw_twonodes.log](https://github.com/user-attachments/files/21696166/put_bw_twonodes.log) [nvdebug.txt](https://github.com/user-attachments/files/21696172/nvdebug.txt)
bsakaguchi commented 2026-10-26 09:57:18 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Aug 8, 2025):

Your NIC IP address may not be configured correctly.

<!-- gh-comment-id:3166838337 --> @sphish commented on GitHub (Aug 8, 2025): Your NIC IP address may not be configured correctly.
bsakaguchi commented 2026-10-26 09:57:19 +00:00
Author
Owner
Copy link

@new-TonyWang commented on GitHub (Aug 8, 2025):

Your NIC IP address may not be configured correctly.

Thanks! Could you please tell me how to check if the IP address is correct? Both node can access to each other using http and 16 gpu allreduce can also be done successfully.

<!-- gh-comment-id:3167378552 --> @new-TonyWang commented on GitHub (Aug 8, 2025): > Your NIC IP address may not be configured correctly. Thanks! Could you please tell me how to check if the IP address is correct? Both node can access to each other using http and 16 gpu allreduce can also be done successfully.
bsakaguchi commented 2026-10-26 09:57:19 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Aug 11, 2025):

I’m not an expert in this area, but I think you could try using ib_write_bw to test connectivity.

<!-- gh-comment-id:3173072145 --> @sphish commented on GitHub (Aug 11, 2025): I’m not an expert in this area, but I think you could try using `ib_write_bw` to test connectivity.
bsakaguchi commented 2026-10-26 09:57:19 +00:00
Author
Owner
Copy link

@new-TonyWang commented on GitHub (Aug 12, 2025):

I’m not an expert in this area, but I think you could try using ib_write_bw to test connectivity.

I tried 'ib_write_bw' on both nodes and I got these:

ib_write_bw 

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x055f PSN 0x32e72e RKey 0x1ffed1 VAddr 0x007f3b0b195000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:17:119:17
 remote address: LID 0000 QPN 0x068b PSN 0x8962ee RKey 0x1ffecf VAddr 0x007fe1df2d5000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:17:95:28
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             46361.71            46339.84                  0.741437
---------------------------------------------------------------------------------------
<!-- gh-comment-id:3178734955 --> @new-TonyWang commented on GitHub (Aug 12, 2025): > I’m not an expert in this area, but I think you could try using `ib_write_bw` to test connectivity. I tried 'ib_write_bw' on both nodes and I got these: ```log ib_write_bw ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------------------------- RDMA_Write BW Test Dual-port : OFF Device : mlx5_0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON CQ Moderation : 1 Mtu : 4096[B] Link type : Ethernet GID index : 3 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0000 QPN 0x055f PSN 0x32e72e RKey 0x1ffed1 VAddr 0x007f3b0b195000 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:17:119:17 remote address: LID 0000 QPN 0x068b PSN 0x8962ee RKey 0x1ffecf VAddr 0x007fe1df2d5000 GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:17:95:28 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 46361.71 46339.84 0.741437 --------------------------------------------------------------------------------------- ```
bsakaguchi commented 2026-10-26 09:57:19 +00:00
Author
Owner
Copy link

@new-TonyWang commented on GitHub (Aug 13, 2025):

I solved this problem by seting env:

 export NVSHMEM_DISABLE_CUDA_VMM=1

but I don't know why this could work.

<!-- gh-comment-id:3182757245 --> @new-TonyWang commented on GitHub (Aug 13, 2025): I solved this problem by seting env: ```shell export NVSHMEM_DISABLE_CUDA_VMM=1 ``` but I don't know why this could work.
bsakaguchi commented 2026-10-26 09:57:20 +00:00
Author
Owner
Copy link

@Skylion007 commented on GitHub (Aug 15, 2025):

I solved this problem by seting env:

export NVSHMEM_DISABLE_CUDA_VMM=1

but I don't know why this could work.

Memory fragmentation of NVSHMEM, did you try increasing the NVSHMEM_SYMMETRIC_SIZE?

<!-- gh-comment-id:3192348138 --> @Skylion007 commented on GitHub (Aug 15, 2025): > I solved this problem by seting env: > > export NVSHMEM_DISABLE_CUDA_VMM=1 > > but I don't know why this could work. Memory fragmentation of NVSHMEM, did you try increasing the NVSHMEM_SYMMETRIC_SIZE?
bsakaguchi commented 2026-10-26 09:57:20 +00:00
Author
Owner
Copy link

@new-TonyWang commented on GitHub (Sep 4, 2025):

I solved this problem by seting env:
export NVSHMEM_DISABLE_CUDA_VMM=1
but I don't know why this could work.

Memory fragmentation of NVSHMEM, did you try increasing the NVSHMEM_SYMMETRIC_SIZE?

No, I haven't . But I can give a try. Thank you !

<!-- gh-comment-id:3252294727 --> @new-TonyWang commented on GitHub (Sep 4, 2025): > > I solved this problem by seting env: > > export NVSHMEM_DISABLE_CUDA_VMM=1 > > but I don't know why this could work. > > Memory fragmentation of NVSHMEM, did you try increasing the NVSHMEM_SYMMETRIC_SIZE? No, I haven't . But I can give a try. Thank you !
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#246
No description provided.