deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #60] RDMA bandwidth is not normal when run test_internode.py. #57

New issue
Closed
opened 2026-10-26 09:53:28 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-26 09:53:28 +00:00
Owner
Copy link

Originally created by @MinhuiWan on GitHub (Mar 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/60

My test environment is 2 nodes, 8 H100 per node, 4 IB per node。 the test result is
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 20, RDMA chunk 32: 4.47 GB/s (RDMA), 9.00 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 24, RDMA chunk 32: 4.72 GB/s (RDMA), 9.51 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 4, RDMA chunk 32: 4.71 GB/s (RDMA), 9.50 GB/s (NVL)
But in readme.md， how many ib cards per node?

Type Dispatch #EP Bottleneck bandwidth Combine #EP Bottleneck bandwidth
Internode 16 43 GB/s (RDMA) 16 43 GB/s (RDMA)

During test, arethere any environment variables that need to be set?

Originally created by @MinhuiWan on GitHub (Mar 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/60 My test environment is 2 nodes, 8 H100 per node, 4 IB per node。 the test result is [tuning] Best dispatch (FP8): SMs 24, NVL chunk 20, RDMA chunk 32: 4.47 GB/s (RDMA), 9.00 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 24, RDMA chunk 32: 4.72 GB/s (RDMA), 9.51 GB/s (NVL) [tuning] Best combine: SMs 24, NVL chunk 4, RDMA chunk 32: 4.71 GB/s (RDMA), 9.50 GB/s (NVL) But in readme.md， how many ib cards per node? | Type | Dispatch #EP | Bottleneck bandwidth | Combine #EP | Bottleneck bandwidth | |:---------:|:------------:|:--------------------:|:-----------:|:--------------------:| | Internode | 16 | 43 GB/s (RDMA) | 16 | 43 GB/s (RDMA) | During test, arethere any environment variables that need to be set?
bsakaguchi commented 2026-10-26 09:53:30 +00:00
Author
Owner
Copy link

@Huixxi commented on GitHub (Mar 9, 2025):

Is it pure roce rdma or through IB?

<!-- gh-comment-id:2708756111 --> @Huixxi commented on GitHub (Mar 9, 2025): Is it pure roce rdma or through IB?
bsakaguchi commented 2026-10-26 09:53:30 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Mar 10, 2025):

Our test results were obtained using 8 H800 and 8 IB nodes. We haven't encountered the issue you're experiencing. Perhaps the 4 IB setup requires some additional configurations. There is someone who encountered the same issue https://github.com/deepseek-ai/DeepEP/issues/38#issuecomment-2703724191 and found a solution, so you can pay attention to his response.

<!-- gh-comment-id:2709233769 --> @sphish commented on GitHub (Mar 10, 2025): Our test results were obtained using 8 H800 and 8 IB nodes. We haven't encountered the issue you're experiencing. Perhaps the 4 IB setup requires some additional configurations. There is someone who encountered the same issue https://github.com/deepseek-ai/DeepEP/issues/38#issuecomment-2703724191 and found a solution, so you can pay attention to his response.
bsakaguchi commented 2026-10-26 09:53:30 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Mar 10, 2025):

Can you run nvidia-smi topo -mp and ibv_devinfo and share the results?

<!-- gh-comment-id:2709245806 --> @sphish commented on GitHub (Mar 10, 2025): Can you run `nvidia-smi topo -mp` and `ibv_devinfo` and share the results?
bsakaguchi commented 2026-10-26 09:53:30 +00:00
Author
Owner
Copy link

@MinhuiWan commented on GitHub (Mar 10, 2025):

Our test results were obtained using 8 H800 and 8 IB nodes. We haven't encountered the issue you're experiencing. Perhaps the 4 IB setup requires some additional configurations. There is someone who encountered the same issue #38 (comment) and found a solution, so you can pay attention to his response.

During test， i set the environment variable with this command:
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_LIST=mlx5_1:1,mlx5_7:1,mlx5_9:1,mlx5_13:1
This configuration makes NVSHMEM use the specified IB ports.

<!-- gh-comment-id:2709255026 --> @MinhuiWan commented on GitHub (Mar 10, 2025): > Our test results were obtained using 8 H800 and 8 IB nodes. We haven't encountered the issue you're experiencing. Perhaps the 4 IB setup requires some additional configurations. There is someone who encountered the same issue [#38 (comment)](https://github.com/deepseek-ai/DeepEP/issues/38#issuecomment-2703724191) and found a solution, so you can pay attention to his response. During test， i set the environment variable with this command: export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_HCA_LIST=mlx5_1:1,mlx5_7:1,mlx5_9:1,mlx5_13:1 This configuration makes NVSHMEM use the specified IB ports.
bsakaguchi commented 2026-10-26 09:53:31 +00:00
Author
Owner
Copy link

@MinhuiWan commented on GitHub (Mar 10, 2025):

Is it pure roce rdma or through IB?

I can't understand what‘s relationship of rdma ans IB。During test， i set the environment variable with this command:
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_LIST=mlx5_1:1,mlx5_7:1,mlx5_9:1,mlx5_13:1
This configuration makes NVSHMEM use the specified IB ports.

<!-- gh-comment-id:2709256153 --> @MinhuiWan commented on GitHub (Mar 10, 2025): > Is it pure roce rdma or through IB? I can't understand what‘s relationship of rdma ans IB。During test， i set the environment variable with this command: export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_HCA_LIST=mlx5_1:1,mlx5_7:1,mlx5_9:1,mlx5_13:1 This configuration makes NVSHMEM use the specified IB ports.
bsakaguchi commented 2026-10-26 09:53:31 +00:00
Author
Owner
Copy link

@MinhuiWan commented on GitHub (Mar 10, 2025):

Can you run and and share the results?nvidia-smi topo -mp``ibv_devinfo

root@f51fec68bf55:~# nvidia-smi topo -mp
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    NIC10   NIC11    NIC12   NIC13   CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS     0-47,96-143     0               N/A
GPU1    NODE     X      NODE    NODE    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS     0-47,96-143     0               N/A
GPU2    NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS     0-47,96-143     0               N/A
GPU3    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS      SYS     SYS     0-47,96-143     0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE     NODE    NODE    48-95,144-191   1               N/A
GPU5    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE     NODE    NODE    48-95,144-191   1               N/A
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE     NODE    NODE    48-95,144-191   1               N/A
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE     NODE    PIX     48-95,144-191   1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      PIX     PXB     PXB     NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC4    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX      X      PXB     PXB     NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC5    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     PXB      X      PIX     NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC6    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     PXB     PIX      X      NODE    SYS     SYS     SYS     SYS      SYS     SYS
NIC7    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS      SYS     SYS
NIC8    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE     NODE    NODE
NIC9    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE     NODE    NODE
NIC10   SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE     NODE    NODE
NIC11   SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X       PIX     NODE
NIC12   SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX       X      NODE
NIC13   SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE     NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11
  NIC12: mlx5_12
  NIC13: mlx5_13

root@f51fec68bf55:~# ibv_devinfo
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:002a:09b8
        sys_image_guid:                 58a2:e103:002a:09b8
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:002a:0e38
        sys_image_guid:                 58a2:e103:002a:0e38
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 515
                        port_lid:               1295
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_2
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:0029:c020
        sys_image_guid:                 58a2:e103:0029:c020
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_3
        transport:                      InfiniBand (0)
        fw_ver:                         22.38.1900
        node_guid:                      a088:c203:0030:966c
        sys_image_guid:                 a088:c203:0030:966c
        vendor_id:                      0x02c9
        vendor_part_id:                 4125
        hw_ver:                         0x0
        board_id:                       MT_0000000436
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_4
        transport:                      InfiniBand (0)
        fw_ver:                         22.38.1900
        node_guid:                      a088:c203:0030:966d
        sys_image_guid:                 a088:c203:0030:966c
        vendor_id:                      0x02c9
        vendor_part_id:                 4125
        hw_ver:                         0x0
        board_id:                       MT_0000000436
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_5
        transport:                      InfiniBand (0)
        fw_ver:                         16.32.1010
        node_guid:                      e8eb:d303:00dd:188e
        sys_image_guid:                 e8eb:d303:00dd:188e
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000425
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_6
        transport:                      InfiniBand (0)
        fw_ver:                         16.32.1010
        node_guid:                      e8eb:d303:00dd:188f
        sys_image_guid:                 e8eb:d303:00dd:188e
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000425
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_7
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:002a:0bd8
        sys_image_guid:                 58a2:e103:002a:0bd8
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 515
                        port_lid:               1292
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_8
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:0029:c7a8
        sys_image_guid:                 58a2:e103:0029:c7a8
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_9
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:0029:c700
        sys_image_guid:                 58a2:e103:0029:c700
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 515
                        port_lid:               1287
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_10
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:002a:1708
        sys_image_guid:                 58a2:e103:002a:1708
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               65535
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_11
        transport:                      InfiniBand (0)
        fw_ver:                         22.38.1900
        node_guid:                      a088:c203:0030:9a0c
        sys_image_guid:                 a088:c203:0030:9a0c
        vendor_id:                      0x02c9
        vendor_part_id:                 4125
        hw_ver:                         0x0
        board_id:                       MT_0000000436
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_12
        transport:                      InfiniBand (0)
        fw_ver:                         22.38.1900
        node_guid:                      a088:c203:0030:9a0d
        sys_image_guid:                 a088:c203:0030:9a0c
        vendor_id:                      0x02c9
        vendor_part_id:                 4125
        hw_ver:                         0x0
        board_id:                       MT_0000000436
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_13
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      58a2:e103:002a:1640
        sys_image_guid:                 58a2:e103:002a:1640
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 515
                        port_lid:               1297
                        port_lmc:               0x00
                        link_layer:             InfiniBand
<!-- gh-comment-id:2709256707 --> @MinhuiWan commented on GitHub (Mar 10, 2025): > Can you run and and share the results?`nvidia-smi topo -mp``ibv_devinfo` ``` root@f51fec68bf55:~# nvidia-smi topo -mp GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE NODE SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NODE X NODE NODE SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NODE NODE X NODE SYS SYS SYS SYS NODE NODE PIX NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NODE NODE NODE X SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE 48-95,144-191 1 N/A GPU5 SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE 48-95,144-191 1 N/A GPU6 SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX NODE NODE NODE 48-95,144-191 1 N/A GPU7 SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X PIX PXB PXB NODE SYS SYS SYS SYS SYS SYS NIC4 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE PIX X PXB PXB NODE SYS SYS SYS SYS SYS SYS NIC5 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE PXB PXB X PIX NODE SYS SYS SYS SYS SYS SYS NIC6 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE PXB PXB PIX X NODE SYS SYS SYS SYS SYS SYS NIC7 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE X SYS SYS SYS SYS SYS SYS NIC8 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE NODE NODE NIC9 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE NODE NODE NIC10 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X NODE NODE NODE NIC11 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE X PIX NODE NIC12 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX X NODE NIC13 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 NIC10: mlx5_10 NIC11: mlx5_11 NIC12: mlx5_12 NIC13: mlx5_13 ``` ``` root@f51fec68bf55:~# ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:002a:09b8 sys_image_guid: 58a2:e103:002a:09b8 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:002a:0e38 sys_image_guid: 58a2:e103:002a:0e38 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 515 port_lid: 1295 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_2 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:0029:c020 sys_image_guid: 58a2:e103:0029:c020 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_3 transport: InfiniBand (0) fw_ver: 22.38.1900 node_guid: a088:c203:0030:966c sys_image_guid: a088:c203:0030:966c vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000436 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_4 transport: InfiniBand (0) fw_ver: 22.38.1900 node_guid: a088:c203:0030:966d sys_image_guid: a088:c203:0030:966c vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000436 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_5 transport: InfiniBand (0) fw_ver: 16.32.1010 node_guid: e8eb:d303:00dd:188e sys_image_guid: e8eb:d303:00dd:188e vendor_id: 0x02c9 vendor_part_id: 4119 hw_ver: 0x0 board_id: MT_0000000425 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_6 transport: InfiniBand (0) fw_ver: 16.32.1010 node_guid: e8eb:d303:00dd:188f sys_image_guid: e8eb:d303:00dd:188e vendor_id: 0x02c9 vendor_part_id: 4119 hw_ver: 0x0 board_id: MT_0000000425 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_7 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:002a:0bd8 sys_image_guid: 58a2:e103:002a:0bd8 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 515 port_lid: 1292 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_8 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:0029:c7a8 sys_image_guid: 58a2:e103:0029:c7a8 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_9 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:0029:c700 sys_image_guid: 58a2:e103:0029:c700 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 515 port_lid: 1287 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_10 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:002a:1708 sys_image_guid: 58a2:e103:002a:1708 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_11 transport: InfiniBand (0) fw_ver: 22.38.1900 node_guid: a088:c203:0030:9a0c sys_image_guid: a088:c203:0030:9a0c vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000436 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_12 transport: InfiniBand (0) fw_ver: 22.38.1900 node_guid: a088:c203:0030:9a0d sys_image_guid: a088:c203:0030:9a0c vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000436 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_13 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:002a:1640 sys_image_guid: 58a2:e103:002a:1640 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 515 port_lid: 1297 port_lmc: 0x00 link_layer: InfiniBand ```
bsakaguchi commented 2026-10-26 09:53:31 +00:00
Author
Owner
Copy link

@MinhuiWan commented on GitHub (Mar 18, 2025):

Ensure 8IB per node， and use nccl_test to test the throughout of alltoall， when environment is normal，run test internode.py，the RDMA bandwidth is normal。

<!-- gh-comment-id:2732706726 --> @MinhuiWan commented on GitHub (Mar 18, 2025): Ensure 8IB per node， and use nccl_test to test the throughout of alltoall， when environment is normal，run test internode.py，the RDMA bandwidth is normal。
bsakaguchi commented 2026-10-26 09:53:31 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (Apr 23, 2025):

Ensure 8IB per node， and use nccl_test to test the throughout of alltoall， when environment is normal，run test internode.py，the RDMA bandwidth is normal。

Hello, could you please explain in detail how Ensure 8IB per node operates?

<!-- gh-comment-id:2823741177 --> @CSEEduanyu commented on GitHub (Apr 23, 2025): > Ensure 8IB per node， and use nccl_test to test the throughout of alltoall， when environment is normal，run test internode.py，the RDMA bandwidth is normal。 Hello, could you please explain in detail how Ensure 8IB per node operates?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#57
No description provided.