deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #155] Lower than expected performance on H100 system #133

New issue
Closed
opened 2026-10-26 09:55:01 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 09:55:01 +00:00
Owner
Copy link

Originally created by @sghazimi on GitHub (May 12, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/155

We're running DeepEP on a system with 2 nodes each with 8 H100 and 8 mlx5Gen NICs and we get the bandwidth of around 25 GB/s for dispatch and combine:

[tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 24.19 GB/s (RDMA), 79.12 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 24.21 GB/s (RDMA), 79.19 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 20, RDMA chunk 32: 25.80 GB/s (RDMA), 84.39 GB/s (NVL)

[tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 22.99 GB/s (RDMA), 75.21 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 24.37 GB/s (RDMA), 79.73 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 28: 25.22 GB/s (RDMA), 82.51 GB/s (NVL)

when running shmem_put_bw between nodes, we get the expected bandwidth of around 47 GB/s. The results from nccl_tests are also as expected.

I looked into some suggestion regarding performance issues in other tickets and set the right mappings to the NIC e.g.,

os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1'
os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1'

Still, the best bandwidth we get is around 25 GB/s.

Here is the output of ibv_devinfo and nvidia-smi topo -mp:

ibv_devinfo
hca_id: mlx5_ib0
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f3
        sys_image_guid:                 3868:dd03:01c0:1908
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               22380
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib1
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f4
        sys_image_guid:                 3868:dd03:02c0:1909
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               22353
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib2
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f5
        sys_image_guid:                 3868:dd03:03c0:190a
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               22868
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib3
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f6
        sys_image_guid:                 3868:dd03:04c0:190b
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               23403
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib4
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f7
        sys_image_guid:                 3868:dd03:05c0:190c
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               24245
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib5
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f8
        sys_image_guid:                 3868:dd03:06c0:190d
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               24270
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib6
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25f9
        sys_image_guid:                 3868:dd03:07c0:190e
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               24252
                        port_lmc:               0x00
                        link_layer:             InfiniBand

hca_id: mlx5_ib7
        transport:                      InfiniBand (0)
        fw_ver:                         28.40.1702
        node_guid:                      0015:5dff:fe34:25fa
        sys_image_guid:                 3868:dd03:08c0:190f
        vendor_id:                      0x02c9
        vendor_part_id:                 4126
        hw_ver:                         0x0
        board_id:                       MSF0000000047
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               24261
                        port_lmc:               0x00
                        link_layer:             InfiniBand

nvidia-smi topo -mp
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47    0               N/A
GPU1    NODE     X      NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47    0               N/A
GPU2    NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47    0               N/A
GPU3    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47    0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    48-95   1               N/A
GPU5    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    48-95   1               N/A
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    48-95   1               N/A
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    48-95   1               N/A
NIC0    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

NIC Legend:

  NIC0: mlx5_ib0
  NIC1: mlx5_ib1
  NIC2: mlx5_ib2
  NIC3: mlx5_ib3
  NIC4: mlx5_ib4
  NIC5: mlx5_ib5
  NIC6: mlx5_ib6
  NIC7: mlx5_ib7

Looking into the Nsight profile, most time is spent on dispatch and combine kernels:

Image

A few questions:

  • Do you have any pointers on what the performance bottlenecks could be? or any recommendation on how to debug this performance issue?
  • I see nvshmem calls are in notify_dispatch but my guess is that the actual communication is happening in dispatch and combine kernels where ibgda control messages are sent, right?
Originally created by @sghazimi on GitHub (May 12, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/155 We're running DeepEP on a system with 2 nodes each with 8 H100 and 8 mlx5Gen NICs and we get the bandwidth of around 25 GB/s for dispatch and combine: ``` [tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 24.19 GB/s (RDMA), 79.12 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 24.21 GB/s (RDMA), 79.19 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 20, RDMA chunk 32: 25.80 GB/s (RDMA), 84.39 GB/s (NVL) ``` ``` [tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 22.99 GB/s (RDMA), 75.21 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 24.37 GB/s (RDMA), 79.73 GB/s (NVL) [tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 28: 25.22 GB/s (RDMA), 82.51 GB/s (NVL) ``` when running ```shmem_put_bw``` between nodes, we get the expected bandwidth of around 47 GB/s. The results from nccl_tests are also as expected. I looked into some suggestion regarding performance issues in other tickets and set the right mappings to the NIC e.g., ``` os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1' os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1' ``` Still, the best bandwidth we get is around 25 GB/s. Here is the output of ibv_devinfo and nvidia-smi topo -mp: ``` ibv_devinfo hca_id: mlx5_ib0 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f3 sys_image_guid: 3868:dd03:01c0:1908 vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 22380 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib1 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f4 sys_image_guid: 3868:dd03:02c0:1909 vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 22353 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib2 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f5 sys_image_guid: 3868:dd03:03c0:190a vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 22868 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib3 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f6 sys_image_guid: 3868:dd03:04c0:190b vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 23403 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib4 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f7 sys_image_guid: 3868:dd03:05c0:190c vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 24245 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib5 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f8 sys_image_guid: 3868:dd03:06c0:190d vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 24270 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib6 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25f9 sys_image_guid: 3868:dd03:07c0:190e vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 24252 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_ib7 transport: InfiniBand (0) fw_ver: 28.40.1702 node_guid: 0015:5dff:fe34:25fa sys_image_guid: 3868:dd03:08c0:190f vendor_id: 0x02c9 vendor_part_id: 4126 hw_ver: 0x0 board_id: MSF0000000047 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 24261 port_lmc: 0x00 link_layer: InfiniBand ``` ``` nvidia-smi topo -mp GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU1 NODE X NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU2 NODE NODE X NODE SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU3 NODE NODE NODE X SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU4 SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE 48-95 1 N/A GPU5 SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE 48-95 1 N/A GPU6 SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NODE NODE NODE NODE 48-95 1 N/A GPU7 SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NODE NODE NODE NODE 48-95 1 N/A NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NIC Legend: NIC0: mlx5_ib0 NIC1: mlx5_ib1 NIC2: mlx5_ib2 NIC3: mlx5_ib3 NIC4: mlx5_ib4 NIC5: mlx5_ib5 NIC6: mlx5_ib6 NIC7: mlx5_ib7 ``` Looking into the Nsight profile, most time is spent on dispatch and combine kernels: ![Image](https://github.com/user-attachments/assets/4dd166f7-392e-4ba7-ba50-fac041942abe) A few questions: - Do you have any pointers on what the performance bottlenecks could be? or any recommendation on how to debug this performance issue? - I see nvshmem calls are in notify_dispatch but my guess is that the actual communication is happening in dispatch and combine kernels where ibgda control messages are sent, right?
bsakaguchi commented 2026-10-26 09:55:02 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (May 14, 2025):

Are you using a dual-port network card?

<!-- gh-comment-id:2879502011 --> @sphish commented on GitHub (May 14, 2025): Are you using a dual-port network card?
bsakaguchi commented 2026-10-26 09:55:02 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (May 14, 2025):

why here is not PIX? @sghazimi

Image

Image
https://docs.nvidia.com/nvshmem/api/gen/env.html#c.NVSHMEM_HCA_PE_MAPPING

<!-- gh-comment-id:2879733393 --> @alpha-baby commented on GitHub (May 14, 2025): why here is not PIX? @sghazimi ![Image](https://github.com/user-attachments/assets/e9e2bfe7-e169-4cde-82f7-a5d97e13de48) ![Image](https://github.com/user-attachments/assets/781b083e-0a58-4f3c-9299-c2781755f53d) https://docs.nvidia.com/nvshmem/api/gen/env.html#c.NVSHMEM_HCA_PE_MAPPING
bsakaguchi commented 2026-10-26 09:55:03 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (May 14, 2025):

maybe you should config:

os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '0'
os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1'

or

os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1'
os.environ['NVSHMEM_HCA_PE_MAPPING'] = 'mlx5_ib0:1:1,mlx5_ib1:1:1,mlx5_ib2:1:1,mlx5_ib3:1:1,mlx5_ib4:1:1,mlx5_ib5:1:1,mlx5_ib6:1:1,mlx5_ib7:1:1'

Image

<!-- gh-comment-id:2879749121 --> @alpha-baby commented on GitHub (May 14, 2025): maybe you should config: ``` os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '0' os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1' ``` or ``` os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1' os.environ['NVSHMEM_HCA_PE_MAPPING'] = 'mlx5_ib0:1:1,mlx5_ib1:1:1,mlx5_ib2:1:1,mlx5_ib3:1:1,mlx5_ib4:1:1,mlx5_ib5:1:1,mlx5_ib6:1:1,mlx5_ib7:1:1' ``` ![Image](https://github.com/user-attachments/assets/996b401e-d4f1-4f34-9fc6-655f6effa20a)
bsakaguchi commented 2026-10-26 09:55:03 +00:00
Author
Owner
Copy link

@sghazimi commented on GitHub (May 14, 2025):

Thanks for your response.

I made the following changes to the code to make sure ranks are bind to the right cores, numa node, and gpu on my system. With these changes, I'm now able to get the bandwidth of 60GB/s for inter-node test

git diff
diff --git a/deep_ep/buffer.py b/deep_ep/buffer.py
index feeb386..d81130e 100644
--- a/deep_ep/buffer.py
+++ b/deep_ep/buffer.py
@@ -72,6 +72,8 @@ class Buffer:
             os.environ['NVSHMEM_IB_ENABLE_IBGDA'] = '1'
             os.environ['NVSHMEM_IBGDA_NIC_HANDLER'] = 'gpu'
             os.environ['NVSHMEM_IBGDA_NUM_RC_PER_PE'] = f'{num_qps_per_rank}'
+            os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1'
+            os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1'
             # Make sure QP depth is always larger than the number of on-flight WRs, so that we can skip WQ slot check
             os.environ['NVSHMEM_QP_DEPTH'] = '1024'
             # NOTES: NVSHMEM initialization requires at least 256 MiB
diff --git a/tests/utils.py b/tests/utils.py
index a574366..fffa905 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,10 +1,34 @@
 import os
 import sys
+import psutil
 import numpy as np
 import torch
 import torch.distributed as dist
 from typing import Optional
-
+import ctypes
+
+# Load libnuma
+libnuma = ctypes.CDLL("libnuma.so")
+libnuma.numa_available.restype = ctypes.c_int
+libnuma.numa_run_on_node.argtypes = [ctypes.c_int]
+libnuma.numa_set_preferred.argtypes = [ctypes.c_int]
+
+def set_numa_affinity(rank):
+    # Example: Assume NUMA node 0 → cores 0-15, node 1 → 16-31, etc.
+    cores_per_numa = 12
+    numa_node = rank // 4
+    core_start = rank * cores_per_numa
+    core_end = core_start + cores_per_numa
+    p = psutil.Process(os.getpid())
+    p.cpu_affinity(list(range(core_start, core_end)))
+    print(f"Rank {rank} numa node {numa_node} bound to cores {core_start}-{core_end - 1}")
+
+       # Bind memory to NUMA node
+    if libnuma.numa_available() != -1:
+        libnuma.numa_set_preferred(numa_node)
+        print(f"Rank {rank}: CPU affinity → cores {core_start}-{core_end - 1}, memory NUMA → node {numa_node}")
+    else:
+        print(f"Rank {rank}: libnuma not available")

 def init_dist(local_rank: int, num_local_ranks: int):
     # NOTES: you may rewrite this function with your own cluster settings
@@ -20,8 +44,10 @@ def init_dist(local_rank: int, num_local_ranks: int):
         world_size=num_nodes * num_local_ranks,
         rank=node_rank * num_local_ranks + local_rank
     )
+    set_numa_affinity(local_rank)
     torch.set_default_dtype(torch.bfloat16)
     torch.set_default_device('cuda')
+
     torch.cuda.set_device(local_rank)

     return dist.get_rank(), dist.get_world_size(), dist.new_group(list(range(num_local_ranks * num_nodes)))

Here is the new results:

Image

<!-- gh-comment-id:2881349612 --> @sghazimi commented on GitHub (May 14, 2025): Thanks for your response. I made the following changes to the code to make sure ranks are bind to the right cores, numa node, and gpu on my system. With these changes, I'm now able to get the bandwidth of 60GB/s for inter-node test ``` git diff diff --git a/deep_ep/buffer.py b/deep_ep/buffer.py index feeb386..d81130e 100644 --- a/deep_ep/buffer.py +++ b/deep_ep/buffer.py @@ -72,6 +72,8 @@ class Buffer: os.environ['NVSHMEM_IB_ENABLE_IBGDA'] = '1' os.environ['NVSHMEM_IBGDA_NIC_HANDLER'] = 'gpu' os.environ['NVSHMEM_IBGDA_NUM_RC_PER_PE'] = f'{num_qps_per_rank}' + os.environ['NVSHMEM_ENABLE_NIC_PE_MAPPING'] = '1' + os.environ['NVSHMEM_HCA_LIST'] = 'mlx5_ib0:1,mlx5_ib1:1,mlx5_ib2:1,mlx5_ib3:1,mlx5_ib4:1,mlx5_ib5:1,mlx5_ib6:1,mlx5_ib7:1' # Make sure QP depth is always larger than the number of on-flight WRs, so that we can skip WQ slot check os.environ['NVSHMEM_QP_DEPTH'] = '1024' # NOTES: NVSHMEM initialization requires at least 256 MiB diff --git a/tests/utils.py b/tests/utils.py index a574366..fffa905 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,10 +1,34 @@ import os import sys +import psutil import numpy as np import torch import torch.distributed as dist from typing import Optional - +import ctypes + +# Load libnuma +libnuma = ctypes.CDLL("libnuma.so") +libnuma.numa_available.restype = ctypes.c_int +libnuma.numa_run_on_node.argtypes = [ctypes.c_int] +libnuma.numa_set_preferred.argtypes = [ctypes.c_int] + +def set_numa_affinity(rank): + # Example: Assume NUMA node 0 → cores 0-15, node 1 → 16-31, etc. + cores_per_numa = 12 + numa_node = rank // 4 + core_start = rank * cores_per_numa + core_end = core_start + cores_per_numa + p = psutil.Process(os.getpid()) + p.cpu_affinity(list(range(core_start, core_end))) + print(f"Rank {rank} numa node {numa_node} bound to cores {core_start}-{core_end - 1}") + + # Bind memory to NUMA node + if libnuma.numa_available() != -1: + libnuma.numa_set_preferred(numa_node) + print(f"Rank {rank}: CPU affinity → cores {core_start}-{core_end - 1}, memory NUMA → node {numa_node}") + else: + print(f"Rank {rank}: libnuma not available") def init_dist(local_rank: int, num_local_ranks: int): # NOTES: you may rewrite this function with your own cluster settings @@ -20,8 +44,10 @@ def init_dist(local_rank: int, num_local_ranks: int): world_size=num_nodes * num_local_ranks, rank=node_rank * num_local_ranks + local_rank ) + set_numa_affinity(local_rank) torch.set_default_dtype(torch.bfloat16) torch.set_default_device('cuda') + torch.cuda.set_device(local_rank) return dist.get_rank(), dist.get_world_size(), dist.new_group(list(range(num_local_ranks * num_nodes))) ``` Here is the new results: ![Image](https://github.com/user-attachments/assets/76779504-e5dd-4daa-a417-d5c1c29528a5)
bsakaguchi commented 2026-10-26 09:55:03 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (May 15, 2025):

@sghazimi Your network card is cx6? cx7? What is the bandwidth of the network card?

<!-- gh-comment-id:2882151792 --> @alpha-baby commented on GitHub (May 15, 2025): @sghazimi Your network card is cx6? cx7? What is the bandwidth of the network card?
bsakaguchi commented 2026-10-26 09:55:04 +00:00
Author
Owner
Copy link

@sghazimi commented on GitHub (May 15, 2025):

it's CX7. the bandwidth is 400Gb/s

<!-- gh-comment-id:2883803956 --> @sghazimi commented on GitHub (May 15, 2025): it's CX7. the bandwidth is 400Gb/s
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#133
No description provided.