deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #268] test_low_latency failes with non-zero status: 22 ibv_modify_qp failed #196

New issue
Closed
opened 2026-10-26 09:56:24 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:56:24 +00:00
Owner
Copy link

Originally created by @SzymonOzog on GitHub (Jun 30, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/268

Hi I'm trying to run DeepEP in sglang`s docker container https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile

My host has NVSHMEM with the DeepEP patch installed and using cuda driver 12.9, kernel modules are also loaded in modeprobe.d/nvidia.conf

test_internode and test_intranode run fine but trying to run test_low_latency it failes with the following log:
https://pastebin.com/raw/Q7Z2KGja

Here is my network setup:

root@gh-3714u06:/sgl-workspace/DeepEP# ibstat
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0x9c63c003005548fe
        System image GUID: 0x9c63c003005548fe
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 975
                LMC: 0
                SM lid: 1165
                Capability mask: 0xa751e848
                Port GUID: 0x9c63c003005548fe
                Link layer: InfiniBand
CA 'mlx5_1'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0x9c63c003005548b6
        System image GUID: 0x9c63c003005548b6
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 976
                LMC: 0
                SM lid: 1165
                Capability mask: 0xa751e848
                Port GUID: 0x9c63c003005548b6
                Link layer: InfiniBand
CA 'mlx5_2'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.38.1900
        Hardware version: 0
        Node GUID: 0xb83fd2030091eb2a
        System image GUID: 0xb83fd2030091eb2a
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xba3fd2fffe91eb2a
                Link layer: Ethernet
CA 'mlx5_3'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.38.1900
        Hardware version: 0
        Node GUID: 0xb83fd2030091eb2b
        System image GUID: 0xb83fd2030091eb2a
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xba3fd2fffe91eb2b
                Link layer: Ethernet
CA 'mlx5_4'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0x9c63c003005546fe
        System image GUID: 0x9c63c003005546fe
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 981
                LMC: 0
                SM lid: 1165
                Capability mask: 0xa751e848
                Port GUID: 0x9c63c003005546fe
                Link layer: InfiniBand
CA 'mlx5_5'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0x9c63c003005b1e14
        System image GUID: 0x9c63c003005b1e14
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 984
                LMC: 0
                SM lid: 1165
                Capability mask: 0xa751e848
                Port GUID: 0x9c63c003005b1e14
                Link layer: InfiniBand
root@gh-3714u06:/sgl-workspace/DeepEP# nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity   NUMA Affinity    GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     0-31,64-95     0N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     0-31,64-95     0N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    SYS     SYS     0-31,64-95     0N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    SYS     SYS     0-31,64-95     0N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    32-63,96-127   1N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    NODE    32-63,96-127   1N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    PIX     32-63,96-127   1N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    32-63,96-127   1N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS
NIC1    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS
NIC2    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      PIX     SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    PIX      X      SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE
NIC5    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE     X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5

The ony thing that I can see in the logs is The system does not support registering the NIC control buffers with DMABUF. Fallback to use either nv_peer_mem or nvidia_peermem. Not sure if that could be the reason

Originally created by @SzymonOzog on GitHub (Jun 30, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/268 Hi I'm trying to run DeepEP in sglang`s docker container https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile My host has NVSHMEM with the DeepEP patch installed and using cuda driver 12.9, kernel modules are also loaded in `modeprobe.d/nvidia.conf` `test_internode` and `test_intranode` run fine but trying to run `test_low_latency` it failes with the following log: https://pastebin.com/raw/Q7Z2KGja Here is my network setup: ``` root@gh-3714u06:/sgl-workspace/DeepEP# ibstat CA 'mlx5_0' CA type: MT4129 Number of ports: 1 Firmware version: 28.41.1000 Hardware version: 0 Node GUID: 0x9c63c003005548fe System image GUID: 0x9c63c003005548fe Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 975 LMC: 0 SM lid: 1165 Capability mask: 0xa751e848 Port GUID: 0x9c63c003005548fe Link layer: InfiniBand CA 'mlx5_1' CA type: MT4129 Number of ports: 1 Firmware version: 28.41.1000 Hardware version: 0 Node GUID: 0x9c63c003005548b6 System image GUID: 0x9c63c003005548b6 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 976 LMC: 0 SM lid: 1165 Capability mask: 0xa751e848 Port GUID: 0x9c63c003005548b6 Link layer: InfiniBand CA 'mlx5_2' CA type: MT4123 Number of ports: 1 Firmware version: 20.38.1900 Hardware version: 0 Node GUID: 0xb83fd2030091eb2a System image GUID: 0xb83fd2030091eb2a Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba3fd2fffe91eb2a Link layer: Ethernet CA 'mlx5_3' CA type: MT4123 Number of ports: 1 Firmware version: 20.38.1900 Hardware version: 0 Node GUID: 0xb83fd2030091eb2b System image GUID: 0xb83fd2030091eb2a Port 1: State: Down Physical state: Disabled Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba3fd2fffe91eb2b Link layer: Ethernet CA 'mlx5_4' CA type: MT4129 Number of ports: 1 Firmware version: 28.41.1000 Hardware version: 0 Node GUID: 0x9c63c003005546fe System image GUID: 0x9c63c003005546fe Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 981 LMC: 0 SM lid: 1165 Capability mask: 0xa751e848 Port GUID: 0x9c63c003005546fe Link layer: InfiniBand CA 'mlx5_5' CA type: MT4129 Number of ports: 1 Firmware version: 28.41.1000 Hardware version: 0 Node GUID: 0x9c63c003005b1e14 System image GUID: 0x9c63c003005b1e14 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 984 LMC: 0 SM lid: 1165 Capability mask: 0xa751e848 Port GUID: 0x9c63c003005b1e14 Link layer: InfiniBand root@gh-3714u06:/sgl-workspace/DeepEP# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS 0-31,64-95 0N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-31,64-95 0N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE SYS SYS 0-31,64-95 0N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-31,64-95 0N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE 32-63,96-127 1N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE NODE 32-63,96-127 1N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE PIX 32-63,96-127 1N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE 32-63,96-127 1N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS NIC1 NODE NODE PIX NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS NIC2 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X PIX SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE PIX X SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NIC5 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 ``` The ony thing that I can see in the logs is `The system does not support registering the NIC control buffers with DMABUF. Fallback to use either nv_peer_mem or nvidia_peermem.` Not sure if that could be the reason
bsakaguchi commented 2026-10-26 09:56:25 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 1, 2025):

You should use NVSHMEM_HCA_LIST and NVSHMEM_HCA_PE_MAPPING to assign correct NIC to each GPU.

<!-- gh-comment-id:3022710128 --> @sphish commented on GitHub (Jul 1, 2025): You should use `NVSHMEM_HCA_LIST` and `NVSHMEM_HCA_PE_MAPPING` to assign correct NIC to each GPU.
bsakaguchi commented 2026-10-26 09:56:26 +00:00
Author
Owner
Copy link

@SzymonOzog commented on GitHub (Jul 1, 2025):

This resolves the issue, thank you very much

<!-- gh-comment-id:3023767026 --> @SzymonOzog commented on GitHub (Jul 1, 2025): This resolves the issue, thank you very much
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#196
No description provided.