deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #139] Running low_latency test on RoCE get IBGDA error #122

New issue
Closed
opened 2026-10-26 09:54:47 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 09:54:47 +00:00
Owner
Copy link

Originally created by @TheRainstorm on GitHub (Apr 27, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/139

I can run test_internode.py and test_intranode.py correctly, but I cannot run the test_low_latency .py script. The error reported is mainly related to IBGDA(more complete output is at the end):

/repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA

I found related issues 38, but it doesn't seem relevant to my error.

How should I run low_latency on RoCE? Are any extra settings required? Any help you can provide would be greatly appreciated.

Environment

  • Two nodes, each with 8 H20 GPUs, connected by CX6 network (200Gb, ~25GB/s) cards using RoCE mode.
  • Using the latest DeepEP code.
  • nvshmem: 3.2.5-1
  • gdrcopy: 2.4.4

I have followed the NVSHMEM install guide, manually compiled and loaded the gdrdrv module, set the nvidia driver IBGDA-related options in /etc/modprobe.d (Even though I use RoCE not IB), and my command to compile nvshmem is as follows:

CUDA_HOME=/opt/lib/cuda-12.4.1_normal/ \
GDRCOPY_HOME=/repo/gdrcopy-2.4.4 \
NVSHMEM_SHMEM_SUPPORT=0 \
NVSHMEM_UCX_SUPPORT=0 \
NVSHMEM_USE_NCCL=0 \
NVSHMEM_MPI_SUPPORT=0 \
NVSHMEM_IBGDA_SUPPORT=1 \
NVSHMEM_PMIX_SUPPORT=0 \
NVSHMEM_TIMEOUT_DEVICE_POLLING=0 \
NVSHMEM_USE_GDRCOPY=1 \
cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=/repo/nvshmem_src/install

ibv_devinfo output

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         20.35.2000
        node_guid:                      946d:ae03:009c:f3cc
        sys_image_guid:                 946d:ae03:009c:f3cc
        vendor_id:                      0x02c9
        vendor_part_id:                 4123
        hw_ver:                         0x0
        board_id:                       MT_0000000223
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         20.35.2000
        node_guid:                      946d:ae03:009c:f454
        sys_image_guid:                 946d:ae03:009c:f454
        vendor_id:                      0x02c9
        vendor_part_id:                 4123
        hw_ver:                         0x0
        board_id:                       MT_0000000223
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

...
(omit mlx5_2 - mlx5_9)

Log

$ MASTER_ADDR=xxx MASTER_PORT=8362 WORLD_SIZE=2 RANK=0 python tests/test_low_latency.py

Allocating buffer size: 2116.290944 MB ...
WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path.

WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path.

WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path.

WARN: GPU cannot map UAR of device mlx5_0. Skipping...

...


WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path.
WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path.

WARN: GPU cannot map UAR of device mlx5_3. Skipping...

WARN: GPU cannot map UAR of device mlx5_8. Skipping...

WARN: GPU cannot map UAR of device mlx5_1. Skipping...

WARN: GPU cannot map UAR of device mlx5_7. Skipping...

WARN: GPU cannot map UAR of device mlx5_6. Skipping...

WARN: GPU cannot map UAR of device mlx5_bond_0. Skipping...

/repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: GPU cannot map UAR of device mlx5_bond_0. Skipping...

/repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path.



WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path.


...

/repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
/repo/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 16 bytes instead of 8

/repo/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /repo/nvshmem_src/src/host/mem/mem_heap.cpp:222: non-zero status: -3 allgather of heap base for all PE failed

/repo/nvshmem_src/src/host/mem/mem_heap.cpp:588: non-zero status: 7 Failed to allgather PEs peer_base values

/repo/nvshmem_src/src/host/init/init.cu:1011: non-zero status: 7 nvshmem register static heaps failed

/repo/nvshmem_src/src/host/team/team.cu:nvshmem_team_split_strided:63: NVSHMEM API called before NVSHMEM initialization has completed
Originally created by @TheRainstorm on GitHub (Apr 27, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/139 I can run `test_internode.py` and `test_intranode.py` correctly, but I cannot run the `test_low_latency .py` script. The error reported is mainly related to IBGDA(more complete output is at the end): ```shell /repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA ``` I found related [issues 38](https://github.com/deepseek-ai/DeepEP/issues/38), but it doesn't seem relevant to my error. How should I run low_latency on RoCE? Are any extra settings required? Any help you can provide would be greatly appreciated. ### Environment - Two nodes, each with 8 H20 GPUs, connected by CX6 network (200Gb, ~25GB/s) cards using RoCE mode. - Using the latest DeepEP code. - nvshmem: 3.2.5-1 - gdrcopy: 2.4.4 I have followed the [NVSHMEM install guide](https://github.com/deepseek-ai/DeepEP/blob/main/third-party/README.md), manually compiled and loaded the `gdrdrv` module, set the nvidia driver IBGDA-related options in `/etc/modprobe.d` (Even though I use RoCE not IB), and my command to compile nvshmem is as follows: ```shell CUDA_HOME=/opt/lib/cuda-12.4.1_normal/ \ GDRCOPY_HOME=/repo/gdrcopy-2.4.4 \ NVSHMEM_SHMEM_SUPPORT=0 \ NVSHMEM_UCX_SUPPORT=0 \ NVSHMEM_USE_NCCL=0 \ NVSHMEM_MPI_SUPPORT=0 \ NVSHMEM_IBGDA_SUPPORT=1 \ NVSHMEM_PMIX_SUPPORT=0 \ NVSHMEM_TIMEOUT_DEVICE_POLLING=0 \ NVSHMEM_USE_GDRCOPY=1 \ cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=/repo/nvshmem_src/install ``` ibv_devinfo output ``` hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 20.35.2000 node_guid: 946d:ae03:009c:f3cc sys_image_guid: 946d:ae03:009c:f3cc vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 20.35.2000 node_guid: 946d:ae03:009c:f454 sys_image_guid: 946d:ae03:009c:f454 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet ... (omit mlx5_2 - mlx5_9) ``` ### Log ``` $ MASTER_ADDR=xxx MASTER_PORT=8362 WORLD_SIZE=2 RANK=0 python tests/test_low_latency.py Allocating buffer size: 2116.290944 MB ... WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path. WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path. WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path. WARN: GPU cannot map UAR of device mlx5_0. Skipping... ... WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path. WARN: ibgda_alloc_and_map_qp_uar with GPU as handler failed. We may need to enter the CPU fallback path. WARN: GPU cannot map UAR of device mlx5_3. Skipping... WARN: GPU cannot map UAR of device mlx5_8. Skipping... WARN: GPU cannot map UAR of device mlx5_1. Skipping... WARN: GPU cannot map UAR of device mlx5_7. Skipping... WARN: GPU cannot map UAR of device mlx5_6. Skipping... WARN: GPU cannot map UAR of device mlx5_bond_0. Skipping... /repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: GPU cannot map UAR of device mlx5_bond_0. Skipping... /repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: cudaHostRegister with IoMemory failed with error=800. We may need to use a fallback path. WARN: ibgda_nic_mem_gpu_map failed. We may need to use the CPU fallback path. ... /repo/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA /repo/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 16 bytes instead of 8 /repo/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /repo/nvshmem_src/src/host/mem/mem_heap.cpp:222: non-zero status: -3 allgather of heap base for all PE failed /repo/nvshmem_src/src/host/mem/mem_heap.cpp:588: non-zero status: 7 Failed to allgather PEs peer_base values /repo/nvshmem_src/src/host/init/init.cu:1011: non-zero status: 7 nvshmem register static heaps failed /repo/nvshmem_src/src/host/team/team.cu:nvshmem_team_split_strided:63: NVSHMEM API called before NVSHMEM initialization has completed ```
bsakaguchi commented 2026-10-26 09:54:49 +00:00
Author
Owner
Copy link

@TheRainstorm commented on GitHub (Apr 27, 2025):

#55 (comment) Would this be helpful?

I remember that it was configured before; I will double-check later as I currently don't have administrator privileges.

However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment?

<!-- gh-comment-id:2833354943 --> @TheRainstorm commented on GitHub (Apr 27, 2025): > [#55 (comment)](https://github.com/deepseek-ai/DeepEP/issues/55#issuecomment-2833247769) Would this be helpful? I remember that it was configured before; I will double-check later as I currently don't have administrator privileges. However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment?
bsakaguchi commented 2026-10-26 09:54:50 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Apr 27, 2025):

I'm not sure whether CX6 supports IBGDA.

<!-- gh-comment-id:2833355070 --> @sphish commented on GitHub (Apr 27, 2025): I'm not sure whether CX6 supports IBGDA.
bsakaguchi commented 2026-10-26 09:54:50 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Apr 27, 2025):

However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment?

The name IBGDA might be a bit confusing. It actually applies to both IB and RoCE. In the latest version of DeepEP, this is necessary for both.

<!-- gh-comment-id:2833356280 --> @sphish commented on GitHub (Apr 27, 2025): > However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment? The name IBGDA might be a bit confusing. It actually applies to both IB and RoCE. In the latest version of DeepEP, this is necessary for both.
bsakaguchi commented 2026-10-26 09:54:50 +00:00
Author
Owner
Copy link

@TheRainstorm commented on GitHub (Apr 27, 2025):

However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment?

The name IBGDA might be a bit confusing. It actually applies to both IB and RoCE. In the latest version of DeepEP, this is necessary for both.

Thank you, I will check both.

<!-- gh-comment-id:2833357129 --> @TheRainstorm commented on GitHub (Apr 27, 2025): > > However, these two options are used to enable IBGDA, and since IBGDA is an IB feature, is it also necessary in a RoCE environment? > > The name IBGDA might be a bit confusing. It actually applies to both IB and RoCE. In the latest version of DeepEP, this is necessary for both. Thank you, I will check both.
bsakaguchi commented 2026-10-26 09:54:50 +00:00
Author
Owner
Copy link

@TheRainstorm commented on GitHub (Apr 29, 2025):

I'm not sure whether CX6 supports IBGDA.

I checked that CX-6 supports IBGDA. Here is an article comparing CX6 performance with IBGDA and IBRC.

And I check the nvidia option is configured by running cat /proc/driver/nvidia/params, which output EnableStreamMemOPs: 1 and RegistryDwords: "PeerMappingOverride=1;"

In the latest version of DeepEP

I'm sorry that what I was running before was not the latest version, but rather a version from before #130 was merged. After updating to the new version, my error message changed. Now I am getting the error:

$ MASTER_ADDR=172.19.33.83 MASTER_PORT=8362 WORLD_SIZE=2 RANK=0 python tests/test_low_latency.py 2>&1 |tee r_lowlatency2.log
Allocating buffer size: 2115.111296 MB ...
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
[/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch
W0429 20:14:08.665000 145833 /miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 145915 via signal SIGTERM
W0429 20:14:08.665000 145833 /miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 145917 via signal SIGTERM
Traceback (most recent call last):
  File "/repo/DeepEP/tests/test_low_latency.py", line 172, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/repo/DeepEP/tests/test_low_latency.py", line 158, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1)
  File "/repo/DeepEP/tests/test_low_latency.py", line 40, in test_main
    buffer.low_latency_dispatch(x, topk_idx, num_tokens, num_experts, use_fp8=dispatch_use_fp8,
  File "/repo/DeepEP/deep_ep/buffer.py", line 488, in low_latency_dispatch
    self.runtime.low_latency_dispatch(x, topk_idx,
RuntimeError: Failed: CUDA error /repo/DeepEP/csrc/kernels/internode_ll.cu:341 'too many blocks in cooperative launch'

My NVSHMEM version is 3.2.5-1, compiled with the latest patch. Do you have any suggestions?

<!-- gh-comment-id:2838630214 --> @TheRainstorm commented on GitHub (Apr 29, 2025): > I'm not sure whether CX6 supports IBGDA. I checked that CX-6 supports IBGDA. Here is an [article](https://zhuanlan.zhihu.com/p/689043376) comparing CX6 performance with IBGDA and IBRC. And I check the nvidia option is configured by running `cat /proc/driver/nvidia/params`, which output `EnableStreamMemOPs: 1` and `RegistryDwords: "PeerMappingOverride=1;"` > In the latest version of DeepEP I'm sorry that what I was running before was not the latest version, but rather a version from before #130 was merged. After updating to the new version, my error message changed. Now I am getting the error: ```shell $ MASTER_ADDR=172.19.33.83 MASTER_PORT=8362 WORLD_SIZE=2 RANK=0 python tests/test_low_latency.py 2>&1 |tee r_lowlatency2.log Allocating buffer size: 2115.111296 MB ... [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch W0429 20:14:08.665000 145833 /miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 145915 via signal SIGTERM W0429 20:14:08.665000 145833 /miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 145917 via signal SIGTERM Traceback (most recent call last): File "/repo/DeepEP/tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/miniconda3/envs/tcu12/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/repo/DeepEP/tests/test_low_latency.py", line 158, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1) File "/repo/DeepEP/tests/test_low_latency.py", line 40, in test_main buffer.low_latency_dispatch(x, topk_idx, num_tokens, num_experts, use_fp8=dispatch_use_fp8, File "/repo/DeepEP/deep_ep/buffer.py", line 488, in low_latency_dispatch self.runtime.low_latency_dispatch(x, topk_idx, RuntimeError: Failed: CUDA error /repo/DeepEP/csrc/kernels/internode_ll.cu:341 'too many blocks in cooperative launch' ``` My NVSHMEM version is 3.2.5-1, compiled with the latest patch. Do you have any suggestions?
bsakaguchi commented 2026-10-26 09:54:50 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Apr 30, 2025):

@TheRainstorm https://github.com/deepseek-ai/DeepEP/issues/15#issuecomment-2709715782

<!-- gh-comment-id:2840634166 --> @alpha-baby commented on GitHub (Apr 30, 2025): @TheRainstorm https://github.com/deepseek-ai/DeepEP/issues/15#issuecomment-2709715782
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#122
No description provided.