deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #321] 【test ll failed】non-zero status: 101 ibv_modify_qp failed #224

New issue
Closed
opened 2026-10-26 09:56:56 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:56:56 +00:00
Owner
Copy link

Originally created by @ZhenshengWu on GitHub (Jul 22, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/321

PS: test internode could be ok，but test ll faild

Environment

gdrcopy：2.4.4
nvshmem: 3.2.5
deep_ep: Comit_id：e6012370b01a86e821ace99193f5d912b3f66d06

4 * 400GB CX7

Infiniband device 'mlx5_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:a088:c203:0020:e496
	base lid:	 0x78
	sm lid:		 0x2
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_1' port 1 status:
	default gid:	 fe80:0000:0000:0000:58a2:e103:001b:2b22
	base lid:	 0x34
	sm lid:		 0x2
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_2' port 1 status:
	default gid:	 fe80:0000:0000:0000:a088:c203:001c:55d2
	base lid:	 0xff
	sm lid:		 0x2
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_3' port 1 status:
	default gid:	 fe80:0000:0000:0000:a088:c203:0020:e0aa
	base lid:	 0x13c
	sm lid:		 0x2
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 400 Gb/sec (4X NDR)
	link_layer:	 InfiniBand

logs

busId 1c000 commId 0x9cd3d1bc38d05e96 - Init COMPLETE
gpu24:5474:6031 [3] NCCL INFO ncclCommInitRank comm 0x55b33b820890 rank 19 nranks 32 cudaDev 3 nvmlDev 3 busId 67000 commId 0x9cd3d1bc38d05e96 - Init COMPLETE
/sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 101 ibv_modify_qp failed
/sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed
/sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed
/sgl-workspace/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed
/sgl-workspace/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed
/sgl-workspace/nvshmem_src/src/host/team/team.cu:nvshmem_team_split_strided:63: NVSHMEM API called before NVSHMEM initialization has completed
gpu24:5476:6078 [5] NCCL INFO [Service thread] Connection closed by localRank 7
gpu24:5474:6085 [3] NCCL INFO [Service thread] Connection closed by localRank 7
gpu24:5475:6079 [4] NCCL INFO [Service thread] Connection closed by localRank 7
gpu24:5473:6076 [2] NCCL INFO [Service thread] Connection closed by localRank 7
W0722 01:57:47.072000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5471 via signal SIGTERM
W0722 01:57:47.074000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5472 via signal SIGTERM
W0722 01:57:47.075000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5473 via signal SIGTERM
W0722 01:57:47.076000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5474 via signal SIGTERM
W0722 01:57:47.076000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5475 via signal SIGTERM
W0722 01:57:47.077000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5476 via signal SIGTERM
W0722 01:57:47.077000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5477 via signal SIGTERM
Traceback (most recent call last):
  File "/home/DeepEP/tests/test_low_latency.py", line 172, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 204, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 7 terminated with exit code 255
Image

how can i solve this problem？

In addition, with the latest commit — commit_id: d72817eb05 — when there are only 4 IB cards, running internode.py causes an error during the combine process: the assertion num_max_rdma_chunked_send_tokens >= num_warps_per_forwarder fails.

Originally created by @ZhenshengWu on GitHub (Jul 22, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/321 PS: **test internode could be ok，but test ll faild** # Environment ``` gdrcopy：2.4.4 nvshmem: 3.2.5 deep_ep: Comit_id：e6012370b01a86e821ace99193f5d912b3f66d06 ``` # 4 * 400GB CX7 ``` Infiniband device 'mlx5_0' port 1 status: default gid: fe80:0000:0000:0000:a088:c203:0020:e496 base lid: 0x78 sm lid: 0x2 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_1' port 1 status: default gid: fe80:0000:0000:0000:58a2:e103:001b:2b22 base lid: 0x34 sm lid: 0x2 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_2' port 1 status: default gid: fe80:0000:0000:0000:a088:c203:001c:55d2 base lid: 0xff sm lid: 0x2 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand Infiniband device 'mlx5_3' port 1 status: default gid: fe80:0000:0000:0000:a088:c203:0020:e0aa base lid: 0x13c sm lid: 0x2 state: 4: ACTIVE phys state: 5: LinkUp rate: 400 Gb/sec (4X NDR) link_layer: InfiniBand ``` # logs ``` busId 1c000 commId 0x9cd3d1bc38d05e96 - Init COMPLETE gpu24:5474:6031 [3] NCCL INFO ncclCommInitRank comm 0x55b33b820890 rank 19 nranks 32 cudaDev 3 nvmlDev 3 busId 67000 commId 0x9cd3d1bc38d05e96 - Init COMPLETE /sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:418: non-zero status: 101 ibv_modify_qp failed /sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1433: non-zero status: 7 ep_connect failed /sgl-workspace/nvshmem_src/src/modules/transport/ibrc/ibrc.cpp:1500: non-zero status: 7 transport create connect failed /sgl-workspace/nvshmem_src/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed /sgl-workspace/nvshmem_src/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed /sgl-workspace/nvshmem_src/src/host/team/team.cu:nvshmem_team_split_strided:63: NVSHMEM API called before NVSHMEM initialization has completed gpu24:5476:6078 [5] NCCL INFO [Service thread] Connection closed by localRank 7 gpu24:5474:6085 [3] NCCL INFO [Service thread] Connection closed by localRank 7 gpu24:5475:6079 [4] NCCL INFO [Service thread] Connection closed by localRank 7 gpu24:5473:6076 [2] NCCL INFO [Service thread] Connection closed by localRank 7 W0722 01:57:47.072000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5471 via signal SIGTERM W0722 01:57:47.074000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5472 via signal SIGTERM W0722 01:57:47.075000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5473 via signal SIGTERM W0722 01:57:47.076000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5474 via signal SIGTERM W0722 01:57:47.076000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5475 via signal SIGTERM W0722 01:57:47.077000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5476 via signal SIGTERM W0722 01:57:47.077000 5406 torch/multiprocessing/spawn.py:169] Terminating process 5477 via signal SIGTERM Traceback (most recent call last): File "/home/DeepEP/tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 204, in join raise ProcessExitedException( torch.multiprocessing.spawn.ProcessExitedException: process 7 terminated with exit code 255 ``` <img width="954" height="215" alt="Image" src="https://github.com/user-attachments/assets/693c7688-46fd-4b9a-848f-79b98d93b3ae" /> how can i solve this problem？ **In addition, with the latest commit — commit_id: d72817eb05c4e7f425972a2c364b11c13da83907 — when there are only 4 IB cards, running internode.py causes an error during the combine process: the assertion num_max_rdma_chunked_send_tokens >= num_warps_per_forwarder fails.**
bsakaguchi commented 2026-10-26 09:56:58 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 22, 2025):

According to the logs, NVSHMEM failed to establish the IBRC connection. You can try to manually specify the correct NIC by using the NVSHMEM_HCA_PE_MAPPING and NVSHMEM_HCA_LIST environment variables.

In addition, with the latest commit — commit_id: github.com/deepseek-ai/DeepEP@d72817eb05 — when there are only 4 IB cards, running internode.py causes an error during the combine process: the assertion num_max_rdma_chunked_send_tokens >= num_warps_per_forwarder fails.

Thanks. Fixed in github.com/deepseek-ai/DeepEP@bdd119f8b2.

<!-- gh-comment-id:3100690956 --> @sphish commented on GitHub (Jul 22, 2025): According to the logs, NVSHMEM failed to establish the IBRC connection. You can try to manually specify the correct NIC by using the `NVSHMEM_HCA_PE_MAPPING` and `NVSHMEM_HCA_LIST` environment variables. > In addition, with the latest commit — commit_id: https://github.com/deepseek-ai/DeepEP/commit/d72817eb05c4e7f425972a2c364b11c13da83907 — when there are only 4 IB cards, running internode.py causes an error during the combine process: the assertion num_max_rdma_chunked_send_tokens >= num_warps_per_forwarder fails. Thanks. Fixed in https://github.com/deepseek-ai/DeepEP/commit/bdd119f8b249953cab366f4d737ad39d4246fd7e.
bsakaguchi commented 2026-10-26 09:56:58 +00:00
Author
Owner
Copy link

@ZhenshengWu commented on GitHub (Jul 22, 2025):

ccording to the logs, NVSHMEM failed to establish the IBRC connection. You can try to manually specify the correct NIC by using the NVSHMEM_HCA_PE_MAPPING and NVSHMEM_HCA_LIST environment variables.

It's work for me！ Thanks a lot！

Thanks. Fixed in github.com/deepseek-ai/DeepEP@bdd119f8b2.

Get it! I will try it again

<!-- gh-comment-id:3101221418 --> @ZhenshengWu commented on GitHub (Jul 22, 2025): > ccording to the logs, NVSHMEM failed to establish the IBRC connection. You can try to manually specify the correct NIC by using the `NVSHMEM_HCA_PE_MAPPING` and `NVSHMEM_HCA_LIST` environment variables. It's work for me！ Thanks a lot！ > Thanks. Fixed in https://github.com/deepseek-ai/DeepEP/commit/bdd119f8b249953cab366f4d737ad39d4246fd7e. Get it! I will try it again
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#224
No description provided.