deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #189] test_low_latency.py get stucked in single node with 8 devices when setting Buffer allow_nvlink_for_low_latency_mode as False #161

New issue
Closed
opened 2026-10-26 09:55:24 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:55:24 +00:00
Owner
Copy link

Originally created by @MARD1NO on GitHub (Jun 5, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/189

I just test low latency mode in single H20 node, and get some strange result.

when the num_processes is 8, with allow_nvlink_for_low_latency_mode = True(In the latest commit, True is the default value), it test success.

but when I set it to False, it will get stuck, like in deadlock, all the gpu utilization is 100%.

then I try num_processes as 4, 2, it all passes, here is my gpu topo

Image

Can you give some suggestions for debug ?...

Originally created by @MARD1NO on GitHub (Jun 5, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/189 I just test low latency mode in single H20 node, and get some strange result. when the num_processes is 8, with `allow_nvlink_for_low_latency_mode` = True(In the latest commit, True is the default value), it test success. but when I set it to False, it will get stuck, like in deadlock, all the gpu utilization is 100%. then I try num_processes as 4, 2, it all passes, here is my gpu topo ![Image](https://github.com/user-attachments/assets/2b236782-7b39-4ee3-bec6-fb079a41cce2) Can you give some suggestions for debug ?...
bsakaguchi commented 2026-10-26 09:55:26 +00:00
Author
Owner
Copy link

@MARD1NO commented on GitHub (Jun 5, 2025):

emmm My env is special, I need to

export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2"
<!-- gh-comment-id:2944771057 --> @MARD1NO commented on GitHub (Jun 5, 2025): emmm My env is special, I need to ``` export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2" ```
bsakaguchi commented 2026-10-26 09:55:26 +00:00
Author
Owner
Copy link

@FrankerWen commented on GitHub (Jun 6, 2025):

emmm My env is special, I need to

export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2"

is it sufficient to set the environment variables before running, without modifying any code? My environment is similar to yours, but setting the environment variables doesn't seem to have any effect. Also, how is the performance in your tests? Thank you.

<!-- gh-comment-id:2951226126 --> @FrankerWen commented on GitHub (Jun 6, 2025): > emmm My env is special, I need to > > ``` > export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 > export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2" > ``` is it sufficient to set the environment variables before running, without modifying any code? My environment is similar to yours, but setting the environment variables doesn't seem to have any effect. Also, how is the performance in your tests? Thank you.
bsakaguchi commented 2026-10-26 09:55:26 +00:00
Author
Owner
Copy link

@SzymonOzog commented on GitHub (Jul 7, 2025):

Did you manage to resolve the issue? I ran into the same problem

<!-- gh-comment-id:3045037501 --> @SzymonOzog commented on GitHub (Jul 7, 2025): Did you manage to resolve the issue? I ran into the same problem
bsakaguchi commented 2026-10-26 09:55:27 +00:00
Author
Owner
Copy link

@MARD1NO commented on GitHub (Jul 16, 2025):

emmm My env is special, I need to

export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2"

is it sufficient to set the environment variables before running, without modifying any code? My environment is similar to yours, but setting the environment variables doesn't seem to have any effect. Also, how is the performance in your tests? Thank you.

mlx5_x is my device name, you can use nvidia-smi topo -m to check your device

<!-- gh-comment-id:3077072092 --> @MARD1NO commented on GitHub (Jul 16, 2025): > > emmm My env is special, I need to > > ``` > > export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 > > export NVSHMEM_HCA_PE_MAPPING="mlx5_0:1:2,mlx5_1:1:2,mlx5_2:1:2,mlx5_4:1:2" > > ``` > > is it sufficient to set the environment variables before running, without modifying any code? My environment is similar to yours, but setting the environment variables doesn't seem to have any effect. Also, how is the performance in your tests? Thank you. mlx5_x is my device name, you can use `nvidia-smi topo -m` to check your device
bsakaguchi commented 2026-10-26 09:55:27 +00:00
Author
Owner
Copy link

@MARD1NO commented on GitHub (Jul 16, 2025):

Did you manage to resolve the issue? I ran into the same problem

Maybe it is NVSHMEM bug, not DeepEP's

<!-- gh-comment-id:3077073284 --> @MARD1NO commented on GitHub (Jul 16, 2025): > Did you manage to resolve the issue? I ran into the same problem Maybe it is NVSHMEM bug, not DeepEP's
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#161
No description provided.