deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #307] Test test_low_latency.py on L20*8 failed #217

New issue
Closed
opened 2026-10-26 09:56:49 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 09:56:49 +00:00
Owner
Copy link

Originally created by @tianchongchong on GitHub (Jul 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/307

env

L20 * 8

cmd and logs

root@node-8:/sgl-workspace/DeepEP# python tests/test_low_latency.py
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Allocating buffer size: 2115.111296 MB ...
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value
Unable to create ah.
Unable to create ah.
Unable to create ah.
Unable to create ah.

/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7
/sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7
ibgda_rc_init2rtr failed on RC #216.
ibgda_rc_init2rtr failed on RC #216.
ibgda_rc_init2rtr failed on RC #216.
ibgda_rc_init2rtr failed on RC #216.
WARN: Failed to initialize all selected devices. Perf may be limited.
WARN: Failed to initialize all selected devices. Perf may be limited.
WARN: Failed to initialize all selected devices. Perf may be limited.
WARN: Failed to initialize all selected devices. Perf may be limited.
/sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7
/sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7
/sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7
/sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7
connect EPS failed
connect EPS failed
connect EPS failed
connect EPS failed
/sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7
/sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7
/sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7
/sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7

nvshmem setup connections failed
nvshmem setup connections failed
nvshmem setup connections failed
nvshmem setup connections failed

Originally created by @tianchongchong on GitHub (Jul 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/307 ## env L20 * 8 ## cmd and logs root@node-8:/sgl-workspace/DeepEP# python tests/test_low_latency.py Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Allocating buffer size: 2115.111296 MB ... Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:1348: NULL value Unable to create ah. Unable to create ah. Unable to create ah. Unable to create ah. /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7 /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7 /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7 /sgl-workspace/nvshmem/src/modules/transport/ibgda/ibgda.cpp:3025: non-zero status: 7 ibgda_rc_init2rtr failed on RC #216. ibgda_rc_init2rtr failed on RC #216. ibgda_rc_init2rtr failed on RC #216. ibgda_rc_init2rtr failed on RC #216. WARN: Failed to initialize all selected devices. Perf may be limited. WARN: Failed to initialize all selected devices. Perf may be limited. WARN: Failed to initialize all selected devices. Perf may be limited. WARN: Failed to initialize all selected devices. Perf may be limited. /sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7 /sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7 /sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7 /sgl-workspace/nvshmem/src/host/transport/transport.cpp:400: non-zero status: 7 connect EPS failed connect EPS failed connect EPS failed connect EPS failed /sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7 /sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7 /sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7 /sgl-workspace/nvshmem/src/host/init/init.cu:1007: non-zero status: 7 nvshmem setup connections failed nvshmem setup connections failed nvshmem setup connections failed nvshmem setup connections failed
bsakaguchi commented 2026-10-26 09:56:50 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 22, 2025):

NVSHMEM failed to create ah(address handler). I haven't encountered this issue before. It is likely caused by a problem with your network configuration.

<!-- gh-comment-id:3100267701 --> @sphish commented on GitHub (Jul 22, 2025): NVSHMEM failed to create ah(address handler). I haven't encountered this issue before. It is likely caused by a problem with your network configuration.
bsakaguchi commented 2026-10-26 09:56:51 +00:00
Author
Owner
Copy link

@tianchongchong commented on GitHub (Jul 23, 2025):

thanks, @sphish this problem has been solved. but now we have encountered a new one.

root@node-8:/sgl-workspace/DeepEP# CUDA_LAUNCH_BLOCKING=1 python tests/test_low_latency.py 
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Allocating buffer size: 2115.111296 MB ...
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Detected non-PCIe device. NVLink assumption holds.
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
terminate called after throwing an instance of 'terminate called after throwing an instance of 'EPExceptionEPException'
'
  what():    what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'

W0723 07:57:33.583000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13945 via signal SIGTERM
W0723 07:57:33.583000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13946 via signal SIGTERM
W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13947 via signal SIGTERM
W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13948 via signal SIGTERM
W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13949 via signal SIGTERM
W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13950 via signal SIGTERM
W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13951 via signal SIGTERM
Traceback (most recent call last):
  File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 187, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 7 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 169, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1)
  File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 43, in test_main
    buffer.low_latency_dispatch(x, topk_idx, num_tokens, num_experts,
  File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 533, in low_latency_dispatch
    self.runtime.low_latency_dispatch(x, topk_idx,
RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure'
<!-- gh-comment-id:3106496103 --> @tianchongchong commented on GitHub (Jul 23, 2025): thanks, @sphish this problem has been solved. but now we have encountered a new one. ``` root@node-8:/sgl-workspace/DeepEP# CUDA_LAUNCH_BLOCKING=1 python tests/test_low_latency.py Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Allocating buffer size: 2115.111296 MB ... Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Detected non-PCIe device. NVLink assumption holds. Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' terminate called after throwing an instance of 'terminate called after throwing an instance of 'EPExceptionEPException' ' what(): what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure'Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:85 'unspecified launch failure' W0723 07:57:33.583000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13945 via signal SIGTERM W0723 07:57:33.583000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13946 via signal SIGTERM W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13947 via signal SIGTERM W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13948 via signal SIGTERM W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13949 via signal SIGTERM W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13950 via signal SIGTERM W0723 07:57:33.584000 13880 torch/multiprocessing/spawn.py:169] Terminating process 13951 via signal SIGTERM Traceback (most recent call last): File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 187, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 7 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 169, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1) File "/sgl-workspace/DeepEP/tests/test_low_latency.py", line 43, in test_main buffer.low_latency_dispatch(x, topk_idx, num_tokens, num_experts, File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 533, in low_latency_dispatch self.runtime.low_latency_dispatch(x, topk_idx, RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/internode_ll.cu:381 'unspecified launch failure' ```
bsakaguchi commented 2026-10-26 09:56:52 +00:00
Author
Owner
Copy link

@klauseshen commented on GitHub (Jul 30, 2025):

Hi @tianchongchong ,
I've encountered the same error when running the ll test
"transport.cpp:420: non-zero status: 7 transport create connect failed connect EPS failed";

Would it be possible for you to share some thoughts on how you address it

<!-- gh-comment-id:3135027073 --> @klauseshen commented on GitHub (Jul 30, 2025): Hi @tianchongchong , I've encountered the same error when running the ll test "transport.cpp:420: non-zero status: 7 transport create connect failed connect EPS failed"; Would it be possible for you to share some thoughts on how you address it
bsakaguchi commented 2026-10-26 09:56:52 +00:00
Author
Owner
Copy link

@tianchongchong commented on GitHub (Aug 2, 2025):

Hi @tianchongchong ,
I've encountered the same error when running the ll test
"transport.cpp:420: non-zero status: 7 transport create connect failed connect EPS failed";

Would it be possible for you to share some thoughts on how you address it

the num of IB devices on my server exceeded the macro MAX_NUM_HCAS:16(defined in ibgda.cpp), you can check the num of IB devices by ibv_devinfo

<!-- gh-comment-id:3146191714 --> @tianchongchong commented on GitHub (Aug 2, 2025): > Hi @tianchongchong , > I've encountered the same error when running the ll test > "transport.cpp:420: non-zero status: 7 transport create connect failed connect EPS failed"; > > Would it be possible for you to share some thoughts on how you address it the num of IB devices on my server exceeded the macro MAX_NUM_HCAS:16(defined in ibgda.cpp), you can check the num of IB devices by `ibv_devinfo`
bsakaguchi commented 2026-10-26 09:56:52 +00:00
Author
Owner
Copy link

@MengYu10151 commented on GitHub (Aug 15, 2025):

@tianchongchong hi! Could I ask how do you compile the latest deepep on L20? I try to modify cuda_arch config in setup.py ./csrc/CMakeLists.txt and ./csrc/kernels/configs.cuh, but still cant work. Thx！: )

<!-- gh-comment-id:3191082896 --> @MengYu10151 commented on GitHub (Aug 15, 2025): @tianchongchong hi! Could I ask how do you compile the latest deepep on L20? I try to modify cuda_arch config in setup.py ./csrc/CMakeLists.txt and ./csrc/kernels/configs.cuh, but still cant work. Thx！: )
bsakaguchi commented 2026-10-26 09:56:52 +00:00
Author
Owner
Copy link

@tianchongchong commented on GitHub (Sep 25, 2025):

@MengYu10151 I'm not using the latest version. you can ignore internode.cu and intranode.cu and only compile internode_ll.cu

<!-- gh-comment-id:3333217887 --> @tianchongchong commented on GitHub (Sep 25, 2025): @MengYu10151 I'm not using the latest version. you can ignore internode.cu and intranode.cu and only compile internode_ll.cu
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#217
No description provided.