deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #398] sglang PD decoding nodes hang in deepep #271

New issue
Closed
opened 2026-10-26 09:57:45 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:57:45 +00:00
Owner
Copy link

Originally created by @llc-kc on GitHub (Sep 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/398

2 H800 as prefill and 4 H800 as decoding, same with https://github.com/sgl-project/sglang/issues/10124

besides prefill, decoding nodes also occasionally hang
cuda-gdb info cuda kernels shows:

(cuda-gdb) info cuda kernels
Kernel Parent Dev Grid Status SMs Mask GridDim BlockDim Invocation

  •  0      -   0 220891 Active 0x0ff0ffffffffffffffffffffffffffffff (2,1,1) (1024,1,1) deep_ep::internode_ll::combine<false, 7168, 9>()
Originally created by @llc-kc on GitHub (Sep 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/398 2 H800 as prefill and 4 H800 as decoding, same with https://github.com/sgl-project/sglang/issues/10124 besides prefill, decoding nodes also occasionally hang cuda-gdb info cuda kernels shows: (cuda-gdb) info cuda kernels Kernel Parent Dev Grid Status SMs Mask GridDim BlockDim Invocation * 0 - 0 220891 Active 0x0ff0ffffffffffffffffffffffffffffff (2,1,1) (1024,1,1) deep_ep::internode_ll::combine<false, 7168, 9>()
bsakaguchi commented 2026-10-26 09:57:47 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Sep 10, 2025):

You can use this tool to troubleshoot problems. https://github.com/antgroup/DeepXTrace

<!-- gh-comment-id:3272965948 --> @alpha-baby commented on GitHub (Sep 10, 2025): You can use this tool to troubleshoot problems. https://github.com/antgroup/DeepXTrace
bsakaguchi commented 2026-10-26 09:57:47 +00:00
Author
Owner
Copy link

@llc-kc commented on GitHub (Sep 10, 2025):

@alpha-baby thank you, I will try it

<!-- gh-comment-id:3273796410 --> @llc-kc commented on GitHub (Sep 10, 2025): @alpha-baby thank you, I will try it
bsakaguchi commented 2026-10-26 09:57:48 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Sep 10, 2025):

It’s rare for the low-latency kernels to get stuck. Could you check whether all ranks have entered the combine kernel?

<!-- gh-comment-id:3273841335 --> @sphish commented on GitHub (Sep 10, 2025): It’s rare for the low-latency kernels to get stuck. Could you check whether all ranks have entered the combine kernel?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#271
No description provided.