deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #314] Internode Combine Timeout #221

New issue
Open
opened 2026-10-26 09:56:53 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-26 09:56:53 +00:00
Owner
Copy link

Originally created by @ZhiyiHu1999 on GitHub (Jul 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/314

Hello! I found under this configuration, the internode combine will timeout: (The variable names I use here are the same as the ones used in test_internode.py)

num_sms = 48
nvl_chunk_size = 32
nvl_buffer_size = 80
rdma_chunk_size = 20
rdma_buffer_size = 128

For my experiment setup:
Two nodes, each with 8 * H20 GPU and 4 * NIC, ROCE

Could you try the config in your environment to confirm or deny this. Thanks a lot!

Originally created by @ZhiyiHu1999 on GitHub (Jul 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/314 Hello! I found under this configuration, the **internode combine** will timeout: (The variable names I use here are the same as the ones used in `test_internode.py`) **num_sms = 48 nvl_chunk_size = 32 nvl_buffer_size = 80 rdma_chunk_size = 20 rdma_buffer_size = 128** For my experiment setup: **Two nodes, each with 8 * H20 GPU and 4 * NIC, ROCE** Could you try the config in your environment to confirm or deny this. Thanks a lot!
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 22, 2025):

I am able to reproduce this issue. It seems to occur when the nvl_buffer_size is set to a small value. I will look into it and investigate further.

<!-- gh-comment-id:3100693648 --> @sphish commented on GitHub (Jul 22, 2025): I am able to reproduce this issue. It seems to occur when the `nvl_buffer_size` is set to a small value. I will look into it and investigate further.
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@kzlxd commented on GitHub (Jul 25, 2025):

how about this problem？@sphish ，this problem Stable reproduction during continuous stress testing

<!-- gh-comment-id:3116548813 --> @kzlxd commented on GitHub (Jul 25, 2025): how about this problem？@sphish ，this problem Stable reproduction during continuous stress testing
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@Skylion007 commented on GitHub (Jul 29, 2025):

Also can replicate unfortunately.

<!-- gh-comment-id:3133480905 --> @Skylion007 commented on GitHub (Jul 29, 2025): Also can replicate unfortunately.
bsakaguchi commented 2026-10-26 09:56:54 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 30, 2025):

@Skylion007 @kzlxd Did you also use configs similar to those mentioned in the issue?

<!-- gh-comment-id:3134608835 --> @sphish commented on GitHub (Jul 30, 2025): @Skylion007 @kzlxd Did you also use configs similar to those mentioned in the issue?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#221
No description provided.