deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #364] [BUG] test_internode hangs and times out on repeated runs #252

New issue
Open
opened 2026-10-26 09:57:24 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-26 09:57:24 +00:00
Owner
Copy link

Originally created by @polarstormx on GitHub (Aug 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/364

When test_internode is run in a tight loop for stress testing, it randomly hangs and eventually times out.
It seems to be related to TMA function.

Screenshot of the hang:
Image

Environment

  • Commit Hash: The latest commit on main (e3908bf5bd)
  • Hardware: Reproducible on both 2/4-node H20 and 2/4-node H200 setups. ConnectX-7 with dual port 200G. RoCEv2 with PFC.
  • Image: ngc 24.02
  • Driver / CUDA Version: <570.148.08 / 12.8>

Steps to Reproduce

I modified the test script for stress testing.

github.com/deepseek-ai/DeepEP@e3908bf5bd/tests/test_internode.py (L174-L176)
Adding a return to skip performance test.

    if local_rank == 0: 
        print('', flush=True) 
    return

github.com/deepseek-ai/DeepEP@e3908bf5bd/tests/test_internode.py (L242-L245)
Running continuously for multiple rounds.

   for i in range(10000):
        test_main(args, num_sms, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
        if local_rank == 0:
            print('', flush=True)

Moving torch.manual_seed(rank) inside the loop to ensure identical inputs for each iteration does not fix the issue.
I changed 'assert' to 'print' to prevent interrupting the testing program, but it still hang then timeout. calc_diff and check_data will get inf or nan.

Last Known Good Commit: 7705f53 (The stress test passes reliably on this commit). This is the last commit before the internode TMA feature was merged.
First Known Bad Commit: a2fa3b7 (The stress test fails on this commit). This is the first commit that introduces the internode TMA functionality.

Originally created by @polarstormx on GitHub (Aug 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/364 When test_internode is run in a tight loop for stress testing, it randomly hangs and eventually times out. It seems to be related to TMA function. Screenshot of the hang: <img width="2344" height="558" alt="Image" src="https://github.com/user-attachments/assets/e02642a1-12a8-4a05-b6e4-d9af05380471" /> ### Environment - Commit Hash: The latest commit on main (e3908bf5bd0cc6265bcb225d15cd8c996d4759ef) - Hardware: Reproducible on both 2/4-node H20 and 2/4-node H200 setups. ConnectX-7 with dual port 200G. RoCEv2 with PFC. - Image: ngc 24.02 - Driver / CUDA Version: <570.148.08 / 12.8> ### Steps to Reproduce I modified the test script for stress testing. https://github.com/deepseek-ai/DeepEP/blob/e3908bf5bd0cc6265bcb225d15cd8c996d4759ef/tests/test_internode.py#L174-L176 Adding a return to skip performance test. ```python if local_rank == 0: print('', flush=True) return ``` https://github.com/deepseek-ai/DeepEP/blob/e3908bf5bd0cc6265bcb225d15cd8c996d4759ef/tests/test_internode.py#L242-L245 Running continuously for multiple rounds. ```python for i in range(10000): test_main(args, num_sms, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group) if local_rank == 0: print('', flush=True) ``` Moving torch.manual_seed(rank) inside the loop to ensure identical inputs for each iteration does not fix the issue. I changed 'assert' to 'print' to prevent interrupting the testing program, but it still hang then timeout. calc_diff and check_data will get inf or nan. Last Known Good Commit: 7705f53 (The stress test passes reliably on this commit). This is the last commit before the internode TMA feature was merged. First Known Bad Commit: a2fa3b7 (The stress test fails on this commit). This is the first commit that introduces the internode TMA functionality.
bsakaguchi commented 2026-10-26 09:57:26 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Aug 18, 2025):

I have tried to reproduce this issue following your method, but I was not successful. Can you add my wechat(Sphizzz)?

<!-- gh-comment-id:3195666166 --> @sphish commented on GitHub (Aug 18, 2025): I have tried to reproduce this issue following your method, but I was not successful. Can you add my wechat(Sphizzz)?
bsakaguchi commented 2026-10-26 09:57:26 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Aug 18, 2025):

another error

[config] num_tokens=4096, hidden=7168, num_topk_groups=4, num_topk=8
[layout] Kernel performance: 0.045 ms

[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with FP8, without top-k (async=False, previous=False) ... passed
[testing] Running with FP8, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=True, previous=False) ... passed
[testing] Running with BF16, with top-k (async=True, previous=False) ...[ERROR] Expert 0 token count mismatch: expected 4105, got 4113
[ERROR] Expert 3 token count mismatch: expected 4144, got 4143
[ERROR] Expert 1 token count mismatch: expected 3923, got 3924
[ERROR] Expert 3 token count mismatch: expected 4134, got 4133
[testing] expert 3 mask: tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]], device='cuda:4')
[testing] expert 0 mask: tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]], device='cuda:0')
[testing] expert 3 mask: tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]], device='cuda:7')
[testing] expert 1 mask: tensor([[False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False]], device='cuda:5')
[ERROR] calc_diff exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
<!-- gh-comment-id:3195788995 --> @polarstormx commented on GitHub (Aug 18, 2025): another error ``` [config] num_tokens=4096, hidden=7168, num_topk_groups=4, num_topk=8 [layout] Kernel performance: 0.045 ms [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with FP8, without top-k (async=False, previous=False) ... passed [testing] Running with FP8, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=True, previous=False) ... passed [testing] Running with BF16, with top-k (async=True, previous=False) ...[ERROR] Expert 0 token count mismatch: expected 4105, got 4113 [ERROR] Expert 3 token count mismatch: expected 4144, got 4143 [ERROR] Expert 1 token count mismatch: expected 3923, got 3924 [ERROR] Expert 3 token count mismatch: expected 4134, got 4133 [testing] expert 3 mask: tensor([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], ..., [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False]], device='cuda:4') [testing] expert 0 mask: tensor([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], ..., [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False]], device='cuda:0') [testing] expert 3 mask: tensor([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], ..., [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False]], device='cuda:7') [testing] expert 1 mask: tensor([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], ..., [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False]], device='cuda:5') [ERROR] calc_diff exception: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ```
bsakaguchi commented 2026-10-26 09:57:26 +00:00
Author
Owner
Copy link

@polarstormx commented on GitHub (Aug 18, 2025):

some other errors. (WIth my print modify)

Image Image Image
<!-- gh-comment-id:3195848375 --> @polarstormx commented on GitHub (Aug 18, 2025): some other errors. (WIth my print modify) <img width="2170" height="242" alt="Image" src="https://github.com/user-attachments/assets/816b7202-bfaa-4808-86c7-dc3f67265642" /> <img width="1808" height="494" alt="Image" src="https://github.com/user-attachments/assets/e82bc814-be70-40db-a3a7-4fb2e1e6bcd7" /> <img width="2014" height="326" alt="Image" src="https://github.com/user-attachments/assets/ee319099-1adf-48d8-be00-5b96a7d47941" />
bsakaguchi commented 2026-10-26 09:57:26 +00:00
Author
Owner
Copy link

@huakainomore commented on GitHub (Dec 30, 2025):

hello, this problem is fixed ? @polarstormx

<!-- gh-comment-id:3698159657 --> @huakainomore commented on GitHub (Dec 30, 2025): hello, this problem is fixed ? @polarstormx
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#252
No description provided.