[GH-ISSUE #423] barrier_block deadlock occurs in notify-dispatch #283

New issue

Closed

opened 2026-10-26 09:57:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:57:58 +00:00

Owner

Originally created by @gzshan on GitHub (Sep 23, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/423

We discovered a strange phenomenon: when using tp=4 in the attention phase and ep=32, executing the barrier_block in notify_dispatch kernel results in a deadlock.

Specifically, the first four ranks (0, 1, 2, and 3) on the machine timed out during the first barrier_block operation, while the second four ranks (4, 5, 6, and 7) on the same machine timed out during the second barrier_block operation. These two ranks waited for each other, resulting in a deadlock.

Thus, it appears that the bug occurred during the first barrier_block operation.

To analyze this issue, we printed the data in the fifo used for the barrier_block operation and found the following data: after atomicAdd_system and atomicSub_system were executed for ranks 4, 5, 6, and 7, the entire signal matrix had returned to zero, thus passing synchronization. However, from the perspective of ranks 0, 1, 2, and 3, it seemed that no corresponding changes were seen, resulting in a timeout after 100 seconds.

We compared the code before the task-fifo modification with the latest code in the main branch and found a similar phenomenon, with a high probability of recurrence. We are using H20 machines. So, I would like to ask if anyone has encountered similar problems or what the possible reasons are. Thank you very much.

Originally created by @gzshan on GitHub (Sep 23, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/423 We discovered a strange phenomenon: when using tp=4 in the attention phase and ep=32, executing the barrier_block in notify_dispatch kernel results in a deadlock. Specifically, the first four ranks (0, 1, 2, and 3) on the machine timed out during the first barrier_block operation, while the second four ranks (4, 5, 6, and 7) on the same machine timed out during the second barrier_block operation. These two ranks waited for each other, resulting in a deadlock. Thus, it appears that the bug occurred during the first barrier_block operation. To analyze this issue, we printed the data in the fifo used for the barrier_block operation and found the following data: after atomicAdd_system and atomicSub_system were executed for ranks 4, 5, 6, and 7, the entire signal matrix had returned to zero, thus passing synchronization. However, from the perspective of ranks 0, 1, 2, and 3, it seemed that no corresponding changes were seen, resulting in a timeout after 100 seconds. <img width="400" height="300" alt="Image" src="https://github.com/user-attachments/assets/ae91ba5c-5e87-4a39-84ff-3ebecdb48a2b" /> We compared the code before the task-fifo modification with the latest code in the main branch and found a similar phenomenon, with a high probability of recurrence. We are using H20 machines. So, I would like to ask if anyone has encountered similar problems or what the possible reasons are. Thank you very much.

bsakaguchi closed this issue

2026-10-26 09:57:58 +00:00

bsakaguchi commented

2026-10-26 09:57:59 +00:00

Author

Owner

@gzshan commented on GitHub (Oct 20, 2025):

The misunderstanding is that this is not a deadlock problem. The problem is caused by the difference in startup time between different ranks exceeding the barrier_block timeout.

@gzshan commented on GitHub (Oct 20, 2025): The misunderstanding is that this is not a deadlock problem. The problem is caused by the difference in startup time between different ranks exceeding the barrier_block timeout.

bsakaguchi referenced this issue

2026-10-27 21:03:18 +00:00

[PR #283] [MERGED] Support cluster size=2 #501