mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #188] Confused about FINISHED_SUM_TAG #157
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#157
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jiangjiang-coder66 on GitHub (Jun 5, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/188
@LyricZhao commented on GitHub (Jun 6, 2025):
You should notify the remote ranks that the current rank has already cleaned the buffer flags for next use, so that the next dispatch/combine is safe to use the flags.
@jiangjiang-coder66 commented on GitHub (Jun 6, 2025):
@LyricZhao Thank you for your reply. In my understanding, atomic_finish_counter_per_expert equals to 2 * FINISHED_SUM_TAG is used to determine whether all communication instructions for tokens have been issued. However, I'm still confused about
atomic_add_release_global(atomic_finish_counter_per_expert + i, FINISHED_SUM_TAG);and the comment// Notify before executing 'int_p'. Is there a part in the code that polls the value of atomic_finish_counter_per_expert until it equals FINISHED_SUM_TAG before proceeding with subsequent actions? Thank you again for your reply.@LyricZhao commented on GitHub (Jun 6, 2025):
Again, this is for notifying other ranks for cleaning the buffer.
Can you contact me by WeChat (ID: LyricZ_THU)? Which may be more convenient.
@jiangjiang-coder66 commented on GitHub (Jun 10, 2025):
Thanks to @LyricZhao for the insightful explanation. I have understood the need for atomicAdd FINISHED_SUM_TAG. By using atomicAdd FINISHED_SUM_TAG, we block the while (ld_acquire_global(atomic_finish_counter_per_expert + responsible_expert_idx) != FINISHED_SUM_TAG * 2) loop, ensuring that the clean next buffer operation completes before data transmission. This prevents the following scenario(two ranks):
t0: rank0 sends data
t1: rank1 receives data; rank0 cleans buffer
t2: rank1 sends data; rank0 cleans buffer
t3: rank0 cleans buffer
Due to rank0's cleaning operation taking too long (either because it started late or because rank1 is too fast), it might still be cleaning when rank1 has already sent new data. In this case, the data sent by rank1 would be cleared by rank0's cleaning operation.
By using atomicAdd FINISHED_SUM_TAG, we ensure that rank0's cleaning operation is completed before rank1 sends new data, thus preventing the data from being inadvertently cleared.