mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #442] Question regarding the atomic add on the per channel tail/head counters #289
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#289
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ricky-lij on GitHub (Oct 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/442
Hi,
In the DeepEP internode kernel design, we can see the tail/head counters on the peer side is updated via the RDMA atomic add operation:
e.g. Dispatch - SenderCoordinator
...
// Update tails
if (lane_id == dst_rdma_rank) {
last_issued_tail += num_tokens_to_issue;
num_tokens_to_send -= num_tokens_to_issue;
nvshmemi_ibgda_amo_nonfetch_add(rdma_channel_tail.buffer(rdma_rank), num_tokens_to_issue,
translate_dst_rdma_rank(dst_rdma_rank, nvl_rank), channel_id, dst_rdma_rank == rdma_rank);
}
...
Looks like this tail update producer is already designed as per rank per channel, and the target GPU memory is also designed as dedicated for per rank per channel in the SymBuffer design, should be the typical SPSC model. In this case, why using RDMA atomic add to update the tail but not using the RDMA write instead? Since the RDMA atomic could introduce extra latency compared with the RDMA write. Can you help share some thoughts on this?
Many Thanks!
Ricky
@ghghliu commented on GitHub (Oct 10, 2025):
It seems that ATOMIC could be used as a fence for the preceding WRITE operation , so that adaptive routing could be enabled. But I wonder why not use WRITE WITH IMM as the fence, which may have lower overhead for rnic compared to atomic operation?
@sphish commented on GitHub (Oct 10, 2025):
Same as the previous comment, the atomic operation here is used as a fence. We don’t use
write_with_immbecause NVSHMEM does not provide that functionality.@ricky-lij commented on GitHub (Oct 10, 2025):
@ghghliu @sphish Thanks for your comments, yes I checked the IB SPEC, only the send/write imm/atomic can fence the previous writes, but the normal write is not in the list: