deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #282] Redundant atomics in internode_ll #206

New issue

Closed

opened 2026-10-26 09:56:37 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-26 09:56:37 +00:00

Owner

Originally created by @itayalroy on GitHub (Jul 6, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/282

Hi guys, just wondering about this part in low latency dispatch kernel:

    // Issue count sends
    if (responsible_expert_idx < num_experts and sub_warp_id == 0 and lane_id == 0) {
        const auto dst_rank = responsible_expert_idx / num_local_experts;
        const auto dst_expert_local_idx = responsible_expert_idx % num_local_experts;
        const auto num_tokens_sent = shared_num_tokens_sent_per_expert[responsible_expert_idx - sm_id * num_warp_groups];

        // Wait local sends issued and send expert counts
        while (ld_acquire_global(atomic_finish_counter_per_expert + responsible_expert_idx) != FINISHED_SUM_TAG * 2);
        auto dst_ptr = reinterpret_cast<uint64_t>(rdma_recv_count + dst_expert_local_idx * num_ranks + rank);
        auto dst_p2p_ptr = nvshmemi_get_p2p_ptr(dst_ptr, rank, dst_rank);
        if (dst_p2p_ptr == 0) {
            nvshmemi_ibgda_amo_nonfetch_add(reinterpret_cast<int*>(dst_ptr), -num_tokens_sent - 1, dst_rank, dst_expert_local_idx);
        } else {
            st_release_sys_global(reinterpret_cast<int*>(dst_p2p_ptr), -num_tokens_sent - 1);
        }

Why do we need nvshmemi_ibgda_amo_nonfetch_add when writing to the remote recv counter? it looks like each counter is written to only once by a single thread- can't we use a regular DMA write instead? Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't?

Originally created by @itayalroy on GitHub (Jul 6, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/282 Hi guys, just wondering about this part in low latency dispatch kernel: ``` // Issue count sends if (responsible_expert_idx < num_experts and sub_warp_id == 0 and lane_id == 0) { const auto dst_rank = responsible_expert_idx / num_local_experts; const auto dst_expert_local_idx = responsible_expert_idx % num_local_experts; const auto num_tokens_sent = shared_num_tokens_sent_per_expert[responsible_expert_idx - sm_id * num_warp_groups]; // Wait local sends issued and send expert counts while (ld_acquire_global(atomic_finish_counter_per_expert + responsible_expert_idx) != FINISHED_SUM_TAG * 2); auto dst_ptr = reinterpret_cast<uint64_t>(rdma_recv_count + dst_expert_local_idx * num_ranks + rank); auto dst_p2p_ptr = nvshmemi_get_p2p_ptr(dst_ptr, rank, dst_rank); if (dst_p2p_ptr == 0) { nvshmemi_ibgda_amo_nonfetch_add(reinterpret_cast<int*>(dst_ptr), -num_tokens_sent - 1, dst_rank, dst_expert_local_idx); } else { st_release_sys_global(reinterpret_cast<int*>(dst_p2p_ptr), -num_tokens_sent - 1); } ```` Why do we need `nvshmemi_ibgda_amo_nonfetch_add` when writing to the remote recv counter? it looks like each counter is written to only once by a single thread- can't we use a regular DMA write instead? Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't?

bsakaguchi closed this issue

2026-10-26 09:56:37 +00:00

bsakaguchi commented

2026-10-26 09:56:38 +00:00

Author

Owner

@sphish commented on GitHub (Jul 7, 2025):

Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't?

Yes, the RDMA atomic operation guarantees that this operation will only become visible after all previous write operations have completed.

@sphish commented on GitHub (Jul 7, 2025): > Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't? Yes, the RDMA atomic operation guarantees that this operation will only become visible after all previous write operations have completed.

bsakaguchi commented

2026-10-26 09:56:38 +00:00

Author

Owner

@zhangboyue commented on GitHub (Jul 8, 2025):

Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't?

Yes, the RDMA atomic operation guarantees that this operation will only become visible after all previous write operations have completed.

Thanks for the reply, I'm caught by memory order questions of this piece of code. And it makes perfect sense if AMO 'fences' all previous write operations. Is it defacto standard or defined by official RDMA memory model ? I've search a while, but got little clues.

@zhangboyue commented on GitHub (Jul 8, 2025): > > Does this atomic write guarantee some remote execution order that a regular RDMA write doesn't? > > Yes, the RDMA atomic operation guarantees that this operation will only become visible after all previous write operations have completed. Thanks for the reply, I'm caught by memory order questions of this piece of code. And it makes perfect sense if AMO 'fences' all previous write operations. Is it defacto standard or defined by official RDMA memory model ? I've search a while, but got little clues.

bsakaguchi commented

2026-10-26 09:56:39 +00:00

Author

Owner

@sphish commented on GitHub (Jul 8, 2025):

@zhangboyue This rule is from InfiniBand Architecture Specification.

o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:

...

Update of a memory element by a subsequent ATOMIC operation.

@sphish commented on GitHub (Jul 8, 2025): @zhangboyue This rule is from _InfiniBand Architecture Specification_. > o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: > - ... > - Update of a memory element by a subsequent ATOMIC operation.

bsakaguchi commented

2026-10-26 09:56:39 +00:00

Author

Owner

@zhangboyue commented on GitHub (Jul 8, 2025):

@zhangboyue This rule is from InfiniBand Architecture Specification.

o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:

...

Update of a memory element by a subsequent ATOMIC operation.

Peta's of thanks.

@zhangboyue commented on GitHub (Jul 8, 2025): > [@zhangboyue](https://github.com/zhangboyue) This rule is from _InfiniBand Architecture Specification_. > > > o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: > > > > * ... > > * Update of a memory element by a subsequent ATOMIC operation. Peta's of thanks.

bsakaguchi commented

2026-10-26 09:56:39 +00:00

Author

Owner

@ghghliu commented on GitHub (Sep 26, 2025):

@zhangboyue This rule is from InfiniBand Architecture Specification.

o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:

...

Update of a memory element by a subsequent ATOMIC operation.

From the relevant parts of IB spec you mentioned, WRITE WITH IMMEDIATE can also serve as a fence for the preceding WRITE. Could you explain the considerations for choosing ATOMIC operations instead of WRITE WITH IMMEDIATE?

o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:
• Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data.

@ghghliu commented on GitHub (Sep 26, 2025): > [@zhangboyue](https://github.com/zhangboyue) This rule is from _InfiniBand Architecture Specification_. > > > o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: > > > > * ... > > * Update of a memory element by a subsequent ATOMIC operation. From the relevant parts of IB spec you mentioned, WRITE WITH IMMEDIATE can also serve as a fence for the preceding WRITE. Could you explain the considerations for choosing ATOMIC operations instead of WRITE WITH IMMEDIATE? o9-20: An application shall not depend on the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: • Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data.

bsakaguchi commented

2026-10-26 09:56:39 +00:00

Author

Owner

@sphish commented on GitHub (Oct 10, 2025):

@ghghliu Mainly because NVSHMEM does not have write_with_imm.

@sphish commented on GitHub (Oct 10, 2025): @ghghliu Mainly because NVSHMEM does not have `write_with_imm`.

bsakaguchi referenced this issue

2026-10-26 10:01:05 +00:00

[PR #206] [MERGED] Support UE8M0 data format. #465