[PR #632] [CLOSED] Fix: Add fence.proxy.async before TMA load in combine recv phase #674

New issue

Closed

opened 2026-10-27 21:05:09 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:09 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/632
Author: @elvircrn
Created: 5/11/2026
Status: ❌ Closed

Base: main ← Head: correctness-fix

📝 Commits (1)

d7d1cdc Add fence.proxy.async before TMA load in combine recv phase. Fixes correctness for batch size > 256 on NVL72

📊 Changes

1 file changed (+1 additions, -0 deletions)

View changed files

📝 csrc/kernels/legacy/internode_ll.cu (+1 -0)

📄 Description

Background

Correctness issues with repeated runs of combine at bath sizes higher than 256 were found and issue was reported.

An investigation was done revealing doing __ldg on the buffer ptr instead of TMA make combine produce consistent results at bach size > 256 on NVL72. A plausible explanation for this is that switching to generic proxy resolves the issue.

Investigating further revealed that the local rank polls the generic proxy but then uses TMA to load data. Without fence.proxy.async, TMA can see stale data.

From https://docs.nvidia.com/cuda/parallel-thread-execution/#async-proxy:

Accessing the same memory location across multiple proxies needs a cross-proxy fence. 
For the async proxy, fence.proxy.async should be used to synchronize memory between 
generic proxy and the async proxy.

With the fence in place, the test scripts made as part of this report correct behavior.

Fence Performance Impact

low-latency benchmark (median of 3 rounds, 1000 iters each)
  T=1024  H=7168  E=256  topk=8  ranks=4
  worst-case rank (max across ranks per round)

  strategies:
    random      = topk over random scores, different experts per token (cross-rank RDMA)
    random-same = topk over random scores, same K experts for all tokens (cross-rank RDMA)
    local-rand  = random offsets mapped to local rank experts (no RDMA, varied experts)
    local-same  = every token picks the same K local experts (no RDMA, hotspot)
    remote-rand = random experts, each from a different remote rank (all cross-rank RDMA)

strategy     fmt       cleg_on  cleg_off   delta     pct
                        (us)      (us)     (us)
------------ ------   -------  --------  ------  ------
random       bf16      178.1     170.1    +8.0   +4.7%
random       fp8       178.2     170.3    +7.9   +4.6%
random-same  bf16      646.8     639.0    +7.7   +1.2%
random-same  fp8       647.8     640.2    +7.6   +1.2%
local-rand   bf16      177.3     172.5    +4.8   +2.8%
local-rand   fp8       181.4     177.2    +4.2   +2.4%
local-same   bf16      656.2     653.2    +3.0   +0.5%
local-same   fp8       657.8     655.1    +2.7   +0.4%
remote-rand  bf16      223.7     215.8    +8.0   +3.7%
remote-rand  fp8       226.0     217.8    +8.2   +3.8%

Benchmark script.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/632 **Author:** [@elvircrn](https://github.com/elvircrn) **Created:** 5/11/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `correctness-fix` --- ### 📝 Commits (1) - [`d7d1cdc`](https://github.com/deepseek-ai/DeepEP/commit/d7d1cdce4abef18f3b9b09150d47cf8df8de1531) Add fence.proxy.async before TMA load in combine recv phase. Fixes correctness for batch size > 256 on NVL72 ### 📊 Changes **1 file changed** (+1 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/internode_ll.cu` (+1 -0) </details> ### 📄 Description # Background Correctness issues with repeated runs of combine at bath sizes higher than 256 were found and [issue](https://github.com/deepseek-ai/DeepEP/issues/621) was reported. An investigation was done revealing doing __ldg on the `buffer` ptr instead of TMA make combine produce consistent results at bach size > 256 on NVL72. A plausible explanation for this is that switching to generic proxy resolves the issue. Investigating further revealed that the local rank polls the generic proxy but then uses TMA to load data. Without `fence.proxy.async`, TMA can see stale data. From [https://docs.nvidia.com/cuda/parallel-thread-execution/#async-proxy](url): ``` Accessing the same memory location across multiple proxies needs a cross-proxy fence. For the async proxy, fence.proxy.async should be used to synchronize memory between generic proxy and the async proxy. ``` With the fence in place, the test scripts made as part of [this](https://github.com/deepseek-ai/DeepEP/issues/621) report correct behavior. # Fence Performance Impact ``` low-latency benchmark (median of 3 rounds, 1000 iters each) T=1024 H=7168 E=256 topk=8 ranks=4 worst-case rank (max across ranks per round) strategies: random = topk over random scores, different experts per token (cross-rank RDMA) random-same = topk over random scores, same K experts for all tokens (cross-rank RDMA) local-rand = random offsets mapped to local rank experts (no RDMA, varied experts) local-same = every token picks the same K local experts (no RDMA, hotspot) remote-rand = random experts, each from a different remote rank (all cross-rank RDMA) strategy fmt cleg_on cleg_off delta pct (us) (us) (us) ------------ ------ ------- -------- ------ ------ random bf16 178.1 170.1 +8.0 +4.7% random fp8 178.2 170.3 +7.9 +4.6% random-same bf16 646.8 639.0 +7.7 +1.2% random-same fp8 647.8 640.2 +7.6 +1.2% local-rand bf16 177.3 172.5 +4.8 +2.8% local-rand fp8 181.4 177.2 +4.2 +2.4% local-same bf16 656.2 653.2 +3.0 +0.5% local-same fp8 657.8 655.1 +2.7 +0.4% remote-rand bf16 223.7 215.8 +8.0 +3.7% remote-rand fp8 226.0 217.8 +8.2 +3.8% ``` Benchmark [script](https://gist.github.com/elvircrn/2e564cc0a53c3a72ab887480686a3456). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:09 +00:00