deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #715] [MERGED] fix: add system-scope release before the GIN barrier when scaleup spans NVLink and RDMA #728

New issue

Closed

opened 2026-10-27 21:05:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:40 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/715
Author: @EvadLee
Created: 8/3/2026
Status: ✅ Merged
Merged: 8/4/2026
Merged by: @sphish

Base: main ← Head: pr/gin-barrier-fence

📝 Commits (2)

7dcf493 fix: Add fence.acq_rel.sys before the GIN barrier for mixed-fabric scaleup
9d170b2 fix: fence mixed-fabric writes in GIN barrier

📊 Changes

1 file changed (+2 additions, -0 deletions)

View changed files

📝 deep_ep/include/deep_ep/common/comm.cuh (+2 -0)

📄 Description

Problem

While integrating ElasticBuffer into SGLang, multi-node direct mode hangs non-deterministically at the GIN barrier:

DeepEP Gin barrier timeout, tag: 4, scaleout: 1, scaleup: 0/8, signal: 14, target: 15

Envrioment: 2 nodes x 8 H200, EP16

Root cause

When the scaleup team spans nodes, kIsScaleupNVLink == false and the data plane is mixed: same-node peers are written by raw ptx::tma_store_1d, cross-node peers by gin.put. The GIN barrier never releases the NVLink-written half to system scope — flush() does not publish (gin.h:310-311) and signal() defaults to cuda::thread_scope_device (gin.h:304-308). Device scope covers RDMA, since the NIC reads local memory, but not stores landing in another GPU.

Fix

comm::gpu_barrier already drains TMA stores before the barrier. Add a system-scope release there, on the path that publishes the raw NVLink writes:

if constexpr (kFlushStores) {
    ptx::tma_store_commit();
    ptx::tma_store_wait();
    __syncwarp();
    if constexpr (not kIsScaleupNVLink)
        ptx::fence_acq_rel_sys();
}

Validation

We could not reproduce this in a standalone unit test — it only shows up under a real multi-node workload, where the timing window is wide enough. Under SGLang on 2 nodes x 8 H200 (EP16, direct), the barrier timeout happens every time without these two lines, and never happens with them.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/715 **Author:** [@EvadLee](https://github.com/EvadLee) **Created:** 8/3/2026 **Status:** ✅ Merged **Merged:** 8/4/2026 **Merged by:** [@sphish](https://github.com/sphish) **Base:** `main` ← **Head:** `pr/gin-barrier-fence` --- ### 📝 Commits (2) - [`7dcf493`](https://github.com/deepseek-ai/DeepEP/commit/7dcf493a8d4f884de81dcb7276c688a6c228a03c) fix: Add fence.acq_rel.sys before the GIN barrier for mixed-fabric scaleup - [`9d170b2`](https://github.com/deepseek-ai/DeepEP/commit/9d170b27f547118a69bf64dd22c1a45ef059206e) fix: fence mixed-fabric writes in GIN barrier ### 📊 Changes **1 file changed** (+2 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/include/deep_ep/common/comm.cuh` (+2 -0) </details> ### 📄 Description ## Problem While integrating ElasticBuffer into SGLang, multi-node **direct** mode hangs non-deterministically at the GIN barrier: ``` DeepEP Gin barrier timeout, tag: 4, scaleout: 1, scaleup: 0/8, signal: 14, target: 15 ``` Envrioment: 2 nodes x 8 H200, EP16 ## Root cause When the scaleup team spans nodes, `kIsScaleupNVLink == false` and the data plane is **mixed**: same-node peers are written by raw `ptx::tma_store_1d`, cross-node peers by `gin.put`. The GIN barrier never releases the NVLink-written half to system scope — `flush()` does not publish (`gin.h:310-311`) and `signal()` defaults to `cuda::thread_scope_device` (`gin.h:304-308`). Device scope covers RDMA, since the NIC reads *local* memory, but not stores landing in another GPU. ## Fix `comm::gpu_barrier` already drains TMA stores before the barrier. Add a system-scope release there, on the path that publishes the raw NVLink writes: ```cpp if constexpr (kFlushStores) { ptx::tma_store_commit(); ptx::tma_store_wait(); __syncwarp(); if constexpr (not kIsScaleupNVLink) ptx::fence_acq_rel_sys(); } ``` ## Validation We could not reproduce this in a standalone unit test — it only shows up under a real multi-node workload, where the timing window is wide enough. Under SGLang on 2 nodes x 8 H200 (EP16, direct), the barrier timeout happens every time without these two lines, and never happens with them. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:05:40 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#728

No description provided.

Rows
Columns