[PR #703] fix(legacy): order work after ready-index publication #718

New issue

Open

opened 2026-10-27 21:05:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:33 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/703
Author: @youchengsong
Created: 7/27/2026
Status: 🔄 Open

Base: main ← Head: fix-legacy-ready-fence

📝 Commits (1)

d49e2c5 fix(legacy): order work after ready-index publication

📊 Changes

1 file changed (+4 additions, -3 deletions)

View changed files

📝 csrc/kernels/legacy/ibgda_device.cuh (+4 -3)

📄 Description

Summary

Restore the post-publication ordering point used by NVSHMEM after
ibgda_submit_requests advances ready_idx.

DeepEP already uses __threadfence() before the CAS to order WQE payload
writes before ready-index publication. This change adds the separate fence
after the successful CAS so subsequent post-send work remains ordered after
that publication.

The added memory_fence_cta() emits the same fence.acq_rel.cta
instruction used by NVSHMEM's IBGDA_MFENCE() at the corresponding point:

https://github.com/NVIDIA/nvshmem/blob/v3.7.2-0/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L1648-L1670

Scope

The functional change is one fence in the legacy IBGDA submission helper.
It does not change:

WQE construction or the existing pre-publication system fence;
ready-index selection or CAS behavior;
synchronous versus asynchronous post-send handling;
CQ polling, QP selection, batching, or public APIs.

This PR is protocol-ordering hardening and does not claim that this omitted
fence alone explains every observed legacy progress failure.

Validation

full build of this commit for sm_103 with CUDA 13.0.88,
NVSHMEM 3.7.2, NCCL 2.30.7, and PyTorch 2.11;
generated extension contains an sm_103 cubin;
bash ./format.sh;
git diff --check upstream/main...HEAD;
verified that the helper's memory_fence_cta() is
fence.acq_rel.cta, matching NVSHMEM's ordering primitive.

The change also merges cleanly with the current head of #677; the two
patches address independent parts of the submission path.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/703 **Author:** [@youchengsong](https://github.com/youchengsong) **Created:** 7/27/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix-legacy-ready-fence` --- ### 📝 Commits (1) - [`d49e2c5`](https://github.com/deepseek-ai/DeepEP/commit/d49e2c5e28c691433096f5fdfa9b291d06d96ca2) fix(legacy): order work after ready-index publication ### 📊 Changes **1 file changed** (+4 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/ibgda_device.cuh` (+4 -3) </details> ### 📄 Description ## Summary Restore the post-publication ordering point used by NVSHMEM after `ibgda_submit_requests` advances `ready_idx`. DeepEP already uses `__threadfence()` before the CAS to order WQE payload writes before ready-index publication. This change adds the separate fence after the successful CAS so subsequent post-send work remains ordered after that publication. The added `memory_fence_cta()` emits the same `fence.acq_rel.cta` instruction used by NVSHMEM's `IBGDA_MFENCE()` at the corresponding point: https://github.com/NVIDIA/nvshmem/blob/v3.7.2-0/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L1648-L1670 ## Scope The functional change is one fence in the legacy IBGDA submission helper. It does not change: - WQE construction or the existing pre-publication system fence; - ready-index selection or CAS behavior; - synchronous versus asynchronous post-send handling; - CQ polling, QP selection, batching, or public APIs. This PR is protocol-ordering hardening and does not claim that this omitted fence alone explains every observed legacy progress failure. ## Validation - full build of this commit for `sm_103` with CUDA 13.0.88, NVSHMEM 3.7.2, NCCL 2.30.7, and PyTorch 2.11; - generated extension contains an `sm_103` cubin; - `bash ./format.sh`; - `git diff --check upstream/main...HEAD`; - verified that the helper's `memory_fence_cta()` is `fence.acq_rel.cta`, matching NVSHMEM's ordering primitive. The change also merges cleanly with the current head of #677; the two patches address independent parts of the submission path. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:33 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:41 +00:00

[PR #719] Fix EP_BUFFER_DEBUG=0 turning on Python-side debug output #730

bsakaguchi referenced this issue

2026-10-27 21:05:50 +00:00

[PR #742] Parse EP_BUFFER_DEBUG as an integer on the Python side #747

bsakaguchi referenced this issue

2026-10-27 21:06:13 +00:00