[PR #696] Fix legacy RC QP indexing for NVSHMEM 3.5.19+ #713

New issue

Open

opened 2026-10-27 21:05:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/696
Author: @caoxiaoyuyuyuyuyu
Created: 7/23/2026
Status: 🔄 Open

Base: main ← Head: fix/nvshmem-rc-qp-index

📝 Commits (2)

d268280 fix: select NVSHMEM RC QPs by runtime layout
2c70ee1 fix: reject incompatible NVSHMEM runtime layouts

📊 Changes

2 files changed (+58 additions, -10 deletions)

View changed files

📝 csrc/kernels/backend/nvshmem.cu (+41 -4)
📝 csrc/kernels/legacy/ibgda_device.cuh (+17 -6)

📄 Description

Background

The legacy IBGDA path currently indexes globalmem.rcs as PE-major. That matches NVSHMEM 3.4.5 and earlier, but NVSHMEM changed RC QP allocation to QP-major, PE-interleaved order in NVIDIA/nvshmem@ce9d487252. The change is included in NVSHMEM 3.5.19 and later.

With NVSHMEM 3.7.2, the old expression can select another PE RC QP. All ranks may initialize successfully, then the first cross-node dispatch stops making progress and can eventually surface a CUDA fault.

Change

Calculate the queue number once.
Use queue * npes + pe for NVSHMEM 3.5.19 and later.
Preserve pe * queues_per_pe + queue for older supported NVSHMEM releases.
Select the device layout at compile time using NVSHMEM vendor version macros.
Query the loaded host-library version before NVSHMEM initialization and reject compile/runtime combinations that cross the 3.5.19 layout transition. This closes the compatibility gap left by the normal NVSHMEM check, which permits an older device module with a newer host library.
Fail compilation if the required vendor version macros are unavailable instead of silently selecting the old layout.

Validation

Built the first current-main change (d268280) successfully for SM 10.3 with CUDA 13, NVSHMEM 3.7.2, and NCCL 2.30.7; cuobjdump confirms arch = sm_103.
Verified from official NVSHMEM 3.4.5, 3.5.19, and 3.7.2 headers that nvshmemx_vendor_get_version_info and all three vendor version macros are available.
Added compile-time boundary assertions for 3.5.18 (old layout) and 3.5.19 (new layout); the same classifier also passed a local C++17 boundary check.
Applied the equivalent device-index fix to DeepEP v1.2.1 built for SM 10.3 with CUDA 13 and NVSHMEM 3.7.2.
Passed multi-node EP32 differential checks for dispatch/combine forward output, input gradients, and route-probability gradients.
Passed two simultaneous EP32 groups and ten consecutive finite full-model optimization steps without a dispatch timeout, CUDA illegal-access error, NCCL error, process restart, or Xid.
The longer CP1 diagnostic stopped before step 10 on a model-loss torch.OutOfMemoryError while materializing FP32 vocabulary logits. The failure was outside DeepEP/NVSHMEM; no transport or device-fault signature preceded it.
Formatted both touched files with clang-format 15.0.7.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/696 **Author:** [@caoxiaoyuyuyuyuyu](https://github.com/caoxiaoyuyuyuyuyu) **Created:** 7/23/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/nvshmem-rc-qp-index` --- ### 📝 Commits (2) - [`d268280`](https://github.com/deepseek-ai/DeepEP/commit/d26828097e1100db82ac1d7698c29b7c4ab95eb0) fix: select NVSHMEM RC QPs by runtime layout - [`2c70ee1`](https://github.com/deepseek-ai/DeepEP/commit/2c70ee1215bfd02299d8c14654572dfa700b169e) fix: reject incompatible NVSHMEM runtime layouts ### 📊 Changes **2 files changed** (+58 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/backend/nvshmem.cu` (+41 -4) 📝 `csrc/kernels/legacy/ibgda_device.cuh` (+17 -6) </details> ### 📄 Description ## Background The legacy IBGDA path currently indexes `globalmem.rcs` as PE-major. That matches NVSHMEM 3.4.5 and earlier, but NVSHMEM changed RC QP allocation to QP-major, PE-interleaved order in NVIDIA/nvshmem@ce9d4872522775c50ee536a7044bc36c688db2b9. The change is included in NVSHMEM 3.5.19 and later. With NVSHMEM 3.7.2, the old expression can select another PE RC QP. All ranks may initialize successfully, then the first cross-node dispatch stops making progress and can eventually surface a CUDA fault. ## Change - Calculate the queue number once. - Use `queue * npes + pe` for NVSHMEM 3.5.19 and later. - Preserve `pe * queues_per_pe + queue` for older supported NVSHMEM releases. - Select the device layout at compile time using NVSHMEM vendor version macros. - Query the loaded host-library version before NVSHMEM initialization and reject compile/runtime combinations that cross the 3.5.19 layout transition. This closes the compatibility gap left by the normal NVSHMEM check, which permits an older device module with a newer host library. - Fail compilation if the required vendor version macros are unavailable instead of silently selecting the old layout. ## Validation - Built the first current-main change (`d268280`) successfully for SM 10.3 with CUDA 13, NVSHMEM 3.7.2, and NCCL 2.30.7; `cuobjdump` confirms `arch = sm_103`. - Verified from official NVSHMEM 3.4.5, 3.5.19, and 3.7.2 headers that `nvshmemx_vendor_get_version_info` and all three vendor version macros are available. - Added compile-time boundary assertions for 3.5.18 (old layout) and 3.5.19 (new layout); the same classifier also passed a local C++17 boundary check. - Applied the equivalent device-index fix to DeepEP v1.2.1 built for SM 10.3 with CUDA 13 and NVSHMEM 3.7.2. - Passed multi-node EP32 differential checks for dispatch/combine forward output, input gradients, and route-probability gradients. - Passed two simultaneous EP32 groups and ten consecutive finite full-model optimization steps without a dispatch timeout, CUDA illegal-access error, NCCL error, process restart, or Xid. - The longer CP1 diagnostic stopped before step 10 on a model-loss `torch.OutOfMemoryError` while materializing FP32 vocabulary logits. The failure was outside DeepEP/NVSHMEM; no transport or device-fault signature preceded it. - Formatted both touched files with clang-format 15.0.7. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:30 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:39 +00:00

[PR #713] Handle missing bandwidth in SM estimation #726