deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[PR #623] [CLOSED] Nvshmem 3 5 runtime fixes (#564) #669

New issue
Closed
opened 2026-10-27 21:05:03 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-27 21:05:03 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/623
Author: @ko3n1g
Created: 5/6/2026
Status: ❌ Closed

Base: hybrid-ep ← Head: ko3n1g/fix/nvshmem-3.5-runtime-fixes-on-hybrid-ep

📝 Commits (6)

  • 379eef7 Nvshmem 3 5 runtime fixes (#564)
  • 4e38d1d Guard NVSHMEM IBGDA header from host compilation to avoid duplicate symbol
  • 1e0dc0d diag: probe IBGDA device-state contents on rank 0 in notify_dispatch
  • 16c4c99 probe: drop num_qps_per_rank default 24 → 8 (match notify_dispatch num_channels)
  • 6e9cb53 probe: force v2 RC indexing (NVSHMEM 3.6.5 uses post-3.5.21 layout despite v1 typedef)
  • cb55a2c revert: force-v2 RC indexing (state IS populated, layout not the cause)

📊 Changes

4 files changed (+46 additions, -5 deletions)

View changed files

📝 csrc/kernels/configs.cuh (+15 -0)
📝 csrc/kernels/ibgda_device.cuh (+19 -4)
📝 csrc/kernels/internode.cu (+11 -0)
📝 deep_ep/buffer.py (+1 -1)

📄 Description

  • DeepEP/csrc: Force CUDA RDC when NVSHMEM is used.

When DeepEP is against a source build of NVSHMEM starting with NVSHMEM 3.3.20, multiple instances of nvshmemi_ibgda_device_state_d are detected. One from NVSHMEM, another from the kernels.

  • DeepEP/csrc: Update RC selection for NVSHMEM 3.5.

The locations of QPs within the device state struct changed with the addition of QP-specific APIs.

This change updates DeepEP to work with the new layout.

The ibgda_get_rc function is also templated so that main remains backwards compatible with previous versions of NVSHMEM.

Tested with both internode and low_latency kernels on 4 H100 nodes.

  • Check Format

(cherry picked from commit 29d31c0957)

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/623 **Author:** [@ko3n1g](https://github.com/ko3n1g) **Created:** 5/6/2026 **Status:** ❌ Closed **Base:** `hybrid-ep` ← **Head:** `ko3n1g/fix/nvshmem-3.5-runtime-fixes-on-hybrid-ep` --- ### 📝 Commits (6) - [`379eef7`](https://github.com/deepseek-ai/DeepEP/commit/379eef7a927d15a092fb3037d359caa5187c007a) Nvshmem 3 5 runtime fixes (#564) - [`4e38d1d`](https://github.com/deepseek-ai/DeepEP/commit/4e38d1d944306a15a009162286062266da87ac4a) Guard NVSHMEM IBGDA header from host compilation to avoid duplicate symbol - [`1e0dc0d`](https://github.com/deepseek-ai/DeepEP/commit/1e0dc0d584e46995959db53c51e03d4a390acfde) diag: probe IBGDA device-state contents on rank 0 in notify_dispatch - [`16c4c99`](https://github.com/deepseek-ai/DeepEP/commit/16c4c99b7925cb236116bfd95cf0ee2e495d9dfc) probe: drop num_qps_per_rank default 24 → 8 (match notify_dispatch num_channels) - [`6e9cb53`](https://github.com/deepseek-ai/DeepEP/commit/6e9cb5325ee3adee6e570c538953e6c6cb74db85) probe: force v2 RC indexing (NVSHMEM 3.6.5 uses post-3.5.21 layout despite v1 typedef) - [`cb55a2c`](https://github.com/deepseek-ai/DeepEP/commit/cb55a2cf090ebaf881046ecfbeade5778264745f) revert: force-v2 RC indexing (state IS populated, layout not the cause) ### 📊 Changes **4 files changed** (+46 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/configs.cuh` (+15 -0) 📝 `csrc/kernels/ibgda_device.cuh` (+19 -4) 📝 `csrc/kernels/internode.cu` (+11 -0) 📝 `deep_ep/buffer.py` (+1 -1) </details> ### 📄 Description * DeepEP/csrc: Force CUDA RDC when NVSHMEM is used. When DeepEP is against a source build of NVSHMEM starting with NVSHMEM 3.3.20, multiple instances of nvshmemi_ibgda_device_state_d are detected. One from NVSHMEM, another from the kernels. * DeepEP/csrc: Update RC selection for NVSHMEM 3.5. The locations of QPs within the device state struct changed with the addition of QP-specific APIs. This change updates DeepEP to work with the new layout. The ibgda_get_rc function is also templated so that main remains backwards compatible with previous versions of NVSHMEM. Tested with both internode and low_latency kernels on 4 H100 nodes. * Check Format --------- (cherry picked from commit 29d31c095796f3c8ece47ee9cdcc167051bbeed9) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-27 21:05:03 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#669
No description provided.