deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #597] [CLOSED] Fix NVLink domain over-count on MNNVL systems in `detect_accessible_ranks` #651

New issue

Closed

opened 2026-10-27 21:04:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/597
Author: @samnordmann
Created: 3/31/2026
Status: ❌ Closed

Base: hybrid-ep ← Head: hybrid_ep_mnnvl_detection

📝 Commits (1)

16a11ac Fix NVLink domain over-count on MNNVL systems in detect_accessible_ranks

📊 Changes

1 file changed (+4 additions, -2 deletions)

View changed files

📝 csrc/hybrid_ep/allocator/allocator.cu (+4 -2)

📄 Description

Summary

Fixes https://github.com/deepseek-ai/DeepEP/issues/596

On GB200 NVL72 (MNNVL) systems, detect_accessible_ranks() returns the full EP group size instead of the physical per-node GPU count because cuMemImportFromShareableHandle succeeds for cross-node peers via the MNNVL fabric. This causes device_sync_kernel to deadlock waiting for atomic increments that never arrive.

The non-fabric path of is_accessible() already guards on hostname; this patch applies the same check to the fabric path so the NVLink domain is correctly bounded to the physical node.

Test plan

Multi-node MNNVL (GB200 NVL72): detect_accessible_ranks returns physical GPUs per node, not group size
Single-node fabric (e.g. DGX H100): no behavior change
NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN env var override still works and warns on mismatch
Multi-node dispatch/combine complete without deadlock

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/597 **Author:** [@samnordmann](https://github.com/samnordmann) **Created:** 3/31/2026 **Status:** ❌ Closed **Base:** `hybrid-ep` ← **Head:** `hybrid_ep_mnnvl_detection` --- ### 📝 Commits (1) - [`16a11ac`](https://github.com/deepseek-ai/DeepEP/commit/16a11ac5e9c84331612f6c26d1155741f8cf93ec) Fix NVLink domain over-count on MNNVL systems in detect_accessible_ranks ### 📊 Changes **1 file changed** (+4 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `csrc/hybrid_ep/allocator/allocator.cu` (+4 -2) </details> ### 📄 Description ## Summary Fixes https://github.com/deepseek-ai/DeepEP/issues/596 On GB200 NVL72 (MNNVL) systems, `detect_accessible_ranks()` returns the full EP group size instead of the physical per-node GPU count because `cuMemImportFromShareableHandle` succeeds for cross-node peers via the MNNVL fabric. This causes `device_sync_kernel` to deadlock waiting for atomic increments that never arrive. The non-fabric path of `is_accessible()` already guards on hostname; this patch applies the same check to the fabric path so the NVLink domain is correctly bounded to the physical node. ## Test plan - [ ] Multi-node MNNVL (GB200 NVL72): `detect_accessible_ranks` returns physical GPUs per node, not group size - [ ] Single-node fabric (e.g. DGX H100): no behavior change - [ ] `NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN` env var override still works and warns on mismatch - [ ] Multi-node dispatch/combine complete without deadlock --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:04:54 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-10-27 21:05:17 +00:00

[PR #653] [CLOSED] fix #651 testing barrier #689

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#651

No description provided.

Rows
Columns

[PR #597] [CLOSED] Fix NVLink domain over-count on MNNVL systems in detect_accessible_ranks #651