[PR #609] fix: skip self NVLink connection check #582 #660

New issue

Open

opened 2026-10-27 21:04:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/609
Author: @pazyork
Created: 4/24/2026
Status: 🔄 Open

Base: main ← Head: main

📝 Commits (1)

4fdd8a2 fix: skip self NVLink check for duplicate physical GPU IDs

📊 Changes

2 files changed (+37 additions, -1 deletions)

View changed files

📝 deep_ep/utils.py (+1 -1)
➕ tests/test_utils.py (+36 -0)

📄 Description

Fixes #582

Problem
When multiple ranks share the same physical GPU (common in local single-GPU debugging, container GPU sharing, MIG vGPU scenarios, even on multi-GPU servers), the NVLink initialization check triggers a false assertion:
AssertionError: No NVLink connection between GPU 0 and GPU 0

Root cause
The original code only skips duplicate checks by index i >= j, but doesn't handle the case where different rank indices map to the same physical GPU ID.

Fix
Add or physical_device_indices[i] == physical_device_indices[j] condition to the check loop, skip NVLink check for same physical GPU pairs.

Test
Added lightweight unit test in tests/test_utils.py with mocked pynvml/distributed, no real multi-GPU hardware required. Verified:

Duplicate GPU scenario no longer raises assertion
Normal multi-GPU scenario works unchanged

Impact
No breaking changes, only affects the NVLink initialization check process. All existing normal deployment scenarios are completely unaffected.

Thanks for reviewing! It's a tiny fix, feel free to let me know if you have any suggestions and I'll adjust promptly. @jershi425 @sphish

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/609 **Author:** [@pazyork](https://github.com/pazyork) **Created:** 4/24/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`4fdd8a2`](https://github.com/deepseek-ai/DeepEP/commit/4fdd8a2e3dd3ed23f90f230b62585961e5886178) fix: skip self NVLink check for duplicate physical GPU IDs ### 📊 Changes **2 files changed** (+37 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/utils.py` (+1 -1) ➕ `tests/test_utils.py` (+36 -0) </details> ### 📄 Description Fixes #582 **Problem** When multiple ranks share the same physical GPU (common in local single-GPU debugging, container GPU sharing, MIG vGPU scenarios, even on multi-GPU servers), the NVLink initialization check triggers a false assertion: `AssertionError: No NVLink connection between GPU 0 and GPU 0` **Root cause** The original code only skips duplicate checks by index `i >= j`, but doesn't handle the case where different rank indices map to the same physical GPU ID. **Fix** Add `or physical_device_indices[i] == physical_device_indices[j]` condition to the check loop, skip NVLink check for same physical GPU pairs. **Test** Added lightweight unit test in `tests/test_utils.py` with mocked pynvml/distributed, no real multi-GPU hardware required. Verified: 1. Duplicate GPU scenario no longer raises assertion 2. Normal multi-GPU scenario works unchanged 3. **Impact** No breaking changes, only affects the NVLink initialization check process. All existing normal deployment scenarios are completely unaffected. Thanks for reviewing! It's a tiny fix, feel free to let me know if you have any suggestions and I'll adjust promptly. @jershi425 @sphish --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:04:58 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:19 +00:00

[PR #660] [CLOSED] Fix PP TMA stage reuse race #693