[PR #783] Match GPUs by UUID in NVLink connection check #782

New issue

Open

opened 2026-10-27 21:06:18 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:06:18 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/783
Author: @lvren1485
Created: 10/15/2026
Status: 🔄 Open

Base: main ← Head: nvlink-check-by-uuid

📝 Commits (1)

7b92bf8 Match GPUs by UUID in NVLink connection check

📊 Changes

1 file changed (+7 additions, -7 deletions)

View changed files

📝 deep_ep/utils/envs.py (+7 -7)

📄 Description

Fixes #582.

check_nvlink_connections maps the current device to a physical index through CUDA_VISIBLE_DEVICES. This goes wrong when:

it's unset in containers, so every rank resolves to GPU 0 and fails with "GPU 0 and GPU 0 are not connected via NVLink"
it contains UUIDs (int() raises)
CUDA and NVML enumerate devices in different orders

This uses torch.cuda.get_device_properties().uuid with nvmlDeviceGetHandleByUUID instead, and gives a clear error when multiple ranks share one GPU.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/783 **Author:** [@lvren1485](https://github.com/lvren1485) **Created:** 10/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `nvlink-check-by-uuid` --- ### 📝 Commits (1) - [`7b92bf8`](https://github.com/deepseek-ai/DeepEP/commit/7b92bf8bc40967d8e9a481855450b835d56b32b7) Match GPUs by UUID in NVLink connection check ### 📊 Changes **1 file changed** (+7 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/utils/envs.py` (+7 -7) </details> ### 📄 Description Fixes #582. `check_nvlink_connections` maps the current device to a physical index through `CUDA_VISIBLE_DEVICES`. This goes wrong when: - it's unset in containers, so every rank resolves to GPU 0 and fails with "GPU 0 and GPU 0 are not connected via NVLink" - it contains UUIDs (`int()` raises) - CUDA and NVML enumerate devices in different orders This uses `torch.cuda.get_device_properties().uuid` with `nvmlDeviceGetHandleByUUID` instead, and gives a clear error when multiple ranks share one GPU. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:06:18 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#782

No description provided.

Rows
Columns