[PR #712] Reject NCCL runtimes older than the build #725

New issue

Open

opened 2026-10-27 21:05:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:38 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/712
Author: @gambletan
Created: 8/1/2026
Status: 🔄 Open

Base: main ← Head: fix/nccl-runtime-version-guard

📝 Commits (4)

7c0b6c4 feat: record the NCCL build version
49474cc fix: reject NCCL runtimes older than the build
6716e23 fix: defer extension import until after NCCL checks
841e4e3 fix: clarify NCCL runtime guard failures

📊 Changes

7 files changed (+414 additions, -6 deletions)

View changed files

📝 deep_ep/__init__.py (+18 -3)
📝 deep_ep/utils/__init__.py (+11 -3)
➕ deep_ep/utils/nccl.py (+40 -0)
📝 setup.py (+7 -0)
➕ tests/test_nccl_import_guard.py (+179 -0)
➕ tests/utils/test_nccl.py (+126 -0)
➕ tests/utils/test_utils_init.py (+33 -0)

📄 Description

Summary

record the NCCL header version used to build each wheel
query the already-loaded NCCL runtime before importing deep_ep._C
reject runtimes older than the build with an actionable versioned error
defer the compatibility EventHandle export so _C is not loaded before the NCCL guard

Why

When another package downgrades NCCL, the existing loaded-vs-installed binary comparison moves both sides to the downgraded library. The check therefore cannot detect that _C was built against a newer NCCL Device API, and users see an opaque undefined-symbol error.

Generated wheel metadata now carries built_nccl_version. At import, DeepEP calls ncclGetVersion on the mapped runtime and allows equal or newer versions. Source checkouts without generated metadata keep the existing behavior, and EP_SUPPRESS_NCCL_CHECK=1 still bypasses the checks.

Testing

PYTHONPYCACHEPREFIX=/tmp/deepep-pycache python3 -m pytest tests/utils/test_nccl.py tests/utils/test_utils_init.py tests/test_nccl_import_guard.py -q
PYTHONPYCACHEPREFIX=/tmp/deepep-pycache python3 -m compileall -q deep_ep tests

Result: 14 tests passed. These are GPU-free unit tests; no CUDA/NCCL integration run was performed.

Fixes #710

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/712 **Author:** [@gambletan](https://github.com/gambletan) **Created:** 8/1/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/nccl-runtime-version-guard` --- ### 📝 Commits (4) - [`7c0b6c4`](https://github.com/deepseek-ai/DeepEP/commit/7c0b6c4b6a62c8f44b8996e747c34003e024e0a0) feat: record the NCCL build version - [`49474cc`](https://github.com/deepseek-ai/DeepEP/commit/49474cc093ccb81cccaa4d4d15d28675585b8b7d) fix: reject NCCL runtimes older than the build - [`6716e23`](https://github.com/deepseek-ai/DeepEP/commit/6716e23a59087854fe2529fc3a1b3889ddabb1b9) fix: defer extension import until after NCCL checks - [`841e4e3`](https://github.com/deepseek-ai/DeepEP/commit/841e4e3be868dbb29a4f509fe823082919da946a) fix: clarify NCCL runtime guard failures ### 📊 Changes **7 files changed** (+414 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/__init__.py` (+18 -3) 📝 `deep_ep/utils/__init__.py` (+11 -3) ➕ `deep_ep/utils/nccl.py` (+40 -0) 📝 `setup.py` (+7 -0) ➕ `tests/test_nccl_import_guard.py` (+179 -0) ➕ `tests/utils/test_nccl.py` (+126 -0) ➕ `tests/utils/test_utils_init.py` (+33 -0) </details> ### 📄 Description ## Summary - record the NCCL header version used to build each wheel - query the already-loaded NCCL runtime before importing `deep_ep._C` - reject runtimes older than the build with an actionable versioned error - defer the compatibility `EventHandle` export so `_C` is not loaded before the NCCL guard ## Why When another package downgrades NCCL, the existing loaded-vs-installed binary comparison moves both sides to the downgraded library. The check therefore cannot detect that `_C` was built against a newer NCCL Device API, and users see an opaque undefined-symbol error. Generated wheel metadata now carries `built_nccl_version`. At import, DeepEP calls `ncclGetVersion` on the mapped runtime and allows equal or newer versions. Source checkouts without generated metadata keep the existing behavior, and `EP_SUPPRESS_NCCL_CHECK=1` still bypasses the checks. ## Testing - `PYTHONPYCACHEPREFIX=/tmp/deepep-pycache python3 -m pytest tests/utils/test_nccl.py tests/utils/test_utils_init.py tests/test_nccl_import_guard.py -q` - `PYTHONPYCACHEPREFIX=/tmp/deepep-pycache python3 -m compileall -q deep_ep tests` Result: 14 tests passed. These are GPU-free unit tests; no CUDA/NCCL integration run was performed. Fixes #710 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:38 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#725

No description provided.

Rows
Columns