mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #710] pip-installing vLLM/torch silently downgrades NCCL below the 2.30.4 floor; import dies at undefined symbol and check_nccl_so() cannot catch it #408
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#408
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @dmvevents on GitHub (Jul 30, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/710
Problem
import deep_epfails with a raw dynamic-linker error after installing vLLM (or anything that pinstorch==2.11.0) into an environment that satisfied the documented NCCL floor:The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):
nvidia-nccl-cu13==2.30.4(the README floor: "NCCL 2.30.4 and above").pip install vllm==0.26.0pinstorch==2.11.0, and torch 2.11.0 pinsnvidia-nccl-cu13==2.28.9(exact
==) — pip silently downgrades NCCL as a side effect._C.sowas built against, so the nextimport deep_epdies at the linker with no version hint.pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4fully heals it — deep_ep + vllm0.26.0 + torch 2.11.0 then import and run together cleanly.
Why
check_nccl_so()cannot catch this caseThe guard in
deep_ep/__init__.pycompares the loadedlibnccl.so(from/proc/self/maps) againstthe lib at
find_nccl_root()— i.e. the NCCL currently installed. A wheel downgrade moves both sidestogether: torch loads 2.28.9 and
find_nccl_root()now also resolves to 2.28.9. Loaded == reference, theguard passes, and the failure surfaces one line later at
import deep_ep._Cas an opaque linker error.The invariant that actually broke is "loaded == what
_C.sowas built against", which the build doesnot record anywhere the guard can read.
Suggested fixes (either or both)
install_requires=["nvidia-nccl-cu13>=2.30.4"](per CUDAvariant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
Today
setup.pylocates NCCL at build time viafind_pkgs.find_nccl_root()but ships no constraint.envs.pynext to the existing persistent envs) and extend
check_nccl_so()to compare the loaded lib againstthat, so this failure mode produces the guard's existing clear message instead of
undefined symbol: ....Environment
Happy to provide the full install transcript.
@dmvevents commented on GitHub (Oct 27, 2026):
Checked against the V2.5 source on
main@f6aa38fand the current PyPI metadata. The documented floor is now NCCL 2.32.3 (README.md:53), and the exact pins that trigger the downgrade are still below it:vllm==0.32.0requirestorch==2.13.0, whose Linux x86_64 wheel requiresnvidia-nccl-cu13==2.29.7torch==2.14.0requiresnvidia-nccl-cu13==2.30.7check_nccl_so()(deep_ep/__init__.py:19-41) still compares the loaded library with the one underfind_nccl_root(), and the downgrade replaces that one too.setup.pystill declares no NCCL requirement. So installing vLLM or torch after DeepEP still moves NCCL below the floor without the guard firing. #712 implements the second suggestion above: it records the build-time NCCL version and rejects older runtimes.