[PR #796] fix(utils): enhance NCCL library directory resolution and environment robustness #793

New issue

Open

opened 2026-10-27 21:06:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:06:26 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/796
Author: @reginaldalfret
Created: 10/25/2026
Status: 🔄 Open

Base: main ← Head: fix/nccl-discovery-lib-dirs

📝 Commits (1)

a1f7cbf fix(utils): enhance NCCL library directory resolution and environment robustness

📊 Changes

4 files changed (+124 additions, -19 deletions)

View changed files

📝 deep_ep/__init__.py (+21 -7)
📝 deep_ep/utils/envs.py (+1 -1)
📝 deep_ep/utils/find_pkgs.py (+16 -11)
➕ tests/utils/test_find_pkgs.py (+86 -0)

📄 Description

Problem

NCCL Library Resolution Across Different Linux Layouts: In deep_ep/utils/find_pkgs.py, get_nccl_lib_name() strictly looked inside Path(root) / 'lib'. If a user set NCCL_DIR=/usr/lib64 or NCCL_DIR=/usr/lib/x86_64-linux-gnu, or when find_pkg_root() returned a library folder directly (due to os.path.basename(lib_dir) != 'lib'), resolution failed with ModuleNotFoundError: libnccl.so not found under .../lib.
Empty Environment Variables: If EP_NCCL_ROOT_DIR="" or NCCL_DIR="" was exported as an empty string, find_pkg_root() checked if env_name in os.environ: and returned "", breaking downstream path construction instead of falling back to distribution discovery.
Metadata & Distribution Attribute Safety: In find_pkg_root(), accessing dist._path.parent and dist.metadata['Name'] directly could raise AttributeError or KeyError on non-path or malformed distributions.
Environment Check Portability: check_nccl_so() unconditionally attempted to read /proc/self/maps, crashing with FileNotFoundError in environments without /proc mounted.

Solution

Update get_nccl_lib_name() to search candidate directories: Path(root) / 'lib', Path(root) / 'lib64', and Path(root).
Use os.environ.get(env_name) to ignore empty environment variables.
Safely guard dist._path and dictionary metadata lookups in find_pkg_root().
Check os.path.exists('/proc/self/maps') in check_nccl_so() and support candidate search paths for linked NCCL candidates.
In EnvOverride.__exit__, use os.environ.pop(name, None) to prevent KeyError if variables were deleted during the context block.
Add unit tests in tests/utils/test_find_pkgs.py covering standard lib, lib64, direct directories, empty environment variables, and EnvOverride safety.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/796 **Author:** [@reginaldalfret](https://github.com/reginaldalfret) **Created:** 10/25/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/nccl-discovery-lib-dirs` --- ### 📝 Commits (1) - [`a1f7cbf`](https://github.com/deepseek-ai/DeepEP/commit/a1f7cbf1ebd3259412bd3e7bef0a5d444a4dab8e) fix(utils): enhance NCCL library directory resolution and environment robustness ### 📊 Changes **4 files changed** (+124 additions, -19 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/__init__.py` (+21 -7) 📝 `deep_ep/utils/envs.py` (+1 -1) 📝 `deep_ep/utils/find_pkgs.py` (+16 -11) ➕ `tests/utils/test_find_pkgs.py` (+86 -0) </details> ### 📄 Description ### Problem 1. **NCCL Library Resolution Across Different Linux Layouts**: In `deep_ep/utils/find_pkgs.py`, `get_nccl_lib_name()` strictly looked inside `Path(root) / 'lib'`. If a user set `NCCL_DIR=/usr/lib64` or `NCCL_DIR=/usr/lib/x86_64-linux-gnu`, or when `find_pkg_root()` returned a library folder directly (due to `os.path.basename(lib_dir) != 'lib'`), resolution failed with `ModuleNotFoundError: libnccl.so not found under .../lib`. 2. **Empty Environment Variables**: If `EP_NCCL_ROOT_DIR=""` or `NCCL_DIR=""` was exported as an empty string, `find_pkg_root()` checked `if env_name in os.environ:` and returned `""`, breaking downstream path construction instead of falling back to distribution discovery. 3. **Metadata & Distribution Attribute Safety**: In `find_pkg_root()`, accessing `dist._path.parent` and `dist.metadata['Name']` directly could raise `AttributeError` or `KeyError` on non-path or malformed distributions. 4. **Environment Check Portability**: `check_nccl_so()` unconditionally attempted to read `/proc/self/maps`, crashing with `FileNotFoundError` in environments without `/proc` mounted. ### Solution 1. Update `get_nccl_lib_name()` to search candidate directories: `Path(root) / 'lib'`, `Path(root) / 'lib64'`, and `Path(root)`. 2. Use `os.environ.get(env_name)` to ignore empty environment variables. 3. Safely guard `dist._path` and dictionary metadata lookups in `find_pkg_root()`. 4. Check `os.path.exists('/proc/self/maps')` in `check_nccl_so()` and support candidate search paths for linked NCCL candidates. 5. In `EnvOverride.__exit__`, use `os.environ.pop(name, None)` to prevent `KeyError` if variables were deleted during the context block. 6. Add unit tests in `tests/utils/test_find_pkgs.py` covering standard `lib`, `lib64`, direct directories, empty environment variables, and `EnvOverride` safety. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:06:26 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#793

No description provided.

Rows
Columns