maps '(deleted)' suffix as filename and raises FileNotFoundError #421

New issue

Open

opened 2026-10-26 10:00:24 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:24 +00:00

Owner

Originally created by @TommyFive on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/777

Summary

When Linux marks a mapped NCCL library as deleted/replaced, /proc/self/maps ends the pathname in " (deleted)". DeepEP check_nccl_so() uses line.strip().split(' ')[-1] to identify its path, which yields literally "(deleted)" instead. filecmp.cmp("(deleted)", linked_nccl_so) then raises FileNotFoundError, obscuring the NCCL library inconsistency.

Error observed live

File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 52
    from deep_ep import Buffer, Config
File ".../site-packages/deep_ep/__init__.py", line 91
    check_nccl_so()
File ".../site-packages/deep_ep/__init__.py", line 74
    assert filecmp.cmp(loaded_nccl_so, linked_nccl_so, shallow=False)
File "/usr/lib/python3.12/filecmp.py", line 53
    s1 = _sig(os.stat(f1))
FileNotFoundError: [Errno 2] No such file or directory: '(deleted)'

SGLang scheduler subsequently exited with code 1, before GPU model loading.

Environment

Image: lmsysorg/sglang:dev-qwen38-27b-dflash2, SGLang 0.0.0.dev1+g5f55db35e
sgl-deep-ep 0.1.1, torch 2.13.0+cu130, nvidia-nccl-cu13 2.29.7, Python 3.12
RTX PRO 6000 Blackwell; NVIDIA driver 595.91.07 (CUDA driver support 13.2)
NCCL wheel lib modification timestamp 2026-10-09 14:13:08 UTC; SGLang crashed at 14:13:15 UTC
The application simultaneously ran an unpinned pip install -q open-webui in the same Python environment.
Afterward pip check found SGLang requirements mismatches: openai 2.6.1 expected / 2.29.0 installed, transformers 5.12.1 expected / 5.5.4 installed, suggesting dependency changes at runtime.

Expected

Please parse the complete /proc/self/maps pathname (including possible spaces), explicitly detect a " (deleted)" mapping, and report an actionable library-replaced/NCCL inconsistency error rather than treating "(deleted)" as a file. The check should not be suppressed.

The concurrent pip installation is a suspected trigger, not yet an isolated minimal reproduction. We are fixing our application bootstrap to stop modifying a live SGLang environment. Related but distinct: #710 reports a downgrade-induced undefined symbol, not this deleted-mapping parser failure.

Originally created by @TommyFive on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/777 ## Summary When Linux marks a mapped NCCL library as deleted/replaced, /proc/self/maps ends the pathname in " (deleted)". DeepEP check_nccl_so() uses line.strip().split(' ')[-1] to identify its path, which yields literally "(deleted)" instead. filecmp.cmp("(deleted)", linked_nccl_so) then raises FileNotFoundError, obscuring the NCCL library inconsistency. ## Error observed live ```text File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 52 from deep_ep import Buffer, Config File ".../site-packages/deep_ep/__init__.py", line 91 check_nccl_so() File ".../site-packages/deep_ep/__init__.py", line 74 assert filecmp.cmp(loaded_nccl_so, linked_nccl_so, shallow=False) File "/usr/lib/python3.12/filecmp.py", line 53 s1 = _sig(os.stat(f1)) FileNotFoundError: [Errno 2] No such file or directory: '(deleted)' ``` SGLang scheduler subsequently exited with code 1, before GPU model loading. ## Environment - Image: lmsysorg/sglang:dev-qwen38-27b-dflash2, SGLang 0.0.0.dev1+g5f55db35e - sgl-deep-ep 0.1.1, torch 2.13.0+cu130, nvidia-nccl-cu13 2.29.7, Python 3.12 - RTX PRO 6000 Blackwell; NVIDIA driver 595.91.07 (CUDA driver support 13.2) - NCCL wheel lib modification timestamp 2026-10-09 14:13:08 UTC; SGLang crashed at 14:13:15 UTC - The application simultaneously ran an unpinned pip install -q open-webui in the same Python environment. - Afterward pip check found SGLang requirements mismatches: openai 2.6.1 expected / 2.29.0 installed, transformers 5.12.1 expected / 5.5.4 installed, suggesting dependency changes at runtime. ## Expected Please parse the complete /proc/self/maps pathname (including possible spaces), explicitly detect a " (deleted)" mapping, and report an actionable library-replaced/NCCL inconsistency error rather than treating "(deleted)" as a file. The check should not be suppressed. The concurrent pip installation is a suspected trigger, not yet an isolated minimal reproduction. We are fixing our application bootstrap to stop modifying a live SGLang environment. Related but distinct: #710 reports a downgrade-induced undefined symbol, not this deleted-mapping parser failure.

bsakaguchi referenced this issue

2026-10-27 21:04:01 +00:00

[PR #421] [MERGED] Support EP48/96 for internode kernels and update some config. #557