[GH-ISSUE #584] deep_ep.cpp:200 init fails on partial CUDA peer access; preflight and report unsupported device pairs (cudaErrorPeerAccessUnsupported) #359

New issue

Open

opened 2026-10-26 09:59:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-26 09:59:07 +00:00

Owner

Copy link

Originally created by @stewtong on GitHub (Feb 20, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/584

Problem

Scope: Not requesting PCIe-only support; proposing an early capability check and a clearer
error message when the NVLink/full-mesh P2P requirement isn't met.

DeepEP initialization (runtime.sync(...) in deep_ep.cpp:200) aborts with a CUDA peer access
error on an 8-GPU PCIe-only host where only 4 undirected GPU pairs support
cudaDeviceCanAccessPeer. The error fires on the first unsupported pair without validating the
full required set upfront or reporting which pairs are incompatible. (The SGLang wrapper path that
triggers this is shown in the stack trace under Evidence.)

RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200
'peer access is not supported between these two devices'

A secondary CUDA error at runtime.cu:29 also appears in the exception chain — see crash
excerpt for details; its relationship to the primary peer-access failure is unclear.

Steps to reproduce

1. Use an 8-GPU host with partial CUDA P2P: only same-PCIe-switch pairs support peer access
   (e.g., {0,1}, {2,3}, {4,5}, {6,7}). All other 48 of 56 off-diagonal pairs return
   cudaErrorPeerAccessUnsupported (err=217).

2. Launch with EP=8 and DeepEP normal mode:

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B-FP8 \
  --tp-size 8 \
  --ep-size 8 \
  --moe-a2a-backend deepep \
  --deepep-mode normal \
  --disable-cuda-graph

Tested with Qwen/Qwen3.5-397B-A17B-FP8; any MoE model with EP support should reproduce on a
partial-P2P topology.

Note: --deepep-mode normal is required to isolate this crash. --deepep-mode auto fails
earlier with an unrelated deep_gemm assertion.

3. All 8 workers crash on first token dispatch during server warmup.

The repro command above is minimal. The exact command used in our run (including additional
flags such as --kv-cache-dtype, --mem-fraction-static, etc.) is in the Environment section.

Evidence

On this host, 4 undirected PIX pairs support P2P ({0,1}, {2,3}, {4,5}, {6,7}); in the
directed 8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed
pairs fail with cudaErrorPeerAccessUnsupported (217).

nvidia-smi topo -m (8-GPU PCIe topology, no NVLink)

        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  CPU Affinity    NUMA Affinity
GPU0     X    PIX   NODE  NODE  SYS   SYS   SYS   SYS   0-47,96-143     0
GPU1    PIX    X    NODE  NODE  SYS   SYS   SYS   SYS   0-47,96-143     0
GPU2    NODE  NODE   X    PIX   SYS   SYS   SYS   SYS   0-47,96-143     0
GPU3    NODE  NODE  PIX    X    SYS   SYS   SYS   SYS   0-47,96-143     0
GPU4    SYS   SYS   SYS   SYS   X    PIX   NODE  NODE   48-95,144-191   1
GPU5    SYS   SYS   SYS   SYS  PIX    X    NODE  NODE   48-95,144-191   1
GPU6    SYS   SYS   SYS   SYS  NODE  NODE   X    PIX    48-95,144-191   1
GPU7    SYS   SYS   SYS   SYS  NODE  NODE  PIX    X     48-95,144-191   1

Legend: PIX = single PCIe bridge  NODE = within NUMA  SYS = cross-NUMA  NV# = NVLink (none present)

CUDA peer access matrix — cudaDeviceCanAccessPeer (8×8)

      GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7
GPU0    X    Y    N    N    N    N    N    N
GPU1    Y    X    N    N    N    N    N    N
GPU2    N    N    X    Y    N    N    N    N
GPU3    N    N    Y    X    N    N    N    N
GPU4    N    N    N    N    X    Y    N    N
GPU5    N    N    N    N    Y    X    N    N
GPU6    N    N    N    N    N    N    X    Y
GPU7    N    N    N    N    N    N    Y    X

Summary: 4 undirected PIX pairs support P2P ({0,1}, {2,3}, {4,5}, {6,7}); in the directed
8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed pairs
fail with cudaErrorPeerAccessUnsupported (217).

Crash log excerpt

[2026-02-19 21:47:12 TP0 EP0] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP1 EP1] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP2 EP2] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP3 EP3] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP4 EP4] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP5 EP5] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP6 EP6] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.
[2026-02-19 21:47:12 TP7 EP7] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior.

  File ".../sglang/srt/models/qwen2_moe.py", line 271, in _forward_deepep
  File ".../sglang/srt/batch_overlap/two_batch_overlap.py", line 1035, in dispatch
  File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 782, in dispatch
  File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 801, in dispatch_b
  File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 410, in dispatch_b
  File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 524, in _get_buffer
    return DeepEPBuffer.get_deepep_buffer(
  File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 221, in get_deepep_buffer
RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200
'peer access is not supported between these two devices'
  what():  Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/runtime.cu:29
'named symbol not found'

Request

Two improvements that would help users on non-NVLink / partial-P2P hardware:

1. Document the peer-access assumption.
It would help to know whether DeepEP requires full-mesh peer access across all intranode EP devices,
or whether partial P2P is supported in some configurations. If full-mesh is required, noting this
in the README would let users rule out DeepEP before attempting it on PCIe hosts.

2. Validate all required pairs at init, report them together.
Rather than calling runtime.sync and aborting on the first unsupported pair, a preflight loop
over all required pairs would let users understand the full scope of incompatibility:

DeepEP P2P preflight failed.
Unsupported pairs: (GPU0,GPU2), (GPU0,GPU3), ... (48/56 directed pairs).
DeepEP requires full CUDA peer access across all participating devices.
Try a different MoE A2A backend or a topology with full P2P support.

The "Only use 20 SMs" warning already fires before the crash, so partial connectivity is detectable
at this stage. An actionable error at the same point — before any token dispatch — would improve
the experience significantly.

Environment

DeepEP:       compiled package bundled with lmsysorg/sglang:nightly-dev-cu13-20260219-46226798
              (no separate git SHA; /sgl-workspace/DeepEP is not a git repo in this container)
Python:       3.12.3
torch:        2.9.1+cu130
CUDA toolkit: 13.0 (nvcc V13.0.88)
Driver:       580.126.09
GPU:          8x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120, CC 12.0, 96 GB)
OS:           Ubuntu 22.04 / kernel 6.8.0-1044-aws
Env:          SGLANG_DISABLE_DEEP_GEMM=1 (SM120-specific; required on this hardware to use
              --deepep-mode normal; not needed for reproduction on hosts where deep_gemm works)

Originally created by @stewtong on GitHub (Feb 20, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/584 ## Problem > **Scope:** Not requesting PCIe-only support; proposing an early capability check and a clearer > error message when the NVLink/full-mesh P2P requirement isn't met. DeepEP initialization (`runtime.sync(...)` in `deep_ep.cpp:200`) aborts with a CUDA peer access error on an 8-GPU PCIe-only host where only 4 undirected GPU pairs support `cudaDeviceCanAccessPeer`. The error fires on the first unsupported pair without validating the full required set upfront or reporting which pairs are incompatible. (The SGLang wrapper path that triggers this is shown in the stack trace under Evidence.) ```text RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200 'peer access is not supported between these two devices' ``` A secondary CUDA error at `runtime.cu:29` also appears in the exception chain — see crash excerpt for details; its relationship to the primary peer-access failure is unclear. ## Steps to reproduce 1. Use an 8-GPU host with partial CUDA P2P: only same-PCIe-switch pairs support peer access (e.g., `{0,1}`, `{2,3}`, `{4,5}`, `{6,7}`). All other 48 of 56 off-diagonal pairs return `cudaErrorPeerAccessUnsupported` (err=217). 2. Launch with EP=8 and DeepEP normal mode: ```bash python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B-FP8 \ --tp-size 8 \ --ep-size 8 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-cuda-graph ``` > Tested with `Qwen/Qwen3.5-397B-A17B-FP8`; any MoE model with EP support should reproduce on a > partial-P2P topology. > **Note:** `--deepep-mode normal` is required to isolate this crash. `--deepep-mode auto` fails > earlier with an unrelated `deep_gemm` assertion. 3. All 8 workers crash on first token dispatch during server warmup. > The repro command above is minimal. The exact command used in our run (including additional > flags such as `--kv-cache-dtype`, `--mem-fraction-static`, etc.) is in the Environment section. ## Evidence On this host, 4 undirected PIX pairs support P2P (`{0,1}`, `{2,3}`, `{4,5}`, `{6,7}`); in the directed 8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed pairs fail with `cudaErrorPeerAccessUnsupported` (217). <details> <summary>nvidia-smi topo -m (8-GPU PCIe topology, no NVLink)</summary> ```text GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU0 X PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 GPU1 PIX X NODE NODE SYS SYS SYS SYS 0-47,96-143 0 GPU2 NODE NODE X PIX SYS SYS SYS SYS 0-47,96-143 0 GPU3 NODE NODE PIX X SYS SYS SYS SYS 0-47,96-143 0 GPU4 SYS SYS SYS SYS X PIX NODE NODE 48-95,144-191 1 GPU5 SYS SYS SYS SYS PIX X NODE NODE 48-95,144-191 1 GPU6 SYS SYS SYS SYS NODE NODE X PIX 48-95,144-191 1 GPU7 SYS SYS SYS SYS NODE NODE PIX X 48-95,144-191 1 Legend: PIX = single PCIe bridge NODE = within NUMA SYS = cross-NUMA NV# = NVLink (none present) ``` </details> <details> <summary>CUDA peer access matrix — cudaDeviceCanAccessPeer (8×8)</summary> ```text GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 X Y N N N N N N GPU1 Y X N N N N N N GPU2 N N X Y N N N N GPU3 N N Y X N N N N GPU4 N N N N X Y N N GPU5 N N N N Y X N N GPU6 N N N N N N X Y GPU7 N N N N N N Y X Summary: 4 undirected PIX pairs support P2P ({0,1}, {2,3}, {4,5}, {6,7}); in the directed 8×8 matrix this appears as 8/56 off-diagonal 'Y' entries, and the remaining directed pairs fail with cudaErrorPeerAccessUnsupported (217). ``` </details> <details> <summary>Crash log excerpt</summary> ```text [2026-02-19 21:47:12 TP0 EP0] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP1 EP1] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP2 EP2] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP3 EP3] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP4 EP4] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP5 EP5] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP6 EP6] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. [2026-02-19 21:47:12 TP7 EP7] Only use 20 SMs for DeepEP communication. This may result in highly suboptimal performance. Consider using --deepep-config to change the behavior. File ".../sglang/srt/models/qwen2_moe.py", line 271, in _forward_deepep File ".../sglang/srt/batch_overlap/two_batch_overlap.py", line 1035, in dispatch File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 782, in dispatch File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 801, in dispatch_b File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 410, in dispatch_b File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 524, in _get_buffer return DeepEPBuffer.get_deepep_buffer( File ".../sglang/srt/layers/moe/token_dispatcher/deepep.py", line 221, in get_deepep_buffer RuntimeError: Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:200 'peer access is not supported between these two devices' what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/kernels/runtime.cu:29 'named symbol not found' ``` </details> ## Request Two improvements that would help users on non-NVLink / partial-P2P hardware: **1. Document the peer-access assumption.** It would help to know whether DeepEP requires full-mesh peer access across all intranode EP devices, or whether partial P2P is supported in some configurations. If full-mesh is required, noting this in the README would let users rule out DeepEP before attempting it on PCIe hosts. **2. Validate all required pairs at init, report them together.** Rather than calling `runtime.sync` and aborting on the first unsupported pair, a preflight loop over all required pairs would let users understand the full scope of incompatibility: ```text DeepEP P2P preflight failed. Unsupported pairs: (GPU0,GPU2), (GPU0,GPU3), ... (48/56 directed pairs). DeepEP requires full CUDA peer access across all participating devices. Try a different MoE A2A backend or a topology with full P2P support. ``` The "Only use 20 SMs" warning already fires before the crash, so partial connectivity is detectable at this stage. An actionable error at the same point — before any token dispatch — would improve the experience significantly. ## Environment ``` DeepEP: compiled package bundled with lmsysorg/sglang:nightly-dev-cu13-20260219-46226798 (no separate git SHA; /sgl-workspace/DeepEP is not a git repo in this container) Python: 3.12.3 torch: 2.9.1+cu130 CUDA toolkit: 13.0 (nvcc V13.0.88) Driver: 580.126.09 GPU: 8x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120, CC 12.0, 96 GB) OS: Ubuntu 22.04 / kernel 6.8.0-1044-aws Env: SGLANG_DISABLE_DEEP_GEMM=1 (SM120-specific; required on this hardware to use --deepep-mode normal; not needed for reproduction on hosts where deep_gemm works) ```

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

hybrid-ep

chore/remove-clang-format-yapf

v2.5

nvDev

try-fix-cu128

mori-ep

antgroup-opt

compat-old-torch

add-linter

eager-rdma

v1.2.1

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#359

No description provided.