[GH-ISSUE #629] EP_DISABLE_GIN=1 does not bypass NCCL Gin preflight check #378

New issue

Closed

opened 2026-10-26 09:59:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:59:44 +00:00

Owner

Originally created by @DanielDanyang on GitHub (May 10, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/629

Summary

DeepEP V2/elastic still fails during ElasticBuffer initialization with EP_DISABLE_GIN=1 when NCCL reports no Gin capability for the communicator.

This blocks pure single-node intra-node EP runs.

Reproduction

Environment used to reproduce:

GPU: 8 x NVIDIA B300 SXM6 AC
CUDA compiler: /usr/local/cuda-13.2/bin/nvcc
PyTorch: 2.13.0.dev20260502+cu130
NCCL: nvidia-nccl-cu13 2.30.4
NVSHMEM: nvidia-nvshmem-cu13 3.4.5

Run a pure intra-node elastic EP smoke test:

export CUDA_VISIBLE_DEVICES=0,1
export MASTER_ADDR=127.0.0.1
export MASTER_PORT=29591
export NCCL_DEBUG=WARN
export EP_DISABLE_GIN=1
export EP_JIT_CACHE_DIR=/tmp/deepep_b300_jit_cache

python tests/elastic/test_ep.py \
  --num-processes 2 \
  --num-tokens 128 \
  --hidden 1024 \
  --num-topk 2 \
  --num-experts 16 \
  --allow-hybrid-mode 0 \
  --test-first-only \
  --skip-perf-test

Actual Result

The original code fails in host-side initialization:

csrc/kernels/backend/nccl.cu:87
(allow_hybrid_mode ? props.railedGinType : props.ginType) != NCCL_GIN_TYPE_NONE

On the reproduced system:

props.ginType == NCCL_GIN_TYPE_NONE
props.railedGinType == NCCL_GIN_TYPE_NONE

Originally created by @DanielDanyang on GitHub (May 10, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/629 ## Summary DeepEP V2/elastic still fails during `ElasticBuffer` initialization with `EP_DISABLE_GIN=1` when NCCL reports no Gin capability for the communicator. This blocks pure single-node intra-node EP runs. ## Reproduction Environment used to reproduce: ```text GPU: 8 x NVIDIA B300 SXM6 AC CUDA compiler: /usr/local/cuda-13.2/bin/nvcc PyTorch: 2.13.0.dev20260502+cu130 NCCL: nvidia-nccl-cu13 2.30.4 NVSHMEM: nvidia-nvshmem-cu13 3.4.5 ``` Run a pure intra-node elastic EP smoke test: ```bash export CUDA_VISIBLE_DEVICES=0,1 export MASTER_ADDR=127.0.0.1 export MASTER_PORT=29591 export NCCL_DEBUG=WARN export EP_DISABLE_GIN=1 export EP_JIT_CACHE_DIR=/tmp/deepep_b300_jit_cache python tests/elastic/test_ep.py \ --num-processes 2 \ --num-tokens 128 \ --hidden 1024 \ --num-topk 2 \ --num-experts 16 \ --allow-hybrid-mode 0 \ --test-first-only \ --skip-perf-test ``` ## Actual Result The original code fails in host-side initialization: ```text csrc/kernels/backend/nccl.cu:87 (allow_hybrid_mode ? props.railedGinType : props.ginType) != NCCL_GIN_TYPE_NONE ``` On the reproduced system: ```text props.ginType == NCCL_GIN_TYPE_NONE props.railedGinType == NCCL_GIN_TYPE_NONE ```

bsakaguchi closed this issue

2026-10-26 09:59:44 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#378

No description provided.

Rows
Columns