[GH-ISSUE #789] EngramBuffer: single-node NVLink support without RDMA devices? #423

New issue

Open

opened 2026-10-26 10:00:24 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 10:00:24 +00:00

Owner

Originally created by @peters on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/789

EngramBuffer initialization fails with CPU- or GPU-backed storage on a single NVLink node. Ordinary NCCL all_reduce passes on both ranks. This was a DeepInfer preflight; no model weights were loaded.

Environment

DeepEP 93eb6eb238127e96c6d7a4a625a6dad158348509, pinned by DeepInfer 8b16149741a4935e1dec8cea27853f273b374c5b.
RunPod container, 2 × B300 SXM6 AC; nvidia-smi topo -m reports NV18 between GPUs.
Python 3.12, PyTorch 2.13.0+cu132, loaded NCCL 2.32.3 (verified with ncclGetVersion).
Driver 595.91.07; CUDA compiler 13.3.33 with cuda-compat-13-3 selected at runtime.
Host NICs appear in /sys/class/infiniband, but the container has no /dev/infiniband device nodes.

Reproducer, reduced to the failing constructor from the tested probe. Save as repro.py in an environment with the pinned DeepEP build:

import sys
from datetime import timedelta
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from deep_ep import EngramBuffer


def worker(rank, cpu_storage):
    torch.cuda.set_device(rank)
    dist.init_process_group(
        "nccl", init_method="tcp://127.0.0.1:29741",
        rank=rank, world_size=2, device_id=torch.device("cuda", rank),
        timeout=timedelta(seconds=90),
    )
    group = dist.group.WORLD
    value = torch.tensor([rank + 1], device="cuda")
    dist.all_reduce(value)
    assert value.item() == 3
    print(f"rank {rank}: all_reduce passed", flush=True)
    gpu_bytes, storage_bytes = EngramBuffer.get_storage_size_hint(
        group, [256], 256, 8, 4, torch.bfloat16, 0
    )
    qps, depth, restrict = EngramBuffer.get_theoretical_config(group, 1, 8, 4)
    buffer = EngramBuffer(
        group, gpu_bytes, storage_bytes, cpu_storage, qps, depth, restrict,
        allow_hybrid_mode=True, explicitly_destroy=True, num_gpu_timeout_secs=30,
    )
    buffer.destroy()
    dist.destroy_process_group()


if __name__ == "__main__":
    mp.spawn(worker, args=(sys.argv[1] == "cpu",), nprocs=2, join=True)

Run separately:

python repro.py cpu
python repro.py gpu

Both tested storage modes fail at csrc/kernels/comm/context.cpp:125:

(allow_hybrid_mode ? props.railedGinType : props.ginType) != NCCL_GIN_TYPE_NONE
NCCL GIN is unavailable.

Is single-node NVLink-only Engram supported, or is exposed RDMA/GIN required even for GPU-backed storage? If required, documenting that prerequisite would help. Related: #629 concerns ElasticBuffer; this reproduces in EngramBuffer with allow_hybrid_mode=True.

Originally created by @peters on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/789 `EngramBuffer` initialization fails with CPU- or GPU-backed storage on a single NVLink node. Ordinary NCCL `all_reduce` passes on both ranks. This was a DeepInfer preflight; no model weights were loaded. **Environment** - DeepEP `93eb6eb238127e96c6d7a4a625a6dad158348509`, pinned by DeepInfer `8b16149741a4935e1dec8cea27853f273b374c5b`. - RunPod container, 2 × B300 SXM6 AC; `nvidia-smi topo -m` reports `NV18` between GPUs. - Python 3.12, PyTorch `2.13.0+cu132`, loaded NCCL **2.32.3** (verified with `ncclGetVersion`). - Driver `595.91.07`; CUDA compiler `13.3.33` with `cuda-compat-13-3` selected at runtime. - Host NICs appear in `/sys/class/infiniband`, but the container has **no `/dev/infiniband` device nodes**. **Reproducer**, reduced to the failing constructor from the tested probe. Save as `repro.py` in an environment with the pinned DeepEP build: ```python import sys from datetime import timedelta import torch import torch.distributed as dist import torch.multiprocessing as mp from deep_ep import EngramBuffer def worker(rank, cpu_storage): torch.cuda.set_device(rank) dist.init_process_group( "nccl", init_method="tcp://127.0.0.1:29741", rank=rank, world_size=2, device_id=torch.device("cuda", rank), timeout=timedelta(seconds=90), ) group = dist.group.WORLD value = torch.tensor([rank + 1], device="cuda") dist.all_reduce(value) assert value.item() == 3 print(f"rank {rank}: all_reduce passed", flush=True) gpu_bytes, storage_bytes = EngramBuffer.get_storage_size_hint( group, [256], 256, 8, 4, torch.bfloat16, 0 ) qps, depth, restrict = EngramBuffer.get_theoretical_config(group, 1, 8, 4) buffer = EngramBuffer( group, gpu_bytes, storage_bytes, cpu_storage, qps, depth, restrict, allow_hybrid_mode=True, explicitly_destroy=True, num_gpu_timeout_secs=30, ) buffer.destroy() dist.destroy_process_group() if __name__ == "__main__": mp.spawn(worker, args=(sys.argv[1] == "cpu",), nprocs=2, join=True) ``` Run separately: ```sh python repro.py cpu python repro.py gpu ``` Both tested storage modes fail at `csrc/kernels/comm/context.cpp:125`: ```text (allow_hybrid_mode ? props.railedGinType : props.ginType) != NCCL_GIN_TYPE_NONE NCCL GIN is unavailable. ``` Is single-node NVLink-only Engram supported, or is exposed RDMA/GIN required even for GPU-backed storage? If required, documenting that prerequisite would help. Related: #629 concerns `ElasticBuffer`; this reproduces in `EngramBuffer` with `allow_hybrid_mode=True`.

bsakaguchi commented

2026-10-26 10:00:26 +00:00

Author

Owner

@kbespalov commented on GitHub (Oct 21, 2026):

Another data point, with RDMA devices present: same failure, different cause.

8×B200 on one node, driver 615.71.10. DeepInfer 9bd9cf2 (DeepEP f6aa38fe), DeepSeek-V4.1-Flash, EP8.
Two ConnectX-7 ports, both Active / LinkUp (InfiniBand); /dev/infiniband is present.
NCCL picks NET/IB with GIN_TYPE=3, then GDAKI context creation fails on every rank:

transport/net_ib/gdaki/gin_host_gdaki.cc:277 (exchange_info) -> 6
transport/net_ib/gdaki/gin_host_gdaki.cc:745 (ncclGinGdakiCreateContext) -> 6
gin/gin_host.cc:356 (ginDevCommSetupWithBackend) -> 6
dev_runtime.cc:2070 (ncclDevCommCreate) -> 3
RuntimeError: NCCL exception (csrc/kernels/comm/context.cpp:164): 3 (GIN: DevComm setup failed on all available backends)

With NCCL_IB_DISABLE=1, NCCL falls back to Socket (globalGinSupport 0), and the GIN assertion in context.cpp:125 fires.
With EP_DISABLE_GIN=1, the first forward pass dies with cudaErrorIllegalAddress / cudaErrorMisalignedAddress at comm/barrier.hpp:19, with or without MLOPart.

The same DeepInfer build ran fine the day before on an 8×B300 VM from the same cloud, so it comes down to whether the host's RDMA setup supports GDAKI. A single NVLink node needs no RDMA, so an NVLink path like deepseek-ai/DeepInfer#99 would cover both cases.

@kbespalov commented on GitHub (Oct 21, 2026): Another data point, with RDMA devices present: same failure, different cause. - 8×B200 on one node, driver 615.71.10. DeepInfer `9bd9cf2` (DeepEP `f6aa38fe`), DeepSeek-V4.1-Flash, EP8. - Two ConnectX-7 ports, both `Active` / `LinkUp` (InfiniBand); `/dev/infiniband` is present. - NCCL picks `NET/IB` with `GIN_TYPE=3`, then GDAKI context creation fails on every rank: ``` transport/net_ib/gdaki/gin_host_gdaki.cc:277 (exchange_info) -> 6 transport/net_ib/gdaki/gin_host_gdaki.cc:745 (ncclGinGdakiCreateContext) -> 6 gin/gin_host.cc:356 (ginDevCommSetupWithBackend) -> 6 dev_runtime.cc:2070 (ncclDevCommCreate) -> 3 RuntimeError: NCCL exception (csrc/kernels/comm/context.cpp:164): 3 (GIN: DevComm setup failed on all available backends) ``` - With `NCCL_IB_DISABLE=1`, NCCL falls back to Socket (`globalGinSupport 0`), and the GIN assertion in `context.cpp:125` fires. - With `EP_DISABLE_GIN=1`, the first forward pass dies with `cudaErrorIllegalAddress` / `cudaErrorMisalignedAddress` at `comm/barrier.hpp:19`, with or without MLOPart. The same DeepInfer build ran fine the day before on an 8×B300 VM from the same cloud, so it comes down to whether the host's RDMA setup supports GDAKI. A single NVLink node needs no RDMA, so an NVLink path like deepseek-ai/DeepInfer#99 would cover both cases.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#423

No description provided.

Rows
Columns