[GH-ISSUE #658] Does NVSHMEM/DeepEP support SR-IOV Virtual Function devices? #390

New issue

Open

opened 2026-10-26 09:59:54 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:59:54 +00:00

Owner

Originally created by @hmthhh on GitHub (Jun 3, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/658

Question

Does NVSHMEM (used by DeepEP) work with SR-IOV Virtual Function (VF) devices in Kubernetes? We're running sglang with --moe-a2a-backend deepep on 4 nodes with SR-IOV VF devices, and NVSHMEM hangs after initialization.

Environment

4 nodes, each with 8x Hopper + 8x ConnectX-6 HCA (SR-IOV enabled, 8 VFs per HCA)
Kubernetes pods with SR-IOV VF devices allocated
sglang with DeepEP: --moe-a2a-backend deepep --deepep-mode normal
Same config works on bare metal with PF devices

Pod YAML

spec:
  hostIPC: true
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

Launch Command

export NVSHMEM_HCA_LIST=$(ls /sys/class/infiniband/ | tr '\n' ',' | sed 's/,$//' | sed 's/,/:1,/g' | sed 's/$/:1/')
export NVSHMEM_IBGDA_SUPPORT=0
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_IB_ADDR_RANGE=100.0.0.0/8
export NVSHMEM_IB_ADDR_FAMILY=AF_INET

python3 -m sglang.launch_server \
    --model-path $MODELPATH \
    --host 0.0.0.0 \
    --port 30000 \
    --disaggregation-mode prefill \
    --disaggregation-bootstrap-port 8998 \
    --disaggregation-transfer-backend mooncake \
    --moe-dense-tp-size 1 \
    --moe-a2a-backend deepep \
    --deepep-mode normal \
    --page-size 64 \
    --dist-init-addr $HEADPODIP:6379 \
    --enable-dp-lm-head \
    --enable-dp-attention \
    --dp-size 32 \
    --tp-size 32 \
    --mem-fraction-static 0.75 \
    --max-total-tokens 10000 \
    --nnodes 4 \
    --node-rank $LOCAL_RANK \
    --watchdog-timeout 100000 \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --enable-eplb \
    --ep-dispatch-algorithm dynamic \
    --chunked-prefill-size 32768 \
    --served-model-name GLM-5.1-FP8

Symptom

NVSHMEM initializes successfully (ibrc + IBGDA), creates teams, completes SHARP cleanup, but then hangs indefinitely without proceeding to cuIpcOpenMemHandle. All 4 nodes hang at the same point.

Key log lines:

Successfully initialized the transport: ibrc
Successfully initialized the transport: IBGDA
NVSHMEMI_TEAM_GPU_LEADERS: start=0, stride=1, size=4
Skipping NVLINK SHARP resource initialized for team ID: 1
Skipping NVLINK SHARP resource initialized for team ID: 2
Skipping NVLINK SHARP resource initialized for team ID: 4
# HANGS HERE - no further output

With PF devices on bare metal, the log continues with cuIpcOpenMemHandle and DeepGEMM compilation.

Originally created by @hmthhh on GitHub (Jun 3, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/658 ## Question Does NVSHMEM (used by DeepEP) work with SR-IOV Virtual Function (VF) devices in Kubernetes? We're running sglang with `--moe-a2a-backend deepep` on 4 nodes with SR-IOV VF devices, and NVSHMEM hangs after initialization. ## Environment - 4 nodes, each with 8x Hopper + 8x ConnectX-6 HCA (SR-IOV enabled, 8 VFs per HCA) - Kubernetes pods with SR-IOV VF devices allocated - sglang with DeepEP: `--moe-a2a-backend deepep --deepep-mode normal` - Same config works on bare metal with PF devices ## Pod YAML ```yaml spec: hostIPC: true hostNetwork: true dnsPolicy: ClusterFirstWithHostNet ``` ## Launch Command ```bash export NVSHMEM_HCA_LIST=$(ls /sys/class/infiniband/ | tr '\n' ',' | sed 's/,$//' | sed 's/,/:1,/g' | sed 's/$/:1/') export NVSHMEM_IBGDA_SUPPORT=0 export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_IB_ADDR_RANGE=100.0.0.0/8 export NVSHMEM_IB_ADDR_FAMILY=AF_INET python3 -m sglang.launch_server \ --model-path $MODELPATH \ --host 0.0.0.0 \ --port 30000 \ --disaggregation-mode prefill \ --disaggregation-bootstrap-port 8998 \ --disaggregation-transfer-backend mooncake \ --moe-dense-tp-size 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --page-size 64 \ --dist-init-addr $HEADPODIP:6379 \ --enable-dp-lm-head \ --enable-dp-attention \ --dp-size 32 \ --tp-size 32 \ --mem-fraction-static 0.75 \ --max-total-tokens 10000 \ --nnodes 4 \ --node-rank $LOCAL_RANK \ --watchdog-timeout 100000 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-eplb \ --ep-dispatch-algorithm dynamic \ --chunked-prefill-size 32768 \ --served-model-name GLM-5.1-FP8 ``` ## Symptom NVSHMEM initializes successfully (ibrc + IBGDA), creates teams, completes SHARP cleanup, but then **hangs indefinitely** without proceeding to `cuIpcOpenMemHandle`. All 4 nodes hang at the same point. Key log lines: ``` Successfully initialized the transport: ibrc Successfully initialized the transport: IBGDA NVSHMEMI_TEAM_GPU_LEADERS: start=0, stride=1, size=4 Skipping NVLINK SHARP resource initialized for team ID: 1 Skipping NVLINK SHARP resource initialized for team ID: 2 Skipping NVLINK SHARP resource initialized for team ID: 4 # HANGS HERE - no further output ```` With PF devices on bare metal, the log continues with `cuIpcOpenMemHandle` and DeepGEMM compilation.

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@hiSandog commented on GitHub (Jun 25, 2026):

This would be useful to separate into “DeepEP configuration” vs “NVSHMEM on SR-IOV VF” evidence. If possible, please include the relevant NVSHMEM environment variables, the HCA names visible inside the pod, and whether a minimal NVSHMEM/NCCL all-to-all test hangs on the same VF devices without sglang/DeepEP in the loop. Since the same workload works on bare-metal PF devices, the VF exposure of GPUDirect RDMA / peer memory is likely an important part of the triage.

@hiSandog commented on GitHub (Jun 25, 2026): This would be useful to separate into “DeepEP configuration” vs “NVSHMEM on SR-IOV VF” evidence. If possible, please include the relevant NVSHMEM environment variables, the HCA names visible inside the pod, and whether a minimal NVSHMEM/NCCL all-to-all test hangs on the same VF devices without sglang/DeepEP in the loop. Since the same workload works on bare-metal PF devices, the VF exposure of GPUDirect RDMA / peer memory is likely an important part of the triage.

bsakaguchi commented

2026-10-26 09:59:57 +00:00

Author

Owner

@neverhook commented on GitHub (Aug 24, 2026):

You can take https://github.com/deepseek-ai/DeepEP/issues/54#issuecomment-3674325983 as a reference.

@neverhook commented on GitHub (Aug 24, 2026): You can take https://github.com/deepseek-ai/DeepEP/issues/54#issuecomment-3674325983 as a reference.

bsakaguchi referenced this issue

2026-10-27 21:03:53 +00:00

[PR #390] [CLOSED] [Feat] Single Batch Overlap (SBO): Overlaping of Down GEMM with Combine Send #546