deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #128] [low latency] test failed：init failed for transport: IBGDA #112

New issue
Closed
opened 2026-10-26 09:54:31 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 09:54:31 +00:00
Owner
Copy link

Originally created by @allan0703 on GitHub (Apr 21, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/128

hi team，
thanks for your excellent job.
When i deploy the DeepEP on the ML platform with 4 nodes. i've encoutered the following issue:

Allocating buffer size: 2115.111296 MB ...
NCCL version 2.21.5+cuda12.6
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping...

WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping...

/share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA
[rank5]:[W421 15:22:03.760631008 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank3]:[W421 15:22:03.761313250 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank6]:[W421 15:22:03.761462516 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank0]:[W421 15:22:03.761629239 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank2]:[W421 15:22:03.761806848 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank7]:[W421 15:22:03.762468993 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank4]:[W421 15:22:03.762760426 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
[rank1]:[W421 15:22:03.762855825 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent)
W0421 15:22:03.955000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94024 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94025 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94026 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94027 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94029 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94030 via signal SIGTERM
W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94031 via signal SIGTERM
Traceback (most recent call last):
  File "/share/backup/DeepEP/tests/test_low_latency.py", line 172, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes
    while not context.join():
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 203, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/share/backup/DeepEP/tests/test_low_latency.py", line 158, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1)
  File "/share/backup/DeepEP/tests/test_low_latency.py", line 55, in test_main
    num_valid_tokens = recv_count.item()
  File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/utils/_device.py", line 106, in __torch_function__
    return func(*args, **kwargs)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

i have installed the GDRCopy and NVSHMEM following the readme doc. And the gdrcopy_copybw output are as follows:

GPU id:0; name: NVIDIA H800; Bus id: 0000:19:00
GPU id:1; name: NVIDIA H800; Bus id: 0000:2a:00
GPU id:2; name: NVIDIA H800; Bus id: 0000:3b:00
GPU id:3; name: NVIDIA H800; Bus id: 0000:5d:00
GPU id:4; name: NVIDIA H800; Bus id: 0000:9b:00
GPU id:5; name: NVIDIA H800; Bus id: 0000:ab:00
GPU id:6; name: NVIDIA H800; Bus id: 0000:bb:00
GPU id:7; name: NVIDIA H800; Bus id: 0000:db:00
selecting device 0
testing size: 131072
rounded size: 131072
gpu alloc fn: cuMemAlloc
device ptr: 7f027be00000
map_d_ptr: 0x7f049c244000
info.va: 7f027be00000
info.mapped_size: 131072
info.page_size: 65536
info.mapped: 1
info.wc_mapping: 1
page offset: 0
user-space pointer:0x7f049c244000
writing test, size=131072 offset=0 num_iters=10000
write BW: 18577.5MB/s
reading test, size=131072 offset=0 num_iters=100
read BW: 620.811MB/s
unmapping buffer
unpinning buffer
closing gdrdrv
  • nvshmem-info -a
NVSHMEM v3.2.5

Build Information:
  CUDA API                     12060
  CUDA Driver                  12060
  Build Timestamp              Apr 21 2025 11:55:16
  Build Variables             
        NVSHMEM_DEBUG=OFF NVSHMEM_DEVEL=OFF NVSHMEM_DEFAULT_PMI2=OFF
        NVSHMEM_DEFAULT_PMIX=OFF NVSHMEM_DEFAULT_UCX=OFF
        NVSHMEM_ENABLE_ALL_DEVICE_INLINING=OFF NVSHMEM_GPU_COLL_USE_LDST=OFF
        NVSHMEM_IBGDA_SUPPORT=ON NVSHMEM_IBGDA_SUPPORT_GPUMEM_ONLY=OFF
        NVSHMEM_IBDEVX_SUPPORT=OFF NVSHMEM_IBRC_SUPPORT=ON
        NVSHMEM_MPI_SUPPORT=OFF NVSHMEM_NVTX=ON NVSHMEM_PMIX_SUPPORT=OFF
        NVSHMEM_SHMEM_SUPPORT=OFF NVSHMEM_TEST_STATIC_LIB=OFF
        NVSHMEM_TIMEOUT_DEVICE_POLLING=OFF NVSHMEM_TRACE=OFF NVSHMEM_UCX_SUPPORT=OFF
        NVSHMEM_USE_DLMALLOC=OFF NVSHMEM_USE_NCCL=OFF NVSHMEM_USE_GDRCOPY=ON
        NVSHMEM_VERBOSE=OFF CUDA_HOME=/usr/local/cuda GDRCOPY_HOME=/usr/local
        LIBFABRIC_HOME=/usr/local/libfabric MPI_HOME=/usr/local/ompi
        NCCL_HOME=/usr/local/nccl NVSHMEM_PREFIX=/usr/local/nvshmem PMIX_HOME=/usr
        SHMEM_HOME=/usr/local/ompi UCX_HOME=/usr/local/ucx

Standard options:
  NVSHMEM_VERSION              false (type: bool, default: false)
        Print library version at startup
  NVSHMEM_INFO                 false (type: bool, default: false)
        Print environment variable options at startup
  NVSHMEM_DISABLE_NVLS         false (type: bool, default: false)
        Disable NVLS SHARP resources for collectives, even if available for platform
  NVSHMEM_SYMMETRIC_SIZE       1073741824 (type: size, default: 1073741824)
        Specifies the size (in bytes) of the symmetric heap memory per PE. The
        size is implementation-defined and must be at least as large as the integer
        ceiling of the product of the numeric prefix and the scaling factor. The
        character suffixes for the scaling factor are as follows:

          *  k or K multiplies by 2^10 (kibibytes)
          *  m or M multiplies by 2^20 (mebibytes)
          *  g or G multiplies by 2^30 (gibibytes)
          *  t or T multiplies by 2^40 (tebibytes)

        For example, string '20m' is equivalent to the integer value 20971520, or 20
        mebibytes. Similarly the string '3.1M' is equivalent to the integer value
        3250586. Only one multiplier is recognized and any characters following the
        multiplier are ignored, so '20kk' will not produce the same result as '20m'.
        Usage of string '.5m' will yield the same result as the string '0.5m'.
        An invalid value for NVSHMEM_SYMMETRIC_SIZE is an error, which the NVSHMEM
        library shall report by either returning a nonzero value from
        nvshmem_init_thread or causing program termination.
  NVSHMEM_DEBUG                "" (type: string, default: "")
        Set to enable debugging messages.
        Optional values: VERSION, WARN, INFO, ABORT, TRACE

Bootstrap options:
  NVSHMEM_BOOTSTRAP            "PMI" (type: string, default: "PMI")
        Name of the default bootstrap that should be used to initialize NVSHMEM.
        Allowed values: PMI, MPI, SHMEM, plugin, UID
  NVSHMEM_BOOTSTRAP_PMI        "PMI" (type: string, default: "PMI")
        Name of the PMI bootstrap that should be used to initialize NVSHMEM.
        Allowed values: PMI, PMI-2, PMIX
  NVSHMEM_BOOTSTRAP_PLUGIN     "" (type: string, default: "")
        Absolute path to or name of the bootstrap plugin file to load when
        NVSHMEM_BOOTSTRAP=plugin is specified
  NVSHMEM_BOOTSTRAP_MPI_PLUGIN "nvshmem_bootstrap_mpi.so.3" (type: string, default: "nvshmem_bootstrap_mpi.so.3")
        Absolute path to or name of the MPI bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen
  NVSHMEM_BOOTSTRAP_SHMEM_PLUGIN "nvshmem_bootstrap_shmem.so.3" (type: string, default: "nvshmem_bootstrap_shmem.so.3")
        Absolute path to or name of the SHMEM bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen
  NVSHMEM_BOOTSTRAP_PMI_PLUGIN "nvshmem_bootstrap_pmi.so.3" (type: string, default: "nvshmem_bootstrap_pmi.so.3")
        Absolute path to or name of the PMI bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen
  NVSHMEM_BOOTSTRAP_PMI2_PLUGIN "nvshmem_bootstrap_pmi2.so.3" (type: string, default: "nvshmem_bootstrap_pmi2.so.3")
        Absolute path to or name of the PMI-2 bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen
  NVSHMEM_BOOTSTRAP_PMIX_PLUGIN "nvshmem_bootstrap_pmix.so.3" (type: string, default: "nvshmem_bootstrap_pmix.so.3")
        Absolute path to or name of the PMIx bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen
  NVSHMEM_BOOTSTRAP_UID_PLUGIN "nvshmem_bootstrap_uid.so.3" (type: string, default: "nvshmem_bootstrap_uid.so.3")
        Absolute path to or name of the UID bootstrap plugin file. 
        NVSHMEM will search for the plugin based on linux linker priorities. See man
        dlopen

Additional options:
  NVSHMEM_CUDA_PATH            "" (type: string, default: "")
        Path to directory containing libcuda.so (for use when not in default location)
  NVSHMEM_DEBUG_ATTACH_DELAY   0 (type: int, default: 0)
        Delay (in seconds) during the first call to NVSHMEM_INIT to allow for attaching
        a debuggger (Default 0)
  NVSHMEM_DEBUG_FILE           "" (type: string, default: "")
        Debugging output filename, may contain %h for hostname and %p for pid
  NVSHMEM_MAX_TEAMS            32 (type: long, default: 32)
        Maximum number of simultaneous teams allowed
  NVSHMEM_MAX_MEMORY_PER_GPU   137438953472 (type: size, default: 137438953472)
        Maximum memory per GPU
  NVSHMEM_DISABLE_CUDA_VMM     false (type: bool, default: false)
        Disable use of CUDA VMM for P2P memory mapping. By default, CUDA VMM is enabled
        on x86 and disabled on P9. CUDA VMM feature in NVSHMEM requires CUDA RT version
        and CUDA Driver version to be greater than or equal to 11.3.
  NVSHMEM_DISABLE_P2P          false (type: bool, default: false)
        Disable P2P connectivity of GPUs even when available
  NVSHMEM_IGNORE_CUDA_MPS_ACTIVE_THREAD_PERCENTAGE false (type: bool, default: false)
        When doing Multi-Process Per GPU (MPG) run, full API support is available only
        if sum of CUDA_MPS_ACTIVE_THREAD_PERCENTAGE of processes running on a GPU is <=
        100%. Through this variable, user can request NVSHMEM runtime to ignore the
        active thread percentage and allow full MPG support. Users enable it at their
        own risk as NVSHMEM might deadlock.
  NVSHMEM_CUMEM_GRANULARITY    536870912 (type: size, default: 536870912)
        Granularity for cuMemAlloc/cuMemCreate
  NVSHMEM_PROXY_REQUEST_BATCH_MAX 32 (type: int, default: 32)
        Maxmum number of requests that the proxy thread processes in a single iteration
        of the progress loop.

Collectives options:
  NVSHMEM_DISABLE_NCCL         false (type: bool, default: false)
        Disable use of NCCL for collective operations
  NVSHMEM_BARRIER_DISSEM_KVAL  2 (type: int, default: 2)
        Radix of the dissemination algorithm used for barriers
  NVSHMEM_BARRIER_TG_DISSEM_KVAL 2 (type: int, default: 2)
        Radix of the dissemination algorithm used for thread group barriers
  NVSHMEM_FCOLLECT_LL_THRESHOLD 2048 (type: size, default: 2048)
        Message size threshold up to which fcollect LL algo will be used

  NVSHMEM_REDUCE_SCRATCH_SIZE  524288 (type: size, default: 524288)
        Amount of symmetric heap memory (minimum 16B, multiple of 8B) reserved by
        runtime for every team to implement reduce and reducescatter collectives

  NVSHMEM_BCAST_ALGO           0 (type: int, default: 0)
        Broadcast algorithm to be used.
          * 0 - use default algorithm selection strategy

  NVSHMEM_REDMAXLOC_ALGO       1 (type: int, default: 1)
        Reduction algorithm to be used for MAXLOC operation.
          * 1 - default, flag alltoall algorithm
          * 2 - flat reduce + flat bcast
          * 3 - topo-aware two-level reduce + topo-aware bcast


Transport options:
  NVSHMEM_REMOTE_TRANSPORT     "ibrc" (type: string, default: "ibrc")
        Selected transport for remote operations: ibrc, ucx, libfabric, ibdevx, none
  NVSHMEM_ENABLE_NIC_PE_MAPPING false (type: bool, default: false)
        When not set or set to 0, a PE is assigned the NIC on the node that is closest
        to it by distance. When set to 1, NVSHMEM either assigns NICs to PEs on a
        round-robin basis or uses NVSHMEM_HCA_PE_MAPPING or NVSHMEM_HCA_LIST when they
        are specified.
  NVSHMEM_DISABLE_LOCAL_ONLY_PROXY false (type: bool, default: false)
        When running on an NVLink-only configuaration (No-IB, No-UCX), completely
        disable the proxy thread. This will disable device side global exit and device
        side wait timeout polling (enabled by NVSHMEM_TIMEOUT_DEVICE_POLLING build-time
        variable) because these are processed by the proxy thread.
  NVSHMEM_IB_ENABLE_IBGDA      false (type: bool, default: false)
        Set to enable GPU-initiated communication transport.

Hidden options:
  NVSHMEM_INFO_HIDDEN          true (type: bool, default: false)
        Print hidden environment variable options at startup
  NVSHMEM_DISABLE_NVLS_SHARING true (type: bool, default: true)
        Disable NVLS SHARP resource sharing for user-defined teams
  NVSHMEM_HEAP_KIND            "DEVICE" (type: string, default: "DEVICE")
        Specify the memory kind used by the NVSHMEM symmetric heap.
        Allowed values: VIDMEM, SYSMEM
  NVSHMEM_ENABLE_RAIL_OPT      false (type: bool, default: false)
        Enable Rail Optimization when heap is in SYSMEM
  NVSHMEM_BOOTSTRAP_TWO_STAGE  false (type: bool, default: false)
        Ignore CUDA device setting during initialization,forcing two-stage
        initialization
  NVSHMEM_DEBUG_SUBSYS         "" (type: string, default: "")
        Comma separated list of debugging message sources. Prefix with '^' to exclude.
        Values: INIT, COLL, P2P, PROXY, TRANSPORT, MEM, BOOTSTRAP, TOPO, UTIL, ALL
  NVSHMEM_ENABLE_ERROR_CHECKS  false (type: bool, default: false)
        Enable error checks
  NVSHMEM_DISABLE_MNNVL        false (type: bool, default: false)
        Disable MNNVL connectivity for GPUs even when available
  NVSHMEM_CUMEM_HANDLE_TYPE    "FILE_DESCRIPTOR" (type: string, default: "FILE_DESCRIPTOR")
        Handle type for cuMemCreate. Supported are - FABRIC or FILE_DESCRIPTOR
  NVSHMEM_BYPASS_ACCESSIBILITY_CHECK false (type: bool, default: false)
        Bypass peer GPU accessbility checks
  NVSHMEM_FCOLLECT_NTHREADS    512 (type: int, default: 512)
        Sets number of threads per block for fcollect collective.
        By default, if no env is set, default value is min(max_occupancy per CTA, msg
        size per PE).
        If env is specified, value overrides the default irrespective of max occupancy
        per CTA

  NVSHMEM_REDUCESCATTER_NTHREADS 512 (type: int, default: 512)
        Sets number of threads per block for reducescatter collective.
        By default, if no env is set, default value is min(max_occupancy per CTA, msg
        size per PE).
        If env is specified, value overrides the default irrespective of max occupancy
        per CTA

  NVSHMEM_MAX_CTAS             1 (type: int, default: 1)
        Sets number of blocks per grid for host onstream collective.
        By default, if no env is set, default value to 1 CTA
        If env is specified, value overrides the default value

  NVSHMEM_REDUCE_RECEXCH_KVAL  2 (type: int, default: 2)
        Radix of the recursive exchange reduction algorithm
  NVSHMEM_FCOLLECT_LL128_THRESHOLD 0 (type: size, default: 0)
        Message size threshold up to which the fcollect LL128 algo will be used.
        LL128 will be used only when FCOLLECT_LL_THRESHOLD < size
  NVSHMEM_FCOLLECT_NVLS_THRESHOLD 16777216 (type: size, default: 16777216)
        Message size threshold up to which fcollect NVLS algo will be used

  NVSHMEM_REDUCESCATTER_NVLS_THRESHOLD 16777216 (type: size, default: 16777216)
        Message size threshold up to which reducescatter NVLS algo will be used

  NVSHMEM_BCAST_TREE_KVAL      2 (type: int, default: 2)
        Radix of the broadcast tree algorithm
  NVSHMEM_FCOLLECT_ALGO        0 (type: int, default: 0)
        Fcollect algorithm to be used. 
          * 0 - use default algorithm selection strategy

  NVSHMEM_REDUCE_ALGO          0 (type: int, default: 0)
        Allreduce algorithm to be used. 
           * 0/1 - use default algorithm selection strategy

  NVSHMEM_REDUCE_NVLS_THRESHOLD 2048 (type: int, default: 2048)
        Message size threshold up to which allreduce one-shot algo will be used

  NVSHMEM_REDUCESCATTER_ALGO   0 (type: int, default: 0)
        Reduce Scatter algorithm to be used. 
          * 0 - use default algorithm selection strategy

  NVSHMEM_ASSERT_ATOMICS_SYNC  false (type: bool, default: false)
        Bypass flush on wait_until at target
  NVSHMEM_BYPASS_FLUSH         false (type: bool, default: false)
        Bypass flush in proxy when enforcing consistency

NVTX options:
  NVSHMEM_NVTX                 "off" (type: string, default: "off")
        Set to enable NVTX instrumentation. Accepts a comma separated list of
        instrumentation groups. By default the NVTX instrumentation is disabled.
          init                : library setup
          alloc               : memory management
          launch              : kernel launch routines
          coll                : collective communications
          wait                : blocking point-to-point synchronization
          wait_on_stream      : point-to-point synchronization (on stream)
          test                : non-blocking point-to-point synchronization
          memorder            : memory ordering (quiet, fence)
          quiet_on_stream     : nvshmemx_quiet_on_stream
          atomic_fetch        : fetching atomic memory operations
          atomic_set          : non-fetchong atomic memory operations
          rma_blocking        : blocking remote memory access operations
          rma_nonblocking     : non-blocking remote memory access operations
          proxy               : activity of the proxy thread
          common              : init,alloc,launch,coll,memorder,wait,atomic_fetch,rma_blocking,proxy
          all                 : all groups
          off                 : disable all NVTX instrumentation
  • nvidia-smi topo -mp
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU1    SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU2    SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU3    SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU4    SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU5    SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     56-111,168-223  1               N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     56-111,168-223  1               N/A
NIC0    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC2    SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC3    SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS
NIC5    SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS
NIC6    SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS
NIC7    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS
NIC8    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS
NIC9    SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  • ibv_devinfo
hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00f2:80be
	sys_image_guid:			a088:c203:00f2:80be
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_1
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00e6:b170
	sys_image_guid:			a088:c203:00e6:b170
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_2
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00f2:ca1e
	sys_image_guid:			a088:c203:00f2:ca1e
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_3
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00e5:4eaa
	sys_image_guid:			a088:c203:00e5:4eaa
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_4
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			58a2:e103:00c1:c414
	sys_image_guid:			58a2:e103:00c1:c414
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_5
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00f2:a12e
	sys_image_guid:			a088:c203:00f2:a12e
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_6
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00e5:46fa
	sys_image_guid:			a088:c203:00e5:46fa
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_7
	transport:			InfiniBand (0)
	fw_ver:				20.40.1000
	node_guid:			9c63:c003:005a:23fa
	sys_image_guid:			9c63:c003:005a:23fa
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000224
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_8
	transport:			InfiniBand (0)
	fw_ver:				20.40.1000
	node_guid:			9c63:c003:005f:4f2c
	sys_image_guid:			9c63:c003:005f:4f2c
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000224
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_9
	transport:			InfiniBand (0)
	fw_ver:				28.39.2048
	node_guid:			a088:c203:00e5:46b2
	sys_image_guid:			a088:c203:00e5:46b2
	vendor_id:			0x02c9
	vendor_part_id:			4129
	hw_ver:				0x0
	board_id:			MT_0000000838
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

Is there anything wrong?

Originally created by @allan0703 on GitHub (Apr 21, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/128 hi team， thanks for your excellent job. When i deploy the DeepEP on the ML platform with 4 nodes. i've encoutered the following issue: ```python tests/test_low_latency.py Allocating buffer size: 2115.111296 MB ... NCCL version 2.21.5+cuda12.6 WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_4 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_5 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_6 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_7 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA WARN: device mlx5_8 cannot allocate buffer on the specified memory type. Skipping... WARN: device mlx5_9 cannot allocate buffer on the specified memory type. Skipping... /share/backup/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA [rank5]:[W421 15:22:03.760631008 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank3]:[W421 15:22:03.761313250 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank6]:[W421 15:22:03.761462516 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank0]:[W421 15:22:03.761629239 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank2]:[W421 15:22:03.761806848 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank7]:[W421 15:22:03.762468993 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank4]:[W421 15:22:03.762760426 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) [rank1]:[W421 15:22:03.762855825 CUDAGuardImpl.h:119] Warning: CUDA warning: an illegal memory access was encountered (function destroyEvent) W0421 15:22:03.955000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94024 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94025 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94026 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94027 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94029 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94030 via signal SIGTERM W0421 15:22:03.956000 93959 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 94031 via signal SIGTERM Traceback (most recent call last): File "/share/backup/DeepEP/tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes while not context.join(): File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 203, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/share/backup/DeepEP/tests/test_low_latency.py", line 158, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, seed=1) File "/share/backup/DeepEP/tests/test_low_latency.py", line 55, in test_main num_valid_tokens = recv_count.item() File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/utils/_device.py", line 106, in __torch_function__ return func(*args, **kwargs) RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` i have installed the GDRCopy and NVSHMEM following the readme doc. And the **gdrcopy_copybw** output are as follows: ``` GPU id:0; name: NVIDIA H800; Bus id: 0000:19:00 GPU id:1; name: NVIDIA H800; Bus id: 0000:2a:00 GPU id:2; name: NVIDIA H800; Bus id: 0000:3b:00 GPU id:3; name: NVIDIA H800; Bus id: 0000:5d:00 GPU id:4; name: NVIDIA H800; Bus id: 0000:9b:00 GPU id:5; name: NVIDIA H800; Bus id: 0000:ab:00 GPU id:6; name: NVIDIA H800; Bus id: 0000:bb:00 GPU id:7; name: NVIDIA H800; Bus id: 0000:db:00 selecting device 0 testing size: 131072 rounded size: 131072 gpu alloc fn: cuMemAlloc device ptr: 7f027be00000 map_d_ptr: 0x7f049c244000 info.va: 7f027be00000 info.mapped_size: 131072 info.page_size: 65536 info.mapped: 1 info.wc_mapping: 1 page offset: 0 user-space pointer:0x7f049c244000 writing test, size=131072 offset=0 num_iters=10000 write BW: 18577.5MB/s reading test, size=131072 offset=0 num_iters=100 read BW: 620.811MB/s unmapping buffer unpinning buffer closing gdrdrv ``` - nvshmem-info -a ``` NVSHMEM v3.2.5 Build Information: CUDA API 12060 CUDA Driver 12060 Build Timestamp Apr 21 2025 11:55:16 Build Variables NVSHMEM_DEBUG=OFF NVSHMEM_DEVEL=OFF NVSHMEM_DEFAULT_PMI2=OFF NVSHMEM_DEFAULT_PMIX=OFF NVSHMEM_DEFAULT_UCX=OFF NVSHMEM_ENABLE_ALL_DEVICE_INLINING=OFF NVSHMEM_GPU_COLL_USE_LDST=OFF NVSHMEM_IBGDA_SUPPORT=ON NVSHMEM_IBGDA_SUPPORT_GPUMEM_ONLY=OFF NVSHMEM_IBDEVX_SUPPORT=OFF NVSHMEM_IBRC_SUPPORT=ON NVSHMEM_MPI_SUPPORT=OFF NVSHMEM_NVTX=ON NVSHMEM_PMIX_SUPPORT=OFF NVSHMEM_SHMEM_SUPPORT=OFF NVSHMEM_TEST_STATIC_LIB=OFF NVSHMEM_TIMEOUT_DEVICE_POLLING=OFF NVSHMEM_TRACE=OFF NVSHMEM_UCX_SUPPORT=OFF NVSHMEM_USE_DLMALLOC=OFF NVSHMEM_USE_NCCL=OFF NVSHMEM_USE_GDRCOPY=ON NVSHMEM_VERBOSE=OFF CUDA_HOME=/usr/local/cuda GDRCOPY_HOME=/usr/local LIBFABRIC_HOME=/usr/local/libfabric MPI_HOME=/usr/local/ompi NCCL_HOME=/usr/local/nccl NVSHMEM_PREFIX=/usr/local/nvshmem PMIX_HOME=/usr SHMEM_HOME=/usr/local/ompi UCX_HOME=/usr/local/ucx Standard options: NVSHMEM_VERSION false (type: bool, default: false) Print library version at startup NVSHMEM_INFO false (type: bool, default: false) Print environment variable options at startup NVSHMEM_DISABLE_NVLS false (type: bool, default: false) Disable NVLS SHARP resources for collectives, even if available for platform NVSHMEM_SYMMETRIC_SIZE 1073741824 (type: size, default: 1073741824) Specifies the size (in bytes) of the symmetric heap memory per PE. The size is implementation-defined and must be at least as large as the integer ceiling of the product of the numeric prefix and the scaling factor. The character suffixes for the scaling factor are as follows: * k or K multiplies by 2^10 (kibibytes) * m or M multiplies by 2^20 (mebibytes) * g or G multiplies by 2^30 (gibibytes) * t or T multiplies by 2^40 (tebibytes) For example, string '20m' is equivalent to the integer value 20971520, or 20 mebibytes. Similarly the string '3.1M' is equivalent to the integer value 3250586. Only one multiplier is recognized and any characters following the multiplier are ignored, so '20kk' will not produce the same result as '20m'. Usage of string '.5m' will yield the same result as the string '0.5m'. An invalid value for NVSHMEM_SYMMETRIC_SIZE is an error, which the NVSHMEM library shall report by either returning a nonzero value from nvshmem_init_thread or causing program termination. NVSHMEM_DEBUG "" (type: string, default: "") Set to enable debugging messages. Optional values: VERSION, WARN, INFO, ABORT, TRACE Bootstrap options: NVSHMEM_BOOTSTRAP "PMI" (type: string, default: "PMI") Name of the default bootstrap that should be used to initialize NVSHMEM. Allowed values: PMI, MPI, SHMEM, plugin, UID NVSHMEM_BOOTSTRAP_PMI "PMI" (type: string, default: "PMI") Name of the PMI bootstrap that should be used to initialize NVSHMEM. Allowed values: PMI, PMI-2, PMIX NVSHMEM_BOOTSTRAP_PLUGIN "" (type: string, default: "") Absolute path to or name of the bootstrap plugin file to load when NVSHMEM_BOOTSTRAP=plugin is specified NVSHMEM_BOOTSTRAP_MPI_PLUGIN "nvshmem_bootstrap_mpi.so.3" (type: string, default: "nvshmem_bootstrap_mpi.so.3") Absolute path to or name of the MPI bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_SHMEM_PLUGIN "nvshmem_bootstrap_shmem.so.3" (type: string, default: "nvshmem_bootstrap_shmem.so.3") Absolute path to or name of the SHMEM bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMI_PLUGIN "nvshmem_bootstrap_pmi.so.3" (type: string, default: "nvshmem_bootstrap_pmi.so.3") Absolute path to or name of the PMI bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMI2_PLUGIN "nvshmem_bootstrap_pmi2.so.3" (type: string, default: "nvshmem_bootstrap_pmi2.so.3") Absolute path to or name of the PMI-2 bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMIX_PLUGIN "nvshmem_bootstrap_pmix.so.3" (type: string, default: "nvshmem_bootstrap_pmix.so.3") Absolute path to or name of the PMIx bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_UID_PLUGIN "nvshmem_bootstrap_uid.so.3" (type: string, default: "nvshmem_bootstrap_uid.so.3") Absolute path to or name of the UID bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen Additional options: NVSHMEM_CUDA_PATH "" (type: string, default: "") Path to directory containing libcuda.so (for use when not in default location) NVSHMEM_DEBUG_ATTACH_DELAY 0 (type: int, default: 0) Delay (in seconds) during the first call to NVSHMEM_INIT to allow for attaching a debuggger (Default 0) NVSHMEM_DEBUG_FILE "" (type: string, default: "") Debugging output filename, may contain %h for hostname and %p for pid NVSHMEM_MAX_TEAMS 32 (type: long, default: 32) Maximum number of simultaneous teams allowed NVSHMEM_MAX_MEMORY_PER_GPU 137438953472 (type: size, default: 137438953472) Maximum memory per GPU NVSHMEM_DISABLE_CUDA_VMM false (type: bool, default: false) Disable use of CUDA VMM for P2P memory mapping. By default, CUDA VMM is enabled on x86 and disabled on P9. CUDA VMM feature in NVSHMEM requires CUDA RT version and CUDA Driver version to be greater than or equal to 11.3. NVSHMEM_DISABLE_P2P false (type: bool, default: false) Disable P2P connectivity of GPUs even when available NVSHMEM_IGNORE_CUDA_MPS_ACTIVE_THREAD_PERCENTAGE false (type: bool, default: false) When doing Multi-Process Per GPU (MPG) run, full API support is available only if sum of CUDA_MPS_ACTIVE_THREAD_PERCENTAGE of processes running on a GPU is <= 100%. Through this variable, user can request NVSHMEM runtime to ignore the active thread percentage and allow full MPG support. Users enable it at their own risk as NVSHMEM might deadlock. NVSHMEM_CUMEM_GRANULARITY 536870912 (type: size, default: 536870912) Granularity for cuMemAlloc/cuMemCreate NVSHMEM_PROXY_REQUEST_BATCH_MAX 32 (type: int, default: 32) Maxmum number of requests that the proxy thread processes in a single iteration of the progress loop. Collectives options: NVSHMEM_DISABLE_NCCL false (type: bool, default: false) Disable use of NCCL for collective operations NVSHMEM_BARRIER_DISSEM_KVAL 2 (type: int, default: 2) Radix of the dissemination algorithm used for barriers NVSHMEM_BARRIER_TG_DISSEM_KVAL 2 (type: int, default: 2) Radix of the dissemination algorithm used for thread group barriers NVSHMEM_FCOLLECT_LL_THRESHOLD 2048 (type: size, default: 2048) Message size threshold up to which fcollect LL algo will be used NVSHMEM_REDUCE_SCRATCH_SIZE 524288 (type: size, default: 524288) Amount of symmetric heap memory (minimum 16B, multiple of 8B) reserved by runtime for every team to implement reduce and reducescatter collectives NVSHMEM_BCAST_ALGO 0 (type: int, default: 0) Broadcast algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_REDMAXLOC_ALGO 1 (type: int, default: 1) Reduction algorithm to be used for MAXLOC operation. * 1 - default, flag alltoall algorithm * 2 - flat reduce + flat bcast * 3 - topo-aware two-level reduce + topo-aware bcast Transport options: NVSHMEM_REMOTE_TRANSPORT "ibrc" (type: string, default: "ibrc") Selected transport for remote operations: ibrc, ucx, libfabric, ibdevx, none NVSHMEM_ENABLE_NIC_PE_MAPPING false (type: bool, default: false) When not set or set to 0, a PE is assigned the NIC on the node that is closest to it by distance. When set to 1, NVSHMEM either assigns NICs to PEs on a round-robin basis or uses NVSHMEM_HCA_PE_MAPPING or NVSHMEM_HCA_LIST when they are specified. NVSHMEM_DISABLE_LOCAL_ONLY_PROXY false (type: bool, default: false) When running on an NVLink-only configuaration (No-IB, No-UCX), completely disable the proxy thread. This will disable device side global exit and device side wait timeout polling (enabled by NVSHMEM_TIMEOUT_DEVICE_POLLING build-time variable) because these are processed by the proxy thread. NVSHMEM_IB_ENABLE_IBGDA false (type: bool, default: false) Set to enable GPU-initiated communication transport. Hidden options: NVSHMEM_INFO_HIDDEN true (type: bool, default: false) Print hidden environment variable options at startup NVSHMEM_DISABLE_NVLS_SHARING true (type: bool, default: true) Disable NVLS SHARP resource sharing for user-defined teams NVSHMEM_HEAP_KIND "DEVICE" (type: string, default: "DEVICE") Specify the memory kind used by the NVSHMEM symmetric heap. Allowed values: VIDMEM, SYSMEM NVSHMEM_ENABLE_RAIL_OPT false (type: bool, default: false) Enable Rail Optimization when heap is in SYSMEM NVSHMEM_BOOTSTRAP_TWO_STAGE false (type: bool, default: false) Ignore CUDA device setting during initialization,forcing two-stage initialization NVSHMEM_DEBUG_SUBSYS "" (type: string, default: "") Comma separated list of debugging message sources. Prefix with '^' to exclude. Values: INIT, COLL, P2P, PROXY, TRANSPORT, MEM, BOOTSTRAP, TOPO, UTIL, ALL NVSHMEM_ENABLE_ERROR_CHECKS false (type: bool, default: false) Enable error checks NVSHMEM_DISABLE_MNNVL false (type: bool, default: false) Disable MNNVL connectivity for GPUs even when available NVSHMEM_CUMEM_HANDLE_TYPE "FILE_DESCRIPTOR" (type: string, default: "FILE_DESCRIPTOR") Handle type for cuMemCreate. Supported are - FABRIC or FILE_DESCRIPTOR NVSHMEM_BYPASS_ACCESSIBILITY_CHECK false (type: bool, default: false) Bypass peer GPU accessbility checks NVSHMEM_FCOLLECT_NTHREADS 512 (type: int, default: 512) Sets number of threads per block for fcollect collective. By default, if no env is set, default value is min(max_occupancy per CTA, msg size per PE). If env is specified, value overrides the default irrespective of max occupancy per CTA NVSHMEM_REDUCESCATTER_NTHREADS 512 (type: int, default: 512) Sets number of threads per block for reducescatter collective. By default, if no env is set, default value is min(max_occupancy per CTA, msg size per PE). If env is specified, value overrides the default irrespective of max occupancy per CTA NVSHMEM_MAX_CTAS 1 (type: int, default: 1) Sets number of blocks per grid for host onstream collective. By default, if no env is set, default value to 1 CTA If env is specified, value overrides the default value NVSHMEM_REDUCE_RECEXCH_KVAL 2 (type: int, default: 2) Radix of the recursive exchange reduction algorithm NVSHMEM_FCOLLECT_LL128_THRESHOLD 0 (type: size, default: 0) Message size threshold up to which the fcollect LL128 algo will be used. LL128 will be used only when FCOLLECT_LL_THRESHOLD < size NVSHMEM_FCOLLECT_NVLS_THRESHOLD 16777216 (type: size, default: 16777216) Message size threshold up to which fcollect NVLS algo will be used NVSHMEM_REDUCESCATTER_NVLS_THRESHOLD 16777216 (type: size, default: 16777216) Message size threshold up to which reducescatter NVLS algo will be used NVSHMEM_BCAST_TREE_KVAL 2 (type: int, default: 2) Radix of the broadcast tree algorithm NVSHMEM_FCOLLECT_ALGO 0 (type: int, default: 0) Fcollect algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_REDUCE_ALGO 0 (type: int, default: 0) Allreduce algorithm to be used. * 0/1 - use default algorithm selection strategy NVSHMEM_REDUCE_NVLS_THRESHOLD 2048 (type: int, default: 2048) Message size threshold up to which allreduce one-shot algo will be used NVSHMEM_REDUCESCATTER_ALGO 0 (type: int, default: 0) Reduce Scatter algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_ASSERT_ATOMICS_SYNC false (type: bool, default: false) Bypass flush on wait_until at target NVSHMEM_BYPASS_FLUSH false (type: bool, default: false) Bypass flush in proxy when enforcing consistency NVTX options: NVSHMEM_NVTX "off" (type: string, default: "off") Set to enable NVTX instrumentation. Accepts a comma separated list of instrumentation groups. By default the NVTX instrumentation is disabled. init : library setup alloc : memory management launch : kernel launch routines coll : collective communications wait : blocking point-to-point synchronization wait_on_stream : point-to-point synchronization (on stream) test : non-blocking point-to-point synchronization memorder : memory ordering (quiet, fence) quiet_on_stream : nvshmemx_quiet_on_stream atomic_fetch : fetching atomic memory operations atomic_set : non-fetchong atomic memory operations rma_blocking : blocking remote memory access operations rma_nonblocking : non-blocking remote memory access operations proxy : activity of the proxy thread common : init,alloc,launch,coll,memorder,wait,atomic_fetch,rma_blocking,proxy all : all groups off : disable all NVTX instrumentation ``` - nvidia-smi topo -mp ``` GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A GPU1 SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A GPU2 SYS SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A GPU3 SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS 0-55,112-167 0 N/A GPU4 SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS 56-111,168-223 1 N/A GPU5 SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS 56-111,168-223 1 N/A GPU6 SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS 56-111,168-223 1 N/A GPU7 SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX 56-111,168-223 1 N/A NIC0 PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS NIC1 SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS NIC2 SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS NIC3 SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS NIC5 SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS NIC6 SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS NIC7 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX SYS NIC8 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X SYS NIC9 SYS SYS SYS SYS SYS SYS SYS PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 ``` - ibv_devinfo ``` hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00f2:80be sys_image_guid: a088:c203:00f2:80be vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00e6:b170 sys_image_guid: a088:c203:00e6:b170 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_2 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00f2:ca1e sys_image_guid: a088:c203:00f2:ca1e vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_3 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00e5:4eaa sys_image_guid: a088:c203:00e5:4eaa vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_4 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: 58a2:e103:00c1:c414 sys_image_guid: 58a2:e103:00c1:c414 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_5 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00f2:a12e sys_image_guid: a088:c203:00f2:a12e vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_6 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00e5:46fa sys_image_guid: a088:c203:00e5:46fa vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_7 transport: InfiniBand (0) fw_ver: 20.40.1000 node_guid: 9c63:c003:005a:23fa sys_image_guid: 9c63:c003:005a:23fa vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000224 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_8 transport: InfiniBand (0) fw_ver: 20.40.1000 node_guid: 9c63:c003:005f:4f2c sys_image_guid: 9c63:c003:005f:4f2c vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000224 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_9 transport: InfiniBand (0) fw_ver: 28.39.2048 node_guid: a088:c203:00e5:46b2 sys_image_guid: a088:c203:00e5:46b2 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet ``` Is there anything wrong?
bsakaguchi commented 2026-10-26 09:54:32 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Apr 22, 2025):

Can you check if nvidia-peermem is correctly installed?

<!-- gh-comment-id:2819926978 --> @sphish commented on GitHub (Apr 22, 2025): Can you check if nvidia-peermem is correctly installed?
bsakaguchi commented 2026-10-26 09:54:32 +00:00
Author
Owner
Copy link

@Cydia2018 commented on GitHub (Apr 24, 2025):

I encountered the same issue "nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA"

<!-- gh-comment-id:2826582362 --> @Cydia2018 commented on GitHub (Apr 24, 2025): I encountered the same issue "nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:275: init failed for transport: IBGDA"
bsakaguchi commented 2026-10-26 09:54:32 +00:00
Author
Owner
Copy link

@allan0703 commented on GitHub (Apr 27, 2025):

Thank you for your reply.

I checked my software versions. I am using an IB network card, but the installation locations of the network card drivers on my 2 of 4 nodes were different, and one of the network cards had not loaded nvidia-peermem modules.

So, I reinstalled the network card drivers and found that nvidia-peermem was loaded (it might not be loaded on some machines).

Finally, I am able to successfully run test_internode.py

<!-- gh-comment-id:2833479329 --> @allan0703 commented on GitHub (Apr 27, 2025): Thank you for your reply. I checked my software versions. I am using an IB network card, but the installation locations of the network card drivers on my 2 of 4 nodes were different, and one of the network cards had not loaded `nvidia-peermem` modules. So, I reinstalled the network card drivers and found that nvidia-peermem was loaded (it might not be loaded on some machines). Finally, I am able to successfully run `test_internode.py`
bsakaguchi commented 2026-10-26 09:54:32 +00:00
Author
Owner
Copy link

@yejingfu commented on GitHub (May 1, 2025):

@allan0703 did you resolved this issue?
I got this problem as well when running test_lowlatency.py in a single node.

<!-- gh-comment-id:2844199415 --> @yejingfu commented on GitHub (May 1, 2025): @allan0703 did you resolved this issue? I got this problem as well when running test_lowlatency.py in a single node.
bsakaguchi commented 2026-10-26 09:54:33 +00:00
Author
Owner
Copy link

@mgw2168-1 commented on GitHub (May 15, 2025):

@allan0703 did you resolved this issue? I got this problem as well when running test_lowlatency.py in a single node.

me too

<!-- gh-comment-id:2882080710 --> @mgw2168-1 commented on GitHub (May 15, 2025): > [@allan0703](https://github.com/allan0703) did you resolved this issue? I got this problem as well when running test_lowlatency.py in a single node. me too
bsakaguchi commented 2026-10-26 09:54:33 +00:00
Author
Owner
Copy link

@oliverYoung2001 commented on GitHub (Jun 13, 2025):

@allan0703 did you resolved this issue? I got this problem as well when running test_lowlatency.py in a single node.

me too

<!-- gh-comment-id:2969656263 --> @oliverYoung2001 commented on GitHub (Jun 13, 2025): > [@allan0703](https://github.com/allan0703) did you resolved this issue? I got this problem as well when running test_lowlatency.py in a single node. me too
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#112
No description provided.