deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #223] [ERROR] Run test_internode.py failed in two nodes #174

New issue
Closed
opened 2026-10-26 09:55:47 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:55:47 +00:00
Owner
Copy link

Originally created by @JoeyYoung on GitHub (Jun 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/223

Description

I use docker container built from https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile.deepep

There are two nodes. Each node has x8 H20 GPUs and can run test_intranode.py normally. But they fail to run test_internode.py.

When I use the latest main codebase, I encounter the error init failed for transport: IBGDA, so I checkout the commit version 82dcf48 to run internode tests, which uses IBRC, according to discussions in #163.

Environment

nvidia-smi topo -m

	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	NIC9	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	NODE	NODE	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NODE	NODE	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NODE	NODE	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NODE	NODE	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	48-95,144-191	1	N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	48-95,144-191	1	N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PIX	NODE	48-95,144-191	1	N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	NODE	PIX	48-95,144-191	1	N/A
NIC0	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	 X 	PIX	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS
NIC1	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	PIX	 X 	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS
NIC2	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	NODE	NODE	 X 	NODE	NODE	NODE	SYS	SYS	SYS	SYS
NIC3	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	NODE	NODE	NODE	 X 	NODE	NODE	SYS	SYS	SYS	SYS
NIC4	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	 X 	NODE	SYS	SYS	SYS	SYS
NIC5	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	NODE	 X 	SYS	SYS	SYS	SYS
NIC6	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	NODE	NODE
NIC7	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	NODE	NODE
NIC8	SYS	SYS	SYS	SYS	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	 X 	NODE
NIC9	SYS	SYS	SYS	SYS	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	NODE	 X

ibv_devinfo

hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				22.39.3004
	node_guid:			c470:bd03:00a7:8bf0
	sys_image_guid:			c470:bd03:00a7:8bf0
	vendor_id:			0x02c9
	vendor_part_id:			4125
	hw_ver:				0x0
	board_id:			MT_0000000327
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_1
	transport:			InfiniBand (0)
	fw_ver:				22.39.3004
	node_guid:			c470:bd03:00a7:8bf1
	sys_image_guid:			c470:bd03:00a7:8bf0
	vendor_id:			0x02c9
	vendor_part_id:			4125
	hw_ver:				0x0
	board_id:			MT_0000000327
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		1024 (3)
			sm_lid:			0
			port_lid:		0
			port_lmc:		0x00
			link_layer:		Ethernet

hca_id:	mlx5_2
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00ba:3524
	sys_image_guid:			b8e9:2403:00ba:3524
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			9
			port_lid:		7
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_3
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00d5:ac84
	sys_image_guid:			b8e9:2403:00d5:ac84
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			9
			port_lid:		8
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_4
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00d5:adbc
	sys_image_guid:			b8e9:2403:00d5:adbc
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_5
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00d5:a30c
	sys_image_guid:			b8e9:2403:00d5:a30c
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_6
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00d5:a2d8
	sys_image_guid:			b8e9:2403:00d5:a2d8
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_7
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00ba:3814
	sys_image_guid:			b8e9:2403:00ba:3814
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_8
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:0036:99d4
	sys_image_guid:			b8e9:2403:0036:99d4
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

hca_id:	mlx5_9
	transport:			InfiniBand (0)
	fw_ver:				20.43.2026
	node_guid:			b8e9:2403:00ba:36b0
	sys_image_guid:			b8e9:2403:00ba:36b0
	vendor_id:			0x02c9
	vendor_part_id:			4123
	hw_ver:				0x0
	board_id:			MT_0000000223
	phys_port_cnt:			1
		port:	1
			state:			PORT_DOWN (1)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

gdrcopy_copybw

GPU id:0; name: NVIDIA H20-3e; Bus id: 0000:2a:00
GPU id:1; name: NVIDIA H20-3e; Bus id: 0000:3a:00
GPU id:2; name: NVIDIA H20-3e; Bus id: 0000:4c:00
GPU id:3; name: NVIDIA H20-3e; Bus id: 0000:5c:00
GPU id:4; name: NVIDIA H20-3e; Bus id: 0000:aa:00
GPU id:5; name: NVIDIA H20-3e; Bus id: 0000:ba:00
GPU id:6; name: NVIDIA H20-3e; Bus id: 0000:ca:00
GPU id:7; name: NVIDIA H20-3e; Bus id: 0000:da:00
selecting device 0
testing size: 131072
rounded size: 131072
gpu alloc fn: cuMemAlloc
device ptr: 7f9461400000
gdr_open error: Is gdrdrv driver installed and loaded?

nvshmem-info -a

NVSHMEM v3.2.5

Build Information:
  CUDA API                     12040
  CUDA Driver                  12040
  Build Timestamp              Apr  2 2025 13:57:48
  Build Variables
	NVSHMEM_DEBUG=OFF NVSHMEM_DEVEL=OFF NVSHMEM_DEFAULT_PMI2=OFF
	NVSHMEM_DEFAULT_PMIX=OFF NVSHMEM_DEFAULT_UCX=OFF
	NVSHMEM_ENABLE_ALL_DEVICE_INLINING=OFF NVSHMEM_GPU_COLL_USE_LDST=OFF
	NVSHMEM_IBGDA_SUPPORT=ON NVSHMEM_IBGDA_SUPPORT_GPUMEM_ONLY=OFF
	NVSHMEM_IBDEVX_SUPPORT=OFF NVSHMEM_IBRC_SUPPORT=ON
	NVSHMEM_MPI_SUPPORT=OFF NVSHMEM_NVTX=ON NVSHMEM_PMIX_SUPPORT=OFF
	NVSHMEM_SHMEM_SUPPORT=OFF NVSHMEM_TEST_STATIC_LIB=OFF
	NVSHMEM_TIMEOUT_DEVICE_POLLING=OFF NVSHMEM_TRACE=OFF NVSHMEM_UCX_SUPPORT=OFF
	NVSHMEM_USE_DLMALLOC=OFF NVSHMEM_USE_NCCL=OFF NVSHMEM_USE_GDRCOPY=ON
	NVSHMEM_VERBOSE=OFF CUDA_HOME=/usr/local/cuda
	LIBFABRIC_HOME=/usr/local/libfabric MPI_HOME=/usr/local/ompi
	NCCL_HOME=/usr/local/nccl NVSHMEM_PREFIX=/usr/local/nvshmem PMIX_HOME=/usr
	SHMEM_HOME=/usr/local/ompi UCX_HOME=/usr/local/ucx

Standard options:
  NVSHMEM_VERSION              false (type: bool, default: false)
	Print library version at startup
  NVSHMEM_INFO                 false (type: bool, default: false)
	Print environment variable options at startup
  NVSHMEM_DISABLE_NVLS         false (type: bool, default: false)
	Disable NVLS SHARP resources for collectives, even if available for platform
  NVSHMEM_SYMMETRIC_SIZE       1073741824 (type: size, default: 1073741824)
	Specifies the size (in bytes) of the symmetric heap memory per PE. The
	size is implementation-defined and must be at least as large as the integer
	ceiling of the product of the numeric prefix and the scaling factor. The
	character suffixes for the scaling factor are as follows:

	  *  k or K multiplies by 2^10 (kibibytes)
	  *  m or M multiplies by 2^20 (mebibytes)
	  *  g or G multiplies by 2^30 (gibibytes)
	  *  t or T multiplies by 2^40 (tebibytes)

	For example, string '20m' is equivalent to the integer value 20971520, or 20
	mebibytes. Similarly the string '3.1M' is equivalent to the integer value
	3250586. Only one multiplier is recognized and any characters following the
	multiplier are ignored, so '20kk' will not produce the same result as '20m'.
	Usage of string '.5m' will yield the same result as the string '0.5m'.
	An invalid value for NVSHMEM_SYMMETRIC_SIZE is an error, which the NVSHMEM
	library shall report by either returning a nonzero value from
	nvshmem_init_thread or causing program termination.
  NVSHMEM_DEBUG                "" (type: string, default: "")
	Set to enable debugging messages.
	Optional values: VERSION, WARN, INFO, ABORT, TRACE

Bootstrap options:
  NVSHMEM_BOOTSTRAP            "PMI" (type: string, default: "PMI")
	Name of the default bootstrap that should be used to initialize NVSHMEM.
	Allowed values: PMI, MPI, SHMEM, plugin, UID
  NVSHMEM_BOOTSTRAP_PMI        "PMI" (type: string, default: "PMI")
	Name of the PMI bootstrap that should be used to initialize NVSHMEM.
	Allowed values: PMI, PMI-2, PMIX
  NVSHMEM_BOOTSTRAP_PLUGIN     "" (type: string, default: "")
	Absolute path to or name of the bootstrap plugin file to load when
	NVSHMEM_BOOTSTRAP=plugin is specified
  NVSHMEM_BOOTSTRAP_MPI_PLUGIN "nvshmem_bootstrap_mpi.so.3" (type: string, default: "nvshmem_bootstrap_mpi.so.3")
	Absolute path to or name of the MPI bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen
  NVSHMEM_BOOTSTRAP_SHMEM_PLUGIN "nvshmem_bootstrap_shmem.so.3" (type: string, default: "nvshmem_bootstrap_shmem.so.3")
	Absolute path to or name of the SHMEM bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen
  NVSHMEM_BOOTSTRAP_PMI_PLUGIN "nvshmem_bootstrap_pmi.so.3" (type: string, default: "nvshmem_bootstrap_pmi.so.3")
	Absolute path to or name of the PMI bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen
  NVSHMEM_BOOTSTRAP_PMI2_PLUGIN "nvshmem_bootstrap_pmi2.so.3" (type: string, default: "nvshmem_bootstrap_pmi2.so.3")
	Absolute path to or name of the PMI-2 bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen
  NVSHMEM_BOOTSTRAP_PMIX_PLUGIN "nvshmem_bootstrap_pmix.so.3" (type: string, default: "nvshmem_bootstrap_pmix.so.3")
	Absolute path to or name of the PMIx bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen
  NVSHMEM_BOOTSTRAP_UID_PLUGIN "nvshmem_bootstrap_uid.so.3" (type: string, default: "nvshmem_bootstrap_uid.so.3")
	Absolute path to or name of the UID bootstrap plugin file.
	NVSHMEM will search for the plugin based on linux linker priorities. See man
	dlopen

Additional options:
  NVSHMEM_CUDA_PATH            "" (type: string, default: "")
	Path to directory containing libcuda.so (for use when not in default location)
  NVSHMEM_DEBUG_ATTACH_DELAY   0 (type: int, default: 0)
	Delay (in seconds) during the first call to NVSHMEM_INIT to allow for attaching
	a debuggger (Default 0)
  NVSHMEM_DEBUG_FILE           "" (type: string, default: "")
	Debugging output filename, may contain %h for hostname and %p for pid
  NVSHMEM_MAX_TEAMS            32 (type: long, default: 32)
	Maximum number of simultaneous teams allowed
  NVSHMEM_MAX_MEMORY_PER_GPU   137438953472 (type: size, default: 137438953472)
	Maximum memory per GPU
  NVSHMEM_DISABLE_CUDA_VMM     false (type: bool, default: false)
	Disable use of CUDA VMM for P2P memory mapping. By default, CUDA VMM is enabled
	on x86 and disabled on P9. CUDA VMM feature in NVSHMEM requires CUDA RT version
	and CUDA Driver version to be greater than or equal to 11.3.
  NVSHMEM_DISABLE_P2P          false (type: bool, default: false)
	Disable P2P connectivity of GPUs even when available
  NVSHMEM_IGNORE_CUDA_MPS_ACTIVE_THREAD_PERCENTAGE false (type: bool, default: false)
	When doing Multi-Process Per GPU (MPG) run, full API support is available only
	if sum of CUDA_MPS_ACTIVE_THREAD_PERCENTAGE of processes running on a GPU is <=
	100%. Through this variable, user can request NVSHMEM runtime to ignore the
	active thread percentage and allow full MPG support. Users enable it at their
	own risk as NVSHMEM might deadlock.
  NVSHMEM_CUMEM_GRANULARITY    536870912 (type: size, default: 536870912)
	Granularity for cuMemAlloc/cuMemCreate
  NVSHMEM_PROXY_REQUEST_BATCH_MAX 32 (type: int, default: 32)
	Maxmum number of requests that the proxy thread processes in a single iteration
	of the progress loop.

Collectives options:
  NVSHMEM_DISABLE_NCCL         false (type: bool, default: false)
	Disable use of NCCL for collective operations
  NVSHMEM_BARRIER_DISSEM_KVAL  2 (type: int, default: 2)
	Radix of the dissemination algorithm used for barriers
  NVSHMEM_BARRIER_TG_DISSEM_KVAL 2 (type: int, default: 2)
	Radix of the dissemination algorithm used for thread group barriers
  NVSHMEM_FCOLLECT_LL_THRESHOLD 2048 (type: size, default: 2048)
	Message size threshold up to which fcollect LL algo will be used

  NVSHMEM_REDUCE_SCRATCH_SIZE  524288 (type: size, default: 524288)
	Amount of symmetric heap memory (minimum 16B, multiple of 8B) reserved by
	runtime for every team to implement reduce and reducescatter collectives

  NVSHMEM_BCAST_ALGO           0 (type: int, default: 0)
	Broadcast algorithm to be used.
	  * 0 - use default algorithm selection strategy

  NVSHMEM_REDMAXLOC_ALGO       1 (type: int, default: 1)
	Reduction algorithm to be used for MAXLOC operation.
	  * 1 - default, flag alltoall algorithm
	  * 2 - flat reduce + flat bcast
	  * 3 - topo-aware two-level reduce + topo-aware bcast


Transport options:
  NVSHMEM_REMOTE_TRANSPORT     "ibrc" (type: string, default: "ibrc")
	Selected transport for remote operations: ibrc, ucx, libfabric, ibdevx, none
  NVSHMEM_ENABLE_NIC_PE_MAPPING false (type: bool, default: false)
	When not set or set to 0, a PE is assigned the NIC on the node that is closest
	to it by distance. When set to 1, NVSHMEM either assigns NICs to PEs on a
	round-robin basis or uses NVSHMEM_HCA_PE_MAPPING or NVSHMEM_HCA_LIST when they
	are specified.
  NVSHMEM_DISABLE_LOCAL_ONLY_PROXY false (type: bool, default: false)
	When running on an NVLink-only configuaration (No-IB, No-UCX), completely
	disable the proxy thread. This will disable device side global exit and device
	side wait timeout polling (enabled by NVSHMEM_TIMEOUT_DEVICE_POLLING build-time
	variable) because these are processed by the proxy thread.
  NVSHMEM_IB_ENABLE_IBGDA      false (type: bool, default: false)
	Set to enable GPU-initiated communication transport.

Hidden options:
  NVSHMEM_INFO_HIDDEN          true (type: bool, default: false)
	Print hidden environment variable options at startup
  NVSHMEM_DISABLE_NVLS_SHARING true (type: bool, default: true)
	Disable NVLS SHARP resource sharing for user-defined teams
  NVSHMEM_HEAP_KIND            "DEVICE" (type: string, default: "DEVICE")
	Specify the memory kind used by the NVSHMEM symmetric heap.
	Allowed values: VIDMEM, SYSMEM
  NVSHMEM_ENABLE_RAIL_OPT      false (type: bool, default: false)
	Enable Rail Optimization when heap is in SYSMEM
  NVSHMEM_BOOTSTRAP_TWO_STAGE  false (type: bool, default: false)
	Ignore CUDA device setting during initialization,forcing two-stage
	initialization
  NVSHMEM_DEBUG_SUBSYS         "" (type: string, default: "")
	Comma separated list of debugging message sources. Prefix with '^' to exclude.
	Values: INIT, COLL, P2P, PROXY, TRANSPORT, MEM, BOOTSTRAP, TOPO, UTIL, ALL
  NVSHMEM_ENABLE_ERROR_CHECKS  false (type: bool, default: false)
	Enable error checks
  NVSHMEM_DISABLE_MNNVL        false (type: bool, default: false)
	Disable MNNVL connectivity for GPUs even when available
  NVSHMEM_CUMEM_HANDLE_TYPE    "FILE_DESCRIPTOR" (type: string, default: "FILE_DESCRIPTOR")
	Handle type for cuMemCreate. Supported are - FABRIC or FILE_DESCRIPTOR
  NVSHMEM_BYPASS_ACCESSIBILITY_CHECK false (type: bool, default: false)
	Bypass peer GPU accessbility checks
  NVSHMEM_FCOLLECT_NTHREADS    512 (type: int, default: 512)
	Sets number of threads per block for fcollect collective.
	By default, if no env is set, default value is min(max_occupancy per CTA, msg
	size per PE).
	If env is specified, value overrides the default irrespective of max occupancy
	per CTA

  NVSHMEM_REDUCESCATTER_NTHREADS 512 (type: int, default: 512)
	Sets number of threads per block for reducescatter collective.
	By default, if no env is set, default value is min(max_occupancy per CTA, msg
	size per PE).
	If env is specified, value overrides the default irrespective of max occupancy
	per CTA

  NVSHMEM_MAX_CTAS             1 (type: int, default: 1)
	Sets number of blocks per grid for host onstream collective.
	By default, if no env is set, default value to 1 CTA
	If env is specified, value overrides the default value

  NVSHMEM_REDUCE_RECEXCH_KVAL  2 (type: int, default: 2)
	Radix of the recursive exchange reduction algorithm
  NVSHMEM_FCOLLECT_LL128_THRESHOLD 0 (type: size, default: 0)
	Message size threshold up to which the fcollect LL128 algo will be used.
	LL128 will be used only when FCOLLECT_LL_THRESHOLD < size
  NVSHMEM_FCOLLECT_NVLS_THRESHOLD 16777216 (type: size, default: 16777216)
	Message size threshold up to which fcollect NVLS algo will be used

  NVSHMEM_REDUCESCATTER_NVLS_THRESHOLD 16777216 (type: size, default: 16777216)
	Message size threshold up to which reducescatter NVLS algo will be used

  NVSHMEM_BCAST_TREE_KVAL      2 (type: int, default: 2)
	Radix of the broadcast tree algorithm
  NVSHMEM_FCOLLECT_ALGO        0 (type: int, default: 0)
	Fcollect algorithm to be used.
	  * 0 - use default algorithm selection strategy

  NVSHMEM_REDUCE_ALGO          0 (type: int, default: 0)
	Allreduce algorithm to be used.
	   * 0/1 - use default algorithm selection strategy

  NVSHMEM_REDUCE_NVLS_THRESHOLD 2048 (type: int, default: 2048)
	Message size threshold up to which allreduce one-shot algo will be used

  NVSHMEM_REDUCESCATTER_ALGO   0 (type: int, default: 0)
	Reduce Scatter algorithm to be used.
	  * 0 - use default algorithm selection strategy

  NVSHMEM_ASSERT_ATOMICS_SYNC  false (type: bool, default: false)
	Bypass flush on wait_until at target
  NVSHMEM_BYPASS_FLUSH         false (type: bool, default: false)
	Bypass flush in proxy when enforcing consistency

NVTX options:
  NVSHMEM_NVTX                 "off" (type: string, default: "off")
	Set to enable NVTX instrumentation. Accepts a comma separated list of
	instrumentation groups. By default the NVTX instrumentation is disabled.
	  init                : library setup
	  alloc               : memory management
	  launch              : kernel launch routines
	  coll                : collective communications
	  wait                : blocking point-to-point synchronization
	  wait_on_stream      : point-to-point synchronization (on stream)
	  test                : non-blocking point-to-point synchronization
	  memorder            : memory ordering (quiet, fence)
	  quiet_on_stream     : nvshmemx_quiet_on_stream
	  atomic_fetch        : fetching atomic memory operations
	  atomic_set          : non-fetchong atomic memory operations
	  rma_blocking        : blocking remote memory access operations
	  rma_nonblocking     : non-blocking remote memory access operations
	  proxy               : activity of the proxy thread
	  common              : init,alloc,launch,coll,memorder,wait,atomic_fetch,rma_blocking,proxy
	  all                 : all groups
	  off                 : disable all NVTX instrumentation

Error Information

Node 1

[1750229955.120448] [k8s-node07:9835 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.208265] [k8s-node07:9838 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.240276] [k8s-node07:9836 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.243667] [k8s-node07:9837 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.244811] [k8s-node07:9842 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.252003] [k8s-node07:9839 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.253382] [k8s-node07:9840 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
[1750229955.255574] [k8s-node07:9841 :f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<59677> failed : Broken pipe
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<38229> failed : Broken pipe
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 58 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<40923> failed : Broken pipe
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 186 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 218 bytes instead of 8

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed

/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<42897> failed : Broken pipe
socketProgressOpt: Call to recv from 10.179.114.196<38745> failed : Broken pipe/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 allgather of ipc handles failed
/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6
/sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 allgather of ipc handles failed
building transport map failed


/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: building transport map failed
nvshmem initialization failed, exiting


/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9835 via signal SIGTERM
W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9836 via signal SIGTERM
W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9837 via signal SIGTERM
W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9838 via signal SIGTERM
W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9839 via signal SIGTERM
W0618 06:59:31.818000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9840 via signal SIGTERM
W0618 06:59:31.818000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9842 via signal SIGTERM

Node 2

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 nvshmem initialization failed, exiting
building transport map failed


/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

/sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport.

/sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ...
/sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed

/sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting

W0618 06:59:32.342000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47763 via signal SIGTERM
W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47764 via signal SIGTERM
W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47765 via signal SIGTERM
W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47768 via signal SIGTERM
W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47769 via signal SIGTERM

I would be very appreciate if anyone could help with that!

Originally created by @JoeyYoung on GitHub (Jun 18, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/223 ## Description I use docker container built from https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile.deepep There are two nodes. Each node has x8 H20 GPUs and can run `test_intranode.py` normally. But they fail to run `test_internode.py`. When I use the latest main codebase, I encounter the error `init failed for transport: IBGDA`, so I checkout the commit version [`82dcf48`](https://github.com/deepseek-ai/DeepEP/commit/82dcf48fd315d7b83f2cd2b4f1d1f1fda6af8ed2) to run internode tests, which uses IBRC, according to discussions in #163. ## Environment **nvidia-smi topo -m** ``` GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE PIX SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE PIX NODE 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE NODE SYS SYS SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE NODE SYS SYS SYS SYS NIC2 PIX NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE NODE NODE SYS SYS SYS SYS NIC3 NODE PIX NODE NODE SYS SYS SYS SYS NODE NODE NODE X NODE NODE SYS SYS SYS SYS NIC4 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE NODE X NODE SYS SYS SYS SYS NIC5 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE NODE X SYS SYS SYS SYS NIC6 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS X NODE NODE NODE NIC7 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS NODE X NODE NODE NIC8 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS NODE NODE X NODE NIC9 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS NODE NODE NODE X ``` **ibv_devinfo** ``` hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 22.39.3004 node_guid: c470:bd03:00a7:8bf0 sys_image_guid: c470:bd03:00a7:8bf0 vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000327 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 22.39.3004 node_guid: c470:bd03:00a7:8bf1 sys_image_guid: c470:bd03:00a7:8bf0 vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000327 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_2 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00ba:3524 sys_image_guid: b8e9:2403:00ba:3524 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 9 port_lid: 7 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_3 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00d5:ac84 sys_image_guid: b8e9:2403:00d5:ac84 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 9 port_lid: 8 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_4 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00d5:adbc sys_image_guid: b8e9:2403:00d5:adbc vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_5 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00d5:a30c sys_image_guid: b8e9:2403:00d5:a30c vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_6 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00d5:a2d8 sys_image_guid: b8e9:2403:00d5:a2d8 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_7 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00ba:3814 sys_image_guid: b8e9:2403:00ba:3814 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_8 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:0036:99d4 sys_image_guid: b8e9:2403:0036:99d4 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand hca_id: mlx5_9 transport: InfiniBand (0) fw_ver: 20.43.2026 node_guid: b8e9:2403:00ba:36b0 sys_image_guid: b8e9:2403:00ba:36b0 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000223 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: InfiniBand ``` **gdrcopy_copybw** ``` GPU id:0; name: NVIDIA H20-3e; Bus id: 0000:2a:00 GPU id:1; name: NVIDIA H20-3e; Bus id: 0000:3a:00 GPU id:2; name: NVIDIA H20-3e; Bus id: 0000:4c:00 GPU id:3; name: NVIDIA H20-3e; Bus id: 0000:5c:00 GPU id:4; name: NVIDIA H20-3e; Bus id: 0000:aa:00 GPU id:5; name: NVIDIA H20-3e; Bus id: 0000:ba:00 GPU id:6; name: NVIDIA H20-3e; Bus id: 0000:ca:00 GPU id:7; name: NVIDIA H20-3e; Bus id: 0000:da:00 selecting device 0 testing size: 131072 rounded size: 131072 gpu alloc fn: cuMemAlloc device ptr: 7f9461400000 gdr_open error: Is gdrdrv driver installed and loaded? ``` **nvshmem-info -a** ``` NVSHMEM v3.2.5 Build Information: CUDA API 12040 CUDA Driver 12040 Build Timestamp Apr 2 2025 13:57:48 Build Variables NVSHMEM_DEBUG=OFF NVSHMEM_DEVEL=OFF NVSHMEM_DEFAULT_PMI2=OFF NVSHMEM_DEFAULT_PMIX=OFF NVSHMEM_DEFAULT_UCX=OFF NVSHMEM_ENABLE_ALL_DEVICE_INLINING=OFF NVSHMEM_GPU_COLL_USE_LDST=OFF NVSHMEM_IBGDA_SUPPORT=ON NVSHMEM_IBGDA_SUPPORT_GPUMEM_ONLY=OFF NVSHMEM_IBDEVX_SUPPORT=OFF NVSHMEM_IBRC_SUPPORT=ON NVSHMEM_MPI_SUPPORT=OFF NVSHMEM_NVTX=ON NVSHMEM_PMIX_SUPPORT=OFF NVSHMEM_SHMEM_SUPPORT=OFF NVSHMEM_TEST_STATIC_LIB=OFF NVSHMEM_TIMEOUT_DEVICE_POLLING=OFF NVSHMEM_TRACE=OFF NVSHMEM_UCX_SUPPORT=OFF NVSHMEM_USE_DLMALLOC=OFF NVSHMEM_USE_NCCL=OFF NVSHMEM_USE_GDRCOPY=ON NVSHMEM_VERBOSE=OFF CUDA_HOME=/usr/local/cuda LIBFABRIC_HOME=/usr/local/libfabric MPI_HOME=/usr/local/ompi NCCL_HOME=/usr/local/nccl NVSHMEM_PREFIX=/usr/local/nvshmem PMIX_HOME=/usr SHMEM_HOME=/usr/local/ompi UCX_HOME=/usr/local/ucx Standard options: NVSHMEM_VERSION false (type: bool, default: false) Print library version at startup NVSHMEM_INFO false (type: bool, default: false) Print environment variable options at startup NVSHMEM_DISABLE_NVLS false (type: bool, default: false) Disable NVLS SHARP resources for collectives, even if available for platform NVSHMEM_SYMMETRIC_SIZE 1073741824 (type: size, default: 1073741824) Specifies the size (in bytes) of the symmetric heap memory per PE. The size is implementation-defined and must be at least as large as the integer ceiling of the product of the numeric prefix and the scaling factor. The character suffixes for the scaling factor are as follows: * k or K multiplies by 2^10 (kibibytes) * m or M multiplies by 2^20 (mebibytes) * g or G multiplies by 2^30 (gibibytes) * t or T multiplies by 2^40 (tebibytes) For example, string '20m' is equivalent to the integer value 20971520, or 20 mebibytes. Similarly the string '3.1M' is equivalent to the integer value 3250586. Only one multiplier is recognized and any characters following the multiplier are ignored, so '20kk' will not produce the same result as '20m'. Usage of string '.5m' will yield the same result as the string '0.5m'. An invalid value for NVSHMEM_SYMMETRIC_SIZE is an error, which the NVSHMEM library shall report by either returning a nonzero value from nvshmem_init_thread or causing program termination. NVSHMEM_DEBUG "" (type: string, default: "") Set to enable debugging messages. Optional values: VERSION, WARN, INFO, ABORT, TRACE Bootstrap options: NVSHMEM_BOOTSTRAP "PMI" (type: string, default: "PMI") Name of the default bootstrap that should be used to initialize NVSHMEM. Allowed values: PMI, MPI, SHMEM, plugin, UID NVSHMEM_BOOTSTRAP_PMI "PMI" (type: string, default: "PMI") Name of the PMI bootstrap that should be used to initialize NVSHMEM. Allowed values: PMI, PMI-2, PMIX NVSHMEM_BOOTSTRAP_PLUGIN "" (type: string, default: "") Absolute path to or name of the bootstrap plugin file to load when NVSHMEM_BOOTSTRAP=plugin is specified NVSHMEM_BOOTSTRAP_MPI_PLUGIN "nvshmem_bootstrap_mpi.so.3" (type: string, default: "nvshmem_bootstrap_mpi.so.3") Absolute path to or name of the MPI bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_SHMEM_PLUGIN "nvshmem_bootstrap_shmem.so.3" (type: string, default: "nvshmem_bootstrap_shmem.so.3") Absolute path to or name of the SHMEM bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMI_PLUGIN "nvshmem_bootstrap_pmi.so.3" (type: string, default: "nvshmem_bootstrap_pmi.so.3") Absolute path to or name of the PMI bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMI2_PLUGIN "nvshmem_bootstrap_pmi2.so.3" (type: string, default: "nvshmem_bootstrap_pmi2.so.3") Absolute path to or name of the PMI-2 bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_PMIX_PLUGIN "nvshmem_bootstrap_pmix.so.3" (type: string, default: "nvshmem_bootstrap_pmix.so.3") Absolute path to or name of the PMIx bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen NVSHMEM_BOOTSTRAP_UID_PLUGIN "nvshmem_bootstrap_uid.so.3" (type: string, default: "nvshmem_bootstrap_uid.so.3") Absolute path to or name of the UID bootstrap plugin file. NVSHMEM will search for the plugin based on linux linker priorities. See man dlopen Additional options: NVSHMEM_CUDA_PATH "" (type: string, default: "") Path to directory containing libcuda.so (for use when not in default location) NVSHMEM_DEBUG_ATTACH_DELAY 0 (type: int, default: 0) Delay (in seconds) during the first call to NVSHMEM_INIT to allow for attaching a debuggger (Default 0) NVSHMEM_DEBUG_FILE "" (type: string, default: "") Debugging output filename, may contain %h for hostname and %p for pid NVSHMEM_MAX_TEAMS 32 (type: long, default: 32) Maximum number of simultaneous teams allowed NVSHMEM_MAX_MEMORY_PER_GPU 137438953472 (type: size, default: 137438953472) Maximum memory per GPU NVSHMEM_DISABLE_CUDA_VMM false (type: bool, default: false) Disable use of CUDA VMM for P2P memory mapping. By default, CUDA VMM is enabled on x86 and disabled on P9. CUDA VMM feature in NVSHMEM requires CUDA RT version and CUDA Driver version to be greater than or equal to 11.3. NVSHMEM_DISABLE_P2P false (type: bool, default: false) Disable P2P connectivity of GPUs even when available NVSHMEM_IGNORE_CUDA_MPS_ACTIVE_THREAD_PERCENTAGE false (type: bool, default: false) When doing Multi-Process Per GPU (MPG) run, full API support is available only if sum of CUDA_MPS_ACTIVE_THREAD_PERCENTAGE of processes running on a GPU is <= 100%. Through this variable, user can request NVSHMEM runtime to ignore the active thread percentage and allow full MPG support. Users enable it at their own risk as NVSHMEM might deadlock. NVSHMEM_CUMEM_GRANULARITY 536870912 (type: size, default: 536870912) Granularity for cuMemAlloc/cuMemCreate NVSHMEM_PROXY_REQUEST_BATCH_MAX 32 (type: int, default: 32) Maxmum number of requests that the proxy thread processes in a single iteration of the progress loop. Collectives options: NVSHMEM_DISABLE_NCCL false (type: bool, default: false) Disable use of NCCL for collective operations NVSHMEM_BARRIER_DISSEM_KVAL 2 (type: int, default: 2) Radix of the dissemination algorithm used for barriers NVSHMEM_BARRIER_TG_DISSEM_KVAL 2 (type: int, default: 2) Radix of the dissemination algorithm used for thread group barriers NVSHMEM_FCOLLECT_LL_THRESHOLD 2048 (type: size, default: 2048) Message size threshold up to which fcollect LL algo will be used NVSHMEM_REDUCE_SCRATCH_SIZE 524288 (type: size, default: 524288) Amount of symmetric heap memory (minimum 16B, multiple of 8B) reserved by runtime for every team to implement reduce and reducescatter collectives NVSHMEM_BCAST_ALGO 0 (type: int, default: 0) Broadcast algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_REDMAXLOC_ALGO 1 (type: int, default: 1) Reduction algorithm to be used for MAXLOC operation. * 1 - default, flag alltoall algorithm * 2 - flat reduce + flat bcast * 3 - topo-aware two-level reduce + topo-aware bcast Transport options: NVSHMEM_REMOTE_TRANSPORT "ibrc" (type: string, default: "ibrc") Selected transport for remote operations: ibrc, ucx, libfabric, ibdevx, none NVSHMEM_ENABLE_NIC_PE_MAPPING false (type: bool, default: false) When not set or set to 0, a PE is assigned the NIC on the node that is closest to it by distance. When set to 1, NVSHMEM either assigns NICs to PEs on a round-robin basis or uses NVSHMEM_HCA_PE_MAPPING or NVSHMEM_HCA_LIST when they are specified. NVSHMEM_DISABLE_LOCAL_ONLY_PROXY false (type: bool, default: false) When running on an NVLink-only configuaration (No-IB, No-UCX), completely disable the proxy thread. This will disable device side global exit and device side wait timeout polling (enabled by NVSHMEM_TIMEOUT_DEVICE_POLLING build-time variable) because these are processed by the proxy thread. NVSHMEM_IB_ENABLE_IBGDA false (type: bool, default: false) Set to enable GPU-initiated communication transport. Hidden options: NVSHMEM_INFO_HIDDEN true (type: bool, default: false) Print hidden environment variable options at startup NVSHMEM_DISABLE_NVLS_SHARING true (type: bool, default: true) Disable NVLS SHARP resource sharing for user-defined teams NVSHMEM_HEAP_KIND "DEVICE" (type: string, default: "DEVICE") Specify the memory kind used by the NVSHMEM symmetric heap. Allowed values: VIDMEM, SYSMEM NVSHMEM_ENABLE_RAIL_OPT false (type: bool, default: false) Enable Rail Optimization when heap is in SYSMEM NVSHMEM_BOOTSTRAP_TWO_STAGE false (type: bool, default: false) Ignore CUDA device setting during initialization,forcing two-stage initialization NVSHMEM_DEBUG_SUBSYS "" (type: string, default: "") Comma separated list of debugging message sources. Prefix with '^' to exclude. Values: INIT, COLL, P2P, PROXY, TRANSPORT, MEM, BOOTSTRAP, TOPO, UTIL, ALL NVSHMEM_ENABLE_ERROR_CHECKS false (type: bool, default: false) Enable error checks NVSHMEM_DISABLE_MNNVL false (type: bool, default: false) Disable MNNVL connectivity for GPUs even when available NVSHMEM_CUMEM_HANDLE_TYPE "FILE_DESCRIPTOR" (type: string, default: "FILE_DESCRIPTOR") Handle type for cuMemCreate. Supported are - FABRIC or FILE_DESCRIPTOR NVSHMEM_BYPASS_ACCESSIBILITY_CHECK false (type: bool, default: false) Bypass peer GPU accessbility checks NVSHMEM_FCOLLECT_NTHREADS 512 (type: int, default: 512) Sets number of threads per block for fcollect collective. By default, if no env is set, default value is min(max_occupancy per CTA, msg size per PE). If env is specified, value overrides the default irrespective of max occupancy per CTA NVSHMEM_REDUCESCATTER_NTHREADS 512 (type: int, default: 512) Sets number of threads per block for reducescatter collective. By default, if no env is set, default value is min(max_occupancy per CTA, msg size per PE). If env is specified, value overrides the default irrespective of max occupancy per CTA NVSHMEM_MAX_CTAS 1 (type: int, default: 1) Sets number of blocks per grid for host onstream collective. By default, if no env is set, default value to 1 CTA If env is specified, value overrides the default value NVSHMEM_REDUCE_RECEXCH_KVAL 2 (type: int, default: 2) Radix of the recursive exchange reduction algorithm NVSHMEM_FCOLLECT_LL128_THRESHOLD 0 (type: size, default: 0) Message size threshold up to which the fcollect LL128 algo will be used. LL128 will be used only when FCOLLECT_LL_THRESHOLD < size NVSHMEM_FCOLLECT_NVLS_THRESHOLD 16777216 (type: size, default: 16777216) Message size threshold up to which fcollect NVLS algo will be used NVSHMEM_REDUCESCATTER_NVLS_THRESHOLD 16777216 (type: size, default: 16777216) Message size threshold up to which reducescatter NVLS algo will be used NVSHMEM_BCAST_TREE_KVAL 2 (type: int, default: 2) Radix of the broadcast tree algorithm NVSHMEM_FCOLLECT_ALGO 0 (type: int, default: 0) Fcollect algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_REDUCE_ALGO 0 (type: int, default: 0) Allreduce algorithm to be used. * 0/1 - use default algorithm selection strategy NVSHMEM_REDUCE_NVLS_THRESHOLD 2048 (type: int, default: 2048) Message size threshold up to which allreduce one-shot algo will be used NVSHMEM_REDUCESCATTER_ALGO 0 (type: int, default: 0) Reduce Scatter algorithm to be used. * 0 - use default algorithm selection strategy NVSHMEM_ASSERT_ATOMICS_SYNC false (type: bool, default: false) Bypass flush on wait_until at target NVSHMEM_BYPASS_FLUSH false (type: bool, default: false) Bypass flush in proxy when enforcing consistency NVTX options: NVSHMEM_NVTX "off" (type: string, default: "off") Set to enable NVTX instrumentation. Accepts a comma separated list of instrumentation groups. By default the NVTX instrumentation is disabled. init : library setup alloc : memory management launch : kernel launch routines coll : collective communications wait : blocking point-to-point synchronization wait_on_stream : point-to-point synchronization (on stream) test : non-blocking point-to-point synchronization memorder : memory ordering (quiet, fence) quiet_on_stream : nvshmemx_quiet_on_stream atomic_fetch : fetching atomic memory operations atomic_set : non-fetchong atomic memory operations rma_blocking : blocking remote memory access operations rma_nonblocking : non-blocking remote memory access operations proxy : activity of the proxy thread common : init,alloc,launch,coll,memorder,wait,atomic_fetch,rma_blocking,proxy all : all groups off : disable all NVTX instrumentation ``` ## Error Information **Node 1** ``` [1750229955.120448] [k8s-node07:9835 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.208265] [k8s-node07:9838 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.240276] [k8s-node07:9836 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.243667] [k8s-node07:9837 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.244811] [k8s-node07:9842 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.252003] [k8s-node07:9839 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.253382] [k8s-node07:9840 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device [1750229955.255574] [k8s-node07:9841 :f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 40 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<59677> failed : Broken pipe /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<38229> failed : Broken pipe /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 58 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<40923> failed : Broken pipe /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 186 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_net_recv:99: Message truncated : received 218 bytes instead of 8 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -3 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -3 allgather of ipc handles failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: /sgl-workspace/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 10.179.114.196<42897> failed : Broken pipe socketProgressOpt: Call to recv from 10.179.114.196<38745> failed : Broken pipe/sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:90: non-zero status: -6 allgather of ipc handles failed /sgl-workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /sgl-workspace/nvshmem/src/host/topo/topo.cpp:477: non-zero status: -6 /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 allgather of ipc handles failed building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 7 /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: building transport map failed nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9835 via signal SIGTERM W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9836 via signal SIGTERM W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9837 via signal SIGTERM W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9838 via signal SIGTERM W0618 06:59:31.817000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9839 via signal SIGTERM W0618 06:59:31.818000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9840 via signal SIGTERM W0618 06:59:31.818000 9770 torch/multiprocessing/spawn.py:169] Terminating process 9842 via signal SIGTERM ``` **Node 2** ``` /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 nvshmem initialization failed, exiting building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting /sgl-workspace/nvshmem/src/modules/transport/ibrc/ibrc.cpp:nvshmemt_init:1851: neither nv_peer_mem, or nvidia_peermem detected. Skipping transport. /sgl-workspace/nvshmem/src/host/topo/topo.cpp:469: [GPU 1] Peer GPU 0 is not accessible, exiting ... /sgl-workspace/nvshmem/src/host/init/init.cu:998: non-zero status: 3 building transport map failed /sgl-workspace/nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1080: nvshmem initialization failed, exiting W0618 06:59:32.342000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47763 via signal SIGTERM W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47764 via signal SIGTERM W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47765 via signal SIGTERM W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47768 via signal SIGTERM W0618 06:59:32.343000 47698 torch/multiprocessing/spawn.py:169] Terminating process 47769 via signal SIGTERM ``` I would be very appreciate if anyone could help with that!
bsakaguchi commented 2026-10-26 09:55:50 +00:00
Author
Owner
Copy link

@JoeyYoung commented on GitHub (Jun 18, 2025):

I use high-throughput mode instead of low-latency

<!-- gh-comment-id:2982960631 --> @JoeyYoung commented on GitHub (Jun 18, 2025): I use high-throughput mode instead of low-latency
bsakaguchi commented 2026-10-26 09:55:51 +00:00
Author
Owner
Copy link

@JoeyYoung commented on GitHub (Jun 18, 2025):

After checking the nvidia-peermem kernel in the host machine, seems it is not loaded successfully.

Solve by loading:
modprobe nvidia-peermem and check: lsmod |grep peer

<!-- gh-comment-id:2983300961 --> @JoeyYoung commented on GitHub (Jun 18, 2025): After checking the `nvidia-peermem` kernel in the host machine, seems it is not loaded successfully. Solve by loading: `modprobe nvidia-peermem` and check: `lsmod |grep peer`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#174
No description provided.