mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #303] H20 intra-node low_latency_test performance #215
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#215
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @LCAI-TIHU on GitHub (Jul 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/303
Hi,
During testing of the low_latency_test on the H20 server, I captured some warnings and performance metrics. I noticed that the combined bandwidth of this server is significantly lower—about half compared to previous test results from other H20 machines. May I ask how I should configure or debug to improve this server's performance?
Thank you!
nvidia-smi topo -m
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE PIX SYS SYS 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS PIX NODE 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS NODE NODE 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS NODE PIX 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS NODE NODE 48-95,144-191 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE SYS SYS
NIC1 NODE NODE PIX NODE SYS SYS SYS SYS NODE X SYS SYS
NIC2 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS X NODE
NIC3 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
python3 test_low_latency.py
Allocating buffer size: 2115.111296 MB ...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_1 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_2 cannot allocate buffer on the specified memory type. Skipping...
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
WARN: device mlx5_3 cannot allocate buffer on the specified memory type. Skipping...
/DeepEP-test/nvshmem_src/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA
[rank 7] Dispatch + combine bandwidth: 99.09 GB/s, avg_t=222.52 us, min_t=217.31 us, max_t=228.74 us
[rank 6] Dispatch + combine bandwidth: 99.20 GB/s, avg_t=222.27 us, min_t=215.90 us, max_t=229.50 us
[rank 1] Dispatch + combine bandwidth: 99.16 GB/s, avg_t=222.35 us, min_t=215.39 us, max_t=228.35 us
[rank 2] Dispatch + combine bandwidth: 99.20 GB/s, avg_t=222.27 us, min_t=216.83 us, max_t=228.32 us
[rank 0] Dispatch + combine bandwidth: 99.09 GB/s, avg_t=222.51 us, min_t=218.34 us, max_t=226.69 us
[rank 5] Dispatch + combine bandwidth: 99.28 GB/s, avg_t=222.08 us, min_t=212.45 us, max_t=229.57 us
[rank 3] Dispatch + combine bandwidth: 99.27 GB/s, avg_t=222.10 us, min_t=216.32 us, max_t=231.81 us
[rank 4] Dispatch + combine bandwidth: 99.20 GB/s, avg_t=222.27 us, min_t=214.14 us, max_t=230.72 us
[rank 1] Dispatch bandwidth: 187.18 GB/s, avg_t=40.13 us | Combine bandwidth: 83.64 GB/s, avg_t=173.79 us
[rank 0] Dispatch bandwidth: 181.39 GB/s, avg_t=41.41 us | Combine bandwidth: 83.70 GB/s, avg_t=173.68 us
[rank 7] Dispatch bandwidth: 186.16 GB/s, avg_t=40.35 us | Combine bandwidth: 83.25 GB/s, avg_t=174.61 us
[rank 6] Dispatch bandwidth: 185.06 GB/s, avg_t=40.59 us | Combine bandwidth: 83.56 GB/s, avg_t=173.96 us
[rank 3] Dispatch bandwidth: 186.00 GB/s, avg_t=40.38 us | Combine bandwidth: 82.69 GB/s, avg_t=175.79 us
[rank 2] Dispatch bandwidth: 181.98 GB/s, avg_t=41.28 us | Combine bandwidth: 84.05 GB/s, avg_t=172.95 us
[rank 5] Dispatch bandwidth: 151.79 GB/s, avg_t=49.49 us | Combine bandwidth: 87.69 GB/s, avg_t=165.78 us
[rank 4] Dispatch bandwidth: 183.05 GB/s, avg_t=41.04 us | Combine bandwidth: 83.57 GB/s, avg_t=173.95 us
[rank 3] Dispatch send/recv time: 31.12 + 8.88 us | Combine send/recv time: 120.76 + 14.85 us
[rank 6] Dispatch send/recv time: 31.25 + 10.75 us | Combine send/recv time: 133.54 + 54.91 us
[rank 5] Dispatch send/recv time: 32.07 + 9.86 us | Combine send/recv time: 123.59 + 14.95 us
[rank 2] Dispatch send/recv time: 31.31 + 10.43 us | Combine send/recv time: 126.01 + 14.52 us
[rank 1] Dispatch send/recv time: 31.34 + 10.38 us | Combine send/recv time: 124.32 + 14.72 us
[rank 7] Dispatch send/recv time: 30.59 + 10.58 us | Combine send/recv time: 133.12 + 14.77 us
[rank 4] Dispatch send/recv time: 31.00 + 10.08 us | Combine send/recv time: 161.68 + 14.67 us
[rank 0] Dispatch send/recv time: 32.11 + 10.02 us | Combine send/recv time: 127.93 + 14.54 us
nvshmem-info -a
NVSHMEM v3.3.9
Build Information:
CUDA API 12060
CUDA Driver 12060
Build Timestamp Jul 14 2025 03:07:10
Build Variables
NVSHMEM_DEBUG=OFF NVSHMEM_DEVEL=OFF NVSHMEM_DEFAULT_PMI2=OFF
NVSHMEM_DEFAULT_PMIX=OFF NVSHMEM_DEFAULT_UCX=OFF
NVSHMEM_ENABLE_ALL_DEVICE_INLINING=OFF NVSHMEM_GPU_COLL_USE_LDST=OFF
NVSHMEM_IBGDA_SUPPORT=ON NVSHMEM_IBGDA_SUPPORT_GPUMEM_ONLY=OFF
NVSHMEM_IBDEVX_SUPPORT=OFF NVSHMEM_IBRC_SUPPORT=OFF
NVSHMEM_LIBFABRIC_SUPPORT=OFF NVSHMEM_MPI_SUPPORT=OFF NVSHMEM_NVTX=ON
NVSHMEM_PMIX_SUPPORT=OFF NVSHMEM_SHMEM_SUPPORT=OFF NVSHMEM_TEST_STATIC_LIB=OFF
NVSHMEM_TIMEOUT_DEVICE_POLLING=OFF NVSHMEM_TRACE=OFF NVSHMEM_UCX_SUPPORT=OFF
NVSHMEM_USE_DLMALLOC=OFF NVSHMEM_USE_NCCL=OFF NVSHMEM_USE_GDRCOPY=OFF
NVSHMEM_VERBOSE=OFF CUDA_HOME=/usr/local/cuda GDRCOPY_HOME=/usr/local/gdrdrv
LIBFABRIC_HOME=/usr/local/libfabric MPI_HOME=/usr/local/ompi
NCCL_HOME=/usr/local/nccl NVSHMEM_PREFIX=/usr/local/nvshmem PMIX_HOME=/usr
SHMEM_HOME=/usr/local/ompi UCX_HOME=/usr/local/ucx
nvidia-smi
Mon Jul 14 09:02:45 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01 Driver Version: 550.163.01 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H20-3e Off | 00000000:0F:00.0 Off | 0 |
| N/A 34C P0 117W / 500W | 127276MiB / 143771MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
lsmod | grep nvidia
nvidia_uvm 4669440 22
nvidia_drm 94208 0
nvidia_modeset 1343488 1 nvidia_drm
nvidia 54325248 934 nvidia_uvm,nv_peer_mem,nvidia_modeset
drm_kms_helper 315392 5 drm_vram_helper,ast,nvidia_drm
drm 622592 8 drm_kms_helper,drm_vram_helper,ast,nvidia,drm_ttm_helper,nvidia_drm,ttm
ibstat
CA 'mlx5_0'
CA type: MT4129
Number of ports: 1
Firmware version: 28.43.2026
Hardware version: 0
Node GUID: 0xa088c2030074cfda
System image GUID: 0xa088c2030074cfda
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 50
LMC: 0
SM lid: 1
Capability mask: 0xa751e848
Port GUID: 0xa088c2030074cfda
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4129
Number of ports: 1
Firmware version: 28.43.2026
Hardware version: 0
Node GUID: 0xa088c2030074cbaa
System image GUID: 0xa088c2030074cbaa
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 49
LMC: 0
SM lid: 1
Capability mask: 0xa751e848
Port GUID: 0xa088c2030074cbaa
Link layer: InfiniBand
CA 'mlx5_2'
CA type: MT4129
Number of ports: 1
Firmware version: 28.43.2026
Hardware version: 0
Node GUID: 0xa088c2030076b78c
System image GUID: 0xa088c2030076b78c
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 48
LMC: 0
SM lid: 1
Capability mask: 0xa751e848
Port GUID: 0xa088c2030076b78c
Link layer: InfiniBand
CA 'mlx5_3'
CA type: MT4129
Number of ports: 1
Firmware version: 28.43.2566
Hardware version: 0
Node GUID: 0xa088c2030059dc28
System image GUID: 0xa088c2030059dc28
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 47
LMC: 0
SM lid: 1
Capability mask: 0xa751e848
Port GUID: 0xa088c2030059dc28
Link layer: InfiniBand
all_reduce_perf_mpi
# nThread 1 nGpus 1 minBytes 4 maxBytes 17179869184 step: 2(factor) warmup iters: 10 iters: 100 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 1848 on ubuntu device 0 [0x0f] NVIDIA H20-3e
# Rank 1 Group 0 Pid 1849 on ubuntu device 1 [0x34] NVIDIA H20-3e
# Rank 2 Group 0 Pid 1850 on ubuntu device 2 [0x48] NVIDIA H20-3e
# Rank 3 Group 0 Pid 1851 on ubuntu device 3 [0x5a] NVIDIA H20-3e
# Rank 4 Group 0 Pid 1852 on ubuntu device 4 [0x87] NVIDIA H20-3e
# Rank 5 Group 0 Pid 1853 on ubuntu device 5 [0xae] NVIDIA H20-3e
# Rank 6 Group 0 Pid 1854 on ubuntu device 6 [0xc2] NVIDIA H20-3e
# Rank 7 Group 0 Pid 1855 on ubuntu device 7 [0xd7] NVIDIA H20-3e
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
4 1 float sum -1 12.49 0.00 0.00 0 12.61 0.00 0.00 0
8 2 float sum -1 12.49 0.00 0.00 0 12.50 0.00 0.00 0
16 4 float sum -1 12.52 0.00 0.00 0 12.51 0.00 0.00 0
32 8 float sum -1 24.63 0.00 0.00 0 13.10 0.00 0.00 0
64 16 float sum -1 14.50 0.00 0.01 0 14.39 0.00 0.01 0
128 32 float sum -1 15.50 0.01 0.01 0 15.31 0.01 0.01 0
256 64 float sum -1 15.70 0.02 0.03 0 15.39 0.02 0.03 0
512 128 float sum -1 16.26 0.03 0.06 0 15.55 0.03 0.06 0
1024 256 float sum -1 16.38 0.06 0.11 0 15.71 0.07 0.11 0
2048 512 float sum -1 16.49 0.12 0.22 0 15.84 0.13 0.23 0
4096 1024 float sum -1 16.69 0.25 0.43 0 16.03 0.26 0.45 0
8192 2048 float sum -1 16.90 0.48 0.85 0 16.36 0.50 0.88 0
16384 4096 float sum -1 17.55 0.93 1.63 0 16.96 0.97 1.69 0
32768 8192 float sum -1 18.97 1.73 3.02 0 18.40 1.78 3.12 0
65536 16384 float sum -1 19.19 3.42 5.98 0 18.52 3.54 6.19 0
131072 32768 float sum -1 19.41 6.75 11.82 0 18.70 7.01 12.27 0
262144 65536 float sum -1 19.78 13.26 23.20 0 19.53 13.42 23.49 0
524288 131072 float sum -1 20.75 25.27 44.22 0 20.86 25.14 43.99 0
1048576 262144 float sum -1 28.15 37.25 65.19 0 27.97 37.49 65.61 0
2097152 524288 float sum -1 41.07 51.07 89.36 0 40.93 51.23 89.66 0
4194304 1048576 float sum -1 53.54 78.33 137.09 0 53.31 78.68 137.69 0
8388608 2097152 float sum -1 81.76 102.60 179.56 0 81.14 103.39 180.93 0
16777216 4194304 float sum -1 122.6 136.85 239.49 0 121.7 137.86 241.26 0
33554432 8388608 float sum -1 195.5 171.60 300.31 0 195.6 171.51 300.14 0
67108864 16777216 float sum -1 318.4 210.77 368.84 0 318.5 210.74 368.79 0
134217728 33554432 float sum -1 566.2 237.06 414.85 0 565.5 237.33 415.32 0
268435456 67108864 float sum -1 1066.1 251.80 440.65 0 1064.0 252.29 441.51 0
536870912 134217728 float sum -1 2075.1 258.72 452.76 0 2074.7 258.77 452.85 0
1073741824 268435456 float sum -1 4023.4 266.87 467.03 0 4025.8 266.71 466.75 0
2147483648 536870912 float sum -1 7935.8 270.61 473.56 0 7935.2 270.63 473.60 0
4294967296 1073741824 float sum -1 15758 272.56 476.99 0 15753 272.64 477.12 0
8589934592 2147483648 float sum -1 31385 273.70 478.97 0 31369 273.84 479.21 0
17179869184 4294967296 float sum -1 62576 274.55 480.45 0 62567 274.58 480.52 0
all_gather_perf_mpi
# nThread 1 nGpus 1 minBytes 4 maxBytes 17179869184 step: 2(factor) warmup iters: 10 iters: 100 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 1989 on ubuntu device 0 [0x0f] NVIDIA H20-3e
# Rank 1 Group 0 Pid 1990 on ubuntu device 1 [0x34] NVIDIA H20-3e
# Rank 2 Group 0 Pid 1991 on ubuntu device 2 [0x48] NVIDIA H20-3e
# Rank 3 Group 0 Pid 1992 on ubuntu device 3 [0x5a] NVIDIA H20-3e
# Rank 4 Group 0 Pid 1993 on ubuntu device 4 [0x87] NVIDIA H20-3e
# Rank 5 Group 0 Pid 1994 on ubuntu device 5 [0xae] NVIDIA H20-3e
# Rank 6 Group 0 Pid 1995 on ubuntu device 6 [0xc2] NVIDIA H20-3e
# Rank 7 Group 0 Pid 1996 on ubuntu device 7 [0xd7] NVIDIA H20-3e
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
0 0 float none -1 3.18 0.00 0.00 0 3.17 0.00 0.00 0
0 0 float none -1 3.12 0.00 0.00 0 3.09 0.00 0.00 0
0 0 float none -1 3.12 0.00 0.00 0 3.13 0.00 0.00 0
0 0 float none -1 3.17 0.00 0.00 0 3.07 0.00 0.00 0
0 0 float none -1 3.20 0.00 0.00 0 3.07 0.00 0.00 0
128 4 float none -1 10.59 0.01 0.01 0 10.32 0.01 0.01 0
256 8 float none -1 10.48 0.02 0.02 0 10.37 0.02 0.02 0
512 16 float none -1 10.56 0.05 0.04 0 10.49 0.05 0.04 0
1024 32 float none -1 10.80 0.09 0.08 0 10.54 0.10 0.09 0
2048 64 float none -1 11.08 0.18 0.16 0 10.80 0.19 0.17 0
4096 128 float none -1 11.13 0.37 0.32 0 10.93 0.37 0.33 0
8192 256 float none -1 11.35 0.72 0.63 0 11.15 0.73 0.64 0
16384 512 float none -1 11.88 1.38 1.21 0 11.65 1.41 1.23 0
32768 1024 float none -1 13.56 2.42 2.11 0 13.25 2.47 2.16 0
65536 2048 float none -1 13.84 4.74 4.14 0 13.39 4.89 4.28 0
131072 4096 float none -1 14.02 9.35 8.18 0 13.57 9.66 8.45 0
262144 8192 float none -1 14.33 18.29 16.00 0 13.90 18.86 16.50 0
524288 16384 float none -1 14.58 35.97 31.47 0 14.41 36.38 31.83 0
1048576 32768 float none -1 18.09 57.96 50.71 0 17.71 59.20 51.80 0
2097152 65536 float none -1 23.56 89.00 77.87 0 22.92 91.49 80.05 0
4194304 131072 float none -1 34.52 121.52 106.33 0 33.67 124.58 109.01 0
8388608 262144 float none -1 44.68 187.77 164.29 0 43.72 191.87 167.88 0
16777216 524288 float none -1 65.56 255.91 223.92 0 64.46 260.26 227.73 0
33554432 1048576 float none -1 120.0 279.59 244.64 0 117.1 286.65 250.82 0
67108864 2097152 float none -1 205.0 327.34 286.43 0 204.3 328.46 287.41 0
134217728 4194304 float none -1 382.0 351.37 307.45 0 385.7 348.03 304.53 0
268435456 8388608 float none -1 733.6 365.89 320.16 0 735.4 365.03 319.40 0
536870912 16777216 float none -1 1413.2 379.90 332.41 0 1400.4 383.38 335.45 0
1073741824 33554432 float none -1 2738.9 392.04 343.03 0 2717.6 395.10 345.72 0
2147483648 67108864 float none -1 5401.1 397.60 347.90 0 5340.3 402.13 351.86 0
4294967296 134217728 float none -1 10657 403.03 352.65 0 10509 408.69 357.60 0
8589934592 268435456 float none -1 21031 408.43 357.38 0 20826 412.46 360.91 0
17179869184 536870912 float none -1 41841 410.60 359.27 0 41341 415.56 363.62 0
alltoall_perf_mpi
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
0 0 float none -1 57.89 0.00 0.00 0 57.90 0.00 0.00 N/A
0 0 float none -1 58.09 0.00 0.00 0 57.32 0.00 0.00 N/A
0 0 float none -1 57.55 0.00 0.00 0 57.57 0.00 0.00 N/A
32 1 float none -1 58.56 0.00 0.00 0 58.38 0.00 0.00 N/A
64 2 float none -1 58.82 0.00 0.00 0 58.38 0.00 0.00 N/A
128 4 float none -1 58.30 0.00 0.00 0 58.07 0.00 0.00 N/A
256 8 float none -1 58.19 0.00 0.00 0 58.20 0.00 0.00 N/A
512 16 float none -1 58.27 0.01 0.01 0 58.20 0.01 0.01 N/A
1024 32 float none -1 57.93 0.02 0.02 0 58.10 0.02 0.02 N/A
2048 64 float none -1 57.91 0.04 0.03 0 58.25 0.04 0.03 N/A
4096 128 float none -1 57.80 0.07 0.06 0 57.68 0.07 0.06 N/A
8192 256 float none -1 58.19 0.14 0.12 0 58.04 0.14 0.12 N/A
16384 512 float none -1 58.16 0.28 0.25 0 57.53 0.28 0.25 N/A
32768 1024 float none -1 58.18 0.56 0.49 0 57.82 0.57 0.50 N/A
65536 2048 float none -1 57.79 1.13 0.99 0 57.92 1.13 0.99 N/A
131072 4096 float none -1 58.36 2.25 1.97 0 58.82 2.23 1.95 N/A
262144 8192 float none -1 58.90 4.45 3.89 0 59.17 4.43 3.88 N/A
524288 16384 float none -1 59.28 8.84 7.74 0 59.21 8.85 7.75 N/A
1048576 32768 float none -1 59.05 17.76 15.54 0 58.92 17.80 15.57 N/A
2097152 65536 float none -1 59.40 35.31 30.89 0 60.31 34.77 30.43 N/A
4194304 131072 float none -1 59.45 70.56 61.74 0 59.38 70.63 61.80 N/A
8388608 262144 float none -1 59.78 140.32 122.78 0 59.47 141.05 123.42 N/A
16777216 524288 float none -1 73.44 228.45 199.90 0 69.17 242.54 212.22 N/A
33554432 1048576 float none -1 127.0 264.20 231.18 0 119.9 279.89 244.90 N/A
67108864 2097152 float none -1 220.1 304.92 266.81 0 206.6 324.85 284.24 N/A
134217728 4194304 float none -1 416.6 322.15 281.88 0 386.4 347.37 303.95 N/A
268435456 8388608 float none -1 806.3 332.94 291.32 0 747.4 359.17 314.27 N/A
536870912 16777216 float none -1 1585.1 338.71 296.37 0 1455.9 368.75 322.66 N/A
1073741824 33554432 float none -1 3251.8 330.20 288.92 0 3283.5 327.02 286.14 N/A
2147483648 67108864 float none -1 6576.7 326.53 285.71 0 6597.5 325.50 284.81 N/A
4294967296 134217728 float none -1 12810 335.28 293.37 0 12775 336.19 294.16 N/A
8589934592 268435456 float none -1 25557 336.11 294.09 0 25462 337.37 295.19 N/A
17179869184 536870912 float none -1 51104 336.18 294.15 0 51122 336.05 294.05 N/A
@sphish commented on GitHub (Jul 15, 2025):
This log line indicates that IBGDA was not successfully enabled. Please check whether nvidia-peermem is loaded and if the driver options are set correctly.
@LCAI-TIHU commented on GitHub (Jul 15, 2025):
I installed nv_peer_memory, can it replace nvidia-peermem?
Also, I used the deepEP Docker image provided by sglang. Although the log messages indicate no changes, the combined bandwidth has improved by 2x. I'm confused—why did this happen?
@sphish commented on GitHub (Jul 15, 2025):
Sorry, my mistake. You are running single-node tests, so this is not related to IBGDA. It looks like the SGLang image is using an older version of DeepEP. I need to check whether this issue is caused by the newer code.
@sphish commented on GitHub (Jul 15, 2025):
The latest TMA changes https://github.com/deepseek-ai/DeepEP/pull/299 have introduced some side effects for pure NVLink cases. You can temporarily use a version before this PR. We will optimize this case in the future.
@LCAI-TIHU commented on GitHub (Jul 15, 2025):
Thanks a lot!