mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #183] [Low Latency with nvlink] H20 2 hosts 16 GPUs low latency with nvlink bandwidth abnormal ? #155
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#155
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @elevenxiang on GitHub (Jun 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/183
Hi,
we tested low latency with nvlink on latest DeepEP on H20. and we found that benched out bandwidth performance abnormal
/workspace/DeepEP# MASTER_ADDR=xxxx WORLD_SIZE=2 RANK=0 python tests/test_low_latency.py
Allocating buffer size: 1880.098944 MB ...
rank 0 nvml_device_index 0, NVSHMEM_HCA_LIST mlx5_bond_0:1
rank 7 nvml_device_index 7, NVSHMEM_HCA_LIST mlx5_bond_3:1
rank 4 nvml_device_index 4, NVSHMEM_HCA_LIST mlx5_bond_2:1
rank 6 nvml_device_index 6, NVSHMEM_HCA_LIST mlx5_bond_3:1
rank 5 nvml_device_index 5, NVSHMEM_HCA_LIST mlx5_bond_2:1
rank 1 nvml_device_index 1, NVSHMEM_HCA_LIST mlx5_bond_0:1
rank 2 nvml_device_index 2, NVSHMEM_HCA_LIST mlx5_bond_1:1
rank 3 nvml_device_index 3, NVSHMEM_HCA_LIST mlx5_bond_1:1
[rank 3] Dispatch + combine bandwidth: 25.26 GB/s, avg_t=873.03 us, min_t=717.92 us, max_t=928.10 us
[rank 5] Dispatch + combine bandwidth: 25.19 GB/s, avg_t=875.15 us, min_t=737.15 us, max_t=912.86 us
[rank 1] Dispatch + combine bandwidth: 25.21 GB/s, avg_t=874.66 us, min_t=730.91 us, max_t=904.70 us
[rank 6] Dispatch + combine bandwidth: 25.24 GB/s, avg_t=873.44 us, min_t=724.45 us, max_t=931.62 us
[rank 4] Dispatch + combine bandwidth: 25.22 GB/s, avg_t=874.13 us, min_t=732.74 us, max_t=922.56 us
[rank 2] Dispatch + combine bandwidth: 25.25 GB/s, avg_t=873.21 us, min_t=724.35 us, max_t=930.59 us
[rank 7] Dispatch + combine bandwidth: 25.23 GB/s, avg_t=873.99 us, min_t=735.36 us, max_t=924.42 us
[rank 0] Dispatch + combine bandwidth: 25.26 GB/s, avg_t=872.91 us, min_t=712.64 us, max_t=929.31 us
[rank 4] Dispatch bandwidth: 133.13 GB/s, avg_t=56.43 us | Combine bandwidth: 22.59 GB/s, avg_t=643.38 us
[rank 6] Dispatch bandwidth: 138.88 GB/s, avg_t=54.09 us | Combine bandwidth: 22.50 GB/s, avg_t=646.02 us
[rank 3] Dispatch bandwidth: 126.95 GB/s, avg_t=59.17 us | Combine bandwidth: 22.67 GB/s, avg_t=641.15 us
[rank 5] Dispatch bandwidth: 114.49 GB/s, avg_t=65.61 us | Combine bandwidth: 22.91 GB/s, avg_t=634.56 us
[rank 0] Dispatch bandwidth: 147.11 GB/s, avg_t=51.06 us | Combine bandwidth: 22.49 GB/s, avg_t=646.44 us
[rank 2] Dispatch bandwidth: 131.37 GB/s, avg_t=57.18 us | Combine bandwidth: 22.63 GB/s, avg_t=642.27 us
[rank 1] Dispatch bandwidth: 129.63 GB/s, avg_t=57.95 us | Combine bandwidth: 22.58 GB/s, avg_t=643.92 us
[rank 7] Dispatch bandwidth: 136.74 GB/s, avg_t=54.93 us | Combine bandwidth: 22.56 GB/s, avg_t=644.42 us
[rank 1] Dispatch send/recv time: 157.02 us | Combine send/recv time: 195.73 us
[rank 2] Dispatch send/recv time: 197.97 us | Combine send/recv time: 273.87 us
[rank 6] Dispatch send/recv time: 512.62 us | Combine send/recv time: 598.74 us
[rank 5] Dispatch send/recv time: 38.96 us | Combine send/recv time: 48.31 us
[rank 3] Dispatch send/recv time: 85.01 us | Combine send/recv time: 131.15 us
[rank 7] Dispatch send/recv time: 527.40 us | Combine send/recv time: 612.89 us
[rank 0] Dispatch send/recv time: 39.14 us | Combine send/recv time: 46.84 us
[rank 4] Dispatch send/recv time: 286.16 us | Combine send/recv time: 362.97 us
MASTER_ADDR=xxx WORLD_SIZE=2 RANK=1 python tests/test_low_latency.py
Allocating buffer size: 1880.098944 MB ...
rank 15 nvml_device_index 7, NVSHMEM_HCA_LIST mlx5_bond_3:1
rank 11 nvml_device_index 3, NVSHMEM_HCA_LIST mlx5_bond_1:1
rank 12 nvml_device_index 4, NVSHMEM_HCA_LIST mlx5_bond_2:1
rank 13 nvml_device_index 5, NVSHMEM_HCA_LIST mlx5_bond_2:1
rank 14 nvml_device_index 6, NVSHMEM_HCA_LIST mlx5_bond_3:1
rank 9 nvml_device_index 1, NVSHMEM_HCA_LIST mlx5_bond_0:1
rank 8 nvml_device_index 0, NVSHMEM_HCA_LIST mlx5_bond_0:1
rank 10 nvml_device_index 2, NVSHMEM_HCA_LIST mlx5_bond_1:1
[rank 8] Dispatch + combine bandwidth: 25.07 GB/s, avg_t=879.48 us, min_t=856.16 us, max_t=924.35 us
[rank 12] Dispatch + combine bandwidth: 25.08 GB/s, avg_t=879.20 us, min_t=856.32 us, max_t=920.77 us
[rank 13] Dispatch + combine bandwidth: 25.08 GB/s, avg_t=879.28 us, min_t=859.84 us, max_t=923.62 us
[rank 14] Dispatch + combine bandwidth: 25.08 GB/s, avg_t=879.30 us, min_t=856.13 us, max_t=921.18 us
[rank 9] Dispatch + combine bandwidth: 25.08 GB/s, avg_t=879.27 us, min_t=850.50 us, max_t=932.64 us
[rank 10] Dispatch + combine bandwidth: 25.07 GB/s, avg_t=879.40 us, min_t=861.79 us, max_t=923.90 us
[rank 15] Dispatch + combine bandwidth: 25.08 GB/s, avg_t=879.26 us, min_t=857.02 us, max_t=921.92 us
[rank 11] Dispatch + combine bandwidth: 25.07 GB/s, avg_t=879.46 us, min_t=856.35 us, max_t=921.12 us
[rank 13] Dispatch bandwidth: 15.83 GB/s, avg_t=474.43 us | Combine bandwidth: 66.59 GB/s, avg_t=218.30 us
[rank 9] Dispatch bandwidth: 15.62 GB/s, avg_t=481.01 us | Combine bandwidth: 67.79 GB/s, avg_t=214.44 us
[rank 11] Dispatch bandwidth: 15.79 GB/s, avg_t=475.85 us | Combine bandwidth: 66.61 GB/s, avg_t=218.24 us
[rank 12] Dispatch bandwidth: 15.63 GB/s, avg_t=480.54 us | Combine bandwidth: 68.49 GB/s, avg_t=212.26 us
[rank 14] Dispatch bandwidth: 15.71 GB/s, avg_t=478.03 us | Combine bandwidth: 67.68 GB/s, avg_t=214.77 us
[rank 15] Dispatch bandwidth: 15.74 GB/s, avg_t=477.18 us | Combine bandwidth: 67.65 GB/s, avg_t=214.89 us
[rank 8] Dispatch bandwidth: 15.59 GB/s, avg_t=481.98 us | Combine bandwidth: 69.16 GB/s, avg_t=210.18 us
[rank 10] Dispatch bandwidth: 15.76 GB/s, avg_t=476.68 us | Combine bandwidth: 65.60 GB/s, avg_t=221.60 us
[rank 15] Dispatch send/recv time: 151.50 us | Combine send/recv time: 145.43 us
[rank 12] Dispatch send/recv time: 425.48 us | Combine send/recv time: 433.57 us
[rank 14] Dispatch send/recv time: 41.35 us | Combine send/recv time: 48.14 us
[rank 11] Dispatch send/recv time: 237.67 us | Combine send/recv time: 261.05 us
[rank 8] Dispatch send/recv time: 43.15 us | Combine send/recv time: 60.94 us
[rank 9] Dispatch send/recv time: 147.78 us | Combine send/recv time: 174.06 us
[rank 13] Dispatch send/recv time: 39.77 us | Combine send/recv time: 48.91 us
[rank 10] Dispatch send/recv time: 39.35 us | Combine send/recv time: 48.60 us
why dispatch and combine has so huge bandwidth difference ?
Thanks
@LyricZhao commented on GitHub (Jun 5, 2025):
I guess the launchs of these two node are not aligned (the CPU is slow and launch the GPU kernel too early, making others ranks waiting), or someone else occupying the node?
@elevenxiang commented on GitHub (Jun 5, 2025):
how to explain some GPUs' dispatch bandwidth will be over 130GB/s ?
consider the network bound, it seemed exceed the theoretical bandwidth ?
Thanks
@LyricZhao commented on GitHub (Jun 6, 2025):
The bandwidth is "overall algorithm bandwidth", see #51 for more discussion. And NVLink is leveraged for local-node copies.