deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #28] test_low_latency.py bandwidth extremely low #29

New issue
Closed
opened 2026-10-26 09:52:46 +00:00 by bsakaguchi · 17 comments
bsakaguchi commented 2026-10-26 09:52:46 +00:00
Owner
Copy link

Originally created by @Qizhi697 on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/28

I tested tests/test_low_latency.py on 2 nodes with a total of 16 H800 (8 H800 per node) with 400G CX7 connected. I wanted to reproduce the situation where Dispatch #EP and Combine #EP are both 16. The test results show that the Dispatch + combine bandwidth is normal, but the Dispatch bandwidth or Combine bandwidth is extremely low. I would like to ask why this is happening.

# mpirun --allow-run-as-root -x NVSHMEM_IB_GID_INDEX=3 -x MASTER_ADDR=10.82.176.210 -mca btl_tcp_if_include bond0 -mca btl self,tcp --mca pml ob1 --host 10.82.176.210:8,10.82.177.166:8 python3 tests/test_low_latency.py 
Allocating buffer size: 2116.291072 MB ...
Allocating buffer size: 2116.291072 MB ...
[rank 1] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=521.97 us, min_t=517.44 us, max_t=527.33 us
[rank 2] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.16 us, min_t=517.09 us, max_t=527.94 us
[rank 3] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.08 us, min_t=518.37 us, max_t=529.86 us
[rank 4] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.21 us, min_t=516.96 us, max_t=526.94 us
[rank 6] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.24 us, min_t=515.81 us, max_t=528.64 us
[rank 0] Dispatch + combine bandwidth: 42.25 GB/s, avg_t=521.91 us, min_t=515.78 us, max_t=526.18 us
[rank 5] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.25 us, min_t=512.99 us, max_t=528.54 us
[rank 7] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.15 us, min_t=516.80 us, max_t=527.36 us
[rank 10] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=521.97 us, min_t=515.62 us, max_t=527.65 us
[rank 14] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.06 us, min_t=517.12 us, max_t=529.54 us
[rank 8] Dispatch + combine bandwidth: 42.25 GB/s, avg_t=521.90 us, min_t=518.27 us, max_t=525.02 us
[rank 9] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.05 us, min_t=517.34 us, max_t=531.46 us
[rank 11] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.06 us, min_t=519.04 us, max_t=525.92 us
[rank 12] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=522.02 us, min_t=515.87 us, max_t=528.35 us
[rank 13] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.11 us, min_t=518.02 us, max_t=526.14 us
[rank 15] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=522.00 us, min_t=516.96 us, max_t=526.14 us
[rank 2] Dispatch bandwidth: 0.25 GB/s, avg_t=30298.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9693.00 us
[rank 4] Dispatch bandwidth: 0.25 GB/s, avg_t=30292.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9694.00 us
[rank 12] Dispatch bandwidth: 0.25 GB/s, avg_t=30284.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9698.00 us
[rank 5] Dispatch bandwidth: 0.25 GB/s, avg_t=30292.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9694.00 us
[rank 8] Dispatch bandwidth: 0.25 GB/s, avg_t=30300.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9695.00 us
[rank 11] Dispatch bandwidth: 0.25 GB/s, avg_t=30302.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9681.00 us
[rank 3] Dispatch bandwidth: 0.25 GB/s, avg_t=30290.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9700.00 us
[rank 0] Dispatch bandwidth: 0.25 GB/s, avg_t=30296.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9701.00 us
[rank 10] Dispatch bandwidth: 0.25 GB/s, avg_t=30290.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9698.00 us
[rank 13] Dispatch bandwidth: 0.25 GB/s, avg_t=30279.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9699.00 us
[rank 1] Dispatch bandwidth: 0.25 GB/s, avg_t=30299.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9693.00 us
[rank 15] Dispatch bandwidth: 0.55 GB/s, avg_t=13738.00 us | Combine bandwidth: 0.54 GB/s, avg_t=26841.00 us
[rank 7] Dispatch bandwidth: 0.25 GB/s, avg_t=30285.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9697.00 us
[rank 6] Dispatch bandwidth: 0.25 GB/s, avg_t=30285.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9699.00 us
[rank 14] Dispatch bandwidth: 0.38 GB/s, avg_t=19511.00 us | Combine bandwidth: 0.70 GB/s, avg_t=20869.00 us
[rank 9] Dispatch bandwidth: 0.82 GB/s, avg_t=9146.00 us | Combine bandwidth: 0.45 GB/s, avg_t=31998.00 us
[rank 13] Dispatch send/recv time: 2700.00 us | Combine send/recv time: 170.45 us
[rank 6] Dispatch send/recv time: 30.70 us | Combine send/recv time: 40.66 us
[rank 1] Dispatch send/recv time: 32.09 us | Combine send/recv time: 41.86 us
[rank 7] Dispatch send/recv time: 30.76 us | Combine send/recv time: 40.69 us
[rank 8] Dispatch send/recv time: 2710.00 us | Combine send/recv time: 158.35 us
[rank 11] Dispatch send/recv time: 2728.00 us | Combine send/recv time: 151.44 us
[rank 12] Dispatch send/recv time: 2708.00 us | Combine send/recv time: 176.30 us
[rank 10] Dispatch send/recv time: 2722.00 us | Combine send/recv time: 181.67 us
[rank 14] Dispatch send/recv time: 29.95 us | Combine send/recv time: 3290.00 us
[rank 15] Dispatch send/recv time: 33.87 us | Combine send/recv time: 3468.00 us
[rank 9] Dispatch send/recv time: 31.15 us | Combine send/recv time: 3634.00 us
[rank 0] Dispatch send/recv time: 30.40 us | Combine send/recv time: 40.99 us
[rank 4] Dispatch send/recv time: 30.92 us | Combine send/recv time: 40.25 us
[rank 2] Dispatch send/recv time: 31.13 us | Combine send/recv time: 41.89 us
[rank 3] Dispatch send/recv time: 31.15 us | Combine send/recv time: 41.22 us
[rank 5] Dispatch send/recv time: 32.73 us | Combine send/recv time: 42.25 us
Originally created by @Qizhi697 on GitHub (Feb 28, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/28 I tested `tests/test_low_latency.py` on 2 nodes with a total of 16 H800 (8 H800 per node) with 400G CX7 connected. I wanted to reproduce the situation where `Dispatch #EP` and `Combine #EP` are both `16`. The test results show that the `Dispatch + combine bandwidth` is normal, but the `Dispatch bandwidth` or `Combine bandwidth` is extremely low. I would like to ask why this is happening. ```shell # mpirun --allow-run-as-root -x NVSHMEM_IB_GID_INDEX=3 -x MASTER_ADDR=10.82.176.210 -mca btl_tcp_if_include bond0 -mca btl self,tcp --mca pml ob1 --host 10.82.176.210:8,10.82.177.166:8 python3 tests/test_low_latency.py Allocating buffer size: 2116.291072 MB ... Allocating buffer size: 2116.291072 MB ... [rank 1] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=521.97 us, min_t=517.44 us, max_t=527.33 us [rank 2] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.16 us, min_t=517.09 us, max_t=527.94 us [rank 3] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.08 us, min_t=518.37 us, max_t=529.86 us [rank 4] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.21 us, min_t=516.96 us, max_t=526.94 us [rank 6] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.24 us, min_t=515.81 us, max_t=528.64 us [rank 0] Dispatch + combine bandwidth: 42.25 GB/s, avg_t=521.91 us, min_t=515.78 us, max_t=526.18 us [rank 5] Dispatch + combine bandwidth: 42.22 GB/s, avg_t=522.25 us, min_t=512.99 us, max_t=528.54 us [rank 7] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.15 us, min_t=516.80 us, max_t=527.36 us [rank 10] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=521.97 us, min_t=515.62 us, max_t=527.65 us [rank 14] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.06 us, min_t=517.12 us, max_t=529.54 us [rank 8] Dispatch + combine bandwidth: 42.25 GB/s, avg_t=521.90 us, min_t=518.27 us, max_t=525.02 us [rank 9] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.05 us, min_t=517.34 us, max_t=531.46 us [rank 11] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.06 us, min_t=519.04 us, max_t=525.92 us [rank 12] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=522.02 us, min_t=515.87 us, max_t=528.35 us [rank 13] Dispatch + combine bandwidth: 42.23 GB/s, avg_t=522.11 us, min_t=518.02 us, max_t=526.14 us [rank 15] Dispatch + combine bandwidth: 42.24 GB/s, avg_t=522.00 us, min_t=516.96 us, max_t=526.14 us [rank 2] Dispatch bandwidth: 0.25 GB/s, avg_t=30298.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9693.00 us [rank 4] Dispatch bandwidth: 0.25 GB/s, avg_t=30292.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9694.00 us [rank 12] Dispatch bandwidth: 0.25 GB/s, avg_t=30284.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9698.00 us [rank 5] Dispatch bandwidth: 0.25 GB/s, avg_t=30292.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9694.00 us [rank 8] Dispatch bandwidth: 0.25 GB/s, avg_t=30300.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9695.00 us [rank 11] Dispatch bandwidth: 0.25 GB/s, avg_t=30302.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9681.00 us [rank 3] Dispatch bandwidth: 0.25 GB/s, avg_t=30290.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9700.00 us [rank 0] Dispatch bandwidth: 0.25 GB/s, avg_t=30296.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9701.00 us [rank 10] Dispatch bandwidth: 0.25 GB/s, avg_t=30290.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9698.00 us [rank 13] Dispatch bandwidth: 0.25 GB/s, avg_t=30279.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9699.00 us [rank 1] Dispatch bandwidth: 0.25 GB/s, avg_t=30299.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9693.00 us [rank 15] Dispatch bandwidth: 0.55 GB/s, avg_t=13738.00 us | Combine bandwidth: 0.54 GB/s, avg_t=26841.00 us [rank 7] Dispatch bandwidth: 0.25 GB/s, avg_t=30285.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9697.00 us [rank 6] Dispatch bandwidth: 0.25 GB/s, avg_t=30285.00 us | Combine bandwidth: 1.50 GB/s, avg_t=9699.00 us [rank 14] Dispatch bandwidth: 0.38 GB/s, avg_t=19511.00 us | Combine bandwidth: 0.70 GB/s, avg_t=20869.00 us [rank 9] Dispatch bandwidth: 0.82 GB/s, avg_t=9146.00 us | Combine bandwidth: 0.45 GB/s, avg_t=31998.00 us [rank 13] Dispatch send/recv time: 2700.00 us | Combine send/recv time: 170.45 us [rank 6] Dispatch send/recv time: 30.70 us | Combine send/recv time: 40.66 us [rank 1] Dispatch send/recv time: 32.09 us | Combine send/recv time: 41.86 us [rank 7] Dispatch send/recv time: 30.76 us | Combine send/recv time: 40.69 us [rank 8] Dispatch send/recv time: 2710.00 us | Combine send/recv time: 158.35 us [rank 11] Dispatch send/recv time: 2728.00 us | Combine send/recv time: 151.44 us [rank 12] Dispatch send/recv time: 2708.00 us | Combine send/recv time: 176.30 us [rank 10] Dispatch send/recv time: 2722.00 us | Combine send/recv time: 181.67 us [rank 14] Dispatch send/recv time: 29.95 us | Combine send/recv time: 3290.00 us [rank 15] Dispatch send/recv time: 33.87 us | Combine send/recv time: 3468.00 us [rank 9] Dispatch send/recv time: 31.15 us | Combine send/recv time: 3634.00 us [rank 0] Dispatch send/recv time: 30.40 us | Combine send/recv time: 40.99 us [rank 4] Dispatch send/recv time: 30.92 us | Combine send/recv time: 40.25 us [rank 2] Dispatch send/recv time: 31.13 us | Combine send/recv time: 41.89 us [rank 3] Dispatch send/recv time: 31.15 us | Combine send/recv time: 41.22 us [rank 5] Dispatch send/recv time: 32.73 us | Combine send/recv time: 42.25 us ```
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Feb 28, 2025):

Dispatch + combine uses CUDA event to profile; while the rest tests use CUPTI (PyTorch Kineto profiler, more accuracy at microsecond level), but maybe CUPTI costs more CPU resources so that some ranks maybe launch the send/recv kernel later than other ranks. In your example, rank 8-10 seem launch eariler than other ones (keep waiting for other ranks).

[rank 8] Dispatch send/recv time: 2710.00 us | Combine send/recv time: 158.35 us
[rank 11] Dispatch send/recv time: 2728.00 us | Combine send/recv time: 151.44 us
[rank 12] Dispatch send/recv time: 2708.00 us | Combine send/recv time: 176.30 us
[rank 10] Dispatch send/recv time: 2722.00 us | Combine send/recv time: 181.67 us

You can try to add the trace_path to check profiling JSON (align the timeline of all ranks) or switch on/off the barrier_comm_profiling (add/remove a barrier before launching the kernel) option in bench_kineto and observe results.

If the problem is from CPU, one possible solution is to bind NUMA for CPU cores.

If the problem is not from CPU, try to check whether other processes run on GPU/PCIe/network card/switch/background traffic.

<!-- gh-comment-id:2689754386 --> @LyricZhao commented on GitHub (Feb 28, 2025): `Dispatch + combine` uses CUDA event to profile; while the rest tests use CUPTI (PyTorch Kineto profiler, more accuracy at microsecond level), but maybe CUPTI costs more CPU resources so that some ranks maybe launch the send/recv kernel later than other ranks. In your example, rank 8-10 seem launch eariler than other ones (keep waiting for other ranks). > [rank 8] Dispatch send/recv time: 2710.00 us | Combine send/recv time: 158.35 us [rank 11] Dispatch send/recv time: 2728.00 us | Combine send/recv time: 151.44 us [rank 12] Dispatch send/recv time: 2708.00 us | Combine send/recv time: 176.30 us [rank 10] Dispatch send/recv time: 2722.00 us | Combine send/recv time: 181.67 us You can try to add the `trace_path` to check profiling JSON (align the timeline of all ranks) or switch on/off the `barrier_comm_profiling` (add/remove a barrier before launching the kernel) option in `bench_kineto` and observe results. If the problem is from CPU, one possible solution is to bind NUMA for CPU cores. If the problem is not from CPU, try to check whether other processes run on GPU/PCIe/network card/switch/background traffic.
bsakaguchi commented 2026-10-26 09:52:48 +00:00
Author
Owner
Copy link

@haswelliris commented on GitHub (Feb 28, 2025):

It appears that the test was conducted in a RoCE network. I recommend checking the following network settings to ensure optimal performance:

  1. GPU:NIC Ratio: Is it configured as 1:1?
  2. Traffic Class: Did you configure NVSHMEM_IB_TRAFFIC_CLASS or set a default DSCP value? These settings are essential for lossless RDMA traffic.
  3. PFC: RoCE networks require PFC to be properly configured to prevent packet loss and ensure reliable communication.
<!-- gh-comment-id:2690203455 --> @haswelliris commented on GitHub (Feb 28, 2025): It appears that the test was conducted in a RoCE network. I recommend checking the following network settings to ensure optimal performance: 1. **GPU:NIC Ratio**: Is it configured as 1:1? 2. **Traffic Class**: Did you configure `NVSHMEM_IB_TRAFFIC_CLASS` or set a default DSCP value? These settings are essential for lossless RDMA traffic. 3. **PFC**: RoCE networks require PFC to be properly configured to prevent packet loss and ensure reliable communication.
bsakaguchi commented 2026-10-26 09:52:48 +00:00
Author
Owner
Copy link

@Qizhi697 commented on GitHub (Mar 2, 2025):

I am truly grateful for the help from both developers. I have tried each of your suggestions and made some progress, but I still encountered some unexplained phenomena. I hope to get your assistance.

@haswelliris

  1. The GPU:NIC ratio is 1:1.
  2. After configuring NVSHMEM_IB_TRAFFIC_CLASS, there was no change in bandwidth.
  3. PFC has been correctly configured.
  4. I use RoCE v2 network.

@LyricZhao

By default, barrier_comm_profiling is switched on, resulting in very low bandwidth in my reproduce. After I turned off barrier_comm_profiling, the situation improved significantly, and the combined bandwidth was basically normal at 42-43GB/s. However, the dispatch bandwidth is still low by about 25% and bind NUMA for CPU cores did not make any difference.

Additionally, I discovered 4 strange phenomena:

  1. After turning off barrier_comm_profiling, both combine and dispatch became much faster. Why do the previous matrix multiplication and all_reduce synchronization operations affect dispatch/combine?
  2. When barrier_comm_profiling is on, the initial few dispatch/combine operations are much faster compared to the later ones.
  3. When barrier_comm_profiling is off, the trace recording starts from combine, resulting in only 59 dispatch modules (one is missing).
  4. When barrier_comm_profiling is off, there is always one dispatch operation that takes an unusually long time.

The experimental data regarding the above phenomena is also provided below for reference. I truly appreciate all the help and support you have provided. Thank you so much for your assistance.

switch on
Image

[rank 8] Dispatch bandwidth: 0.25 GB/s, avg_t=29868.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12480.00 us
[rank 12] Dispatch bandwidth: 0.25 GB/s, avg_t=29862.00 us | Combine bandwidth: 1.17 GB/s, avg_t=12477.00 us
[rank 0] Dispatch bandwidth: 0.25 GB/s, avg_t=29870.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12478.00 us
[rank 9] Dispatch bandwidth: 0.25 GB/s, avg_t=29871.00 us | Combine bandwidth: 1.17 GB/s, avg_t=12472.00 us
[rank 4] Dispatch bandwidth: 0.25 GB/s, avg_t=30349.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12484.00 us
[rank 10] Dispatch bandwidth: 0.25 GB/s, avg_t=29862.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12482.00 us
[rank 15] Dispatch bandwidth: 0.25 GB/s, avg_t=29850.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12483.00 us
[rank 2] Dispatch bandwidth: 0.25 GB/s, avg_t=29860.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12483.00 us
[rank 6] Dispatch bandwidth: 0.25 GB/s, avg_t=30348.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12486.00 us
[rank 5] Dispatch bandwidth: 0.24 GB/s, avg_t=30874.00 us | Combine bandwidth: 1.15 GB/s, avg_t=12681.00 us
[rank 14] Dispatch bandwidth: 0.34 GB/s, avg_t=21850.00 us | Combine bandwidth: 0.70 GB/s, avg_t=20785.00 us
[rank 1] Dispatch bandwidth: 0.25 GB/s, avg_t=29861.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12484.00 us
[rank 11] Dispatch bandwidth: 0.63 GB/s, avg_t=11965.00 us | Combine bandwidth: 0.47 GB/s, avg_t=30992.00 us
[rank 13] Dispatch bandwidth: 0.61 GB/s, avg_t=12249.00 us | Combine bandwidth: 0.47 GB/s, avg_t=31152.00 us
[rank 3] Dispatch bandwidth: 0.25 GB/s, avg_t=29863.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12478.00 us
[rank 7] Dispatch bandwidth: 0.25 GB/s, avg_t=29848.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12486.00 us
[rank 5] Dispatch send/recv time: 32.77 us | Combine send/recv time: 42.20 us
[rank 3] Dispatch send/recv time: 31.22 us | Combine send/recv time: 41.40 us
[rank 1] Dispatch send/recv time: 32.42 us | Combine send/recv time: 42.02 us
[rank 7] Dispatch send/recv time: 30.81 us | Combine send/recv time: 40.85 us
[rank 14] Dispatch send/recv time: 31.10 us | Combine send/recv time: 43.14 us
[rank 13] Dispatch send/recv time: 30.56 us | Combine send/recv time: 348.94 us
[rank 6] Dispatch send/recv time: 30.97 us | Combine send/recv time: 40.69 us
[rank 11] Dispatch send/recv time: 31.03 us | Combine send/recv time: 359.33 us
[rank 8] Dispatch send/recv time: 30.94 us | Combine send/recv time: 40.10 us
[rank 9] Dispatch send/recv time: 32.28 us | Combine send/recv time: 42.87 us
[rank 0] Dispatch send/recv time: 30.55 us | Combine send/recv time: 40.95 us
[rank 10] Dispatch send/recv time: 31.52 us | Combine send/recv time: 42.64 us
[rank 15] Dispatch send/recv time: 31.00 us | Combine send/recv time: 41.24 us
[rank 4] Dispatch send/recv time: 30.82 us | Combine send/recv time: 40.14 us
[rank 2] Dispatch send/recv time: 31.23 us | Combine send/recv time: 41.86 us
[rank 12] Dispatch send/recv time: 32.34 us | Combine send/recv time: 41.85 us

switch off
Image

[rank 9] Dispatch bandwidth: 40.12 GB/s, avg_t=187.22 us | Combine bandwidth: 42.25 GB/s, avg_t=344.05 us
[rank 7] Dispatch bandwidth: 33.05 GB/s, avg_t=227.28 us | Combine bandwidth: 42.70 GB/s, avg_t=340.42 us
[rank 11] Dispatch bandwidth: 33.52 GB/s, avg_t=224.11 us | Combine bandwidth: 42.19 GB/s, avg_t=344.53 us
[rank 1] Dispatch bandwidth: 33.31 GB/s, avg_t=225.53 us | Combine bandwidth: 42.22 GB/s, avg_t=344.34 us
[rank 14] Dispatch bandwidth: 36.13 GB/s, avg_t=207.91 us | Combine bandwidth: 42.08 GB/s, avg_t=345.49 us
[rank 3] Dispatch bandwidth: 33.14 GB/s, avg_t=226.68 us | Combine bandwidth: 42.55 GB/s, avg_t=341.64 us
[rank 15] Dispatch bandwidth: 33.73 GB/s, avg_t=222.69 us | Combine bandwidth: 41.81 GB/s, avg_t=347.68 us
[rank 4] Dispatch bandwidth: 33.28 GB/s, avg_t=225.75 us | Combine bandwidth: 42.20 GB/s, avg_t=344.46 us
[rank 12] Dispatch bandwidth: 33.44 GB/s, avg_t=224.63 us | Combine bandwidth: 42.18 GB/s, avg_t=344.66 us
[rank 2] Dispatch bandwidth: 32.58 GB/s, avg_t=230.57 us | Combine bandwidth: 42.89 GB/s, avg_t=338.95 us
[rank 5] Dispatch bandwidth: 33.58 GB/s, avg_t=223.72 us | Combine bandwidth: 42.00 GB/s, avg_t=346.11 us
[rank 8] Dispatch bandwidth: 36.79 GB/s, avg_t=204.18 us | Combine bandwidth: 41.56 GB/s, avg_t=349.80 us
[rank 6] Dispatch bandwidth: 33.62 GB/s, avg_t=223.45 us | Combine bandwidth: 42.04 GB/s, avg_t=345.82 us
[rank 0] Dispatch bandwidth: 32.39 GB/s, avg_t=231.90 us | Combine bandwidth: 43.08 GB/s, avg_t=337.47 us
[rank 13] Dispatch bandwidth: 33.43 GB/s, avg_t=224.69 us | Combine bandwidth: 42.19 GB/s, avg_t=344.53 us
[rank 10] Dispatch bandwidth: 43.07 GB/s, avg_t=174.42 us | Combine bandwidth: 42.25 GB/s, avg_t=344.04 us
[rank 9] Dispatch send/recv time: 31.98 us | Combine send/recv time: 65.98 us
[rank 14] Dispatch send/recv time: 30.79 us | Combine send/recv time: 40.30 us
[rank 15] Dispatch send/recv time: 30.89 us | Combine send/recv time: 40.80 us
[rank 2] Dispatch send/recv time: 31.10 us | Combine send/recv time: 41.25 us
[rank 4] Dispatch send/recv time: 30.59 us | Combine send/recv time: 40.18 us
[rank 1] Dispatch send/recv time: 32.09 us | Combine send/recv time: 41.58 us
[rank 7] Dispatch send/recv time: 30.57 us | Combine send/recv time: 40.42 us
[rank 12] Dispatch send/recv time: 31.94 us | Combine send/recv time: 41.02 us
[rank 8] Dispatch send/recv time: 30.75 us | Combine send/recv time: 40.00 us
[rank 3] Dispatch send/recv time: 31.01 us | Combine send/recv time: 41.13 us
[rank 5] Dispatch send/recv time: 32.57 us | Combine send/recv time: 42.27 us
[rank 11] Dispatch send/recv time: 30.87 us | Combine send/recv time: 40.97 us
[rank 6] Dispatch send/recv time: 30.54 us | Combine send/recv time: 40.32 us
[rank 10] Dispatch send/recv time: 31.32 us | Combine send/recv time: 42.23 us
[rank 0] Dispatch send/recv time: 30.35 us | Combine send/recv time: 40.65 us
[rank 13] Dispatch send/recv time: 30.38 us | Combine send/recv time: 39.83 us
<!-- gh-comment-id:2692832213 --> @Qizhi697 commented on GitHub (Mar 2, 2025): I am truly grateful for the help from both developers. I have tried each of your suggestions and made some progress, but I still encountered some unexplained phenomena. I hope to get your assistance. @haswelliris 1. The GPU:NIC ratio is `1:1`. 2. After configuring `NVSHMEM_IB_TRAFFIC_CLASS`, there was no change in bandwidth. 3. `PFC` has been correctly configured. 4. I use `RoCE v2` network. @LyricZhao By default, `barrier_comm_profiling` is switched on, resulting in very low bandwidth in my reproduce. After I turned off `barrier_comm_profiling`, the situation improved significantly, and the combined bandwidth was basically normal at `42-43GB/s`. However, the dispatch bandwidth is still low by about `25%` and bind NUMA for CPU cores did not make any difference. Additionally, I discovered 4 strange phenomena: 1. After turning off `barrier_comm_profiling`, both combine and dispatch became much faster. Why do the previous `matrix multiplication` and `all_reduce` synchronization operations affect `dispatch/combine`? 2. When `barrier_comm_profiling` is on, the initial few `dispatch/combine` operations are much faster compared to the later ones. 3. When `barrier_comm_profiling` is off, the trace recording starts from combine, resulting in only `59` dispatch modules (one is missing). 4. When `barrier_comm_profiling` is off, there is `always one dispatch` operation that takes an unusually long time. The experimental data regarding the above phenomena is also provided below for reference. I truly appreciate all the help and support you have provided. Thank you so much for your assistance. `switch on` ![Image](https://github.com/user-attachments/assets/b57a4dba-ba21-43cc-8c13-bdb69845f368) ```shell [rank 8] Dispatch bandwidth: 0.25 GB/s, avg_t=29868.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12480.00 us [rank 12] Dispatch bandwidth: 0.25 GB/s, avg_t=29862.00 us | Combine bandwidth: 1.17 GB/s, avg_t=12477.00 us [rank 0] Dispatch bandwidth: 0.25 GB/s, avg_t=29870.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12478.00 us [rank 9] Dispatch bandwidth: 0.25 GB/s, avg_t=29871.00 us | Combine bandwidth: 1.17 GB/s, avg_t=12472.00 us [rank 4] Dispatch bandwidth: 0.25 GB/s, avg_t=30349.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12484.00 us [rank 10] Dispatch bandwidth: 0.25 GB/s, avg_t=29862.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12482.00 us [rank 15] Dispatch bandwidth: 0.25 GB/s, avg_t=29850.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12483.00 us [rank 2] Dispatch bandwidth: 0.25 GB/s, avg_t=29860.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12483.00 us [rank 6] Dispatch bandwidth: 0.25 GB/s, avg_t=30348.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12486.00 us [rank 5] Dispatch bandwidth: 0.24 GB/s, avg_t=30874.00 us | Combine bandwidth: 1.15 GB/s, avg_t=12681.00 us [rank 14] Dispatch bandwidth: 0.34 GB/s, avg_t=21850.00 us | Combine bandwidth: 0.70 GB/s, avg_t=20785.00 us [rank 1] Dispatch bandwidth: 0.25 GB/s, avg_t=29861.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12484.00 us [rank 11] Dispatch bandwidth: 0.63 GB/s, avg_t=11965.00 us | Combine bandwidth: 0.47 GB/s, avg_t=30992.00 us [rank 13] Dispatch bandwidth: 0.61 GB/s, avg_t=12249.00 us | Combine bandwidth: 0.47 GB/s, avg_t=31152.00 us [rank 3] Dispatch bandwidth: 0.25 GB/s, avg_t=29863.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12478.00 us [rank 7] Dispatch bandwidth: 0.25 GB/s, avg_t=29848.00 us | Combine bandwidth: 1.16 GB/s, avg_t=12486.00 us [rank 5] Dispatch send/recv time: 32.77 us | Combine send/recv time: 42.20 us [rank 3] Dispatch send/recv time: 31.22 us | Combine send/recv time: 41.40 us [rank 1] Dispatch send/recv time: 32.42 us | Combine send/recv time: 42.02 us [rank 7] Dispatch send/recv time: 30.81 us | Combine send/recv time: 40.85 us [rank 14] Dispatch send/recv time: 31.10 us | Combine send/recv time: 43.14 us [rank 13] Dispatch send/recv time: 30.56 us | Combine send/recv time: 348.94 us [rank 6] Dispatch send/recv time: 30.97 us | Combine send/recv time: 40.69 us [rank 11] Dispatch send/recv time: 31.03 us | Combine send/recv time: 359.33 us [rank 8] Dispatch send/recv time: 30.94 us | Combine send/recv time: 40.10 us [rank 9] Dispatch send/recv time: 32.28 us | Combine send/recv time: 42.87 us [rank 0] Dispatch send/recv time: 30.55 us | Combine send/recv time: 40.95 us [rank 10] Dispatch send/recv time: 31.52 us | Combine send/recv time: 42.64 us [rank 15] Dispatch send/recv time: 31.00 us | Combine send/recv time: 41.24 us [rank 4] Dispatch send/recv time: 30.82 us | Combine send/recv time: 40.14 us [rank 2] Dispatch send/recv time: 31.23 us | Combine send/recv time: 41.86 us [rank 12] Dispatch send/recv time: 32.34 us | Combine send/recv time: 41.85 us ``` `switch off` ![Image](https://github.com/user-attachments/assets/9313279b-bd66-4a66-9d96-e21baf3a7384) ```shell [rank 9] Dispatch bandwidth: 40.12 GB/s, avg_t=187.22 us | Combine bandwidth: 42.25 GB/s, avg_t=344.05 us [rank 7] Dispatch bandwidth: 33.05 GB/s, avg_t=227.28 us | Combine bandwidth: 42.70 GB/s, avg_t=340.42 us [rank 11] Dispatch bandwidth: 33.52 GB/s, avg_t=224.11 us | Combine bandwidth: 42.19 GB/s, avg_t=344.53 us [rank 1] Dispatch bandwidth: 33.31 GB/s, avg_t=225.53 us | Combine bandwidth: 42.22 GB/s, avg_t=344.34 us [rank 14] Dispatch bandwidth: 36.13 GB/s, avg_t=207.91 us | Combine bandwidth: 42.08 GB/s, avg_t=345.49 us [rank 3] Dispatch bandwidth: 33.14 GB/s, avg_t=226.68 us | Combine bandwidth: 42.55 GB/s, avg_t=341.64 us [rank 15] Dispatch bandwidth: 33.73 GB/s, avg_t=222.69 us | Combine bandwidth: 41.81 GB/s, avg_t=347.68 us [rank 4] Dispatch bandwidth: 33.28 GB/s, avg_t=225.75 us | Combine bandwidth: 42.20 GB/s, avg_t=344.46 us [rank 12] Dispatch bandwidth: 33.44 GB/s, avg_t=224.63 us | Combine bandwidth: 42.18 GB/s, avg_t=344.66 us [rank 2] Dispatch bandwidth: 32.58 GB/s, avg_t=230.57 us | Combine bandwidth: 42.89 GB/s, avg_t=338.95 us [rank 5] Dispatch bandwidth: 33.58 GB/s, avg_t=223.72 us | Combine bandwidth: 42.00 GB/s, avg_t=346.11 us [rank 8] Dispatch bandwidth: 36.79 GB/s, avg_t=204.18 us | Combine bandwidth: 41.56 GB/s, avg_t=349.80 us [rank 6] Dispatch bandwidth: 33.62 GB/s, avg_t=223.45 us | Combine bandwidth: 42.04 GB/s, avg_t=345.82 us [rank 0] Dispatch bandwidth: 32.39 GB/s, avg_t=231.90 us | Combine bandwidth: 43.08 GB/s, avg_t=337.47 us [rank 13] Dispatch bandwidth: 33.43 GB/s, avg_t=224.69 us | Combine bandwidth: 42.19 GB/s, avg_t=344.53 us [rank 10] Dispatch bandwidth: 43.07 GB/s, avg_t=174.42 us | Combine bandwidth: 42.25 GB/s, avg_t=344.04 us [rank 9] Dispatch send/recv time: 31.98 us | Combine send/recv time: 65.98 us [rank 14] Dispatch send/recv time: 30.79 us | Combine send/recv time: 40.30 us [rank 15] Dispatch send/recv time: 30.89 us | Combine send/recv time: 40.80 us [rank 2] Dispatch send/recv time: 31.10 us | Combine send/recv time: 41.25 us [rank 4] Dispatch send/recv time: 30.59 us | Combine send/recv time: 40.18 us [rank 1] Dispatch send/recv time: 32.09 us | Combine send/recv time: 41.58 us [rank 7] Dispatch send/recv time: 30.57 us | Combine send/recv time: 40.42 us [rank 12] Dispatch send/recv time: 31.94 us | Combine send/recv time: 41.02 us [rank 8] Dispatch send/recv time: 30.75 us | Combine send/recv time: 40.00 us [rank 3] Dispatch send/recv time: 31.01 us | Combine send/recv time: 41.13 us [rank 5] Dispatch send/recv time: 32.57 us | Combine send/recv time: 42.27 us [rank 11] Dispatch send/recv time: 30.87 us | Combine send/recv time: 40.97 us [rank 6] Dispatch send/recv time: 30.54 us | Combine send/recv time: 40.32 us [rank 10] Dispatch send/recv time: 31.32 us | Combine send/recv time: 42.23 us [rank 0] Dispatch send/recv time: 30.35 us | Combine send/recv time: 40.65 us [rank 13] Dispatch send/recv time: 30.38 us | Combine send/recv time: 39.83 us ```
bsakaguchi commented 2026-10-26 09:52:48 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

After turning off barrier_comm_profiling, both combine and dispatch became much faster. Why do the previous matrix multiplication and all_reduce synchronization operations affect dispatch/combine?

The GPU all_reduce is used to eliminate the CPU launch overhead, consider two ranks launch the kernel with a huge time gap, the fast rank will wait for the slower one, will may introduce inaccurate timing.

But still, if all_reduce kernel execution is too fast and cannot cover the different launch overhead (or the kernel gap between all_reduce and later communication kernels) of the dispatch/combine kernel, so we need a GEMM here to make sure the kernel gap is consistent. For example, after the all-reduce is finished (rank 0 and rank 1 have no gap launching the all-reduce), rank 0 costs 1 CPU secs to launch dispatch, rank 1 cost 2 CPU seconds to launch, the rank 1 has to wait rank 0 for another 2 - 1 secs, which is also not accurate. So if the GEMM is long enough, e.g. 10 secs, then the CPU gap is eliminated.

When barrier_comm_profiling is on, the initial few dispatch/combine operations are much faster compared to the later ones.

It is not the expected behavior, maybe you can try to do a large GEMM before every run (currently the GEMM and all-reduce is done only before the tests for once). I guess the CPU launching or GPU freq is getting worse during your run. You can align the timeline of different ranks and test.

After I turned off barrier_comm_profiling, the situation improved significantly, and the combined bandwidth was basically normal at 42-43GB/s.

This is also related to the previous question, dispatch is a natural barrier (just like the all-reduce) for combine. Maybe always only dispatch suffers from the launch gap problem.

When barrier_comm_profiling is off, the trace recording starts from combine, resulting in only 59 dispatch modules (one is missing).

I didn't check the code, but I guess it doesn't matter. As the profile table counts for average time. If you are interested in fixing it, please submit a PR.

When barrier_comm_profiling is off, there is always one dispatch operation that takes an unusually long time.

Similar to above answers, suffering from launching gap.

<!-- gh-comment-id:2693181630 --> @LyricZhao commented on GitHub (Mar 3, 2025): > After turning off barrier_comm_profiling, both combine and dispatch became much faster. Why do the previous matrix multiplication and all_reduce synchronization operations affect dispatch/combine? The GPU `all_reduce` is used to eliminate the CPU launch overhead, consider two ranks launch the kernel with a huge time gap, the fast rank will wait for the slower one, will may introduce inaccurate timing. But still, if `all_reduce` kernel execution is too fast and cannot cover the different launch overhead (or the kernel gap between `all_reduce` and later communication kernels) of the dispatch/combine kernel, so we need a GEMM here to make sure the kernel gap is consistent. For example, **after the all-reduce is finished** (rank 0 and rank 1 have no gap launching the all-reduce), rank 0 costs 1 CPU secs to launch dispatch, rank 1 cost 2 CPU seconds to launch, the rank 1 has to wait rank 0 for another `2 - 1` secs, which is also not accurate. So if the GEMM is long enough, e.g. 10 secs, then the CPU gap is eliminated. > When barrier_comm_profiling is on, the initial few dispatch/combine operations are much faster compared to the later ones. It is not the expected behavior, maybe you can try to do a large GEMM before **every run** (currently the GEMM and all-reduce is done only before the tests for once). I guess the CPU launching or GPU freq is getting worse during your run. You can align the timeline of different ranks and test. > After I turned off barrier_comm_profiling, the situation improved significantly, and the combined bandwidth was basically normal at 42-43GB/s. This is also related to the previous question, dispatch is a natural barrier (just like the all-reduce) for combine. Maybe always only dispatch suffers from the launch gap problem. > When barrier_comm_profiling is off, the trace recording starts from combine, resulting in only 59 dispatch modules (one is missing). I didn't check the code, but I guess it doesn't matter. As the profile table counts for average time. If you are interested in fixing it, please submit a PR. > When barrier_comm_profiling is off, there is always one dispatch operation that takes an unusually long time. Similar to above answers, suffering from launching gap.
bsakaguchi commented 2026-10-26 09:52:48 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

BTW, I also suggest you to test in your model run, as the real cases are also different from the tests. Testing only matters when it is close to the real online scenarios.

<!-- gh-comment-id:2693183383 --> @LyricZhao commented on GitHub (Mar 3, 2025): BTW, I also suggest you to test in your model run, as the real cases are also different from the tests. Testing only matters when it is close to the real online scenarios.
bsakaguchi commented 2026-10-26 09:52:49 +00:00
Author
Owner
Copy link

@Qizhi697 commented on GitHub (Mar 3, 2025):

When barrier_comm_profiling is off, only the first dispatch kernel takes a longer time, which might be due to the lack of synchronization between different ranks. After that, the dispatch/combine times are quite normal (around 170us).

However, when barrier_comm_profiling is on, I don't understand why all dispatch/combine kernel execution times become extremely long (around 30ms). Do you have any insights on this phenomenon?

Image
<!-- gh-comment-id:2693728130 --> @Qizhi697 commented on GitHub (Mar 3, 2025): When `barrier_comm_profiling` is off, only the first dispatch kernel takes a longer time, which might be due to the lack of synchronization between different ranks. After that, the `dispatch/combine` times are quite normal (around 170us). However, when `barrier_comm_profiling` is on, I don't understand why all `dispatch/combine` kernel execution times become extremely long (around 30ms). Do you have any insights on this phenomenon? <img width="1504" alt="Image" src="https://github.com/user-attachments/assets/8033d2cd-3221-44ad-bd8f-cc3651890d3f" />
bsakaguchi commented 2026-10-26 09:52:49 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

However, when barrier_comm_profiling is on, I don't understand why all dispatch/combine kernel execution times become extremely long (around 30ms). Do you have any insights on this phenomenon?

Could you please check the GEMM and all-reduce time in different ranks? e.g. Are they aligned in wall clock?

Or try to remove the GEMM call?

<!-- gh-comment-id:2693736713 --> @LyricZhao commented on GitHub (Mar 3, 2025): > However, when barrier_comm_profiling is on, I don't understand why all dispatch/combine kernel execution times become extremely long (around 30ms). Do you have any insights on this phenomenon? Could you please check the GEMM and all-reduce time in different ranks? e.g. Are they aligned in wall clock? Or try to remove the GEMM call?
bsakaguchi commented 2026-10-26 09:52:49 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

And, overall, I think it is an issue only related to testing, maybe not a problem while model running.

<!-- gh-comment-id:2693738253 --> @LyricZhao commented on GitHub (Mar 3, 2025): And, overall, I think it is an issue only related to testing, maybe not a problem while model running.
bsakaguchi commented 2026-10-26 09:52:50 +00:00
Author
Owner
Copy link

@Qizhi697 commented on GitHub (Mar 3, 2025):

Or try to remove the GEMM call?

I have removed the GEMM above, and having only all-reduce causes the dispatch/combine execution time to be extremely long(around 30ms).

Could you please check the GEMM and all-reduce time in different ranks? e.g. Are they aligned in wall clock?

How can we compare the wall clock times of different nodes, given that the system times of the two nodes themselves may differ?

I want to observe the kernel execution purely from the communication perspective.

I want to observe the kernel execution purely from the communication perspective. Haha~

<!-- gh-comment-id:2693776221 --> @Qizhi697 commented on GitHub (Mar 3, 2025): >Or try to remove the GEMM call? I have removed the GEMM above, and having only all-reduce causes the dispatch/combine execution time to be extremely long(around 30ms). >Could you please check the GEMM and all-reduce time in different ranks? e.g. Are they aligned in wall clock? How can we compare the wall clock times of different nodes, given that the system times of the two nodes themselves may differ? > I want to observe the kernel execution purely from the communication perspective. I want to observe the kernel execution purely from the communication perspective. Haha~
bsakaguchi commented 2026-10-26 09:52:50 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

How can we compare the wall clock times of different nodes, given that the system times of the two nodes themselves may differ?

You can align the end of all-reduce kernels as the same timestamp.

<!-- gh-comment-id:2693789494 --> @LyricZhao commented on GitHub (Mar 3, 2025): > How can we compare the wall clock times of different nodes, given that the system times of the two nodes themselves may differ? You can align the **end** of all-reduce kernels as the same timestamp.
bsakaguchi commented 2026-10-26 09:52:50 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

In conclusion, I guess it must be related to your GEMM time (different node different time, someone else occuping your node?); NCCL all-reduce issues (network issues? slow traffic for some certain node?); CPU launch issues (CPU of some nodes is slowing down). All you have to do is to find something reliable to align the wall clock, and figure out which rank is slow and the slowing reason.

<!-- gh-comment-id:2693800288 --> @LyricZhao commented on GitHub (Mar 3, 2025): In conclusion, I guess it must be related to your GEMM time (different node different time, someone else occuping your node?); NCCL all-reduce issues (network issues? slow traffic for some certain node?); CPU launch issues (CPU of some nodes is slowing down). All you have to do is to find something reliable to align the wall clock, and figure out which rank is slow and the slowing reason.
bsakaguchi commented 2026-10-26 09:52:50 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 3, 2025):

BTW, the low-latency kernels can run on a 8-rank node (no need for two nodes), NVLink is automatically disabled and traffic goes via IBGDA.

<!-- gh-comment-id:2693805328 --> @LyricZhao commented on GitHub (Mar 3, 2025): BTW, the low-latency kernels can run on a 8-rank node (no need for two nodes), NVLink is automatically disabled and traffic goes via IBGDA.
bsakaguchi commented 2026-10-26 09:52:50 +00:00
Author
Owner
Copy link

@Qizhi697 commented on GitHub (Mar 3, 2025):

You can align the end of all-reduce kernels as the same timestamp.

I recorded the execution duration of the kernel for all_reduce and the first dispatch after all_reduce. In theory, the execution time of the first dispatch kernel immediately following all_reduce should be the same. However, I found that there are always some abnormally slow nodes, and these nodes are not consistent.

           all_reduce             dispatch
-------------------------------------------------
rank0   33ms 670us 466ns         653us 50ns
rank1   33ms 768us 906ns         506us 365ns
rank2   33ms 829us 611ns         507us 644ns
rank3   33ms 879us 64ns          460us 926ns
rank4   33ms 986us 737ns         362us 332ns
rank5   34ms 34us  125ns         297us 694ns
rank6   34ms 41us  117ns         287us 229ns
rank7   34ms 89us  463ns         110us 942ns
-------------------------------------------------
rank8   33ms 660us 654ns         680us 255ns
rank9   33ms 627us 444ns         688us 412ns
rank10  33ms 723us 853ns         621us 375ns
rank11       545us 886ns    16ms 165us 279ns
rank12  33ms 837us 337ns         535us 67ns
rank13  33ms 934us 173ns         390us 205ns
rank14  33ms 995us 212ns         320us 191ns
rank15  17ms 157us 811ns    22ms 106us 807ns

BTW, the low-latency kernels can run on a 8-rank node (no need for two nodes), NVLink is automatically disabled and traffic goes via IBGDA.

The results of my tests on a single machine with 8 GPUs showed no issues.

This is also related to the previous question, dispatch is a natural barrier (just like the all-reduce) for combine. Maybe always only dispatch suffers from the launch gap problem.

Since dispatch is a natural barrier (just like the all-reduce) for combine, which eliminates the impact of the launch gap, the bandwidth of combine still becomes extremely low after I swtch on all_reduce.

<!-- gh-comment-id:2694291713 --> @Qizhi697 commented on GitHub (Mar 3, 2025): > You can align the end of all-reduce kernels as the same timestamp. I recorded the execution duration of the kernel for `all_reduce` and the first dispatch after `all_reduce`. In theory, the execution time of the `first dispatch kernel` immediately following `all_reduce` should be the same. However, I found that there are always some abnormally `slow nodes`, and these nodes are not `consistent`. ```table all_reduce dispatch ------------------------------------------------- rank0 33ms 670us 466ns 653us 50ns rank1 33ms 768us 906ns 506us 365ns rank2 33ms 829us 611ns 507us 644ns rank3 33ms 879us 64ns 460us 926ns rank4 33ms 986us 737ns 362us 332ns rank5 34ms 34us 125ns 297us 694ns rank6 34ms 41us 117ns 287us 229ns rank7 34ms 89us 463ns 110us 942ns ------------------------------------------------- rank8 33ms 660us 654ns 680us 255ns rank9 33ms 627us 444ns 688us 412ns rank10 33ms 723us 853ns 621us 375ns rank11 545us 886ns 16ms 165us 279ns rank12 33ms 837us 337ns 535us 67ns rank13 33ms 934us 173ns 390us 205ns rank14 33ms 995us 212ns 320us 191ns rank15 17ms 157us 811ns 22ms 106us 807ns ``` >BTW, the low-latency kernels can run on a 8-rank node (no need for two nodes), NVLink is automatically disabled and traffic goes via IBGDA. The results of my tests on a single machine with 8 GPUs showed no issues. >This is also related to the previous question, dispatch is a natural barrier (just like the all-reduce) for combine. Maybe always only dispatch suffers from the launch gap problem. Since `dispatch` is a natural barrier (just like the `all-reduce`) for `combine`, which eliminates the impact of the launch gap, the bandwidth of `combine` still becomes extremely low after I swtch on `all_reduce`.
bsakaguchi commented 2026-10-26 09:52:51 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 4, 2025):

I recorded the execution duration of the kernel for all_reduce and the first dispatch after all_reduce. In theory, the execution time of the first dispatch kernel immediately following all_reduce should be the same. However, I found that there are always some abnormally slow nodes, and these nodes are not consistent.

How about the gap between all-reduce and dispatch? Is CPU launching costing too much?

Anyway, we didn't do anything special for each rank, every rank executes the same code. I strongly suggest you to investigate your cluster network settings carefully.

<!-- gh-comment-id:2695976120 --> @LyricZhao commented on GitHub (Mar 4, 2025): > I recorded the execution duration of the kernel for all_reduce and the first dispatch after all_reduce. In theory, the execution time of the first dispatch kernel immediately following all_reduce should be the same. However, I found that there are always some abnormally slow nodes, and these nodes are not consistent. How about the gap between all-reduce and dispatch? Is CPU launching costing too much? Anyway, we didn't do anything special for each rank, every rank executes the same code. I strongly suggest you to investigate your cluster network settings carefully.
bsakaguchi commented 2026-10-26 09:52:51 +00:00
Author
Owner
Copy link

@MarkLiJing commented on GitHub (Mar 7, 2025):

barrier_comm_profiling

how to turn off this feature？

<!-- gh-comment-id:2705387932 --> @MarkLiJing commented on GitHub (Mar 7, 2025): > barrier_comm_profiling how to turn off this feature？
bsakaguchi commented 2026-10-26 09:52:51 +00:00
Author
Owner
Copy link

@Qizhi697 commented on GitHub (Mar 7, 2025):

@MarkLiJing
github.com/deepseek-ai/DeepEP@1fc40d50f3/tests/test_low_latency.py (L120-L125)

By set barrier_comm_profiling=False

<!-- gh-comment-id:2705615964 --> @Qizhi697 commented on GitHub (Mar 7, 2025): @MarkLiJing https://github.com/deepseek-ai/DeepEP/blob/1fc40d50f3b199758b57b775e85c1d01080482d9/tests/test_low_latency.py#L120-L125 By set `barrier_comm_profiling=False`
bsakaguchi commented 2026-10-26 09:52:51 +00:00
Author
Owner
Copy link

@HodBadichi commented on GitHub (Aug 22, 2025):

Following up on the discussion with @Qizhi697: I'm also seeing a ~10ms end-to-end latency despite GEMM , all reduce operations having uniform and fast timings across all ranks. Any ideas on what could be causing this bottleneck or a methodologic way to check it out ?

<!-- gh-comment-id:3215777098 --> @HodBadichi commented on GitHub (Aug 22, 2025): Following up on the discussion with @Qizhi697: I'm also seeing a ~10ms end-to-end latency despite GEMM , all reduce operations having uniform and fast timings across all ranks. Any ideas on what could be causing this bottleneck or a methodologic way to check it out ?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#29
No description provided.