deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #75] curious about the difference between latency and throughput #69

New issue
Open
opened 2026-10-26 09:53:39 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-26 09:53:39 +00:00
Owner
Copy link

Originally created by @Eutenacity on GitHub (Mar 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/75

It seems that EP-DISPATCH and EP-COMBINE both work well in throughput tests while EP-DISPATCH works much better than EP-COMBINE in latency tests.
Why?

Originally created by @Eutenacity on GitHub (Mar 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/75 It seems that EP-DISPATCH and EP-COMBINE both work well in throughput tests while EP-DISPATCH works much better than EP-COMBINE in latency tests. Why?
bsakaguchi commented 2026-10-26 09:53:40 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 17, 2025):

Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s

<!-- gh-comment-id:2727794855 --> @LyricZhao commented on GitHub (Mar 17, 2025): Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s
bsakaguchi commented 2026-10-26 09:53:41 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (Apr 13, 2025):

Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s

Allocating buffer size: 2115.111296 MB ...
[rank 10] Dispatch + combine bandwidth: 27.38 GB/s, avg_t=805.21 us, min_t=791.81 us, max_t=820.42 us
[rank 11] Dispatch + combine bandwidth: 27.38 GB/s, avg_t=805.34 us, min_t=795.87 us, max_t=819.01 us
[rank 12] Dispatch + combine bandwidth: 27.40 GB/s, avg_t=804.79 us, min_t=778.53 us, max_t=823.58 us
[rank 9] Dispatch + combine bandwidth: 27.52 GB/s, avg_t=801.18 us, min_t=685.22 us, max_t=863.49 us
[rank 8] Dispatch + combine bandwidth: 27.53 GB/s, avg_t=800.91 us, min_t=674.40 us, max_t=812.06 us
[rank 13] Dispatch + combine bandwidth: 27.52 GB/s, avg_t=801.07 us, min_t=682.46 us, max_t=854.40 us
[rank 14] Dispatch + combine bandwidth: 27.46 GB/s, avg_t=802.90 us, min_t=740.06 us, max_t=851.87 us
[rank 15] Dispatch + combine bandwidth: 27.51 GB/s, avg_t=801.59 us, min_t=732.38 us, max_t=813.70 us
[rank 10] Dispatch bandwidth: 11.44 GB/s, avg_t=656.57 us | Combine bandwidth: 35.01 GB/s, avg_t=415.20 us
[rank 11] Dispatch bandwidth: 11.27 GB/s, avg_t=666.61 us | Combine bandwidth: 35.37 GB/s, avg_t=410.97 us
[rank 12] Dispatch bandwidth: 12.65 GB/s, avg_t=593.74 us | Combine bandwidth: 29.97 GB/s, avg_t=484.99 us
[rank 8] Dispatch bandwidth: 15.29 GB/s, avg_t=491.44 us | Combine bandwidth: 24.35 GB/s, avg_t=597.03 us
[rank 13] Dispatch bandwidth: 15.65 GB/s, avg_t=479.88 us | Combine bandwidth: 24.19 GB/s, avg_t=601.04 us
[rank 9] Dispatch bandwidth: 15.27 GB/s, avg_t=491.91 us | Combine bandwidth: 24.52 GB/s, avg_t=592.86 us
[rank 15] Dispatch bandwidth: 20.90 GB/s, avg_t=359.33 us | Combine bandwidth: 20.05 GB/s, avg_t=725.19 us
[rank 14] Dispatch bandwidth: 15.18 GB/s, avg_t=494.82 us | Combine bandwidth: 24.95 GB/s, avg_t=582.62 us
[rank 15] Dispatch send/recv time: 29.73 us | Combine send/recv time: 32.42 us
[rank 8] Dispatch send/recv time: 29.92 us | Combine send/recv time: 31.86 us
[rank 11] Dispatch send/recv time: 29.47 us | Combine send/recv time: 32.35 us
[rank 12] Dispatch send/recv time: 30.22 us | Combine send/recv time: 32.38 us
[rank 10] Dispatch send/recv time: 29.96 us | Combine send/recv time: 32.63 us
[rank 9] Dispatch send/recv time: 30.23 us | Combine send/recv time: 32.08 us
[rank 13] Dispatch send/recv time: 28.93 us | Combine send/recv time: 32.53 us
[rank 14] Dispatch send/recv time: 29.43 us | Combine send/recv time: 32.62 us

<!-- gh-comment-id:2799879678 --> @CSEEduanyu commented on GitHub (Apr 13, 2025): > Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s Allocating buffer size: 2115.111296 MB ... [rank 10] Dispatch + combine bandwidth: 27.38 GB/s, avg_t=805.21 us, min_t=791.81 us, max_t=820.42 us [rank 11] Dispatch + combine bandwidth: 27.38 GB/s, avg_t=805.34 us, min_t=795.87 us, max_t=819.01 us [rank 12] Dispatch + combine bandwidth: 27.40 GB/s, avg_t=804.79 us, min_t=778.53 us, max_t=823.58 us [rank 9] Dispatch + combine bandwidth: 27.52 GB/s, avg_t=801.18 us, min_t=685.22 us, max_t=863.49 us [rank 8] Dispatch + combine bandwidth: 27.53 GB/s, avg_t=800.91 us, min_t=674.40 us, max_t=812.06 us [rank 13] Dispatch + combine bandwidth: 27.52 GB/s, avg_t=801.07 us, min_t=682.46 us, max_t=854.40 us [rank 14] Dispatch + combine bandwidth: 27.46 GB/s, avg_t=802.90 us, min_t=740.06 us, max_t=851.87 us [rank 15] Dispatch + combine bandwidth: 27.51 GB/s, avg_t=801.59 us, min_t=732.38 us, max_t=813.70 us [rank 10] Dispatch bandwidth: 11.44 GB/s, avg_t=656.57 us | Combine bandwidth: 35.01 GB/s, avg_t=415.20 us [rank 11] Dispatch bandwidth: 11.27 GB/s, avg_t=666.61 us | Combine bandwidth: 35.37 GB/s, avg_t=410.97 us [rank 12] Dispatch bandwidth: 12.65 GB/s, avg_t=593.74 us | Combine bandwidth: 29.97 GB/s, avg_t=484.99 us [rank 8] Dispatch bandwidth: 15.29 GB/s, avg_t=491.44 us | Combine bandwidth: 24.35 GB/s, avg_t=597.03 us [rank 13] Dispatch bandwidth: 15.65 GB/s, avg_t=479.88 us | Combine bandwidth: 24.19 GB/s, avg_t=601.04 us [rank 9] Dispatch bandwidth: 15.27 GB/s, avg_t=491.91 us | Combine bandwidth: 24.52 GB/s, avg_t=592.86 us [rank 15] Dispatch bandwidth: 20.90 GB/s, avg_t=359.33 us | Combine bandwidth: 20.05 GB/s, avg_t=725.19 us [rank 14] Dispatch bandwidth: 15.18 GB/s, avg_t=494.82 us | Combine bandwidth: 24.95 GB/s, avg_t=582.62 us [rank 15] Dispatch send/recv time: 29.73 us | Combine send/recv time: 32.42 us [rank 8] Dispatch send/recv time: 29.92 us | Combine send/recv time: 31.86 us [rank 11] Dispatch send/recv time: 29.47 us | Combine send/recv time: 32.35 us [rank 12] Dispatch send/recv time: 30.22 us | Combine send/recv time: 32.38 us [rank 10] Dispatch send/recv time: 29.96 us | Combine send/recv time: 32.63 us [rank 9] Dispatch send/recv time: 30.23 us | Combine send/recv time: 32.08 us [rank 13] Dispatch send/recv time: 28.93 us | Combine send/recv time: 32.53 us [rank 14] Dispatch send/recv time: 29.43 us | Combine send/recv time: 32.62 us
bsakaguchi commented 2026-10-26 09:53:41 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (Apr 13, 2025):

[config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8
[layout] Kernel performance: 0.053 ms

[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with FP8, without top-k (async=False, previous=False) ... passed
[testing] Running with FP8, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=True, previous=False) ... passed
[testing] Running with BF16, with top-k (async=True, previous=False) ... passed
[testing] Running with BF16, without top-k (async=True, previous=False) ... passed
[testing] Running with BF16, with top-k (async=True, previous=False) ... passed
[testing] Running with FP8, without top-k (async=True, previous=False) ... passed
[testing] Running with FP8, with top-k (async=True, previous=False) ... passed
[testing] Running with BF16, without top-k (async=False, previous=True) ... passed
[testing] Running with BF16, with top-k (async=False, previous=True) ... passed
[testing] Running with BF16, without top-k (async=False, previous=True) ... passed
[testing] Running with BF16, with top-k (async=False, previous=True) ... passed
[testing] Running with FP8, without top-k (async=False, previous=True) ... passed
[testing] Running with FP8, with top-k (async=False, previous=True) ... passed
[testing] Running with BF16, without top-k (async=True, previous=True) ... passed
[testing] Running with BF16, with top-k (async=True, previous=True) ... passed
[testing] Running with BF16, without top-k (async=True, previous=True) ... passed
[testing] Running with BF16, with top-k (async=True, previous=True) ... passed
[testing] Running with FP8, without top-k (async=True, previous=True) ... passed
[testing] Running with FP8, with top-k (async=True, previous=True) ... passed

[tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 19.33 GB/s (RDMA), 63.23 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 31.84 GB/s (RDMA), 104.16 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 36.66 GB/s (RDMA), 119.94 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 37.58 GB/s (RDMA), 122.95 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 38.33 GB/s (RDMA), 125.39 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 38.42 GB/s (RDMA), 125.68 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 38.49 GB/s (RDMA), 125.92 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 38.21 GB/s (RDMA), 124.98 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 4: 19.35 GB/s (RDMA), 63.29 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 8: 31.43 GB/s (RDMA), 102.83 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 12: 36.21 GB/s (RDMA), 118.45 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 16: 37.09 GB/s (RDMA), 121.34 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 20: 38.65 GB/s (RDMA), 126.44 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 24: 39.49 GB/s (RDMA), 129.20 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 28: 39.89 GB/s (RDMA), 130.48 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 32: 39.26 GB/s (RDMA), 128.44 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 4: 19.34 GB/s (RDMA), 63.28 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 8: 31.34 GB/s (RDMA), 102.51 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 12: 36.66 GB/s (RDMA), 119.92 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 16: 37.29 GB/s (RDMA), 121.98 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 20: 38.70 GB/s (RDMA), 126.62 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 24: 39.00 GB/s (RDMA), 127.58 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 28: 39.41 GB/s (RDMA), 128.93 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 32: 39.48 GB/s (RDMA), 129.14 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 8: 31.41 GB/s (RDMA), 102.76 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 12: 36.62 GB/s (RDMA), 119.80 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 16: 37.17 GB/s (RDMA), 121.60 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 20: 38.85 GB/s (RDMA), 127.09 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 24: 39.59 GB/s (RDMA), 129.51 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 28: 39.68 GB/s (RDMA), 129.79 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 32: 39.50 GB/s (RDMA), 129.21 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 8: 31.47 GB/s (RDMA), 102.94 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 12: 36.82 GB/s (RDMA), 120.46 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 16: 37.78 GB/s (RDMA), 123.60 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 20: 39.03 GB/s (RDMA), 127.67 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 24: 39.53 GB/s (RDMA), 129.32 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 28: 39.80 GB/s (RDMA), 130.19 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 32: 40.27 GB/s (RDMA), 131.73 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 8: 31.46 GB/s (RDMA), 102.93 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 12: 36.84 GB/s (RDMA), 120.51 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 16: 37.94 GB/s (RDMA), 124.10 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 20: 38.96 GB/s (RDMA), 127.45 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 24: 39.44 GB/s (RDMA), 129.03 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 28: 39.55 GB/s (RDMA), 129.38 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 32: 40.24 GB/s (RDMA), 131.63 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 4: 19.35 GB/s (RDMA), 63.29 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 8: 31.41 GB/s (RDMA), 102.75 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 12: 36.66 GB/s (RDMA), 119.93 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 16: 37.90 GB/s (RDMA), 123.97 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 20: 38.50 GB/s (RDMA), 125.96 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 24: 39.05 GB/s (RDMA), 127.75 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 28: 39.21 GB/s (RDMA), 128.26 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 32: 39.94 GB/s (RDMA), 130.66 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 4: 19.35 GB/s (RDMA), 63.30 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 8: 31.35 GB/s (RDMA), 102.54 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 12: 36.32 GB/s (RDMA), 118.80 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 16: 37.32 GB/s (RDMA), 122.08 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 20: 38.72 GB/s (RDMA), 126.65 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 24: 39.43 GB/s (RDMA), 129.00 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 39.59 GB/s (RDMA), 129.52 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 39.53 GB/s (RDMA), 129.31 GB/s (NVL)
[tuning] Best dispatch (FP8): SMs 24, NVL chunk 20, RDMA chunk 32: 40.27 GB/s (RDMA), 131.73 GB/s (NVL)

[tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 31.25 GB/s (RDMA), 102.24 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 39.85 GB/s (RDMA), 130.35 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 40.90 GB/s (RDMA), 133.79 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 41.75 GB/s (RDMA), 136.57 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 42.83 GB/s (RDMA), 140.09 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 42.70 GB/s (RDMA), 139.70 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 42.30 GB/s (RDMA), 138.39 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 41.53 GB/s (RDMA), 135.85 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 4: 31.26 GB/s (RDMA), 102.26 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 8: 40.34 GB/s (RDMA), 131.96 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 12: 41.30 GB/s (RDMA), 135.12 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 16: 41.43 GB/s (RDMA), 135.55 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 20: 42.56 GB/s (RDMA), 139.23 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 24: 42.77 GB/s (RDMA), 139.92 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 28: 42.63 GB/s (RDMA), 139.45 GB/s (NVL)
[tuning] SMs 24, NVL chunk 8, RDMA chunk 32: 41.74 GB/s (RDMA), 136.56 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 4: 31.23 GB/s (RDMA), 102.15 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 8: 40.19 GB/s (RDMA), 131.49 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 12: 41.58 GB/s (RDMA), 136.03 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 16: 41.69 GB/s (RDMA), 136.38 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 20: 42.92 GB/s (RDMA), 140.42 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 24: 42.55 GB/s (RDMA), 139.21 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 28: 42.30 GB/s (RDMA), 138.38 GB/s (NVL)
[tuning] SMs 24, NVL chunk 12, RDMA chunk 32: 41.86 GB/s (RDMA), 136.95 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 4: 31.26 GB/s (RDMA), 102.25 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 8: 40.52 GB/s (RDMA), 132.57 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 12: 41.66 GB/s (RDMA), 136.29 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 16: 41.99 GB/s (RDMA), 137.35 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 20: 42.89 GB/s (RDMA), 140.30 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 24: 43.00 GB/s (RDMA), 140.66 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 28: 43.06 GB/s (RDMA), 140.85 GB/s (NVL)
[tuning] SMs 24, NVL chunk 16, RDMA chunk 32: 41.92 GB/s (RDMA), 137.13 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 4: 31.29 GB/s (RDMA), 102.36 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 8: 40.67 GB/s (RDMA), 133.03 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 12: 41.69 GB/s (RDMA), 136.38 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 16: 42.17 GB/s (RDMA), 137.95 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 20: 42.66 GB/s (RDMA), 139.56 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 24: 42.58 GB/s (RDMA), 139.28 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 28: 43.18 GB/s (RDMA), 141.25 GB/s (NVL)
[tuning] SMs 24, NVL chunk 20, RDMA chunk 32: 42.12 GB/s (RDMA), 137.78 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 4: 31.25 GB/s (RDMA), 102.21 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 8: 40.43 GB/s (RDMA), 132.27 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 12: 41.74 GB/s (RDMA), 136.54 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 16: 41.63 GB/s (RDMA), 136.20 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 20: 42.43 GB/s (RDMA), 138.81 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 24: 42.91 GB/s (RDMA), 140.37 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 28: 42.90 GB/s (RDMA), 140.36 GB/s (NVL)
[tuning] SMs 24, NVL chunk 24, RDMA chunk 32: 41.82 GB/s (RDMA), 136.79 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 4: 31.22 GB/s (RDMA), 102.12 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 8: 40.66 GB/s (RDMA), 133.00 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 12: 41.74 GB/s (RDMA), 136.54 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 16: 41.82 GB/s (RDMA), 136.81 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 20: 42.71 GB/s (RDMA), 139.70 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 24: 42.91 GB/s (RDMA), 140.38 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 28: 42.68 GB/s (RDMA), 139.63 GB/s (NVL)
[tuning] SMs 24, NVL chunk 28, RDMA chunk 32: 41.81 GB/s (RDMA), 136.78 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 4: 31.25 GB/s (RDMA), 102.23 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 8: 40.58 GB/s (RDMA), 132.74 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 12: 42.16 GB/s (RDMA), 137.93 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 16: 41.94 GB/s (RDMA), 137.21 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 20: 42.47 GB/s (RDMA), 138.93 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 24: 42.58 GB/s (RDMA), 139.28 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 42.84 GB/s (RDMA), 140.15 GB/s (NVL)
[tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 41.41 GB/s (RDMA), 135.46 GB/s (NVL)
[tuning] Best dispatch (BF16): SMs 24, NVL chunk 20, RDMA chunk 28: 43.18 GB/s (RDMA), 141.25 GB/s (NVL)

[tuning] SMs 24, NVL chunk 1, RDMA chunk 8: 39.99 GB/s (RDMA), 130.81 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 12: 39.58 GB/s (RDMA), 129.46 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 16: 39.90 GB/s (RDMA), 130.51 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 20: 40.09 GB/s (RDMA), 131.15 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 24: 39.76 GB/s (RDMA), 130.07 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 28: 39.45 GB/s (RDMA), 129.05 GB/s (NVL)
[tuning] SMs 24, NVL chunk 1, RDMA chunk 32: 39.43 GB/s (RDMA), 128.98 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 8: 39.94 GB/s (RDMA), 130.65 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 12: 40.12 GB/s (RDMA), 131.24 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 16: 40.18 GB/s (RDMA), 131.44 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 20: 40.29 GB/s (RDMA), 131.82 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 24: 40.44 GB/s (RDMA), 132.29 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 28: 39.98 GB/s (RDMA), 130.78 GB/s (NVL)
[tuning] SMs 24, NVL chunk 2, RDMA chunk 32: 39.48 GB/s (RDMA), 129.15 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 8: 37.10 GB/s (RDMA), 121.35 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 12: 38.48 GB/s (RDMA), 125.86 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 16: 37.50 GB/s (RDMA), 122.69 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 20: 37.88 GB/s (RDMA), 123.91 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 24: 37.70 GB/s (RDMA), 123.34 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 28: 37.32 GB/s (RDMA), 122.07 GB/s (NVL)
[tuning] SMs 24, NVL chunk 3, RDMA chunk 32: 38.20 GB/s (RDMA), 124.97 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 36.83 GB/s (RDMA), 120.48 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 37.07 GB/s (RDMA), 121.27 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 37.11 GB/s (RDMA), 121.41 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 37.51 GB/s (RDMA), 122.69 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 36.69 GB/s (RDMA), 120.03 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 36.67 GB/s (RDMA), 119.95 GB/s (NVL)
[tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 35.94 GB/s (RDMA), 117.58 GB/s (NVL)
[tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 24: 40.44 GB/s (RDMA), 132.29 GB/s (NVL)

<!-- gh-comment-id:2799879823 --> @CSEEduanyu commented on GitHub (Apr 13, 2025): [config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8 [layout] Kernel performance: 0.053 ms [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with FP8, without top-k (async=False, previous=False) ... passed [testing] Running with FP8, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=True, previous=False) ... passed [testing] Running with BF16, with top-k (async=True, previous=False) ... passed [testing] Running with BF16, without top-k (async=True, previous=False) ... passed [testing] Running with BF16, with top-k (async=True, previous=False) ... passed [testing] Running with FP8, without top-k (async=True, previous=False) ... passed [testing] Running with FP8, with top-k (async=True, previous=False) ... passed [testing] Running with BF16, without top-k (async=False, previous=True) ... passed [testing] Running with BF16, with top-k (async=False, previous=True) ... passed [testing] Running with BF16, without top-k (async=False, previous=True) ... passed [testing] Running with BF16, with top-k (async=False, previous=True) ... passed [testing] Running with FP8, without top-k (async=False, previous=True) ... passed [testing] Running with FP8, with top-k (async=False, previous=True) ... passed [testing] Running with BF16, without top-k (async=True, previous=True) ... passed [testing] Running with BF16, with top-k (async=True, previous=True) ... passed [testing] Running with BF16, without top-k (async=True, previous=True) ... passed [testing] Running with BF16, with top-k (async=True, previous=True) ... passed [testing] Running with FP8, without top-k (async=True, previous=True) ... passed [testing] Running with FP8, with top-k (async=True, previous=True) ... passed [tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 19.33 GB/s (RDMA), 63.23 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 31.84 GB/s (RDMA), 104.16 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 36.66 GB/s (RDMA), 119.94 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 37.58 GB/s (RDMA), 122.95 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 38.33 GB/s (RDMA), 125.39 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 38.42 GB/s (RDMA), 125.68 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 38.49 GB/s (RDMA), 125.92 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 38.21 GB/s (RDMA), 124.98 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 4: 19.35 GB/s (RDMA), 63.29 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 8: 31.43 GB/s (RDMA), 102.83 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 12: 36.21 GB/s (RDMA), 118.45 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 16: 37.09 GB/s (RDMA), 121.34 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 20: 38.65 GB/s (RDMA), 126.44 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 24: 39.49 GB/s (RDMA), 129.20 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 28: 39.89 GB/s (RDMA), 130.48 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 32: 39.26 GB/s (RDMA), 128.44 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 4: 19.34 GB/s (RDMA), 63.28 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 8: 31.34 GB/s (RDMA), 102.51 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 12: 36.66 GB/s (RDMA), 119.92 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 16: 37.29 GB/s (RDMA), 121.98 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 20: 38.70 GB/s (RDMA), 126.62 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 24: 39.00 GB/s (RDMA), 127.58 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 28: 39.41 GB/s (RDMA), 128.93 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 32: 39.48 GB/s (RDMA), 129.14 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 8: 31.41 GB/s (RDMA), 102.76 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 12: 36.62 GB/s (RDMA), 119.80 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 16: 37.17 GB/s (RDMA), 121.60 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 20: 38.85 GB/s (RDMA), 127.09 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 24: 39.59 GB/s (RDMA), 129.51 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 28: 39.68 GB/s (RDMA), 129.79 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 32: 39.50 GB/s (RDMA), 129.21 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 8: 31.47 GB/s (RDMA), 102.94 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 12: 36.82 GB/s (RDMA), 120.46 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 16: 37.78 GB/s (RDMA), 123.60 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 20: 39.03 GB/s (RDMA), 127.67 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 24: 39.53 GB/s (RDMA), 129.32 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 28: 39.80 GB/s (RDMA), 130.19 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 32: 40.27 GB/s (RDMA), 131.73 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 4: 19.35 GB/s (RDMA), 63.31 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 8: 31.46 GB/s (RDMA), 102.93 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 12: 36.84 GB/s (RDMA), 120.51 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 16: 37.94 GB/s (RDMA), 124.10 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 20: 38.96 GB/s (RDMA), 127.45 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 24: 39.44 GB/s (RDMA), 129.03 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 28: 39.55 GB/s (RDMA), 129.38 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 32: 40.24 GB/s (RDMA), 131.63 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 4: 19.35 GB/s (RDMA), 63.29 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 8: 31.41 GB/s (RDMA), 102.75 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 12: 36.66 GB/s (RDMA), 119.93 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 16: 37.90 GB/s (RDMA), 123.97 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 20: 38.50 GB/s (RDMA), 125.96 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 24: 39.05 GB/s (RDMA), 127.75 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 28: 39.21 GB/s (RDMA), 128.26 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 32: 39.94 GB/s (RDMA), 130.66 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 4: 19.35 GB/s (RDMA), 63.30 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 8: 31.35 GB/s (RDMA), 102.54 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 12: 36.32 GB/s (RDMA), 118.80 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 16: 37.32 GB/s (RDMA), 122.08 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 20: 38.72 GB/s (RDMA), 126.65 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 24: 39.43 GB/s (RDMA), 129.00 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 39.59 GB/s (RDMA), 129.52 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 39.53 GB/s (RDMA), 129.31 GB/s (NVL) [tuning] Best dispatch (FP8): SMs 24, NVL chunk 20, RDMA chunk 32: 40.27 GB/s (RDMA), 131.73 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 31.25 GB/s (RDMA), 102.24 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 39.85 GB/s (RDMA), 130.35 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 40.90 GB/s (RDMA), 133.79 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 41.75 GB/s (RDMA), 136.57 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 42.83 GB/s (RDMA), 140.09 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 42.70 GB/s (RDMA), 139.70 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 42.30 GB/s (RDMA), 138.39 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 41.53 GB/s (RDMA), 135.85 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 4: 31.26 GB/s (RDMA), 102.26 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 8: 40.34 GB/s (RDMA), 131.96 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 12: 41.30 GB/s (RDMA), 135.12 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 16: 41.43 GB/s (RDMA), 135.55 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 20: 42.56 GB/s (RDMA), 139.23 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 24: 42.77 GB/s (RDMA), 139.92 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 28: 42.63 GB/s (RDMA), 139.45 GB/s (NVL) [tuning] SMs 24, NVL chunk 8, RDMA chunk 32: 41.74 GB/s (RDMA), 136.56 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 4: 31.23 GB/s (RDMA), 102.15 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 8: 40.19 GB/s (RDMA), 131.49 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 12: 41.58 GB/s (RDMA), 136.03 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 16: 41.69 GB/s (RDMA), 136.38 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 20: 42.92 GB/s (RDMA), 140.42 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 24: 42.55 GB/s (RDMA), 139.21 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 28: 42.30 GB/s (RDMA), 138.38 GB/s (NVL) [tuning] SMs 24, NVL chunk 12, RDMA chunk 32: 41.86 GB/s (RDMA), 136.95 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 4: 31.26 GB/s (RDMA), 102.25 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 8: 40.52 GB/s (RDMA), 132.57 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 12: 41.66 GB/s (RDMA), 136.29 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 16: 41.99 GB/s (RDMA), 137.35 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 20: 42.89 GB/s (RDMA), 140.30 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 24: 43.00 GB/s (RDMA), 140.66 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 28: 43.06 GB/s (RDMA), 140.85 GB/s (NVL) [tuning] SMs 24, NVL chunk 16, RDMA chunk 32: 41.92 GB/s (RDMA), 137.13 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 4: 31.29 GB/s (RDMA), 102.36 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 8: 40.67 GB/s (RDMA), 133.03 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 12: 41.69 GB/s (RDMA), 136.38 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 16: 42.17 GB/s (RDMA), 137.95 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 20: 42.66 GB/s (RDMA), 139.56 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 24: 42.58 GB/s (RDMA), 139.28 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 28: 43.18 GB/s (RDMA), 141.25 GB/s (NVL) [tuning] SMs 24, NVL chunk 20, RDMA chunk 32: 42.12 GB/s (RDMA), 137.78 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 4: 31.25 GB/s (RDMA), 102.21 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 8: 40.43 GB/s (RDMA), 132.27 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 12: 41.74 GB/s (RDMA), 136.54 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 16: 41.63 GB/s (RDMA), 136.20 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 20: 42.43 GB/s (RDMA), 138.81 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 24: 42.91 GB/s (RDMA), 140.37 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 28: 42.90 GB/s (RDMA), 140.36 GB/s (NVL) [tuning] SMs 24, NVL chunk 24, RDMA chunk 32: 41.82 GB/s (RDMA), 136.79 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 4: 31.22 GB/s (RDMA), 102.12 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 8: 40.66 GB/s (RDMA), 133.00 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 12: 41.74 GB/s (RDMA), 136.54 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 16: 41.82 GB/s (RDMA), 136.81 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 20: 42.71 GB/s (RDMA), 139.70 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 24: 42.91 GB/s (RDMA), 140.38 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 28: 42.68 GB/s (RDMA), 139.63 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 32: 41.81 GB/s (RDMA), 136.78 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 4: 31.25 GB/s (RDMA), 102.23 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 8: 40.58 GB/s (RDMA), 132.74 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 12: 42.16 GB/s (RDMA), 137.93 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 16: 41.94 GB/s (RDMA), 137.21 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 20: 42.47 GB/s (RDMA), 138.93 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 24: 42.58 GB/s (RDMA), 139.28 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 28: 42.84 GB/s (RDMA), 140.15 GB/s (NVL) [tuning] SMs 24, NVL chunk 32, RDMA chunk 32: 41.41 GB/s (RDMA), 135.46 GB/s (NVL) [tuning] Best dispatch (BF16): SMs 24, NVL chunk 20, RDMA chunk 28: 43.18 GB/s (RDMA), 141.25 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 8: 39.99 GB/s (RDMA), 130.81 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 12: 39.58 GB/s (RDMA), 129.46 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 16: 39.90 GB/s (RDMA), 130.51 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 20: 40.09 GB/s (RDMA), 131.15 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 24: 39.76 GB/s (RDMA), 130.07 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 28: 39.45 GB/s (RDMA), 129.05 GB/s (NVL) [tuning] SMs 24, NVL chunk 1, RDMA chunk 32: 39.43 GB/s (RDMA), 128.98 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 8: 39.94 GB/s (RDMA), 130.65 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 12: 40.12 GB/s (RDMA), 131.24 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 16: 40.18 GB/s (RDMA), 131.44 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 20: 40.29 GB/s (RDMA), 131.82 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 24: 40.44 GB/s (RDMA), 132.29 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 28: 39.98 GB/s (RDMA), 130.78 GB/s (NVL) [tuning] SMs 24, NVL chunk 2, RDMA chunk 32: 39.48 GB/s (RDMA), 129.15 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 8: 37.10 GB/s (RDMA), 121.35 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 12: 38.48 GB/s (RDMA), 125.86 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 16: 37.50 GB/s (RDMA), 122.69 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 20: 37.88 GB/s (RDMA), 123.91 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 24: 37.70 GB/s (RDMA), 123.34 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 28: 37.32 GB/s (RDMA), 122.07 GB/s (NVL) [tuning] SMs 24, NVL chunk 3, RDMA chunk 32: 38.20 GB/s (RDMA), 124.97 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 36.83 GB/s (RDMA), 120.48 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 37.07 GB/s (RDMA), 121.27 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 37.11 GB/s (RDMA), 121.41 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 37.51 GB/s (RDMA), 122.69 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 36.69 GB/s (RDMA), 120.03 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 28: 36.67 GB/s (RDMA), 119.95 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 32: 35.94 GB/s (RDMA), 117.58 GB/s (NVL) [tuning] Best combine: SMs 24, NVL chunk 2, RDMA chunk 24: 40.44 GB/s (RDMA), 132.29 GB/s (NVL)
bsakaguchi commented 2026-10-26 09:53:42 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (Apr 13, 2025):

Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s

I tested on the H800*16 ROCE network and found that test_internode.py worked as expected, but test_low_latency.py seemed to be very slow. What could be the reason for this?

<!-- gh-comment-id:2799880236 --> @CSEEduanyu commented on GitHub (Apr 13, 2025): > Can you please share the test results? Normally, the bandwidth should both be around 40-50 GB/s I tested on the H800*16 ROCE network and found that test_internode.py worked as expected, but test_low_latency.py seemed to be very slow. What could be the reason for this?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#69
No description provided.