[GH-ISSUE #407] NVLink and RDMA traffic distinction in internode kernels #275

New issue

Closed

opened 2026-10-26 09:57:49 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:57:49 +00:00

Owner

Originally created by @aahouzi on GitHub (Sep 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/407

Type of issue

In your test scripts, I was wondering abt some variables that you defined in the internode kernel test:

dispatch_bf16_rdma_send_bytes = num_rdma_token_sent * hidden * 2
dispatch_bf16_nvl_recv_bytes = recv_x.numel() * 2
combine_bf16_nvl_send_bytes = dispatch_bf16_nvl_recv_bytes
combine_bf16_rdma_recv_bytes = dispatch_bf16_rdma_send_bytes

In the above, you are defining dispatch_bf16_nvl_recv_bytes based on the number of elements received after the dispatch operation, but this number of elements is not only NVLink traffic; it would also include internode traffic, but you are using it later to compute the NVL BW:

nvl_recv_bytes = (dispatch_bf16_nvl_recv_bytes * fp8_factor) if isinstance(current_x, tuple) else dispatch_bf16_nvl_recv_bytes
print(f'[tuning] Best dispatch ({"FP8" if isinstance(current_x, tuple) else "BF16"}): SMs {best_results[0]}, NVL chunk {best_results[1]}, RDMA chunk {best_results[2]}, transmit: {best_time * 1e6:.2f} us, notify: {best_results[3] * 1e6:.2f} us, BW: {rdma_send_bytes / 1e9 / best_time:.2f} GB/s (RDMA), {nvl_recv_bytes / 1e9 / best_time:.2f} GB/s (NVL)', flush=True)

@sphish Can you please provide some explanation about this choice ? Thanks :)

Originally created by @aahouzi on GitHub (Sep 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/407 ## Type of issue - In your test scripts, I was wondering abt some variables that you defined in the internode kernel test: ``` dispatch_bf16_rdma_send_bytes = num_rdma_token_sent * hidden * 2 dispatch_bf16_nvl_recv_bytes = recv_x.numel() * 2 combine_bf16_nvl_send_bytes = dispatch_bf16_nvl_recv_bytes combine_bf16_rdma_recv_bytes = dispatch_bf16_rdma_send_bytes ``` - In the above, you are defining ```dispatch_bf16_nvl_recv_bytes``` based on the number of elements received after the dispatch operation, but this number of elements is not only NVLink traffic; it would also include internode traffic, but you are using it later to compute the NVL BW: ``` nvl_recv_bytes = (dispatch_bf16_nvl_recv_bytes * fp8_factor) if isinstance(current_x, tuple) else dispatch_bf16_nvl_recv_bytes print(f'[tuning] Best dispatch ({"FP8" if isinstance(current_x, tuple) else "BF16"}): SMs {best_results[0]}, NVL chunk {best_results[1]}, RDMA chunk {best_results[2]}, transmit: {best_time * 1e6:.2f} us, notify: {best_results[3] * 1e6:.2f} us, BW: {rdma_send_bytes / 1e9 / best_time:.2f} GB/s (RDMA), {nvl_recv_bytes / 1e9 / best_time:.2f} GB/s (NVL)', flush=True) ``` - @sphish Can you please provide some explanation about this choice ? Thanks :)

bsakaguchi closed this issue

2026-10-26 09:57:49 +00:00

bsakaguchi commented

2026-10-26 09:57:50 +00:00

Author

Owner

@sphish commented on GitHub (Sep 24, 2025):

For simplicity, the bandwidth we measure is the algorithm bandwidth. You can refer to: https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md

@sphish commented on GitHub (Sep 24, 2025): For simplicity, the bandwidth we measure is the algorithm bandwidth. You can refer to: https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md

bsakaguchi referenced this issue

2026-10-27 21:03:15 +00:00

[PR #275] [MERGED] Tiny allow testing in MNNVL environments #496