[GH-ISSUE #84] Questions on Profiling `dispatch_t` for Low-Latency Kernel Dispatch Bandwidth Calculation

bsakaguchi commented

2026-10-26 09:53:53 +00:00

Owner

Originally created by @ghghliu on GitHub (Mar 20, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/84

Hi,

I have a question regarding the calculation of low-latency kernel dispatch bandwidth. In the separate profiling part, the dispatch bandwidth is calculated as:

dispatch_bandwidth = num_dispatch_comm_bytes / 1e9 / dispatch_t

Could you please clarify how dispatch_t is profiled when return_recv_hook is False? Specifically:

Does dispatch_t start at the dispatch kernel launch and end when the dispatch send phase completes?
Or does it include the time until the dispatched tokens are received by the destination and the sender is notified?
I have this question because some of the bw result exceed the nic's max bandwidth, so I guess maybe the end timestamp is not all the token dispatched are received, but estimated by all the token for the expert in this rank are received?

Thanks a lot!

Originally created by @ghghliu on GitHub (Mar 20, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/84 Hi, I have a question regarding the calculation of low-latency kernel dispatch bandwidth. In the separate profiling part, the dispatch bandwidth is calculated as: ``` dispatch_bandwidth = num_dispatch_comm_bytes / 1e9 / dispatch_t ``` Could you please clarify how `dispatch_t` is profiled when `return_recv_hook` is `False`? Specifically: 1. Does `dispatch_t` start at the dispatch kernel launch and end when the dispatch send phase completes? 2. Or does it include the time until the dispatched tokens are received by the destination and the sender is notified? I have this question because some of the bw result exceed the nic's max bandwidth, so I guess maybe the end timestamp is not all the token dispatched are received, but estimated by all the token for the expert in this rank are received? Thanks a lot!

bsakaguchi closed this issue

2026-10-26 09:53:53 +00:00

bsakaguchi commented

2026-10-26 09:53:54 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 21, 2025):

Does dispatch_t start at the dispatch kernel launch and end when the dispatch send phase completes?

It starts at launch, ends after receiving all data, i.e. send + recv phases, when return_recv_hook=False.

Or does it include the time until the dispatched tokens are received by the destination and the sender is notified?

Yes.

I have this question because some of the bw result exceed the nic's max bandwidth, so I guess maybe the end timestamp is not all the token dispatched are received, but estimated by all the token for the expert in this rank are received?

How many ranks are you using? For local-to-local transfer, we are using direct memory copy but not RDMA. For example, if you are using 8 ranks, 1/8 bandwidth is sending to the local rank, so the real bandwidth is 7/8 * the printed bandwidth.

Moreover, even for ranks in the same node, we are not using NVLink but using RDMA. So the only thing you have to consider is to the local GPU (rank) copy.

@LyricZhao commented on GitHub (Mar 21, 2025): > Does dispatch_t start at the dispatch kernel launch and end when the dispatch send phase completes? It starts at launch, ends **after** receiving all data, i.e. `send + recv` phases, when `return_recv_hook=False`. > Or does it include the time until the dispatched tokens are received by the destination and the sender is notified? Yes. > I have this question because some of the bw result exceed the nic's max bandwidth, so I guess maybe the end timestamp is not all the token dispatched are received, but estimated by all the token for the expert in this rank are received? How many ranks are you using? For local-to-local transfer, we are using direct memory copy but not RDMA. For example, if you are using 8 ranks, 1/8 bandwidth is sending to the local rank, so the real bandwidth is `7/8 * the printed bandwidth`. Moreover, even for ranks in the same node, we are not using NVLink but using RDMA. So the only thing you have to consider is to the local GPU (rank) copy.

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@ghghliu commented on GitHub (Mar 21, 2025):

Thank you for the quick reply,

We did a simple test in one server with 8 gpu and 8 cx7 nic. After get the topk_idx tensor as the default random way, we mask the expert index belong to gpu0 with the fellowing code, so that gpu 0 receive zero token from other ranks(including itself, so no local gpu mem copy), but send tokens to other rank as usual.

    for i in range(num_tokens):  # num_tokens=128
        for j in range(8):
            if topk_idx[i,j]//(num_experts//8) == 0:  # num_experts=288
                topk_idx[i,j] = -1

and the result are as fellows, rank 0 dispatch bandwidth is 79.21GB/s. It looks like the time ends when other rank told rank0 zero expert is sent to you.
[rank 3] Dispatch bandwidth: 41.64 GB/s, avg_t=153.35 us | Combine bandwidth: 38.82 GB/s, avg_t=318.30 us
[rank 1] Dispatch bandwidth: 41.61 GB/s, avg_t=158.27 us | Combine bandwidth: 40.54 GB/s, avg_t=314.36 us
[rank 0] Dispatch bandwidth: 79.21 GB/s, avg_t=83.24 us | Combine bandwidth: 32.74 GB/s, avg_t=389.67 us
[rank 7] Dispatch bandwidth: 39.92 GB/s, avg_t=165.17 us | Combine bandwidth: 41.46 GB/s, avg_t=307.76 us
[rank 2] Dispatch bandwidth: 43.53 GB/s, avg_t=156.21 us | Combine bandwidth: 41.62 GB/s, avg_t=316.21 us
[rank 4] Dispatch bandwidth: 41.81 GB/s, avg_t=158.22 us | Combine bandwidth: 40.65 GB/s, avg_t=314.93 us
[rank 6] Dispatch bandwidth: 42.19 GB/s, avg_t=154.88 us | Combine bandwidth: 39.78 GB/s, avg_t=317.84 us
[rank 5] Dispatch bandwidth: 42.24 GB/s, avg_t=156.95 us | Combine bandwidth: 40.57 GB/s, avg_t=316.24 us

Does dispatch_t start at the dispatch kernel launch and end when the dispatch send phase completes?

It starts at launch, ends after receiving all data, i.e. send + recv phases, when return_recv_hook=False.

Or does it include the time until the dispatched tokens are received by the destination and the sender is notified?

Yes.

about this part,
1、Are the receive phase and send phases executed sequentially or asynchronously? Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?
2、Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

@ghghliu commented on GitHub (Mar 21, 2025): Thank you for the quick reply, We did a simple test in one server with 8 gpu and 8 cx7 nic. After get the topk_idx tensor as the default random way, we mask the expert index belong to gpu0 with the fellowing code, so that gpu 0 receive zero token from other ranks(including itself, so no local gpu mem copy), but send tokens to other rank as usual. ```python for i in range(num_tokens): # num_tokens=128 for j in range(8): if topk_idx[i,j]//(num_experts//8) == 0: # num_experts=288 topk_idx[i,j] = -1 ``` and the result are as fellows, rank 0 dispatch bandwidth is 79.21GB/s. It looks like the time ends when other rank told rank0 zero expert is sent to you. [rank 3] Dispatch bandwidth: 41.64 GB/s, avg_t=153.35 us | Combine bandwidth: 38.82 GB/s, avg_t=318.30 us [rank 1] Dispatch bandwidth: 41.61 GB/s, avg_t=158.27 us | Combine bandwidth: 40.54 GB/s, avg_t=314.36 us [rank 0] Dispatch bandwidth: 79.21 GB/s, avg_t=83.24 us | Combine bandwidth: 32.74 GB/s, avg_t=389.67 us [rank 7] Dispatch bandwidth: 39.92 GB/s, avg_t=165.17 us | Combine bandwidth: 41.46 GB/s, avg_t=307.76 us [rank 2] Dispatch bandwidth: 43.53 GB/s, avg_t=156.21 us | Combine bandwidth: 41.62 GB/s, avg_t=316.21 us [rank 4] Dispatch bandwidth: 41.81 GB/s, avg_t=158.22 us | Combine bandwidth: 40.65 GB/s, avg_t=314.93 us [rank 6] Dispatch bandwidth: 42.19 GB/s, avg_t=154.88 us | Combine bandwidth: 39.78 GB/s, avg_t=317.84 us [rank 5] Dispatch bandwidth: 42.24 GB/s, avg_t=156.95 us | Combine bandwidth: 40.57 GB/s, avg_t=316.24 us > > Does dispatch_t start at the dispatch kernel launch and end when the dispatch send phase completes? > > It starts at launch, ends **after** receiving all data, i.e. `send + recv` phases, when `return_recv_hook=False`. > > > Or does it include the time until the dispatched tokens are received by the destination and the sender is notified? > > Yes. > about this part, 1、Are the receive phase and send phases executed sequentially or asynchronously? Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? 2、Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 21, 2025):

It looks like the time ends when other rank told rank0 zero expert is sent to you.

Yes, so the printed bandwidth is uni-directional send bandwidth.

Are the receive phase and send phases executed sequentially or asynchronously?

Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized.

Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?

Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends.

Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

The main branch has now removed CQE mechanism.

BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

@LyricZhao commented on GitHub (Mar 21, 2025): > It looks like the time ends when other rank told rank0 zero expert is sent to you. Yes, so the printed bandwidth is uni-directional send bandwidth. > Are the receive phase and send phases executed sequentially or asynchronously? Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized. > Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends. > Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens? The main branch has now removed CQE mechanism. BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@ghghliu commented on GitHub (Mar 21, 2025):

Get it, thanks a lot!

@ghghliu commented on GitHub (Mar 21, 2025): Get it, thanks a lot!

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@wangshaochuang commented on GitHub (Apr 2, 2025):

It looks like the time ends when other rank told rank0 zero expert is sent to you.

Yes, so the printed bandwidth is uni-directional send bandwidth.

Are the receive phase and send phases executed sequentially or asynchronously?

Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized.

Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?

Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends.

Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

The main branch has now removed CQE mechanism.
@LyricZhao 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？

BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

@wangshaochuang commented on GitHub (Apr 2, 2025): > > It looks like the time ends when other rank told rank0 zero expert is sent to you. > > Yes, so the printed bandwidth is uni-directional send bandwidth. > > > Are the receive phase and send phases executed sequentially or asynchronously? > > Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized. > > > Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? > > Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends. > > > Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens? > > The main branch has now removed CQE mechanism. @LyricZhao 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？ > > BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@sphish commented on GitHub (Apr 3, 2025):

It looks like the time ends when other rank told rank0 zero expert is sent to you.

Yes, so the printed bandwidth is uni-directional send bandwidth.

Are the receive phase and send phases executed sequentially or asynchronously?

Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized.

Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?

Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends.

Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

The main branch has now removed CQE mechanism.
@LyricZhao 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？
BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

@wangshaochuang nvshmem 在创建 cq 的时候设置了 overrun 模式。

@sphish commented on GitHub (Apr 3, 2025): > > > It looks like the time ends when other rank told rank0 zero expert is sent to you. > > > > > > Yes, so the printed bandwidth is uni-directional send bandwidth. > > > Are the receive phase and send phases executed sequentially or asynchronously? > > > > > > Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized. > > > Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? > > > > > > Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends. > > > Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens? > > > > > > The main branch has now removed CQE mechanism. > > [@LyricZhao](https://github.com/LyricZhao) 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？ > > BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese. @wangshaochuang nvshmem 在创建 cq 的时候设置了 overrun 模式。

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@wangshaochuang commented on GitHub (Apr 3, 2025):

It looks like the time ends when other rank told rank0 zero expert is sent to you.

Yes, so the printed bandwidth is uni-directional send bandwidth.

Are the receive phase and send phases executed sequentially or asynchronously?

Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized.

Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?

Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends.

Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

The main branch has now removed CQE mechanism.
@LyricZhao 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？
BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

@wangshaochuang nvshmem 在创建 cq 的时候设置了 overrun 模式。

明白了，谢谢

@wangshaochuang commented on GitHub (Apr 3, 2025): > > > > It looks like the time ends when other rank told rank0 zero expert is sent to you. > > > > > > > > > Yes, so the printed bandwidth is uni-directional send bandwidth. > > > > Are the receive phase and send phases executed sequentially or asynchronously? > > > > > > > > > Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized. > > > > Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? > > > > > > > > > Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends. > > > > Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens? > > > > > > > > > The main branch has now removed CQE mechanism. > > > [@LyricZhao](https://github.com/LyricZhao) 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？ > > > BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese. > > [@wangshaochuang](https://github.com/wangshaochuang) nvshmem 在创建 cq 的时候设置了 overrun 模式。明白了，谢谢

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@wangshaochuang commented on GitHub (Apr 3, 2025):

It looks like the time ends when other rank told rank0 zero expert is sent to you.

Yes, so the printed bandwidth is uni-directional send bandwidth.

Are the receive phase and send phases executed sequentially or asynchronously?

Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized.

Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends?

Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends.

Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens?

The main branch has now removed CQE mechanism.
@LyricZhao 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？
BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese.

@wangshaochuang nvshmem 在创建 cq 的时候设置了 overrun 模式。
@sphish
对于low latency的dispatch函数，还有一点疑问，发送侧在进行数据的拷贝和发送之后，没有进行cqe的检查（这时候本地的send buffer还在使用，不能进行覆盖），然后就是recv阶段来check memory进行确认数据是否收到，收到之后进行copy，dispatch结束；如果dispatch进行了第二轮，这时候进行数据拷贝，但是没有机制保证第一次的数据发送已经完成，send buffer可以被重用；当在一个rank上发送比较慢的时候，或者同步不太好的时候是否有send buffer会被覆盖写，导致数据出错的问题？

@wangshaochuang commented on GitHub (Apr 3, 2025): > > > > It looks like the time ends when other rank told rank0 zero expert is sent to you. > > > > > > > > > Yes, so the printed bandwidth is uni-directional send bandwidth. > > > > Are the receive phase and send phases executed sequentially or asynchronously? > > > > > > > > > Asynchronously, as there is no barrier before launching the dispatch kernel. But as all the ranks will try to read the number of tokens from all ranks, it is a natural "barrier", after several rounds, all ranks should be synchronized. > > > > Is it possible the rank receiving all data dispatched to it before its dispatched token are received by others, and if this happens, will the profile ends? > > > > > > > > > Yes, possible. But receiving have two "stages", the first is the data transmission (hidden in network), the second is organizating the data using GPU kernels. The first may be not counted before profile starts, but the second will always be counted before the profile ends. > > > > Is the sender determining the completion of the sending phase by CQE of the message that sends the number of tokens? > > > > > > > > > The main branch has now removed CQE mechanism. > > > [@LyricZhao](https://github.com/LyricZhao) 这里我有一个问题想要请教下，在tx queue中我看到很多wqe都是加了update cqe的flag，但是没有看到对于cqe的处理，这样在理论上会导致cq的overrun，是我的代码没有看全，还是有什么其他机制保证了对于cq的处理是正确的？ > > > BTW, for more convenient communication between us, can you have my WeChat (LyricZ_THU)? If you can speak Chinese. > > [@wangshaochuang](https://github.com/wangshaochuang) nvshmem 在创建 cq 的时候设置了 overrun 模式。 @sphish 对于low latency的dispatch函数，还有一点疑问，发送侧在进行数据的拷贝和发送之后，没有进行cqe的检查（这时候本地的send buffer还在使用，不能进行覆盖），然后就是recv阶段来check memory进行确认数据是否收到，收到之后进行copy，dispatch结束；如果dispatch进行了第二轮，这时候进行数据拷贝，但是没有机制保证第一次的数据发送已经完成，send buffer可以被重用；当在一个rank上发送比较慢的时候，或者同步不太好的时候是否有send buffer会被覆盖写，导致数据出错的问题？

bsakaguchi commented

2026-10-26 09:53:55 +00:00

Author

Owner

@LyricZhao commented on GitHub (Apr 3, 2025):

一共有两套 buffer，不会有重用的情况。

@LyricZhao commented on GitHub (Apr 3, 2025): 一共有两套 buffer，不会有重用的情况。

Rows
Columns

[GH-ISSUE #84] Questions on Profiling dispatch_t for Low-Latency Kernel Dispatch Bandwidth Calculation #77

[GH-ISSUE #84] Questions on Profiling `dispatch_t` for Low-Latency Kernel Dispatch Bandwidth Calculation #77