[GH-ISSUE #59] AllToAll performance not improving

bsakaguchi commented

2026-10-26 09:53:28 +00:00

Owner

Originally created by @cryoco on GitHub (Mar 8, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/59

Hello DeepEP developers, appreciate the solid works!
I conducted an all-to-all communication performance comparison experiment on 8 H100 nodes, using Megatron-LM's DeepEP Integration.
To eliminate the impact of pipeline parallelism bubbles and focus on comm performances, I use the first pipeline stage (8 layers with first-3-dense) of the DeepSeek V3 model only, instead of the full model.
I ran pretraining experiments with DeepEP enabled/disabled(other configurations are identical), and the throughputs were roughly the same, which is below the expectation.
I collected the nsys profile reports as follows:
DeepEP(with normal kernels) enabled:

DeepEP disabled:

There are no PPs so the SendRecv kernels represent the alltoall kernels.
It seems the combining alltoall comms are faster(13ms->5ms) with DeepEP, but dispatching becomes slower(13ms->16~18ms), and notify_dispatch/cached_notify costs roughly the same time as combining, which makes the e2e throughput not improve.
Any suggestions or insights are appreciated. Thanks!

Originally created by @cryoco on GitHub (Mar 8, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/59 Hello DeepEP developers, appreciate the solid works! I conducted an all-to-all communication performance comparison experiment on 8 H100 nodes, using [Megatron-LM's DeepEP Integration](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core/transformer/moe#performance-optimizations). To eliminate the impact of pipeline parallelism bubbles and focus on comm performances, I use the first pipeline stage (8 layers with first-3-dense) of the DeepSeek V3 model only, instead of the full model. I ran pretraining experiments with DeepEP enabled/disabled(other configurations are identical), and the throughputs were roughly the same, which is below the expectation. I collected the nsys profile reports as follows: DeepEP(with normal kernels) enabled: ![Image](https://github.com/user-attachments/assets/0eb030f9-adf2-40a2-b114-f2a926fd52ed) DeepEP disabled: ![Image](https://github.com/user-attachments/assets/18a6e8c7-7642-40cb-ba1a-0a9f8c1d1e63) There are no PPs so the SendRecv kernels represent the alltoall kernels. It seems the combining alltoall comms are faster(13ms->5ms) with DeepEP, but dispatching becomes slower(13ms->16~18ms), and notify_dispatch/cached_notify costs roughly the same time as combining, which makes the e2e throughput not improve. Any suggestions or insights are appreciated. Thanks!

bsakaguchi commented

2026-10-26 09:53:30 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 10, 2025):

Can you please run the performance test in https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_internode.py? You may not use the best config in your cluster.
Or you can check your batch size (DeepEP is not very good when bsz=2K, good at 4K) and calculate the bandwidth in your figure.
notify_dispatch/cached_notify is slow and we may optimize later. But they are also actually barriers, extreme longer time may be due to the computation kernel imbalance. If you are using random data to test, you may change it to a guaranteed balanced gate. Or you can check whether the arrival time of all 64 ranks is aligned.
Or, DeepEP is faster than NCCL when the gate has the property that, there is a broadcasting process between the NVLink forwarding (selecting more than one expert on a node, so that the RDMA transmission is less).

@LyricZhao commented on GitHub (Mar 10, 2025): 1. Can you please run the performance test in `https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_internode.py`? You may not use the best config in your cluster. 2. Or you can check your batch size (DeepEP is not very good when bsz=2K, good at 4K) and calculate the bandwidth in your figure. 3. `notify_dispatch/cached_notify` is slow and we may optimize later. But they are also actually barriers, **extreme longer time** may be due to the computation kernel imbalance. If you are using random data to test, you may change it to a **guaranteed** balanced gate. Or you can check whether the arrival time of all 64 ranks is aligned. 4. Or, DeepEP is faster than NCCL when the gate has the property that, there is a broadcasting process between the NVLink forwarding (selecting more than one expert on a node, so that the RDMA transmission is less).

bsakaguchi commented

2026-10-26 09:53:30 +00:00

Author

Owner

@cryoco commented on GitHub (Mar 13, 2025):

Can you please run the performance test in https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_internode.py? You may not use the best config in your cluster.

Or you can check your batch size (DeepEP is not very good when bsz=2K, good at 4K) and calculate the bandwidth in your figure.

notify_dispatch/cached_notify is slow and we may optimize later. But they are also actually barriers, extreme longer time may be due to the computation kernel imbalance. If you are using random data to test, you may change it to a guaranteed balanced gate. Or you can check whether the arrival time of all 64 ranks is aligned.

Or, DeepEP is faster than NCCL when the gate has the property that, there is a broadcasting process between the NVLink forwarding (selecting more than one expert on a node, so that the RDMA transmission is less).

@LyricZhao Thanks for quick reply!

I did perform tuning beforehand and use tuned configs.
By bsz=4K, do you mean seq_len * bsz=4K? That fits my case.
I set moe_expert_capacity_factor=1.0 to guarantee token dispatching balance. Still, actual arriving time is sure worth diving into. I'll check that.
That fits my case too.

BTW, I noticed unexpected PCIe flow(about 20GiB/s) in cluster DCGM monitor. Wonder the reason.

@cryoco commented on GitHub (Mar 13, 2025): > 1. Can you please run the performance test in `https://github.com/deepseek-ai/DeepEP/blob/main/tests/test_internode.py`? You may not use the best config in your cluster. > 2. Or you can check your batch size (DeepEP is not very good when bsz=2K, good at 4K) and calculate the bandwidth in your figure. > 3. `notify_dispatch/cached_notify` is slow and we may optimize later. But they are also actually barriers, **extreme longer time** may be due to the computation kernel imbalance. If you are using random data to test, you may change it to a **guaranteed** balanced gate. Or you can check whether the arrival time of all 64 ranks is aligned. > 4. Or, DeepEP is faster than NCCL when the gate has the property that, there is a broadcasting process between the NVLink forwarding (selecting more than one expert on a node, so that the RDMA transmission is less). @LyricZhao Thanks for quick reply! 1. I did perform tuning beforehand and use tuned configs. 2. By bsz=4K, do you mean seq_len * bsz=4K? That fits my case. 3. I set moe_expert_capacity_factor=1.0 to guarantee token dispatching balance. Still, actual arriving time is sure worth diving into. I'll check that. 4. That fits my case too. BTW, I noticed unexpected PCIe flow(about 20GiB/s) in cluster DCGM monitor. Wonder the reason. ![Image](https://github.com/user-attachments/assets/4bc60c2a-dd21-4ac5-8b6d-2c6042f8d37f)

bsakaguchi commented

2026-10-26 09:53:30 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 13, 2025):

I did perform tuning beforehand and use tuned configs.

Is the performance correct (should be around ~40 GB/s)?

I set moe_expert_capacity_factor=1.0 to guarantee token dispatching balance. Still, actual arriving time is sure worth diving into. I'll check that.

Yep, I am not familar with Megatron. You better dive into that to see how many tokens each rank is receiving.

BTW, I witnessed unexpected PCIe flow(about 20GiB/s) in cluster DCGM monitor. Wonder the reason.

RDMA transmission path: GPU -> PCIe -> network card (send), network card -> PCIe -> GPU (recv). If the PCIe bandwidth is not enough for all GPUs/network card, it can be the bottleneck.

@LyricZhao commented on GitHub (Mar 13, 2025): > I did perform tuning beforehand and use tuned configs. Is the performance correct (should be around ~40 GB/s)? > I set moe_expert_capacity_factor=1.0 to guarantee token dispatching balance. Still, actual arriving time is sure worth diving into. I'll check that. Yep, I am not familar with Megatron. You better dive into that to see how many tokens each rank is receiving. > BTW, I witnessed unexpected PCIe flow(about 20GiB/s) in cluster DCGM monitor. Wonder the reason. RDMA transmission path: GPU -> PCIe -> network card (send), network card -> PCIe -> GPU (recv). If the PCIe bandwidth is not enough for all GPUs/network card, it can be the bottleneck.

bsakaguchi commented

2026-10-26 09:53:30 +00:00

Author

Owner

@cryoco commented on GitHub (Mar 15, 2025):

@LyricZhao

Is the performance correct (should be around ~40 GB/s)?

Nope, latency is around 8~12ms(both dispatch and combine) and BW around 5GB/s(not precise due to sampling interval but likely lower than 40GB/s)

Yep, I am not familar with Megatron. You better dive into that to see how many tokens each rank is receiving.

I've done the check. The balancing is good. New discovery: Added while loop to dispatching in training, roughly like:

while True:
    timer.start()
    recv_x, ... = buffer.dispatch(x, ...)
    torch.cuda.synchronize()
    timer.stop()

In the first few cycles, the latencies are around 10ms as before; But in the following loops, the latencies are as good as in the unittests(around 2ms). It seems in each dispatching and combining, it requires some "warming up" to reach ideal performance. Perhaps it is somehow related to the buffer caching mechanism in DeepEP?

@cryoco commented on GitHub (Mar 15, 2025): @LyricZhao > Is the performance correct (should be around ~40 GB/s)? Nope, latency is around 8~12ms(both dispatch and combine) and BW around 5GB/s(not precise due to sampling interval but likely lower than 40GB/s) > Yep, I am not familar with Megatron. You better dive into that to see how many tokens each rank is receiving. I've done the check. The balancing is good. New discovery: Added while loop to dispatching in training, roughly like: ```python while True: timer.start() recv_x, ... = buffer.dispatch(x, ...) torch.cuda.synchronize() timer.stop() ``` In the first few cycles, the latencies are around 10ms as before; But in the following loops, the latencies are as good as in the unittests(around 2ms). It seems in each dispatching and combining, it requires some "warming up" to reach ideal performance. Perhaps it is somehow related to the buffer caching mechanism in DeepEP?

bsakaguchi commented

2026-10-26 09:53:30 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 17, 2025):

In the first few cycles, the latencies are around 10ms as before; But in the following loops, the latencies are as good as in the unittests(around 2ms). It seems in each dispatching and combining, it requires some "warming up" to reach ideal performance. Perhaps it is somehow related to the buffer caching mechanism in DeepEP?

No explicit warmup in DeepEP, the __init__ function will do all buffer setup at once. The only reason I come up with is that your network needs some warmup, e.g. each rank arrives not at the same timestamp (or the CPU speed is not the same, launching the kernels at different timestamp).

@LyricZhao commented on GitHub (Mar 17, 2025): > In the first few cycles, the latencies are around 10ms as before; But in the following loops, the latencies are as good as in the unittests(around 2ms). It seems in each dispatching and combining, it requires some "warming up" to reach ideal performance. Perhaps it is somehow related to the buffer caching mechanism in DeepEP? No explicit warmup in DeepEP, the `__init__` function will do all buffer setup at once. The only reason I come up with is that your network needs some warmup, e.g. each rank arrives not at the same timestamp (or the CPU speed is not the same, launching the kernels at different timestamp).

Rows
Columns

[GH-ISSUE #59] AllToAll performance not improving #56