deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #3] NCCL baseline performance #2

New issue

Closed

opened 2026-10-26 09:52:05 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:52:05 +00:00

Owner

Originally created by @YangZhou1997 on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/3

Hi DeepEP maintainers,

Really excited project! I wonder how does the in-stock NCCL would perform under the same testbed you have. Releasing this number would help the community understand the performance gain you achieved.

For example, in my local 2 DGX-H100 nodes with 400G NIC, I can easily achieve 46 GBps busbw with 30 microseconds small message latency. This is measured using standard nccl-tests alltoall with NVLink disabled to stress test the network. My testbed has both CC and adaptive routing enabled.

However, I do not have more nodes than two, thus cannot measure the NCCL baseline performance beyond that. So I wonder how would NCCL perform. My guess is the low-latency mode might cut the time of CPU issuing RDMA verbs like 2-4 microseconds, as it lets the GPU issue directly; but the overall achieved bandwidth would be pretty much the same as in-stock NCCL, given the network won't have congestion due to adaptive routing.

Best,
Yang

Originally created by @YangZhou1997 on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/3 Hi DeepEP maintainers, Really excited project! I wonder how does the in-stock NCCL would perform under the same testbed you have. Releasing this number would help the community understand the performance gain you achieved. For example, in my local 2 DGX-H100 nodes with 400G NIC, I can easily achieve 46 GBps busbw with 30 microseconds small message latency. This is measured using standard nccl-tests alltoall with NVLink disabled to stress test the network. My testbed has both CC and adaptive routing enabled. <img width="1548" alt="Image" src="https://github.com/user-attachments/assets/c3d2528c-791b-46cb-b28d-3e65fce21d0a" /> However, I do not have more nodes than two, thus cannot measure the NCCL baseline performance beyond that. So I wonder how would NCCL perform. My guess is the low-latency mode might cut the time of CPU issuing RDMA verbs like 2-4 microseconds, as it lets the GPU issue directly; but the overall achieved bandwidth would be pretty much the same as in-stock NCCL, given the network won't have congestion due to adaptive routing. Best, Yang

bsakaguchi closed this issue

2026-10-26 09:52:06 +00:00

bsakaguchi commented

2026-10-26 09:52:07 +00:00

Author

Owner

@sphish commented on GitHub (Feb 25, 2025):

The dispatch and combine operations in EP are more sophisticated than classical all-to-all scenarios. Each token needs to be routed to different GPU ranks based on topk_idx, involving complex control and layout logic. Our kernel has fused these operations together for optimal performance. Furthermore, EP's single message sizes are significantly smaller, and traffic patterns can be asymmetric. Therefore, we think that NCCL's all_to_all_test results cannot be directly compared to our implementation.

We welcome contributions from the community if anyone would like to develop an EP communication benchmark using NCCL and contribute it to our repository.

@sphish commented on GitHub (Feb 25, 2025): The dispatch and combine operations in EP are more sophisticated than classical all-to-all scenarios. Each token needs to be routed to different GPU ranks based on topk_idx, involving complex control and layout logic. Our kernel has fused these operations together for optimal performance. Furthermore, EP's single message sizes are significantly smaller, and traffic patterns can be asymmetric. Therefore, we think that NCCL's all_to_all_test results cannot be directly compared to our implementation. We welcome contributions from the community if anyone would like to develop an EP communication benchmark using NCCL and contribute it to our repository.

bsakaguchi commented

2026-10-26 09:52:07 +00:00

Author

Owner

@youkaichao commented on GitHub (Feb 25, 2025):

NCCL is quite inflexible in the expert-parallel case, because the number of tokens routed to each expert is dynamic. However, NCCL can only communicate tensors with known sizes, see the api https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/p2p.html for example.

If you want to use NCCL, you need at least 2 communication passes:

one pass to aggregate metadata, i.e. number of tokens received by each expert
prepare buffers with corresponding size, and then launch the communication to send/recv tensors.

NOTE: technically we can use NCCL send/recv to achieve all-to-all, however, if the sizes of all-to-all are dynamic, they will not be compatible with cudagraph.

@youkaichao commented on GitHub (Feb 25, 2025): NCCL is quite inflexible in the expert-parallel case, because the number of tokens routed to each expert is dynamic. However, NCCL can only communicate tensors with known sizes, see the api https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/p2p.html for example. If you want to use NCCL, you need at least 2 communication passes: - one pass to aggregate metadata, i.e. number of tokens received by each expert - prepare buffers with corresponding size, and then launch the communication to send/recv tensors. NOTE: technically we can use NCCL send/recv to achieve all-to-all, however, if the sizes of all-to-all are dynamic, they will not be compatible with cudagraph.

bsakaguchi commented

2026-10-26 09:52:07 +00:00

Author

Owner

@demonatic commented on GitHub (Mar 10, 2025):

@youkaichao @sphish I wonder can latency of "nccl 2 communication passes" tested using nccl-test be comparable to "Dispatch avg_t" in test_low_latency.py ?

@demonatic commented on GitHub (Mar 10, 2025): @youkaichao @sphish I wonder can latency of "nccl 2 communication passes" tested using nccl-test be comparable to "Dispatch avg_t" in test_low_latency.py ?

bsakaguchi commented

2026-10-26 09:52:07 +00:00

Author

Owner

@alokprasad commented on GitHub (Oct 23, 2025):

i think it will be good to have a benchmark with recent NCCL they Recently added alltoall and with symmetrical memory support
https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-28-3.html#rel_2-28-3

@alokprasad commented on GitHub (Oct 23, 2025): i think it will be good to have a benchmark with recent NCCL they Recently added alltoall and with symmetrical memory support https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-28-3.html#rel_2-28-3

bsakaguchi referenced this issue

2026-10-26 09:53:39 +00:00

[GH-ISSUE #74] test_internode.py hangs at internode.cu:dispatch(...) #68

bsakaguchi referenced this issue

2026-10-26 09:54:16 +00:00

[GH-ISSUE #106] how can I run the test across multi nodes？ #97

bsakaguchi referenced this issue

2026-10-26 09:56:29 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200

bsakaguchi referenced this issue

2026-10-26 09:56:40 +00:00

[GH-ISSUE #290] Running the test case on a 2-node H20 3E machine failed with internode.cu:1029 'unspecified launch failure' #208