deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #322] Low latency kernels deliver low BW for 256#EP on 32N #223

New issue

Closed

opened 2026-10-26 09:56:56 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:56:56 +00:00

Owner

Originally created by @aahouzi on GitHub (Jul 22, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/322

Type of issue

I performed few tests with low latency kernels on RoCEv2 cluster, with multiple configurations as shown below, and notice that in all my tests num_qps_per_rank =1:

See how in the 256#EP case, performance drops sharply. When I tried running 256#EP on 16N so num_qps_per_rank =2, I obtained way better results:

Can you guys explain why there is such drop in performance when I run it on 32N vs 16N ? Thx :)

Originally created by @aahouzi on GitHub (Jul 22, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/322 ## Type of issue - I performed few tests with low latency kernels on RoCEv2 cluster, with multiple configurations as shown below, and notice that in all my tests num_qps_per_rank =1: <img width="916" height="223" alt="Image" src="https://github.com/user-attachments/assets/bd366ff2-a860-410c-800c-95118b5c3138" /> - See how in the 256#EP case, performance drops sharply. When I tried running 256#EP on 16N so num_qps_per_rank =2, I obtained way better results: <img width="914" height="99" alt="Image" src="https://github.com/user-attachments/assets/22374a6a-371d-4fab-9c6f-2933951cff0e" /> - Can you guys explain why there is such drop in performance when I run it on 32N vs 16N ? Thx :)

bsakaguchi closed this issue

2026-10-26 09:56:56 +00:00

bsakaguchi commented

2026-10-26 09:56:58 +00:00

Author

Owner

@sphish commented on GitHub (Jul 29, 2025):

Based on my experience, it's likely related to the network congestion. You can check the congestion-related counters when running with 32N.

@sphish commented on GitHub (Jul 29, 2025): Based on my experience, it's likely related to the network congestion. You can check the congestion-related counters when running with 32N.

bsakaguchi commented

2026-10-26 09:56:58 +00:00

Author

Owner

@aahouzi commented on GitHub (Aug 7, 2025):

@sphish Thank you for ur reply. I saw in the internode tests script that you enforce number of local ranks to be 8 with an assert statement, can you explain why ? What if I want to run 16N 1ppn test with internode kernels ?

@aahouzi commented on GitHub (Aug 7, 2025): @sphish Thank you for ur reply. I saw in the internode tests script that you enforce number of local ranks to be 8 with an assert statement, can you explain why ? What if I want to run 16N 1ppn test with internode kernels ?

bsakaguchi commented

2026-10-26 09:56:59 +00:00

Author

Owner

@sphish commented on GitHub (Aug 8, 2025):

In the current version, our design only considers the case of 8 GPUs per node.

@sphish commented on GitHub (Aug 8, 2025): In the current version, our design only considers the case of 8 GPUs per node.

bsakaguchi commented

2026-10-26 09:56:59 +00:00

Author

Owner

@aahouzi commented on GitHub (Aug 26, 2025):

Thanks @sphish, I confirmed it's a network congestion issue.

@aahouzi commented on GitHub (Aug 26, 2025): Thanks @sphish, I confirmed it's a network congestion issue.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#223

No description provided.

Rows
Columns