deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #101] Why not DeepSeek-V3 choose EP128*PP16 during pretraining #91

New issue

Closed

opened 2026-10-26 09:54:08 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:54:08 +00:00

Owner

Originally created by @sneaxiy on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/101

Compared to EP64 * PP16 * ZeRO-Stage-1 2, EP128 * PP16 does not need the ZeRO communication of expert parameters. If the communication performance between EP64 and EP128 is almost the same, EP128 * PP16 may be better than EP64 * PP16 * ZeRO-Stage-1 2. Why not DeepSeek-V3 choose EP128*PP16 during pretraining? Is it because the communication performance between EP64 and EP128 differ a lot?

Originally created by @sneaxiy on GitHub (Mar 31, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/101 Compared to EP64 * PP16 * ZeRO-Stage-1 2, EP128 * PP16 does not need the ZeRO communication of expert parameters. If the communication performance between EP64 and EP128 is almost the same, EP128 * PP16 may be better than EP64 * PP16 * ZeRO-Stage-1 2. Why not DeepSeek-V3 choose EP128*PP16 during pretraining? Is it because the communication performance between EP64 and EP128 differ a lot?

bsakaguchi closed this issue

2026-10-26 09:54:08 +00:00

bsakaguchi commented

2026-10-26 09:54:09 +00:00

Author

Owner

@LyricZhao commented on GitHub (Apr 1, 2025):

The gate divides experts into 8 groups and selects 3 groups, so EP64 (8 nodes) is a must. Or to say, the EP number and gate is co-designed.

And the cost of ZeRO is not to much.

@LyricZhao commented on GitHub (Apr 1, 2025): The gate divides experts into 8 groups and selects 3 groups, so EP64 (8 nodes) is a must. Or to say, the EP number and gate is co-designed. And the cost of ZeRO is not to much.

bsakaguchi commented

2026-10-26 09:54:10 +00:00

Author

Owner

@sneaxiy commented on GitHub (Apr 1, 2025):

The gate divides experts into 8 groups and selects 3 groups, so EP64 (8 nodes) is a must. Or to say, the EP number and gate is co-designed.

And the cost of ZeRO is not to much.

Do you mean the gate selects experts on 3 nodes atmost? If we use EP128 and the gate selects 6 nodes atmost, what would be the performance in this case?

@sneaxiy commented on GitHub (Apr 1, 2025): > The gate divides experts into 8 groups and selects 3 groups, so EP64 (8 nodes) is a must. Or to say, the EP number and gate is co-designed. > > And the cost of ZeRO is not to much. Do you mean the gate selects experts on 3 nodes atmost? If we use EP128 and the gate selects 6 nodes atmost, what would be the performance in this case?

bsakaguchi commented

2026-10-26 09:54:10 +00:00

Author

Owner

@LyricZhao commented on GitHub (Apr 1, 2025):

The RDMA communication volume will be doubled, so the performance is 2x worse.

@LyricZhao commented on GitHub (Apr 1, 2025): The RDMA communication volume will be doubled, so the performance is 2x worse.

bsakaguchi commented

2026-10-26 09:54:10 +00:00

Author

Owner

@sneaxiy commented on GitHub (Apr 1, 2025):

The RDMA communication volume will be doubled, so the performance is 2x worse.

Thanks a lot.

@sneaxiy commented on GitHub (Apr 1, 2025): > The RDMA communication volume will be doubled, so the performance is 2x worse. Thanks a lot.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#91

No description provided.

Rows
Columns