mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #99] How to avoid OOM in the early stage of training using DeepEP #90
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#90
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ForFishes on GitHub (Mar 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/99
Using DeepEP can efficiently implement MOE model training without dropping tokens. However, in the early stages of training, or cold start training, the model has not reached a good convergence state. Using DeepEP will cause a GPU device to accept more tokens, causing subsequent calculations to be out of memory(OOM). I am curious about how this situation is avoided in the early stages of actual training.
@LyricZhao commented on GitHub (Mar 31, 2025):
DeepEP is doing its best to allocate less memory. The OOM is mainly caused by expert selection / framework level staffs.
To solve OOM, you can try more TPs (adjust your parallel strategies at early phase of training), or implement CPU offloading (if receiving much tokens), or adjust gate (drop at
topk_idxcomputation), or do more tensor recomputation (SwiGLU, RMSNorm).@isayoften commented on GitHub (Apr 10, 2025):
@LyricZhao Hi! Explain please, what do you mean by "adjust gate (drop at topk_idx computation)"?
@LyricZhao commented on GitHub (Apr 11, 2025):
Drop some top-k selections if a rank has too many tokens to receive.