mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #58] output tokens in intranode::dispatch for each expert isn't packed? #55
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#55
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zhang662817 on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/58
The output token in the intranode::dispatch kernel offsets is channel_offset + rank_offsets + recv_token_idx;
The tokens for each expert isn't contiguous?
If yes, do you have plan to optimize these?
Thanks.
@LyricZhao commented on GitHub (Mar 10, 2025):
Yes, and it is by design. Some MoE models will make one token averagely select more than one expert in a GPU rank. So if we automatically expand/copy these tokens into different experts, the GPU memory consumption will be
\timesexpert-per-rank-per-token-selected, and make training OOM.So if you want to do grouped GEMM later, you have to expand the tokens and control the GPU memory precisely by yourself. We currently don't have plans to refactor this design.