mirror of
https://github.com/deepseek-ai/DeepSeek-V3.git
synced 2026-11-05 00:47:48 +00:00
[GH-ISSUE #13] How MLA 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP) is implemented? #11
Labels
No labels
closed-as-stale
enhancement
question
spam
stale
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-V3#11
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @FC-Li on GitHub (Dec 27, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/13
Great work, Deepseekers!
The model and the tech report are literally masterpiece. But I still have some doubts about inference.
Could you reveal more details about how MLA 4-way Tensor Parallelism (TP4) with Sequence Parallelism (SP) is implemented?
How the attention weights are partitioned between different GPUs?
@Cccei000 commented on GitHub (Feb 7, 2025):
have you figured out how tp sp and dp are combined? I do believe the sp in the paper refers to the technique in https://arxiv.org/pdf/2205.05198 which is an extension to tp, and that's why the paper does not go into details. But still it confuses me, what is dp doing here with the following ep32?
with ep32 and dp8, doesn't it bring replicated computation 'cause each every 4 gpus hold the same data?
@adolli commented on GitHub (Feb 12, 2025):
I believe so, there is nothing about the sequence parallelism in their open-sourced inference code. For training purpose, their TP+SP is working similar as the Megatron-LM ways.
@YyWangCS commented on GitHub (Feb 23, 2025):
This is my idea about how to implement TP+SP for MLA, https://yywangcs.notion.site/How-to-Perform-Multi-GPU-Parallelization-for-DeepSeek-MLA-attention-during-Decoding-1a3fc9f5d805807bb9a4df057839163b
A Chinese version can be found on https://zhuanlan.zhihu.com/p/25573883266
@github-actions[bot] commented on GitHub (Mar 31, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
@github-actions[bot] commented on GitHub (Apr 15, 2025):
false