deepseek-ai/DeepSeek-V3
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V3.git synced 2026-11-05 00:47:48 +00:00
Code Issues 25 Projects Packages Wiki Activity

[GH-ISSUE #18] Any comparison of MLA vs MQA? #13

New issue
Closed
opened 2026-10-26 21:10:55 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:10:55 +00:00
Owner
Copy link

Originally created by @chaoyan1037 on GitHub (Dec 28, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/18

The V2 paper only directly compared MLA with MHA. Table 8 in V2 paper only compares dense models with MHA, MQA, and GQA, without MLA. Table 9 compares MoE models with MHA and MLA, without MQA or GQA.

I feel the MLA is kind of similar to MQA in the essence. I am curious if there are any apple-to-apple ablation studies on MLA vs MQA?

Originally created by @chaoyan1037 on GitHub (Dec 28, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/18 The V2 paper only directly compared MLA with MHA. Table 8 in V2 paper only compares dense models with MHA, MQA, and GQA, without MLA. Table 9 compares MoE models with MHA and MLA, without MQA or GQA. I feel the MLA is kind of similar to MQA in the essence. I am curious if there are any apple-to-apple ablation studies on MLA vs MQA?
bsakaguchi 2026-10-26 21:10:55 +00:00
bsakaguchi commented 2026-10-26 21:10:56 +00:00
Author
Owner
Copy link

@MonolithFoundation commented on GitHub (Jan 9, 2025):

MLA didn't have obvious performance enhance compare with MQA, however, not worse either, but faster.

<!-- gh-comment-id:2579363332 --> @MonolithFoundation commented on GitHub (Jan 9, 2025): MLA didn't have obvious performance enhance compare with MQA, however, not worse either, but faster.
bsakaguchi commented 2026-10-26 21:10:56 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Feb 9, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:2645994609 --> @github-actions[bot] commented on GitHub (Feb 9, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:10:57 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Feb 25, 2025):

false

<!-- gh-comment-id:2680004958 --> @github-actions[bot] commented on GitHub (Feb 25, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
closed-as-stale

   
enhancement

   
question

   
spam

   
stale
No labels
closed-as-stale
enhancement
question
spam
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-V3#13
No description provided.