mirror of
https://github.com/deepseek-ai/DeepSeek-V2.git
synced 2026-11-05 00:47:23 +00:00
[GH-ISSUE #23] Failure to reproduce MLA > MHA #22
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-V2#22
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @faresobeid on GitHub (May 12, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/23
I tried out MLA and it was a good amount worse than MHA and wanted to try to find out why. Firstly, I am using a hybrid model therefore I am not using any Rope in either MLA or MHA, and therefore use the basic version of MLA. I suspect the issue could be due to the part saying:
"In addition, the low-rank compression and fine-grained expert segmentation
will impact the output scale of a layer. Therefore, in practice, we employ additional RMS Norm
layers after the compressed latent vectors, and multiply additional scaling factors at the width
bottlenecks (i.e., the compressed latent vectors and the intermediate hidden states of routed experts) to ensure stable training."
It is unclear if the additional scaling factor is done before or after the RMSNorm, also what this factor would be. Another reason could be that the rope version of MLA gives it a performance boost.
Any clarification on this scaling factor and its placement would be great, thanks!
@luofuli commented on GitHub (May 14, 2024):
The following are factors that affect the final result:
@faresobeid
@faresobeid commented on GitHub (May 14, 2024):
Thank you!
@faresobeid commented on GitHub (May 23, 2024):
Sorry to reopen this issue but I have been having some issues with stability at scale with MLA. Like I said before I am using a hybrid model and therefore the MLA that I am using is the basic version with no Rope.
kv = W_b(rms_norm(W_a(x)))I have also tried having
kv = W_b(rms_norm(W_a(ln(x))))but that also has some issues with performance and stability. To be more specific, this is a 24 layer model with model dimension 2048 with the last third of layers using MLA (~300M params). Is there any recommended scaling factors or ways to mitigate this issue? Thank you
@luofuli commented on GitHub (May 27, 2024):
24-layer dense model? @faresobeid
@faresobeid commented on GitHub (May 27, 2024):
Yes, although stability has been fine without the inner rms norm but still any recommendations would be helpful
@LPeng625 commented on GitHub (Mar 3, 2025):
矩阵吸收的计算量反而比正常计算(不使用矩阵结合律)高？代码中的MLA似乎并没有使用矩阵吸收，请问是有什么特殊的考虑吗？