mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #124] Is Multi-head Latent Attention used in DeepSeek-VL2 tiny? #102
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#102
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @niktheod on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/124
Hi and thank you for your great work,
In the paper you published in Table 1 it says that the tiny version of the model uses the original Multi-Head Attention and not MLA:
However, when I run it locally I see the following module in the architecture:
which is part of the vision model, which makes it even more confusing for me as I thought that MLA is applied in the language model.
Could you please clarify what this module is, as it is not part of the original siglip-so400m-patch14-384, which is supposed to be your vision backbone.