mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #73] does deepseekVL2 tiny and small support flash attention2? #58
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#58
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @h-ccc on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/73
Great work here!
I am trying to fine-tune DeepSeekVL2 on my dataset, but the training speed is too slow without flashattention2. When I modify the code to activate flashattention2, I encounter the following error:
"Error(s) in loading state_dict for DeepseekVLV2ForCausalLM:
size mismatch for language.model.embed_tokens.weight: copying a param with shape torch.Size([129280, 1280]) from checkpoint, the shape in current model is torch.Size([102400, 4096])."
How can I train using Flash Attention 2?