deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #13] Need help with evaluation #10

New issue
Closed
opened 2026-10-26 21:10:07 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:10:07 +00:00
Owner
Copy link

Originally created by @yyl494 on GitHub (Dec 19, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/13

Hello. When testing deepseek-vl2 and deepseek-vl2-small, the scores I got were much lower than those in the paper. I don't know if there are any configurations that need attention. The test code is rewritten based on deepseek_vl in VLMEvalKit, and the obtained benchmark scores are as follows. The score of deepseek-vl2-tiny is consistent with the paper results

model MMBench_TEST_EN_V11 MMStar OCRBench AI2D_TEST
deepseek-vl2 67.7 54.2 68.8 68.9
deepseek-vl2-small 62.3 49.0 63.2 63.7
deepseek-vl2-tiny 71.3 49.0 81.8 74.6
Originally created by @yyl494 on GitHub (Dec 19, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/13 Hello. When testing deepseek-vl2 and deepseek-vl2-small, the scores I got were much lower than those in the paper. I don't know if there are any configurations that need attention. The test code is rewritten based on deepseek_vl in VLMEvalKit, and the obtained benchmark scores are as follows. The score of deepseek-vl2-tiny is consistent with the paper results | model | MMBench_TEST_EN_V11 | MMStar | OCRBench | AI2D_TEST | | ----------- | ----------- |----------- |----------- |----------- | | deepseek-vl2 | 67.7 |54.2| 68.8 | 68.9| | deepseek-vl2-small | 62.3 |49.0|63.2|63.7| | deepseek-vl2-tiny | 71.3 | 49.0 | 81.8 | 74.6 |
bsakaguchi commented 2026-10-26 21:10:09 +00:00
Author
Owner
Copy link

@gnobitab commented on GitHub (Dec 19, 2024):

We are working on VLMEvalKit now. We will update after we align the results and merge our version with the VLMEvalKit repo.

<!-- gh-comment-id:2552724825 --> @gnobitab commented on GitHub (Dec 19, 2024): We are working on VLMEvalKit now. We will update after we align the results and merge our version with the VLMEvalKit repo.
bsakaguchi commented 2026-10-26 21:10:09 +00:00
Author
Owner
Copy link

@gnobitab commented on GitHub (Dec 27, 2024):

Please try evaluation again after our official PR gets merged by VLMEvalKit.

https://github.com/open-compass/VLMEvalKit/pull/697

<!-- gh-comment-id:2563251049 --> @gnobitab commented on GitHub (Dec 27, 2024): Please try evaluation again after our official PR gets merged by VLMEvalKit. https://github.com/open-compass/VLMEvalKit/pull/697
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#10
No description provided.