mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #117] Request for Official MMMU-val Evalution Script #96
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#96
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @willburwwb on GitHub (Mar 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/117
Hello, I evaluated the inference accuracy of the DeepSeek-VL2-smal model using the MMMU (https://github.com/MMMU-Benchmark/MMMU) official benchmark script. However, I noticed that the results differ significantly from the accuracy reported in the paper.
Details are as follows:
To better reproduce the results in the paper, could you please provide the official evalution script for the MMMU-val dataset or clarify if there are any special configurations or preprocessing steps required during evalution?