deepseek-ai/Janus
1
0
Fork 0
mirror of https://github.com/deepseek-ai/Janus.git synced 2026-11-05 00:57:18 +00:00
Code Issues 183 Projects Packages Wiki Activity

[GH-ISSUE #239] Cannot reproduce Janus-Pro-1B understanding benchmark results with VLMEvalKit #192

New issue
Closed
opened 2026-10-26 21:13:50 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:13:50 +00:00
Owner
Copy link

Originally created by @CnFaker on GitHub (Jun 9, 2026).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/239

Hi, thanks for releasing Janus-Pro.

I am trying to reproduce the understanding benchmark results of Janus-Pro-1B using VLMEvalKit, but my reproduced numbers are lower than the numbers reported in the technical report on several benchmarks. Could you please clarify the exact evaluation settings used for the report, or confirm whether the reported numbers were produced with a different evaluation pipeline/configuration?

Here are my current results:

Model / setting POPE MMBench SEED MMMU MM-Vet
Janus-Pro-1B, paper 86.2 75.5 68.3 36.3 39.8, GPT-4o
Janus-Pro-1B, reproduced, match 84.11 74.2 67.28 34.89 36.7, GPT-4.1
Janus-Pro-1B, reproduced, deepseek-v4-flash 84.11 74.3 67.28 35.22 35.32 / 37.52, GPT-4o

The gaps are not very large for some metrics, but they are consistent, especially on POPE, SEED, MMMU, and MM-Vet.

Could you please share or confirm the following details?

Thanks in advance!

Originally created by @CnFaker on GitHub (Jun 9, 2026). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/239 Hi, thanks for releasing Janus-Pro. I am trying to reproduce the understanding benchmark results of **Janus-Pro-1B** using **VLMEvalKit**, but my reproduced numbers are lower than the numbers reported in the technical report on several benchmarks. Could you please clarify the exact evaluation settings used for the report, or confirm whether the reported numbers were produced with a different evaluation pipeline/configuration? Here are my current results: | Model / setting | POPE | MMBench | SEED | MMMU | MM-Vet | | --- | ---: | ---: | ---: | ---: | ---: | | Janus-Pro-1B, paper | 86.2 | 75.5 | 68.3 | 36.3 | 39.8, GPT-4o | | Janus-Pro-1B, reproduced, match | 84.11 | 74.2 | 67.28 | 34.89 | 36.7, GPT-4.1 | | Janus-Pro-1B, reproduced, deepseek-v4-flash | 84.11 | 74.3 | 67.28 | 35.22 | 35.32 / 37.52, GPT-4o | The gaps are not very large for some metrics, but they are consistent, especially on POPE, SEED, MMMU, and MM-Vet. Could you please share or confirm the following details? Thanks in advance!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/Janus#192
No description provided.