deepseek-ai/Janus
1
0
Fork 0
mirror of https://github.com/deepseek-ai/Janus.git synced 2026-11-05 00:57:18 +00:00
Code Issues 183 Projects Packages Wiki Activity

[GH-ISSUE #205] Cannot Reproduce MMBench-EN Results for Janus-Pro-7B #164

New issue
Open
opened 2026-10-26 21:13:17 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:13:17 +00:00
Owner
Copy link

Originally created by @SHENZHEBEI on GitHub (Mar 22, 2025).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/205

I am unable to reproduce the MMBench-EN results reported for Janus-Pro-7B. Specifically, when evaluating the model using lmms-eval, I obtained the following scores:

  • mmbench_en_dev: 65.81
  • mmbench_en_test: 65.98

These are significantly lower than the score of 79.2 reported in the paper:

"Janus-Pro-7B achieved a score of 79.2 on the multimodal understanding benchmark MMBench."

Additionally, the score for Janus-Pro-7B on the MMBench leaderboard and opencompass is also relatively low, consistent with my observations.

Could you please clarify:

  • What evaluation settings or prompts were used to achieve the reported score of 79.2?
  • Is there any preprocessing or special configuration required?
  • Are the leaderboard and paper scores based on different evaluation protocols or updated model versions?

Thank you in advance for your help.

Originally created by @SHENZHEBEI on GitHub (Mar 22, 2025). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/205 I am unable to reproduce the MMBench-EN results reported for Janus-Pro-7B. Specifically, when evaluating the model using `lmms-eval`, I obtained the following scores: - `mmbench_en_dev`: **65.81** - `mmbench_en_test`: **65.98** These are significantly lower than the score of **79.2** reported in the paper: > "Janus-Pro-7B achieved a score of 79.2 on the multimodal understanding benchmark MMBench." Additionally, the score for Janus-Pro-7B on the [MMBench leaderboard](https://mmbench.opencompass.org.cn/leaderboard) and [opencompass](https://rank.opencompass.org.cn/leaderboard-multimodal) is also relatively low, consistent with my observations. Could you please clarify: - What evaluation settings or prompts were used to achieve the reported score of 79.2? - Is there any preprocessing or special configuration required? - Are the leaderboard and paper scores based on different evaluation protocols or updated model versions? Thank you in advance for your help.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/Janus#164
No description provided.