deepseek-ai/Janus
1
0
Fork 0
mirror of https://github.com/deepseek-ai/Janus.git synced 2026-11-05 00:57:18 +00:00
Code Issues 183 Projects Packages Wiki Activity

[GH-ISSUE #212] evaluation #174

New issue
Open
opened 2026-10-26 21:13:32 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-26 21:13:32 +00:00
Owner
Copy link

Originally created by @125jx on GitHub (Apr 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/212

是否可以分享一下understanding 和 generation的评估代码

Originally created by @125jx on GitHub (Apr 15, 2025). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/212 是否可以分享一下understanding 和 generation的评估代码
bsakaguchi commented 2026-10-26 21:13:34 +00:00
Author
Owner
Copy link

@david-learning commented on GitHub (Apr 15, 2025):

Code for understanding can be found here, but there is no that for generation.
I tried the setting in generation_inference.py to test on GenEval, but can not reproduce the results.

<!-- gh-comment-id:2804104936 --> @david-learning commented on GitHub (Apr 15, 2025): Code for understanding can be found [here](https://github.com/deepseek-ai/Janus/issues/15), but there is no that for generation. I tried the setting in `generation_inference.py` to test on `GenEval`, but can not reproduce the results.
bsakaguchi commented 2026-10-26 21:13:35 +00:00
Author
Owner
Copy link

@125jx commented on GitHub (Apr 15, 2025):

理解代码可以在这里找到，但没有生成代码。 我尝试了 中的设置generation_inference.py进行测试GenEval，但无法重现结果。

他这里是强制使用VLMEvalKit库吗？

<!-- gh-comment-id:2804128396 --> @125jx commented on GitHub (Apr 15, 2025): > 理解代码可以[在这里](https://github.com/deepseek-ai/Janus/issues/15)找到，但没有生成代码。 我尝试了 中的设置`generation_inference.py`进行测试`GenEval`，但无法重现结果。 他这里是强制使用VLMEvalKit库吗？
bsakaguchi commented 2026-10-26 21:13:35 +00:00
Author
Owner
Copy link

@david-learning commented on GitHub (Apr 15, 2025):

Maybe. The authors proposed that PR, integrating their evaluation code for understanding into VLMEvalKit.

<!-- gh-comment-id:2804141989 --> @david-learning commented on GitHub (Apr 15, 2025): Maybe. The authors proposed that PR, integrating their evaluation code for understanding into `VLMEvalKit`.
bsakaguchi commented 2026-10-26 21:13:35 +00:00
Author
Owner
Copy link

@david-learning commented on GitHub (Apr 15, 2025):

I am also waiting for the scripts or settings for evaluation for generation. #210

<!-- gh-comment-id:2804145161 --> @david-learning commented on GitHub (Apr 15, 2025): I am also waiting for the scripts or settings for evaluation for `generation`. #210
bsakaguchi commented 2026-10-26 21:13:36 +00:00
Author
Owner
Copy link

@125jx commented on GitHub (Apr 15, 2025):

好，谢谢您的帮助，你也可以看一下pku的align-anything的eval分支中貌似也加入了evaluation的代码,也许会有帮助

<!-- gh-comment-id:2804150115 --> @125jx commented on GitHub (Apr 15, 2025): 好，谢谢您的帮助，你也可以看一下pku的align-anything的eval分支中貌似也加入了evaluation的代码,也许会有帮助
bsakaguchi commented 2026-10-26 21:13:36 +00:00
Author
Owner
Copy link

@125jx commented on GitHub (Apr 17, 2025):

请问GenEval是在他给出的prompt基础上，基于JanusPro生成fake图片，然后基于他本身的框架生成groundtruth图片,然后计算fid吗？

<!-- gh-comment-id:2811519678 --> @125jx commented on GitHub (Apr 17, 2025): 请问GenEval是在他给出的prompt基础上，基于JanusPro生成fake图片，然后基于他本身的框架生成groundtruth图片,然后计算fid吗？
bsakaguchi commented 2026-10-26 21:13:36 +00:00
Author
Owner
Copy link

@david-learning commented on GitHub (Apr 17, 2025):

You can follow the steps in the readme in geneval.
First, generate images using Janus-Pro based on the prompts in Geneval. Then evaluate them. From my understanding, no groundtruth or fid computation.

<!-- gh-comment-id:2811633184 --> @david-learning commented on GitHub (Apr 17, 2025): You can follow the steps in the readme in [geneval](https://github.com/djghosh13/geneval). First, generate images using Janus-Pro based on the prompts in Geneval. Then evaluate them. From my understanding, no groundtruth or fid computation.
bsakaguchi commented 2026-10-26 21:13:36 +00:00
Author
Owner
Copy link

@Eli-YiLi commented on GitHub (May 18, 2025):

Here is a new benchmark designed for unified models, supporting overall evaluation for both understanding and generation.
It's easy to follow and offers a much more diverse attributes compared to GenEval.
Results of Janus series are also reported.

https://xmed-lab.github.io/UniEval/
https://github.com/xmed-lab/UniEval

<!-- gh-comment-id:2888866756 --> @Eli-YiLi commented on GitHub (May 18, 2025): Here is a new benchmark designed for unified models, supporting overall evaluation for both understanding and generation. It's easy to follow and offers a much more diverse attributes compared to GenEval. Results of Janus series are also reported. https://xmed-lab.github.io/UniEval/ https://github.com/xmed-lab/UniEval
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/Janus#174
No description provided.