[GH-ISSUE #99] Proposal to Evaluate on the HumanEval-V Benchmark for Enhanced Visual Reasoning and Code Generation #80

New issue

Open

opened 2026-10-26 21:11:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:11:33 +00:00

Owner

Originally created by @zfj1998 on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/99

Congratulations on the impressive work!

I would like to suggest expanding the evaluation of visual reasoning to the HumanEval-V benchmark. This benchmark provides a more challenging set of tasks by introducing complex diagrams paired with coding challenges. Unlike traditional visual reasoning tasks that focus on answering multiple-choice questions or providing short answers, HumanEval-V requires models to generate code based on visual input, which better tests both instruction-following and open-ended generation abilities.

Key points for consideration:

HumanEval-V expands the reasoning scenarios with complex diagrams, pushing the limits of visual understanding.
The task format is tailored to code generation, making it a suitable benchmark for testing MLLMs’ ability to handle more structured, generative tasks.
Evaluating this benchmark will provide valuable insights into how well it handles visual reasoning combined with coding, which can be evaluated and rewarded through execution feedback.

You can find more information about the benchmark here: HumanEval-V Homepage.

Originally created by @zfj1998 on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/99 Congratulations on the impressive work! I would like to suggest expanding the evaluation of visual reasoning to the **HumanEval-V** benchmark. This benchmark provides a more challenging set of tasks by introducing **complex diagrams** paired with **coding challenges**. Unlike traditional visual reasoning tasks that focus on answering multiple-choice questions or providing short answers, HumanEval-V requires models to **generate code** based on visual input, which better tests both **instruction-following** and **open-ended generation abilities**. Key points for consideration: - **HumanEval-V** expands the reasoning scenarios with **complex diagrams**, pushing the limits of visual understanding. - The task format is tailored to **code generation**, making it a suitable benchmark for testing MLLMs’ ability to handle more structured, generative tasks. - Evaluating this benchmark will provide valuable insights into how well it handles **visual reasoning combined with coding**, which can be evaluated and rewarded through execution feedback. You can find more information about the benchmark here: [HumanEval-V Homepage](https://humaneval-v.github.io/).

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-VL2#80

No description provided.

Rows
Columns