mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #123] When evaluating the RefCOCO dataset using the example code, the accuracy is very low. #100
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#100
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @haoxiangzhao12138 on GitHub (Mar 30, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/123
I tried both the tiny and small models, but the accuracy on RefCOCO is extremely low and completely inconsistent with the results reported in the paper. Additionally, the model often fails to output a bounding box or produces bounding boxes in the wrong format. Could you please clarify how the results in the paper were obtained? Are there any special settings that need to be added to the example code during evaluation?
Here is my conversation
conversation = [ { "role": "<|User|>", "content": f"<image>\n<|ref|>{expression}<|/ref|>.", "images": [full_image_path], }, {"role": "<|Assistant|>", "content": ""}, ]and answer is like this
'<|ref|>A girl in stands between two boys, focused on game.<|/ref|><|det|>2, 311, 99999, 98<|/det|><｜end▁of▁sentence｜>'
'<|ref|>The decorated cake sits the table, ready for the celebration.<|/ref|><|det|>470,<|/det|><｜end▁of▁sentence｜>'
'<|ref|>Woman with short bangs holds a plate near the kitchen.<|/ref|><|det|>0, 460, 891, 9999<|/det|><｜end▁of▁sentence｜>'
'<|ref|>Individual at the workstation on the workstation banana in their hand<|/ref|><|det|>0, 5329, 5310, 9, 99999<|/det|><｜end▁of▁sentence｜>'
I noticed that another issue mentioned the need to normalize to 999, but in my case, that doesn’t work either.
@WangYilin210 commented on GitHub (Apr 2, 2025):
try resizing the input image to 336*336, this works to me
@haoxiangzhao12138 commented on GitHub (Apr 2, 2025):
Can you please share your code? I used pil_images = [img.resize((336, 336)) for img in pil_images] to resize, but the result is still the same. The output doesn’t match the bbox format. However, DeepSeek-VL 2 is currently the SOTA on REC, so I don’t even know how to write this part in my paper anymore. It’s driving me crazy—can someone please save me?
@WangYilin210 commented on GitHub (Apr 2, 2025):
确认一下，你用的推理code是不是和inference.py的代码一致。readme中的代码有点问题
@haoxiangzhao12138 commented on GitHub (Apr 2, 2025):
太感谢了，我知道为什么了，在inference.py中设置了几个参数，temperature=0.4, top_p=0.9, repetition_penalty=1.1,把这些加上就好了！
@Ko0smos commented on GitHub (Jan 16, 2026):
Added reference to Qwen2.5-VL in README.md