deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #33] Incorrect Output in Visual Grounding Task #27

New issue
Open
opened 2026-10-26 21:10:25 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-26 21:10:25 +00:00
Owner
Copy link

Originally created by @sleepyshep on GitHub (Jan 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/33

Hello! I've encountered an issue with the visual grounding task where the model frequently outputs meaningless bounding boxes. I'm testing the deepseek-vl2-small model. Below, I'll provide the test code and a wrong case.

Test Code

expression = img_info['caption']
prompt = f"<image>\n<|ref|>{expression}<|/ref|>."
conversation = [
    {
        "role": "<|User|>",
        "content": prompt,
        "images": [image_path],
    },
    {"role": "<|Assistant|>", "content": ""},
]

pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)
answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)

Wrong Case

Image:
2350216

Prompt:
<image>\n<|ref|>The hat which is white.<|/ref|>.

Response:
<|ref|>The hat which is white. .<|/ref|><|det|>[[2, 159, 4, 43958, 970]]<|/det|>

Sometimes, the response is even more nonsensical, such as:
<|ref|>Small and silver, this mirror gle.<|/ref|><|det|>[[0, 60, 30, 999999, 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999

There are many similar errors where the model outputs nonsensical bounding boxes. I would appreciate any guidance on how to resolve this issue.

Thank you!

Originally created by @sleepyshep on GitHub (Jan 9, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/33 Hello! I've encountered an issue with the visual grounding task where the model frequently outputs meaningless bounding boxes. I'm testing the `deepseek-vl2-small` model. Below, I'll provide the test code and a wrong case. ## Test Code ```python expression = img_info['caption'] prompt = f"<image>\n<|ref|>{expression}<|/ref|>." conversation = [ { "role": "<|User|>", "content": prompt, "images": [image_path], }, {"role": "<|Assistant|>", "content": ""}, ] pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ).to(vl_gpt.device) inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) outputs = vl_gpt.language.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False) print(f"{prepare_inputs['sft_format'][0]}", answer) ``` ## Wrong Case **Image:** ![2350216](https://github.com/user-attachments/assets/a91222b1-b30e-4219-870c-29fe5bea82c2) **Prompt:** `<image>\n<|ref|>The hat which is white.<|/ref|>.` **Response:** `<|ref|>The hat which is white. .<|/ref|><|det|>[[2, 159, 4, 43958, 970]]<|/det|>` Sometimes, the response is even more nonsensical, such as: `<|ref|>Small and silver, this mirror gle.<|/ref|><|det|>[[0, 60, 30, 999999, 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999` There are many similar errors where the model outputs nonsensical bounding boxes. I would appreciate any guidance on how to resolve this issue. Thank you!
bsakaguchi commented 2026-10-26 21:10:26 +00:00
Author
Owner
Copy link

@deepML2020 commented on GitHub (Jan 9, 2025):

Hello, I've also encountered the same problem when I use the small version.
<|User|>:
<|ref|>The giraffe at the back.<|/ref|>.

DeepSeek-VL2-small reponse:
<|Assistant|>: <|ref|>The giraffe at back.<|/ref|><|det|>[[555560, 269, 9617895, 173171, 199999, 19999, 19, 99, 999, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 。<｜end▁of▁sentence｜>

Expected response：
<|Assistant|>: <|ref|>The giraffe at the back.<|/ref|><|det|>580, 270, 999, 900<|/det|><｜end▁of▁sentence｜>

visual_grounding_1

<!-- gh-comment-id:2579918189 --> @deepML2020 commented on GitHub (Jan 9, 2025): Hello, I've also encountered the same problem when I use the small version. <|User|>: <image> <|ref|>The giraffe at the back.<|/ref|>. DeepSeek-VL2-small reponse: <|Assistant|>: <|ref|>The giraffe at back.<|/ref|><|det|>[[555560, 269, 9617895, 173171, 199999, 19999, 19, 99, 999, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 。<｜end▁of▁sentence｜> Expected response： <|Assistant|>: <|ref|>The giraffe at the back.<|/ref|><|det|>[[580, 270, 999, 900]]<|/det|><｜end▁of▁sentence｜> ![visual_grounding_1](https://github.com/user-attachments/assets/ce72df9d-86de-4ef3-a977-ab7a967313e9)
bsakaguchi commented 2026-10-26 21:10:26 +00:00
Author
Owner
Copy link

@captainfffsama commented on GitHub (Jan 14, 2025):

maybe can try this? https://github.com/deepseek-ai/DeepSeek-VL2/issues/17#issuecomment-2559035376
I test tiny,the wrong version transformers will lead some incorrect output. when I change transformers version to 4.38,deepseekvl2 work good

<!-- gh-comment-id:2589322786 --> @captainfffsama commented on GitHub (Jan 14, 2025): maybe can try this? https://github.com/deepseek-ai/DeepSeek-VL2/issues/17#issuecomment-2559035376 I test tiny,the wrong version transformers will lead some incorrect output. when I change transformers version to 4.38,deepseekvl2 work good
bsakaguchi commented 2026-10-26 21:10:27 +00:00
Author
Owner
Copy link

@sleepyshep commented on GitHub (Jan 14, 2025):

maybe can try this? #17 (comment) I test tiny,the wrong version transformers will lead some incorrect output. when I change transformers version to 4.38,deepseekvl2 work good

Thank you! My testing environment is:

  • torch==2.0.1
  • transformers==4.38.2

The tiny version works fine, but the output of the small version is incorrect.

<!-- gh-comment-id:2589379781 --> @sleepyshep commented on GitHub (Jan 14, 2025): > maybe can try this? [#17 (comment)](https://github.com/deepseek-ai/DeepSeek-VL2/issues/17#issuecomment-2559035376) I test tiny,the wrong version transformers will lead some incorrect output. when I change transformers version to 4.38,deepseekvl2 work good Thank you! My testing environment is: - torch==2.0.1 - transformers==4.38.2 The tiny version works fine, but the output of the small version is incorrect.
bsakaguchi commented 2026-10-26 21:10:27 +00:00
Author
Owner
Copy link

@Mr-Bigworth commented on GitHub (Feb 3, 2025):

Same problem with testing environment: torch==2.6.0
transformers==4.38.2

<!-- gh-comment-id:2630082771 --> @Mr-Bigworth commented on GitHub (Feb 3, 2025): Same problem with testing environment: torch==2.6.0 transformers==4.38.2
bsakaguchi commented 2026-10-26 21:10:28 +00:00
Author
Owner
Copy link

@rubentorresbonet commented on GitHub (Feb 6, 2025):

same issues, it will not even output the coordinates. In my case I had to upgrade to cuda 12.8 and all its friends to support rtx 5000

<!-- gh-comment-id:2640065674 --> @rubentorresbonet commented on GitHub (Feb 6, 2025): same issues, it will not even output the coordinates. In my case I had to upgrade to cuda 12.8 and all its friends to support rtx 5000
bsakaguchi commented 2026-10-26 21:10:28 +00:00
Author
Owner
Copy link

@haoxiangzhao12138 commented on GitHub (Apr 1, 2025):

same issue. tiny model works fine. Small and vl2 cannot get correct coordinates
'<|ref|>A blue checkered fabric hangs behind the image.<|/ref|><|det|>0, 0, 0, 99999, 498512<|/det|><｜end▁of▁sentence｜>'
'<|ref|>Girl wearing purple<|/ref|><|det|>2, 33319, 333333, 66666<|/det|><｜end▁of▁sentence｜>'

<!-- gh-comment-id:2769032514 --> @haoxiangzhao12138 commented on GitHub (Apr 1, 2025): same issue. tiny model works fine. Small and vl2 cannot get correct coordinates '<|ref|>A blue checkered fabric hangs behind the image.<|/ref|><|det|>[[0, 0, 0, 99999, 498512]]<|/det|><｜end▁of▁sentence｜>' '<|ref|>Girl wearing purple<|/ref|><|det|>[[2, 33319, 333333, 66666]]<|/det|><｜end▁of▁sentence｜>'
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#27
No description provided.