[GH-ISSUE #58] Unable to get the expected answer #43

New issue

Open

opened 2026-10-26 21:10:46 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:10:46 +00:00

Owner

Originally created by @GryffindorZz on GitHub (Feb 6, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/58

Use the official code:

import torch
from transformers import AutoModelForCausalLM

from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.utils.io import load_pil_images


# specify the path to the model
model_path = "deepseek-ai/deepseek-vl2-tiny"
vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

## single image conversation example
## Please note that <|ref|> and <|/ref|> are designed specifically for the object localization feature. These special tokens are not required for normal conversations.
## If you would like to experience the grounded captioning functionality (responses that include both object localization and reasoning), you need to add the special token <|grounding|> at the beginning of the prompt. Examples could be found in Figure 9 of our paper.
conversation = [
    {
        "role": "<|User|>",
        "content": "<image>\n<|ref|>describe the image<|/ref|>.",
        "images": ["./images/visual_grounding_1.jpeg"],
    },
    {"role": "<|Assistant|>", "content": ""},
]

# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)

When i use "describe the image" as the input, I can get the right answer:

But when I change the text to "Is there a giraffe in the picture?" , the answer is incorrect.

I tried many kinds of input, and only the one like "describe the picture" had the correct output, and the other input answers were wrong. And I changed other models(deepseek vl2, not tiny version) and got similar results.

My env: torch==2.0.1 transformers==4.38.2

Originally created by @GryffindorZz on GitHub (Feb 6, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/58 Use the official code: ``` import torch from transformers import AutoModelForCausalLM from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM from deepseek_vl2.utils.io import load_pil_images # specify the path to the model model_path = "deepseek-ai/deepseek-vl2-tiny" vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True) vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval() ## single image conversation example ## Please note that <|ref|> and <|/ref|> are designed specifically for the object localization feature. These special tokens are not required for normal conversations. ## If you would like to experience the grounded captioning functionality (responses that include both object localization and reasoning), you need to add the special token <|grounding|> at the beginning of the prompt. Examples could be found in Figure 9 of our paper. conversation = [ { "role": "<|User|>", "content": "<image>\n<|ref|>describe the image<|/ref|>.", "images": ["./images/visual_grounding_1.jpeg"], }, {"role": "<|Assistant|>", "content": ""}, ] # load images and prepare for inputs pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ).to(vl_gpt.device) # run image encoder to get the image embeddings inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) # run the model to get the response outputs = vl_gpt.language.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False) print(f"{prepare_inputs['sft_format'][0]}", answer) ``` When i use "describe the image" as the input, I can get the right answer: ![Image](https://github.com/user-attachments/assets/81ac313a-c527-4309-97f4-1bb100989d8b) But when I change the text to "Is there a giraffe in the picture?" , the answer is incorrect. ![Image](https://github.com/user-attachments/assets/19411dd9-7168-4dfa-95da-985ca6f32df9) I tried many kinds of input, and only the one like "describe the picture" had the correct output, and the other input answers were wrong. And I changed other models(deepseek vl2, not tiny version) and got similar results. My env: torch==2.0.1 transformers==4.38.2

bsakaguchi commented

2026-10-26 21:10:47 +00:00

Author

Owner

@HubHop commented on GitHub (Feb 6, 2025):

Hi @GryffindorZz , note that "<|ref|> xxx <|/ref|>" and "<|grounding|>" are special tokens for visual grounding, where "xxx" should be a specific description of an object, e.g. "apple", "dog", "cat", etc.

Otherwise, please simply ask the model via natural language. For example,

conversation = [
    {
        "role": "<|User|>",
        "content": "<image>\nDescribe the image in detail.",   # remove special tokens <|ref|> <|/ref|>
        "images": ["./images/visual_grounding_1.jpeg"],
    },
    {"role": "<|Assistant|>", "content": ""},
]

@HubHop commented on GitHub (Feb 6, 2025): Hi @GryffindorZz , note that "<|ref|> xxx <|/ref|>" and "<|grounding|>" are special tokens for visual grounding, where "xxx" should be a specific description of an object, e.g. "apple", "dog", "cat", etc. Otherwise, please simply ask the model via natural language. For example, ```python conversation = [ { "role": "<|User|>", "content": "<image>\nDescribe the image in detail.", # remove special tokens <|ref|> <|/ref|> "images": ["./images/visual_grounding_1.jpeg"], }, {"role": "<|Assistant|>", "content": ""}, ] ```

bsakaguchi commented

2026-10-26 21:10:47 +00:00

Author

Owner

@rubentorresbonet commented on GitHub (Feb 6, 2025):

Yeah I am also struggling to get results.
```Enter your prompt (or 'quit' to exit): <|grounding|>Find the Edit menu item`

<|User|>:
<|grounding|>Find the Edit menu item.

<|Assistant|>: The edit menu is located at the top left corner of the screen.<｜end▁of▁sentence｜>```

bsakaguchi commented

2026-10-26 21:10:48 +00:00

Author

Owner

@GryffindorZz commented on GitHub (Feb 7, 2025):

Hi @GryffindorZz , note that "<|ref|> xxx <|/ref|>" and "<|grounding|>" are special tokens for visual grounding, where "xxx" should be a specific description of an object, e.g. "apple", "dog", "cat", etc.

Otherwise, please simply ask the model via natural language. For example,

conversation = [
{
"role": "<|User|>",
"content": "\nDescribe the image in detail.", # remove special tokens <|ref|> <|/ref|>
"images": ["./images/visual_grounding_1.jpeg"],
},
{"role": "<|Assistant|>", "content": ""},
]

@HubHop Thanks a lot!, it works now.

@GryffindorZz commented on GitHub (Feb 7, 2025): > Hi [@GryffindorZz](https://github.com/GryffindorZz) , note that "<|ref|> xxx <|/ref|>" and "<|grounding|>" are special tokens for visual grounding, where "xxx" should be a specific description of an object, e.g. "apple", "dog", "cat", etc. > > Otherwise, please simply ask the model via natural language. For example, > > conversation = [ > { > "role": "<|User|>", > "content": "<image>\nDescribe the image in detail.", # remove special tokens <|ref|> <|/ref|> > "images": ["./images/visual_grounding_1.jpeg"], > }, > {"role": "<|Assistant|>", "content": ""}, > ] @HubHop Thanks a lot!, it works now.

bsakaguchi referenced this issue

2026-10-26 21:12:32 +00:00

[PR #43] Fixed a typo in Update inference.py #134