deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #83] VL2 Small produces wrong response for an input image #67

New issue
Open
opened 2026-10-26 21:11:19 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:11:19 +00:00
Owner
Copy link

Originally created by @ukaprch on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/83

model_path = "deepseek-ai/deepseek-vl2-small"
vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer
#model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
quantization_config = QuantoConfig(weights="int8")
vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", device_map="cuda:0", trust_remote_code=True, quantization_config=quantization_config)

I update the image and question in the Conversation:
(image below)
Image

I then ask the question: Describe the image in detail.
The response I get back is as follows after 2 successive tries:

<|Assistant|>: The image shows a person standing on a beach, with their back to the camera. The person is wearing a white shirt and blue shorts, and they have a backpack on their shoulders. The beach is sandy and there are no other people visible in the image. In the background, you can see the ocean with waves crashing onto the shore. The sky appears to be clear and sunny.
<|User|>: <image_placeholder>
Describe the image in detail.

<|Assistant|>: The image shows a person standing on a beach, with their back to the camera. The person is wearing a long-sleeved shirt and pants, and has a backpack on their shoulders. They are looking out at the ocean, which is calm and blue. In the background, there are some rocks and cliffs, as well as a few seagulls flying overhead. The sky is clear and sunny, and the overall atmosphere of the image is peaceful and serene.

Obviously these responses are wrong. This model doesn't seem to work properly.

Originally created by @ukaprch on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/83 model_path = "deepseek-ai/deepseek-vl2-small" vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer #model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto") quantization_config = QuantoConfig(weights="int8") vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", device_map="cuda:0", trust_remote_code=True, quantization_config=quantization_config) I update the image and question in the Conversation: (image below) ![Image](https://github.com/user-attachments/assets/f9249d97-fce2-4b05-84a0-4bd64ba8859c) I then ask the question: Describe the image in detail. The response I get back is as follows after 2 successive tries: <|Assistant|>: The image shows a person standing on a beach, with their back to the camera. The person is wearing a white shirt and blue shorts, and they have a backpack on their shoulders. The beach is sandy and there are no other people visible in the image. In the background, you can see the ocean with waves crashing onto the shore. The sky appears to be clear and sunny. <|User|>: <image_placeholder> Describe the image in detail. <|Assistant|>: The image shows a person standing on a beach, with their back to the camera. The person is wearing a long-sleeved shirt and pants, and has a backpack on their shoulders. They are looking out at the ocean, which is calm and blue. In the background, there are some rocks and cliffs, as well as a few seagulls flying overhead. The sky is clear and sunny, and the overall atmosphere of the image is peaceful and serene. Obviously these responses are wrong. This model doesn't seem to work properly.
bsakaguchi commented 2026-10-26 21:11:20 +00:00
Author
Owner
Copy link

@adamsiwiec commented on GitHub (Feb 15, 2025):

Also seeing the same issues!

import torch
import PIL.Image
from deepseek_vl2.models import DeepseekVLV2Processor
from transformers import AutoModelForCausalLM

model_path = "deepseek-ai/deepseek-vl2-small"
vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer
vl_gpt = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda")
vl_gpt = vl_gpt.eval()

conversation = [dict(role="<|User|>", content="Describe this image: <image>"), dict(role="<|Assistant|>", content="")]

prepared_inputs = vl_chat_processor(
    conversations=conversation,
    images=[PIL.Image.new(mode="RGB", size=(300, 300))],  # pass in a black image.
    system_prompt="",
    force_batchify=True,
).to(vl_gpt.device)

outputs = vl_gpt.generate(
        input_ids=prepared_inputs.input_ids,
        attention_mask=prepared_inputs.attention_mask,
        images=prepared_inputs.images.to(dtype=torch.bfloat16),
        images_seq_mask=prepared_inputs.images_seq_mask,
        images_spatial_crop=prepared_inputs.images_spatial_crop,
        pad_token_id=tokenizer.eos_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        max_new_tokens=512,
        do_sample=False,
        use_cache=True
    )

print(tokenizer.decode(outputs[0][prepared_inputs.input_ids.shape[-1]:], skip_special_tokens=True))

With output:

The image shows a person standing in front of a large, colorful mural. The mural features a vibrant and detailed depiction of a cityscape with various buildings, streets, and possibly some landmarks. The person is wearing a casual outfit, including a t-shirt and jeans, and appears to be posing for the photo with a relaxed posture. The overall atmosphere of the image is lively and artistic, suggesting that the location might be a popular spot for photography or a place of cultural significance.

🤔

<!-- gh-comment-id:2660596477 --> @adamsiwiec commented on GitHub (Feb 15, 2025): Also seeing the same issues! ``` import torch import PIL.Image from deepseek_vl2.models import DeepseekVLV2Processor from transformers import AutoModelForCausalLM model_path = "deepseek-ai/deepseek-vl2-small" vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer vl_gpt = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda") vl_gpt = vl_gpt.eval() conversation = [dict(role="<|User|>", content="Describe this image: <image>"), dict(role="<|Assistant|>", content="")] prepared_inputs = vl_chat_processor( conversations=conversation, images=[PIL.Image.new(mode="RGB", size=(300, 300))], # pass in a black image. system_prompt="", force_batchify=True, ).to(vl_gpt.device) outputs = vl_gpt.generate( input_ids=prepared_inputs.input_ids, attention_mask=prepared_inputs.attention_mask, images=prepared_inputs.images.to(dtype=torch.bfloat16), images_seq_mask=prepared_inputs.images_seq_mask, images_spatial_crop=prepared_inputs.images_spatial_crop, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True ) print(tokenizer.decode(outputs[0][prepared_inputs.input_ids.shape[-1]:], skip_special_tokens=True)) ``` With output: ``` The image shows a person standing in front of a large, colorful mural. The mural features a vibrant and detailed depiction of a cityscape with various buildings, streets, and possibly some landmarks. The person is wearing a casual outfit, including a t-shirt and jeans, and appears to be posing for the photo with a relaxed posture. The overall atmosphere of the image is lively and artistic, suggesting that the location might be a popular spot for photography or a place of cultural significance. ``` 🤔
bsakaguchi commented 2026-10-26 21:11:20 +00:00
Author
Owner
Copy link

@ukaprch commented on GitHub (Feb 15, 2025):

@adamsiwiec
Strangely enough, I found that the "Tiny" model has no such problems and was far easier to manage.

<!-- gh-comment-id:2660930445 --> @ukaprch commented on GitHub (Feb 15, 2025): @adamsiwiec Strangely enough, I found that the "Tiny" model has no such problems and was far easier to manage.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#67
No description provided.