[GH-ISSUE #17] The inference output is meaningless #14

New issue

Open

opened 2026-10-26 21:10:10 +00:00 by bsakaguchi · 15 comments

bsakaguchi commented 2026-10-26 21:10:10 +00:00

Owner

Copy link

Originally created by @yydxlv on GitHub (Dec 23, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/17

When i input an image , the inference output is repetitive and not well-organized.

`conversation = [
{
"role": "<|User|>",
"content": "\n Describe the image.",
"images": ["../data/cat.png"],
},
{"role": "<|Assistant|>", "content": ""},
]

pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(conversations=conversation, images=pil_images, force_batchify=True, system_prompt="Describe the image.").to(vl_gpt.device)
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

outputs = vl_gpt.language.generate(
inputs_embeds=inputs_embeds,
attention_mask=prepare_inputs.attention_mask,
pad_token_id=tokenizer.eos_token_id,
bos_token_id=tokenizer.bos_token_id,
eos_token_id=tokenizer.eos_token_id,
max_new_tokens=128,
do_sample=False,
use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)`

The output is follow:
Describe the image.

<|User|>:
Describe the image.

<|Assistant|>: The image shows a cat wearing a shirt with a shirt with a design that resembles a shirt with a design that resembles a shirt with a design that resembles a shirt with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with

Originally created by @yydxlv on GitHub (Dec 23, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/17 When i input an image , the inference output is repetitive and not well-organized. `conversation = [ { "role": "<|User|>", "content": "<image>\n Describe the image.", "images": ["../data/cat.png"], }, {"role": "<|Assistant|>", "content": ""}, ] pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor(conversations=conversation, images=pil_images, force_batchify=True, system_prompt="Describe the image.").to(vl_gpt.device) inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) outputs = vl_gpt.language.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=128, do_sample=False, use_cache=True ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True) print(f"{prepare_inputs['sft_format'][0]}", answer)` The output is follow: Describe the image. <|User|>: <image> Describe the image. <|Assistant|>: The image shows a cat wearing a shirt with a shirt with a design that resembles a shirt with a design that resembles a shirt with a design that resembles a shirt with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with with

bsakaguchi commented 2026-10-26 21:10:11 +00:00

Author

Owner

Copy link

@HubHop commented on GitHub (Dec 23, 2024):

Hi @yydxlv, can you let me know which model you are testing? If possible, can you also share this failure case (image) with us?

<!-- gh-comment-id:2558921995 --> @HubHop commented on GitHub (Dec 23, 2024): Hi @yydxlv, can you let me know which model you are testing? If possible, can you also share this failure case (image) with us?

bsakaguchi commented 2026-10-26 21:10:11 +00:00

Author

Owner

Copy link

@yydxlv commented on GitHub (Dec 23, 2024):

Hi @yydxlv, can you let me know which model you are testing? If possible, can you also share this failure case (image) with us?

The model name is deepseek-vl2-small

I use the following code to load the model

`import torch
from transformers import AutoModelForCausalLM
from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.utils.io import load_pil_images
from transformers import AutoProcessor

model_name = "models/deepseek-vl2-small"

vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_name)
tokenizer = vl_chat_processor.tokenizer
vl_gpt = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(torch.bfloat16).cuda().eval()`

The input img is

Thank you for help

<!-- gh-comment-id:2558966225 --> @yydxlv commented on GitHub (Dec 23, 2024): > Hi @yydxlv, can you let me know which model you are testing? If possible, can you also share this failure case (image) with us? The model name is deepseek-vl2-small I use the following code to load the model `import torch from transformers import AutoModelForCausalLM from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM from deepseek_vl2.utils.io import load_pil_images from transformers import AutoProcessor model_name = "models/deepseek-vl2-small" vl_chat_processor = DeepseekVLV2Processor.from_pretrained(model_name) tokenizer = vl_chat_processor.tokenizer vl_gpt = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(torch.bfloat16).cuda().eval()` The input img is  Thank you for help

bsakaguchi commented 2026-10-26 21:10:11 +00:00

Author

Owner

Copy link

@HubHop commented on GitHub (Dec 23, 2024):

Hi @yydxlv, I tested it with DeepSeek-VL2-Small and your image. The results seem to be good:

<|User|>: <image>
 Describe the image.

<|Assistant|>: The image shows a cat wearing a shirt with various technical diagrams and text related to machine learning and neural networks. The shirt has words like "TRANSFORMER," "ATTENTION MECHANISM," "ENCODER," and "DECODER." The cat is sitting on a rug in a room with a wooden chair and a bookshelf in the background.<｜end▁of▁sentence｜>

Please try our demo code in the readme and directly replace the prompts/images, to see if the results are correct or not.

conversation = [
{
"role": "<|User|>",
"content": "<image>\n Describe the image.",
"images": ["./images/cat.png"],
},
{"role": "<|Assistant|>", "content": ""},
]


# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=128,
    do_sample=False,
    use_cache=True
)

<!-- gh-comment-id:2558990490 --> @HubHop commented on GitHub (Dec 23, 2024): Hi @yydxlv, I tested it with DeepSeek-VL2-Small and your image. The results seem to be good: ```bash <|User|>: <image> Describe the image. <|Assistant|>: The image shows a cat wearing a shirt with various technical diagrams and text related to machine learning and neural networks. The shirt has words like "TRANSFORMER," "ATTENTION MECHANISM," "ENCODER," and "DECODER." The cat is sitting on a rug in a room with a wooden chair and a bookshelf in the background.<｜end▁of▁sentence｜> ``` Please try our demo code in the readme and directly replace the prompts/images, to see if the results are correct or not. ```python conversation = [ { "role": "<|User|>", "content": "<image>\n Describe the image.", "images": ["./images/cat.png"], }, {"role": "<|Assistant|>", "content": ""}, ] # load images and prepare for inputs pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ).to(vl_gpt.device) # run image encoder to get the image embeddings inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) # run the model to get the response outputs = vl_gpt.language.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=128, do_sample=False, use_cache=True ) ```

bsakaguchi commented 2026-10-26 21:10:12 +00:00

Author

Owner

Copy link

@yydxlv commented on GitHub (Dec 23, 2024):

Yeah, I use the same code as the demo, but the result is strange.

Is there anything wrong with me ?

<!-- gh-comment-id:2559020207 --> @yydxlv commented on GitHub (Dec 23, 2024): Yeah, I use the same code as the demo, but the result is strange.   Is there anything wrong with me ?

bsakaguchi commented 2026-10-26 21:10:12 +00:00

Author

Owner

Copy link

@HubHop commented on GitHub (Dec 23, 2024):

The code looks good to me. I'm not entirely sure but you may check if it is due to different versions of python packages. Below is our testing environment.

torch==2.0.1
transformers==4.38.2

<!-- gh-comment-id:2559035376 --> @HubHop commented on GitHub (Dec 23, 2024): The code looks good to me. I'm not entirely sure but you may check if it is due to different versions of python packages. Below is our testing environment. ```text torch==2.0.1 transformers==4.38.2 ```

bsakaguchi commented 2026-10-26 21:10:13 +00:00

Author

Owner

Copy link

@yydxlv commented on GitHub (Dec 23, 2024):

The code looks good to me. I'm not entirely sure but you may check if it is due to different versions of python packages. Below is our testing environment.

torch==2.0.1
transformers==4.38.2

Yeah, i have just reinstalled and verified torch==2.0.1 and transformers==4.38.2.,
the model is runed on A800, but the output is still the same like this

"
<|User|>:
Describe the image.

<|Assistant|>: The image shows a cat wearing a shirt with a shirt with a design that resembles a transformer. The shirt has a transformer design on it. The shirt has a transformer. The shirt has a transformer. The shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt
"

<!-- gh-comment-id:2559144072 --> @yydxlv commented on GitHub (Dec 23, 2024): > The code looks good to me. I'm not entirely sure but you may check if it is due to different versions of python packages. Below is our testing environment. > > ``` > torch==2.0.1 > transformers==4.38.2 > ``` Yeah, i have just reinstalled and verified torch==2.0.1 and transformers==4.38.2., the model is runed on A800, but the output is still the same like this " <|User|>: <image> Describe the image. <|Assistant|>: The image shows a cat wearing a shirt with a shirt with a design that resembles a transformer. The shirt has a transformer design on it. The shirt has a transformer. The shirt has a transformer. The shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt shirt "

bsakaguchi commented 2026-10-26 21:10:14 +00:00

Author

Owner

Copy link

@Blaizzy commented on GitHub (Dec 23, 2024):

You need to pass input_ids.

There is a PR addressing this issue #11

<!-- gh-comment-id:2559377902 --> @Blaizzy commented on GitHub (Dec 23, 2024): You need to pass input_ids. There is a PR addressing this issue #11

bsakaguchi commented 2026-10-26 21:10:14 +00:00

Author

Owner

Copy link

@MrRace commented on GitHub (Dec 24, 2024):

It was truly disappointing. Nothing is perfect everywhere. Maybe it's time to let go!

<!-- gh-comment-id:2560742559 --> @MrRace commented on GitHub (Dec 24, 2024): It was truly disappointing. Nothing is perfect everywhere. Maybe it's time to let go!

bsakaguchi commented 2026-10-26 21:10:15 +00:00

Author

Owner

Copy link

@tracyCzf commented on GitHub (Dec 27, 2024):

@yydxlv @HubHop Using the same code, deepseek-vl2-small generated the same bad result with yydxlv. But deepssek-vl2-tiny and deepssek-vl2 generated results looks good.

<!-- gh-comment-id:2563359900 --> @tracyCzf commented on GitHub (Dec 27, 2024): @yydxlv @HubHop Using the same code, deepseek-vl2-small generated the same bad result with yydxlv. But deepssek-vl2-tiny and deepssek-vl2 generated results looks good.

bsakaguchi commented 2026-10-26 21:10:15 +00:00

Author

Owner

Copy link

@p1x33l commented on GitHub (Jan 8, 2025):

+1

<!-- gh-comment-id:2577615379 --> @p1x33l commented on GitHub (Jan 8, 2025): +1

bsakaguchi commented 2026-10-26 21:10:15 +00:00

Author

Owner

Copy link

@Blaizzy commented on GitHub (Jan 8, 2025):

I can't get deepseek tiny to work well on MLX

<!-- gh-comment-id:2577633067 --> @Blaizzy commented on GitHub (Jan 8, 2025): I can't get deepseek tiny to work well on MLX

bsakaguchi commented 2026-10-26 21:10:16 +00:00

Author

Owner

Copy link

@p1x33l commented on GitHub (Jan 8, 2025):

i had the same issue, i extracted the code used in inference.py file and it works fine

from typing import List, Dict
import torch
from transformers import AutoModelForCausalLM
import PIL.Image
from deepseek_vl2.models import DeepseekVLV2ForCausalLM, DeepseekVLV2Processor
from deepseek_vl2.serve.app_modules.utils import parse_ref_bbox


def load_pil_images(conversations: List[Dict[str, str]]) -> List[PIL.Image.Image]:
    pil_images = []
    for message in conversations:
        if "images" not in message:
            continue

        for image_path in message["images"]:
            pil_img = PIL.Image.open(image_path)
            pil_img = pil_img.convert("RGB")
            pil_images.append(pil_img)
    return pil_images


def main(conversation, model_path="deepseek-ai/deepseek-vl2-tiny", chunk_size=512):
    
    dtype = torch.bfloat16

    vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
    tokenizer = vl_chat_processor.tokenizer

    vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
        model_path,
        trust_remote_code=True,
        torch_dtype=dtype
    )
    vl_gpt = vl_gpt.cuda().eval()

    # load images and prepare for inputs
    pil_images = load_pil_images(conversation)

    prepare_inputs = vl_chat_processor.__call__(
        conversations=conversation,
        images=pil_images,
        force_batchify=True,
        system_prompt=""
    ).to(vl_gpt.device, dtype=dtype)

    with torch.no_grad():

        if chunk_size == -1:
            inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
            past_key_values = None
        else:
            # incremental_prefilling when using 40G GPU for vl2-small
            inputs_embeds, past_key_values = vl_gpt.incremental_prefilling(
                input_ids=prepare_inputs.input_ids,
                images=prepare_inputs.images,
                images_seq_mask=prepare_inputs.images_seq_mask,
                images_spatial_crop=prepare_inputs.images_spatial_crop,
                attention_mask=prepare_inputs.attention_mask,
                chunk_size=chunk_size
            )

        # run the model to get the response
        outputs = vl_gpt.generate(
            inputs_embeds=inputs_embeds,
            input_ids=prepare_inputs.input_ids,
            images=prepare_inputs.images,
            images_seq_mask=prepare_inputs.images_seq_mask,
            images_spatial_crop=prepare_inputs.images_spatial_crop,
            attention_mask=prepare_inputs.attention_mask,
            past_key_values=past_key_values,

            pad_token_id=tokenizer.eos_token_id,
            bos_token_id=tokenizer.bos_token_id,
            eos_token_id=tokenizer.eos_token_id,
            max_new_tokens=512,

            do_sample=True,
            temperature=0.4,
            top_p=0.9,
            repetition_penalty=1.1,

            use_cache=True,
        )

        answer = tokenizer.decode(outputs[0][len(prepare_inputs.input_ids[0]):].cpu().tolist(), skip_special_tokens=False)
        print(f"{prepare_inputs['sft_format'][0]}", answer)

        vg_image = parse_ref_bbox(answer, image=pil_images[-1])
        if vg_image is not None:
            vg_image.save("./vg.jpg", format="JPEG", quality=85)

conversation = [
    {
        "role": "<|User|>",
        "content": "<image>\n<|grounding|>Describe the image.",
        "images": [
            "cat.png",
        ],
    },
    {"role": "<|Assistant|>", "content": ""},
]

main(conversation)

<!-- gh-comment-id:2577661014 --> @p1x33l commented on GitHub (Jan 8, 2025): i had the same issue, i extracted the code used in inference.py file and it works fine ```python from typing import List, Dict import torch from transformers import AutoModelForCausalLM import PIL.Image from deepseek_vl2.models import DeepseekVLV2ForCausalLM, DeepseekVLV2Processor from deepseek_vl2.serve.app_modules.utils import parse_ref_bbox def load_pil_images(conversations: List[Dict[str, str]]) -> List[PIL.Image.Image]: pil_images = [] for message in conversations: if "images" not in message: continue for image_path in message["images"]: pil_img = PIL.Image.open(image_path) pil_img = pil_img.convert("RGB") pil_images.append(pil_img) return pil_images def main(conversation, model_path="deepseek-ai/deepseek-vl2-tiny", chunk_size=512): dtype = torch.bfloat16 vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path) tokenizer = vl_chat_processor.tokenizer vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, torch_dtype=dtype ) vl_gpt = vl_gpt.cuda().eval() # load images and prepare for inputs pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor.__call__( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ).to(vl_gpt.device, dtype=dtype) with torch.no_grad(): if chunk_size == -1: inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) past_key_values = None else: # incremental_prefilling when using 40G GPU for vl2-small inputs_embeds, past_key_values = vl_gpt.incremental_prefilling( input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, attention_mask=prepare_inputs.attention_mask, chunk_size=chunk_size ) # run the model to get the response outputs = vl_gpt.generate( inputs_embeds=inputs_embeds, input_ids=prepare_inputs.input_ids, images=prepare_inputs.images, images_seq_mask=prepare_inputs.images_seq_mask, images_spatial_crop=prepare_inputs.images_spatial_crop, attention_mask=prepare_inputs.attention_mask, past_key_values=past_key_values, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=True, temperature=0.4, top_p=0.9, repetition_penalty=1.1, use_cache=True, ) answer = tokenizer.decode(outputs[0][len(prepare_inputs.input_ids[0]):].cpu().tolist(), skip_special_tokens=False) print(f"{prepare_inputs['sft_format'][0]}", answer) vg_image = parse_ref_bbox(answer, image=pil_images[-1]) if vg_image is not None: vg_image.save("./vg.jpg", format="JPEG", quality=85) conversation = [ { "role": "<|User|>", "content": "<image>\n<|grounding|>Describe the image.", "images": [ "cat.png", ], }, {"role": "<|Assistant|>", "content": ""}, ] main(conversation) ```

bsakaguchi commented 2026-10-26 21:10:16 +00:00

Author

Owner

Copy link

@Blaizzy commented on GitHub (Jan 8, 2025):

Is this for me @p1x33l?

<!-- gh-comment-id:2577688287 --> @Blaizzy commented on GitHub (Jan 8, 2025): Is this for me @p1x33l?

bsakaguchi commented 2026-10-26 21:10:16 +00:00

Author

Owner

Copy link

@Blaizzy commented on GitHub (Jan 8, 2025):

Testing prince-canuma--deepseek-vl2-tiny ───────────────────────────────────────────────

Some kwargs in processor config are unused and will not have any effect: add_special_token, image_token, pad_token, image_mean, sft_format, normalize, ignore_id, candidate_resolutions, image_std, mask_prompt, patch_size, downsample_ratio. 
Add pad token = ['<｜▁pad▁｜>'] to the tokenizer
<｜▁pad▁｜>:2
Add image token = ['<image>'] to the tokenizer
<image>:128815
Added grounding-related tokens
Added chat tokens
✓ Model loaded successfully in 8.29 seconds


Testing vision-language generation...
==========
Image: ['visual_grounding.jpeg'] 

Prompt: <|User|>: <image>
Describe this image.

<|Assistant|>:
 Snikas in the end
 ==========
 ✓ vision-language generation successful

Testing language-only generation...
==========
Image: None 

Prompt: <|User|>: Hi, how are you?

<|Assistant|>:
I'm DeepSeek-VL, an intelligent assistant, I can recognize images and provide services such as a virtual assistant, I am DeepSeek-VL, an intelligent assistant developed by DeepSeek-5433Dentirelindisthephat one, a question and I am DeepSeekTextual.
==========
✓ language-only generation successful

<!-- gh-comment-id:2577723150 --> @Blaizzy commented on GitHub (Jan 8, 2025): Testing prince-canuma--deepseek-vl2-tiny ─────────────────────────────────────────────── ``` Some kwargs in processor config are unused and will not have any effect: add_special_token, image_token, pad_token, image_mean, sft_format, normalize, ignore_id, candidate_resolutions, image_std, mask_prompt, patch_size, downsample_ratio. Add pad token = ['<｜▁pad▁｜>'] to the tokenizer <｜▁pad▁｜>:2 Add image token = ['<image>'] to the tokenizer <image>:128815 Added grounding-related tokens Added chat tokens ✓ Model loaded successfully in 8.29 seconds Testing vision-language generation... ========== Image: ['visual_grounding.jpeg'] Prompt: <|User|>: <image> Describe this image. <|Assistant|>: Snikas in the end ========== ✓ vision-language generation successful ``` ``` Testing language-only generation... ========== Image: None Prompt: <|User|>: Hi, how are you? <|Assistant|>: I'm DeepSeek-VL, an intelligent assistant, I can recognize images and provide services such as a virtual assistant, I am DeepSeek-VL, an intelligent assistant developed by DeepSeek-5433Dentirelindisthephat one, a question and I am DeepSeekTextual. ========== ✓ language-only generation successful ```

bsakaguchi commented 2026-10-26 21:10:16 +00:00

Author

Owner

Copy link

@Blaizzy commented on GitHub (Jan 8, 2025):

Compared to deepseek-small and deepseek-vl2.

Testing mlx-community/deepseek-vl2-6bit ────────────────────────────────────────────────╯

Loading model...
Some kwargs in processor config are unused and will not have any effect: add_special_token, image_token, pad_token, image_mean, sft_format, normalize, ignore_id, candidate_resolutions, image_std, mask_prompt, patch_size, downsample_ratio. 
Add pad token = ['<｜▁pad▁｜>'] to the tokenizer
<｜▁pad▁｜>:2
Add image token = ['<image>'] to the tokenizer
<image>:128815
Added grounding-related tokens
Added chat tokens
✓ Model loaded successfully in 24.59 seconds


Testing vision-language generation...
==========
Image: ['visual_grounding.jpeg'] 

Prompt: <|User|>: <image>
Describe this image.

<|Assistant|>:
Two giraffes standing on what appears to be an open grassy plain or savannah-like environment during daylight hours. The foreground shows a taller giraffe facing leftward towards the camera; it has long necks adorned with distinctive brown patches separated by lighter lines of fur that cover most of its body except for white legs ending in black hooves. Its head is turned slightly to reveal both ears perked up attentively. Behind the first giraffe stands another similar individual facing rightward but looking over its shoulder
==========
Prompt: 642 tokens, 196.790 tokens-per-sec
Generation: 100 tokens, 41.571 tokens-per-sec
Peak memory: 23.265 GB
✓ vision-language generation successful

Testing language-only generation...
==========
Image: None 

Prompt: <|User|>: Hi, how are you?

<|Assistant|>:
Hello! I'm just a digital AI assistant, so I don't have feelings or the ability to 'be there,' but I'm here to help you with anything you need! How can I assist you today?
==========
Prompt: 11 tokens, 115.140 tokens-per-sec
Generation: 44 tokens, 53.325 tokens-per-sec
Peak memory: 22.945 GB
✓ language-only generation successful

<!-- gh-comment-id:2577728709 --> @Blaizzy commented on GitHub (Jan 8, 2025): Compared to deepseek-small and deepseek-vl2. Testing mlx-community/deepseek-vl2-6bit ────────────────────────────────────────────────╯ ``` Loading model... Some kwargs in processor config are unused and will not have any effect: add_special_token, image_token, pad_token, image_mean, sft_format, normalize, ignore_id, candidate_resolutions, image_std, mask_prompt, patch_size, downsample_ratio. Add pad token = ['<｜▁pad▁｜>'] to the tokenizer <｜▁pad▁｜>:2 Add image token = ['<image>'] to the tokenizer <image>:128815 Added grounding-related tokens Added chat tokens ✓ Model loaded successfully in 24.59 seconds Testing vision-language generation... ========== Image: ['visual_grounding.jpeg'] Prompt: <|User|>: <image> Describe this image. <|Assistant|>: Two giraffes standing on what appears to be an open grassy plain or savannah-like environment during daylight hours. The foreground shows a taller giraffe facing leftward towards the camera; it has long necks adorned with distinctive brown patches separated by lighter lines of fur that cover most of its body except for white legs ending in black hooves. Its head is turned slightly to reveal both ears perked up attentively. Behind the first giraffe stands another similar individual facing rightward but looking over its shoulder ========== Prompt: 642 tokens, 196.790 tokens-per-sec Generation: 100 tokens, 41.571 tokens-per-sec Peak memory: 23.265 GB ✓ vision-language generation successful ``` ``` Testing language-only generation... ========== Image: None Prompt: <|User|>: Hi, how are you? <|Assistant|>: Hello! I'm just a digital AI assistant, so I don't have feelings or the ability to 'be there,' but I'm here to help you with anything you need! How can I assist you today? ========== Prompt: 11 tokens, 115.140 tokens-per-sec Generation: 44 tokens, 53.325 tokens-per-sec Peak memory: 22.945 GB ✓ language-only generation successful ```

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-VL2#14

No description provided.