[GH-ISSUE #77] DeepSeek-VL2 tokenizer_config.json文件兼容vllm多模态接口修复建议

bsakaguchi commented

2026-10-26 21:11:08 +00:00

Owner

Originally created by @songafu on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/77

环境要求：
vllm版本：>=0.7.2

注意vllm运行必须带architectures参数：python -m vllm.entrypoints.openai.api_server --hf_overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'

涉及文件https://huggingface.co/deepseek-ai/deepseek-vl2/blob/main/tokenizer_config.json
建议tokenizer_config.json文件补充增加如下内容：
"chat_template": "{%- set found_item = false -%}\n{%- for message in messages -%}\n {%- if message['role'] == 'system' -%}\n {%- set found_item = true -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if not found_item -%}\n{{'You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n'}}\n{%- endif %}\n{%- for message in messages %}\n {%- if message['role'] == 'system' %}\n{{ message['content'] }}\n {%- else %}\n {%- if message['role'] == 'user' %}\n{{'### Instruction:\n' + message['content'] + '\n'}}\n {%- else %}\n{{'### Response:\n' + message['content'] + '\n<|EOT|>\n'}}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{{'### Response:\n'}}\n",

此时可以利用vllm原生的多模态接口可访问成功：
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "deepseek-ai/deepseek-vl2",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
}
},
{
"type": "text",
"text": "Describe this image in one sentence."
}
]
}
]

}'

Originally created by @songafu on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/77 环境要求： vllm版本：>=0.7.2 注意vllm运行必须带architectures参数：python -m vllm.entrypoints.openai.api_server --hf_overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' 涉及文件https://huggingface.co/deepseek-ai/deepseek-vl2/blob/main/tokenizer_config.json 建议tokenizer_config.json文件补充增加如下内容： "chat_template": "{%- set found_item = false -%}\n{%- for message in messages -%}\n {%- if message['role'] == 'system' -%}\n {%- set found_item = true -%}\n {%- endif -%}\n{%- endfor -%}\n{%- if not found_item -%}\n{{'You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\\n'}}\n{%- endif %}\n{%- for message in messages %}\n {%- if message['role'] == 'system' %}\n{{ message['content'] }}\n {%- else %}\n {%- if message['role'] == 'user' %}\n{{'### Instruction:\\n' + message['content'] + '\\n'}}\n {%- else %}\n{{'### Response:\\n' + message['content'] + '\\n<|EOT|>\\n'}}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{{'### Response:\\n'}}\n", 此时可以利用vllm原生的多模态接口可访问成功： curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-ai/deepseek-vl2", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } }, { "type": "text", "text": "Describe this image in one sentence." } ] } ] }'

bsakaguchi commented

2026-10-26 21:11:10 +00:00

Author

Owner

@HubHop commented on GitHub (Feb 13, 2025):

Thanks @songafu, we will consider making an update.

@HubHop commented on GitHub (Feb 13, 2025): Thanks @songafu, we will consider making an update.

bsakaguchi commented

2026-10-26 21:11:11 +00:00

Author

Owner

@VincentYCYao commented on GitHub (Jul 21, 2025):

Hey! Same issue when using lmms-eval

I add chat template support to lmms-eval here.

Inference:

chat_template_file = "template_deepseek_vl2.jinja"
subprocess.run(
    f"wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja -O {chat_template_file}",
    check=True,
    shell=True,
)
cmd = [
    "python3",
    "-m",
    "lmms_eval",
    "--model",
    "vllm",
    "--model_args",
    "model_version=deepseek-ai/deepseek-vl2,"
    'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},'
    f"chat_template={chat_template_file},"
    "tensor_parallel_size=2,"
    "dtype=bfloat16",
    "--tasks",
    task,
    "--batch_size",
    "1",
    "--limit",
    "1000",
    "--log_samples",
    "--output_path",
    "logs",
]
cmd_result = subprocess.run(cmd, check=False)

We can use the chat template file: chat template: https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja

It's definitely better to have the chat template in tokenizer so that we can get the template with

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("deepseek-ai/deepseek-vl2")
chat_template= processor.tokenizer.chat_template

@VincentYCYao commented on GitHub (Jul 21, 2025): Hey! Same issue when using lmms-eval I add chat template support to lmms-eval [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/652#issue-3018353281). Inference: ```python chat_template_file = "template_deepseek_vl2.jinja" subprocess.run( f"wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja -O {chat_template_file}", check=True, shell=True, ) cmd = [ "python3", "-m", "lmms_eval", "--model", "vllm", "--model_args", "model_version=deepseek-ai/deepseek-vl2," 'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},' f"chat_template={chat_template_file}," "tensor_parallel_size=2," "dtype=bfloat16", "--tasks", task, "--batch_size", "1", "--limit", "1000", "--log_samples", "--output_path", "logs", ] cmd_result = subprocess.run(cmd, check=False) ``` We can use the chat template file: chat template: https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja It's definitely better to have the chat template in tokenizer so that we can get the template with ```python from transformers import AutoProcessor processor = AutoProcessor.from_pretrained("deepseek-ai/deepseek-vl2") chat_template= processor.tokenizer.chat_template ```

bsakaguchi commented

2026-10-26 21:11:11 +00:00

Author

Owner

@VincentYCYao commented on GitHub (Jul 21, 2025):

Hey! Same issue when using lmms-eval

I add chat template support to lmms-eval here.

Inference:

chat_template_file = "template_deepseek_vl2.jinja"
subprocess.run(
f"wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja -O {chat_template_file}",
check=True,
shell=True,
)
cmd = [
"python3",
"-m",
"lmms_eval",
"--model",
"vllm",
"--model_args",
"model_version=deepseek-ai/deepseek-vl2,"
'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},'
f"chat_template={chat_template_file},"
"tensor_parallel_size=2,"
"dtype=bfloat16",
"--tasks",
task,
"--batch_size",
"1",
"--limit",
"1000",
"--log_samples",
"--output_path",
"logs",
]
cmd_result = subprocess.run(cmd, check=False)
We can use the chat template file: chat template: https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja

It's definitely better to have the chat template in tokenizer so that we can get the template with

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("deepseek-ai/deepseek-vl2")
chat_template= processor.tokenizer.chat_template

The chat template file is removed from vllm repo, see the original PR (https://github.com/vllm-project/vllm/pull/12143/files).

Alternative:

chat_template = "
    {%- if messages[0]['role'] == 'system' -%}
        {%- set system_message = messages[0]['content'] -%}
        {%- set messages = messages[1:] -%}
    {%- else -%}
        {% set system_message = '' -%}
    {%- endif -%}
    
    {{ bos_token + system_message }}
    {%- for message in messages -%}
        {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
            {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
        {%- endif -%}
    
        {%- if message['role'] == 'user' -%}
            {{ '<|User|>: ' + message['content'] + '\n' }}
        {%- elif message['role'] == 'assistant' -%}
            {{ '<|Assistant|>: ' + message['content'] + eos_token + '\n' }}
        {%- endif -%}
    {%- endfor -%}
    
    {%- if add_generation_prompt -%}
        {{ '<|Assistant|>: ' }}
    {% endif %}
    "
cmd = [
    "python3",
    "-m",
    "lmms_eval",
    "--model",
    "vllm",
    "--model_args",
    "model_version=deepseek-ai/deepseek-vl2,"
    'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},'
    f"chat_template={chat_template},"
    "tensor_parallel_size=2,"
    "dtype=bfloat16",
    "--tasks",
    task,
    "--batch_size",
    "1",
    "--limit",
    "1000",
    "--log_samples",
    "--output_path",
    "logs",
]
cmd_result = subprocess.run(cmd, check=False)

@VincentYCYao commented on GitHub (Jul 21, 2025): > Hey! Same issue when using lmms-eval > > I add chat template support to lmms-eval [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/652#issue-3018353281). > > Inference: > > chat_template_file = "template_deepseek_vl2.jinja" > subprocess.run( > f"wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja -O {chat_template_file}", > check=True, > shell=True, > ) > cmd = [ > "python3", > "-m", > "lmms_eval", > "--model", > "vllm", > "--model_args", > "model_version=deepseek-ai/deepseek-vl2," > 'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},' > f"chat_template={chat_template_file}," > "tensor_parallel_size=2," > "dtype=bfloat16", > "--tasks", > task, > "--batch_size", > "1", > "--limit", > "1000", > "--log_samples", > "--output_path", > "logs", > ] > cmd_result = subprocess.run(cmd, check=False) > We can use the chat template file: chat template: https://raw.githubusercontent.com/vllm-project/vllm/main/examples/template_deepseek_vl2.jinja > > It's definitely better to have the chat template in tokenizer so that we can get the template with > > from transformers import AutoProcessor > > processor = AutoProcessor.from_pretrained("deepseek-ai/deepseek-vl2") > chat_template= processor.tokenizer.chat_template The chat template file is removed from vllm repo, see the original PR (https://github.com/vllm-project/vllm/pull/12143/files). Alternative: ```python chat_template = " {%- if messages[0]['role'] == 'system' -%} {%- set system_message = messages[0]['content'] -%} {%- set messages = messages[1:] -%} {%- else -%} {% set system_message = '' -%} {%- endif -%} {{ bos_token + system_message }} {%- for message in messages -%} {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%} {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }} {%- endif -%} {%- if message['role'] == 'user' -%} {{ '<|User|>: ' + message['content'] + '\n' }} {%- elif message['role'] == 'assistant' -%} {{ '<|Assistant|>: ' + message['content'] + eos_token + '\n' }} {%- endif -%} {%- endfor -%} {%- if add_generation_prompt -%} {{ '<|Assistant|>: ' }} {% endif %} " cmd = [ "python3", "-m", "lmms_eval", "--model", "vllm", "--model_args", "model_version=deepseek-ai/deepseek-vl2," 'hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},' f"chat_template={chat_template}," "tensor_parallel_size=2," "dtype=bfloat16", "--tasks", task, "--batch_size", "1", "--limit", "1000", "--log_samples", "--output_path", "logs", ] cmd_result = subprocess.run(cmd, check=False) ```

Rows
Columns

[GH-ISSUE #77] DeepSeek-VL2 tokenizer_config.json文件兼容vllm多模态接口修复建议 #63