[GH-ISSUE #75] Unable to load deepseek-vl2-tiny using vllm.

bsakaguchi commented

2026-10-26 21:11:03 +00:00

Owner

Originally created by @WindLWQ on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/75

from vllm import LLM
llm = LLM(model="/home/cisdi/workspace/DeepSeek-VL2-main/model")  # Name or path of your model

An error occurred while running:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/utils.py", line 1051, in inner
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 242, in __init__
    self.llm_engine = self.engine_class.from_engine_args(
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 481, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/arg_utils.py", line 1075, in create_engine_config
    model_config = self.create_model_config()
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/arg_utils.py", line 998, in create_model_config
    return ModelConfig(
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/config.py", line 364, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/config.py", line 424, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 445, in is_multimodal_model
    model_cls, _ = self.inspect_model_cls(architectures)
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 398, in inspect_model_cls
    architectures = self._normalize_archs(architectures)
  File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 388, in _normalize_archs
    for model in architectures:
TypeError: 'NoneType' object is not iterable

It seems that the structure format of config.json is not being recognized correctly.

Originally created by @WindLWQ on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/75 ``` from vllm import LLM llm = LLM(model="/home/cisdi/workspace/DeepSeek-VL2-main/model") # Name or path of your model ``` An error occurred while running: ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/utils.py", line 1051, in inner return fn(*args, **kwargs) File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 242, in __init__ self.llm_engine = self.engine_class.from_engine_args( File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 481, in from_engine_args engine_config = engine_args.create_engine_config(usage_context) File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/arg_utils.py", line 1075, in create_engine_config model_config = self.create_model_config() File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/arg_utils.py", line 998, in create_model_config return ModelConfig( File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/config.py", line 364, in __init__ self.multimodal_config = self._init_multimodal_config( File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/config.py", line 424, in _init_multimodal_config if ModelRegistry.is_multimodal_model(architectures): File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 445, in is_multimodal_model model_cls, _ = self.inspect_model_cls(architectures) File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 398, in inspect_model_cls architectures = self._normalize_archs(architectures) File "/root/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/registry.py", line 388, in _normalize_archs for model in architectures: TypeError: 'NoneType' object is not iterable ``` It seems that the structure format of config.json is not being recognized correctly.

bsakaguchi commented

2026-10-26 21:11:05 +00:00

Author

Owner

@coco-droid commented on GitHub (Feb 12, 2025):

It's because you use the llm pipeline to load a vision model

@coco-droid commented on GitHub (Feb 12, 2025): It's because you use the llm pipeline to load a vision model

bsakaguchi commented

2026-10-26 21:11:05 +00:00

Author

Owner

@WindLWQ commented on GitHub (Feb 13, 2025):

What other inference frameworks, besides Transformers, can currently be used to run this model?

@WindLWQ commented on GitHub (Feb 13, 2025): What other inference frameworks, besides Transformers, can currently be used to run this model?

bsakaguchi commented

2026-10-26 21:11:06 +00:00

Author

Owner

@cjjimmy commented on GitHub (Feb 17, 2025):

Ollama

@cjjimmy commented on GitHub (Feb 17, 2025): Ollama

bsakaguchi commented

2026-10-26 21:11:06 +00:00

Author

Owner

@bharathy-atlantic commented on GitHub (Feb 17, 2025):

@WindLWQ , Try adding: --hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' to your model entrypoint. More details here:
https://docs.vllm.ai/en/latest/models/supported_models.html#id2
Please make sure you use ampere or greater architecture for GPU.
This worked for me.

@bharathy-atlantic commented on GitHub (Feb 17, 2025): @WindLWQ , Try adding: `--hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'` to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me.

bsakaguchi commented

2026-10-26 21:11:06 +00:00

Author

Owner

@WindLWQ commented on GitHub (Feb 19, 2025):

@WindLWQ , Try adding: --hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me.

Can you provide a demo of the call?

@WindLWQ commented on GitHub (Feb 19, 2025): > [@WindLWQ](https://github.com/WindLWQ) , Try adding: `--hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'` to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me. Can you provide a demo of the call?

bsakaguchi commented

2026-10-26 21:11:06 +00:00

Author

Owner

@bharathy-atlantic commented on GitHub (Feb 19, 2025):

@WindLWQ , Try adding: --hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me.

Can you provide a demo of the call?

Sure, here is how I call in my dockerfile:
ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \ --model "$MODEL_NAME" \ $VLLM_ARGS \ --hf_overrides '{"architectures":["DeepseekVLV2ForCausalLM"]}'

@bharathy-atlantic commented on GitHub (Feb 19, 2025): > > [@WindLWQ](https://github.com/WindLWQ) , Try adding: `--hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'` to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me. > > Can you provide a demo of the call? Sure, here is how I call in my dockerfile: ` ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \ --model "$MODEL_NAME" \ $VLLM_ARGS \ --hf_overrides '{"architectures":["DeepseekVLV2ForCausalLM"]}' `

bsakaguchi commented

2026-10-26 21:11:07 +00:00

Author

Owner

@WindLWQ commented on GitHub (Feb 19, 2025):

@WindLWQ , Try adding: --hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me.

Can you provide a demo of the call?

Sure, here is how I call in my dockerfile: ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \ --model "$MODEL_NAME" \ $VLLM_ARGS \ --hf_overrides '{"architectures":["DeepseekVLV2ForCausalLM"]}'

Thank you, I have successfully deployed it.

@WindLWQ commented on GitHub (Feb 19, 2025): > > > [@WindLWQ](https://github.com/WindLWQ) , Try adding: `--hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'` to your model entrypoint. More details here: https://docs.vllm.ai/en/latest/models/supported_models.html#id2 Please make sure you use ampere or greater architecture for GPU. This worked for me. > > > > > > Can you provide a demo of the call? > > Sure, here is how I call in my dockerfile: `ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \ --model "$MODEL_NAME" \ $VLLM_ARGS \ --hf_overrides '{"architectures":["DeepseekVLV2ForCausalLM"]}'` Thank you, I have successfully deployed it.

Rows
Columns

[GH-ISSUE #75] Unable to load deepseek-vl2-tiny using vllm. #60