[GH-ISSUE #68] model type multi_modality but Transformers does not recognize this architecture #53

New issue

Open

opened 2026-10-26 21:11:07 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented 2026-10-26 21:11:07 +00:00

Owner

Copy link

Originally created by @Huowuge on GitHub (Jan 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/68

Anyone encountered this error?

ValueError: The checkpoint you are trying to load has model type multi_modality but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

System: ubuntu 2204
python:3.10
transformers:latest
torch:2.2.2
models: 1b and 1.3b

Originally created by @Huowuge on GitHub (Jan 28, 2025). Original GitHub issue: https://github.com/deepseek-ai/Janus/issues/68 Anyone encountered this error? ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date. System: ubuntu 2204 python:3.10 transformers:latest torch:2.2.2 models: 1b and 1.3b

bsakaguchi commented 2026-10-26 21:11:08 +00:00

Author

Owner

Copy link

@ghost commented on GitHub (Jan 28, 2025):

Same problem here

<!-- gh-comment-id:2620278234 --> @ghost commented on GitHub (Jan 28, 2025): Same problem here

bsakaguchi commented 2026-10-26 21:11:10 +00:00

Author

Owner

Copy link

@ghost commented on GitHub (Jan 28, 2025):

(TTS) lco@lco2:~/Programming/LLM/DeepSeek$ python Janus-Pro-1B.py 
Traceback (most recent call last):
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1073, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 775, in __getitem__
    raise KeyError(key)
KeyError: 'multi_modality'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/data1/protected/Programming/LLM/DeepSeek/Janus-Pro-1B.py", line 7, in <module>
    model = AutoModel.from_pretrained(model_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1075, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

for following attempt:

from transformers import AutoModel, AutoTokenizer
import torch
import torch.quantization

# Load the Janus-Pro-1B model and tokenizer
model_name = 'Janus-Pro-1B'  # Path to the Janus-Pro-1B directory
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Apply dynamic quantization to the model
# Quantize only the linear layers to reduce memory usage
quantized_model = torch.quantization.quantize_dynamic(
    model,  # The original model
    {torch.nn.Linear},  # Layers to quantize (e.g., linear layers)
    dtype=torch.qint8  # Quantization type (8-bit integers)
)

# Move the quantized model to the GPU (if available)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
quantized_model.to(device)

# Example input
input_text = "Hello, how are you?"
inputs = tokenizer(input_text, return_tensors='pt').to(device)  # Move inputs to the same device as the model

# Run the quantized model
with torch.no_grad():
    outputs = quantized_model(**inputs)

# Print the outputs
print(outputs)

<!-- gh-comment-id:2620279240 --> @ghost commented on GitHub (Jan 28, 2025): ``` (TTS) lco@lco2:~/Programming/LLM/DeepSeek$ python Janus-Pro-1B.py Traceback (most recent call last): File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1073, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 775, in __getitem__ raise KeyError(key) KeyError: 'multi_modality' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/data1/protected/Programming/LLM/DeepSeek/Janus-Pro-1B.py", line 7, in <module> model = AutoModel.from_pretrained(model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/data1/protected/TTS/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1075, in from_pretrained raise ValueError( ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date. You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git` ``` for following attempt: ```python from transformers import AutoModel, AutoTokenizer import torch import torch.quantization # Load the Janus-Pro-1B model and tokenizer model_name = 'Janus-Pro-1B' # Path to the Janus-Pro-1B directory model = AutoModel.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) # Apply dynamic quantization to the model # Quantize only the linear layers to reduce memory usage quantized_model = torch.quantization.quantize_dynamic( model, # The original model {torch.nn.Linear}, # Layers to quantize (e.g., linear layers) dtype=torch.qint8 # Quantization type (8-bit integers) ) # Move the quantized model to the GPU (if available) device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') quantized_model.to(device) # Example input input_text = "Hello, how are you?" inputs = tokenizer(input_text, return_tensors='pt').to(device) # Move inputs to the same device as the model # Run the quantized model with torch.no_grad(): outputs = quantized_model(**inputs) # Print the outputs print(outputs) ```

bsakaguchi commented 2026-10-26 21:11:10 +00:00

Author

Owner

Copy link

@LeeABarron commented on GitHub (Feb 1, 2025):

Same here with

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "deepseek-ai/Janus-Pro-7B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True)

<!-- gh-comment-id:2629143812 --> @LeeABarron commented on GitHub (Feb 1, 2025): Same here with ``` from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "deepseek-ai/Janus-Pro-7B" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", load_in_8bit=True) ```

bsakaguchi commented 2026-10-26 21:11:11 +00:00

Author

Owner

Copy link

@dineshchitlangia commented on GitHub (Feb 6, 2025):

Same issue here with code similar to @gnusupport and @LeeABarron.
I also tried the example inference.py file and that produces the same error too!

github.com/deepseek-ai/Janus@1daa72fa40/inference.py

@hills-code @charlesCXK - could you pls help?

<!-- gh-comment-id:2638721291 --> @dineshchitlangia commented on GitHub (Feb 6, 2025): Same issue here with code similar to @gnusupport and @LeeABarron. I also tried the example inference.py file and that produces the same error too! https://github.com/deepseek-ai/Janus/blob/1daa72fa409002d40931bd7b36a9280362469ead/inference.py @hills-code @charlesCXK - could you pls help?

bsakaguchi commented 2026-10-26 21:11:11 +00:00

Author

Owner

Copy link

@YoshitoMurayama commented on GitHub (Feb 6, 2025):

I think we have to register custom model classes to "transformers" before loading the model like line below
github.com/deepseek-ai/Janus@1daa72fa40/janus/models/modeling_vlm.py (L272)

maybe importing janus.models execute above line

otherwise I solve this by adding custom model code to model repository(local folder), then I add "auto_map" settings to config.json...

<!-- gh-comment-id:2639036283 --> @YoshitoMurayama commented on GitHub (Feb 6, 2025): I think we have to register custom model classes to "transformers" before loading the model like line below https://github.com/deepseek-ai/Janus/blob/1daa72fa409002d40931bd7b36a9280362469ead/janus/models/modeling_vlm.py#L272 maybe importing janus.models execute above line otherwise I solve this by adding custom model code to model repository(local folder), then I add "auto_map" settings to config.json...

bsakaguchi commented 2026-10-26 21:11:11 +00:00

Author

Owner

Copy link

@creasy2010 commented on GitHub (Feb 6, 2025):

Same problem

System: mac m1
python:3.8
transformers:latest
torch:2.4.1
models: Janus-Pro-7B and Janus-Pro-1B and Janus-1.3B

<!-- gh-comment-id:2639439786 --> @creasy2010 commented on GitHub (Feb 6, 2025): Same problem System: mac m1 python:3.8 transformers:latest torch:2.4.1 models: Janus-Pro-7B and Janus-Pro-1B and Janus-1.3B

bsakaguchi commented 2026-10-26 21:11:11 +00:00

Author

Owner

Copy link

@LeeABarron commented on GitHub (Feb 6, 2025):

@YoshitoMurayama can you go into detail about your solution please? That would be great. Thanks man.

<!-- gh-comment-id:2640584932 --> @LeeABarron commented on GitHub (Feb 6, 2025): @YoshitoMurayama can you go into detail about your solution please? That would be great. Thanks man.

bsakaguchi commented 2026-10-26 21:11:11 +00:00

Author

Owner

Copy link

@DevPaulLiu commented on GitHub (Feb 8, 2025):

Seems HuggingFace team are working on it: https://github.com/huggingface/transformers/issues/35928

<!-- gh-comment-id:2644733732 --> @DevPaulLiu commented on GitHub (Feb 8, 2025): Seems HuggingFace team are working on it: https://github.com/huggingface/transformers/issues/35928

bsakaguchi commented 2026-10-26 21:11:12 +00:00

Author

Owner

Copy link

@Wuwenji18 commented on GitHub (Apr 21, 2025):

同问

<!-- gh-comment-id:2818353417 --> @Wuwenji18 commented on GitHub (Apr 21, 2025): 同问

bsakaguchi commented 2026-10-26 21:11:12 +00:00

Author

Owner

Copy link

@popcornkiller1088 commented on GitHub (May 12, 2025):

this worked for me

<!-- gh-comment-id:2871893326 --> @popcornkiller1088 commented on GitHub (May 12, 2025):  this worked for me

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/Janus#53

No description provided.