[GH-ISSUE #113] load deepseek-vl2模型设定device_map="balanced"时，inference报错 #92

New issue

Open

opened 2026-10-26 21:11:48 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 21:11:48 +00:00

Owner

Originally created by @Chord-Chen-30 on GitHub (Mar 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/113

Scripts:

model_path = 'deepseek-ai/deepseek-vl2'

vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
IMAGE_TOKEN = '<image>'
conversation = [
    {
        "role": "<|User|>",
        "content": "hi",
        "images": ['some_image.jpg'],
    },
    {"role": "<|Assistant|>", "content": ""}
]

tokenizer = vl_chat_processor.tokenizer

# with init_empty_weights():
vl_gpt = AutoModelForCausalLM.from_pretrained(
    model_path, 
    trust_remote_code=True, 
    device_map="balanced",
    torch_dtype=torch.bfloat16
)

vl_gpt = vl_gpt.eval()


pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)

Error:

Traceback (most recent call last):
  File "eval_ds_vl2_multi_gpu.py", line 222, in <module>
    res = inference(text, image)
  File "eval_ds_vl2_multi_gpu.py", line 92, in inference
    outputs = vl_gpt.language.generate(
  File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/transformers/generation/utils.py", line 1544, in generate
    return self.greedy_search(
  File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/transformers/generation/utils.py", line 2446, in greedy_search
    next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!

实测输入较长时，单卡A100-80G跑不动deepseek-ai/deepseek-vl2，多卡时有以上报错。

cuda version 12.1
deepseek_vl 1.0.0
deepseek_vl2 1.0.0

Originally created by @Chord-Chen-30 on GitHub (Mar 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/113 Scripts: ```python model_path = 'deepseek-ai/deepseek-vl2' vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path) IMAGE_TOKEN = '<image>' conversation = [ { "role": "<|User|>", "content": "hi", "images": ['some_image.jpg'], }, {"role": "<|Assistant|>", "content": ""} ] tokenizer = vl_chat_processor.tokenizer # with init_empty_weights(): vl_gpt = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, device_map="balanced", torch_dtype=torch.bfloat16 ) vl_gpt = vl_gpt.eval() pil_images = load_pil_images(conversation) prepare_inputs = vl_chat_processor( conversations=conversation, images=pil_images, force_batchify=True, system_prompt="" ).to(vl_gpt.device) # run image encoder to get the image embeddings inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs) # run the model to get the response outputs = vl_gpt.language.generate( inputs_embeds=inputs_embeds, attention_mask=prepare_inputs.attention_mask, pad_token_id=tokenizer.eos_token_id, bos_token_id=tokenizer.bos_token_id, eos_token_id=tokenizer.eos_token_id, max_new_tokens=512, do_sample=False, use_cache=True ) answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False) print(f"{prepare_inputs['sft_format'][0]}", answer) ``` Error: ```txt Traceback (most recent call last): File "eval_ds_vl2_multi_gpu.py", line 222, in <module> res = inference(text, image) File "eval_ds_vl2_multi_gpu.py", line 92, in inference outputs = vl_gpt.language.generate( File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/transformers/generation/utils.py", line 1544, in generate return self.greedy_search( File "/mnt/nas-alinlp/zhuochen.zc/anaconda3/envs/deepseek/lib/python3.8/site-packages/transformers/generation/utils.py", line 2446, in greedy_search next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0! ``` 实测输入较长时，单卡A100-80G跑不动deepseek-ai/deepseek-vl2，多卡时有以上报错。 cuda version 12.1 deepseek_vl 1.0.0 deepseek_vl2 1.0.0

bsakaguchi commented

2026-10-26 21:11:50 +00:00

Author

Owner

@zhaowei-wang-nlp commented on GitHub (Mar 26, 2025):

解决了吗，也有这个问题

@zhaowei-wang-nlp commented on GitHub (Mar 26, 2025): 解决了吗，也有这个问题

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepSeek-VL2#92

No description provided.

Rows
Columns