deepseek-ai/DeepSeek-VL2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-VL2.git synced 2026-11-05 01:07:23 +00:00
Code Issues 123 Projects Packages Wiki Activity

[GH-ISSUE #132] Running Inference on Multiple GPU #111

New issue
Open
opened 2026-10-26 21:12:08 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:12:08 +00:00
Owner
Copy link

Originally created by @Mukil07 on GitHub (Apr 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/132

Im trying to run the Deepseek-vl2 on 2 A6000 GPUs (48GB). When I modify the model loading from,

  vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path,
      trust_remote_code=True,
      torch_dtype=dtype,
  )

to,

  vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path,
      trust_remote_code=True,
      torch_dtype=dtype,
      device_map="auto"
  )

Its throwing error that tensors should be in the same device;

  File "DeepSeek-VL2/deepseek_vl2/models/modeling_deepseek.py", line 108, in forward
    return self.weight * hidden_states.to(input_dtype)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Please let me know if there is a way to run the multigpu inference. I have already tried the method split_model function in an earlier issue, but still didnt work.

Originally created by @Mukil07 on GitHub (Apr 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/132 Im trying to run the Deepseek-vl2 on 2 A6000 GPUs (48GB). When I modify the model loading from, ``` vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, torch_dtype=dtype, ) ``` to, ``` vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, torch_dtype=dtype, device_map="auto" ) ``` Its throwing error that tensors should be in the same device; ``` File "DeepSeek-VL2/deepseek_vl2/models/modeling_deepseek.py", line 108, in forward return self.weight * hidden_states.to(input_dtype) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! ``` Please let me know if there is a way to run the multigpu inference. I have already tried the method split_model function in an earlier issue, but still didnt work.
bsakaguchi commented 2026-10-26 21:12:10 +00:00
Author
Owner
Copy link

@deadpipe commented on GitHub (May 15, 2025):

try:


import os

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"

  vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path,
      trust_remote_code=True,
      torch_dtype=dtype,
      device_map="sequential",
      offload_state_dict=True,
      max_memory={0: "44GIB", 1: "44GIB", "cpu": "20GIB"},
  )
<!-- gh-comment-id:2885133396 --> @deadpipe commented on GitHub (May 15, 2025): try: ``` import os os.environ['CUDA_VISIBLE_DEVICES'] = '0,1' os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128" vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, torch_dtype=dtype, device_map="sequential", offload_state_dict=True, max_memory={0: "44GIB", 1: "44GIB", "cpu": "20GIB"}, ) ```
bsakaguchi commented 2026-10-26 21:12:10 +00:00
Author
Owner
Copy link

@1215thebqtic commented on GitHub (Jun 3, 2025):

try:


import os

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"

  vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path,
      trust_remote_code=True,
      torch_dtype=dtype,
      device_map="sequential",
      offload_state_dict=True,
      max_memory={0: "44GIB", 1: "44GIB", "cpu": "20GIB"},
  )

Hi, I'm using V100 to do the inference. And I use the code you provided, it reports error below, how can I solve this? thanks!

code

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3,4'

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"

vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
torch_dtype=torch.float32,
device_map="auto",
offload_state_dict=True,
max_memory={0: "28GIB", 1: "28GIB", 2: "28GIB", 3: "28GIB", 4: "28GIB", "cpu": "40GIB"},
)

#vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.float32).cuda().eval()

error

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:27<00:00, 6.81s/it]
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
You shouldn't move a model that is dispatched using accelerate hooks.
Traceback (most recent call last):
File "run_infer.py", line 31, in
vl_gpt = vl_gpt.to(torch.float32).cuda().eval()
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/accelerate/big_modeling.py", line 457, in wrapper
return fn(*args, **kwargs)
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2528, in cuda
return super().cuda(*args, **kwargs)
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in cuda
return self._apply(lambda t: t.cuda(device))
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
[Previous line repeated 5 more times]
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
param_applied = fn(param)
File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in
return self._apply(lambda t: t.cuda(device))
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 31.73 GiB total capacity; 31.30 GiB already allocated; 2.44 MiB free; 31.37 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

<!-- gh-comment-id:2933518746 --> @1215thebqtic commented on GitHub (Jun 3, 2025): > try: > > ``` > > import os > > os.environ['CUDA_VISIBLE_DEVICES'] = '0,1' > > os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128" > > vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( > model_path, > trust_remote_code=True, > torch_dtype=dtype, > device_map="sequential", > offload_state_dict=True, > max_memory={0: "44GIB", 1: "44GIB", "cpu": "20GIB"}, > ) > ``` Hi, I'm using V100 to do the inference. And I use the code you provided, it reports error below, how can I solve this? thanks! # code os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3,4' os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128" vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, torch_dtype=torch.float32, device_map="auto", offload_state_dict=True, max_memory={0: "28GIB", 1: "28GIB", 2: "28GIB", 3: "28GIB", 4: "28GIB", "cpu": "40GIB"}, ) #vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True) vl_gpt = vl_gpt.to(torch.float32).cuda().eval() # error Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:27<00:00, 6.81s/it] huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) You shouldn't move a model that is dispatched using accelerate hooks. Traceback (most recent call last): File "run_infer.py", line 31, in <module> vl_gpt = vl_gpt.to(torch.float32).cuda().eval() File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/accelerate/big_modeling.py", line 457, in wrapper return fn(*args, **kwargs) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2528, in cuda return super().cuda(*args, **kwargs) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in cuda return self._apply(lambda t: t.cuda(device)) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply module._apply(fn) [Previous line repeated 5 more times] File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply param_applied = fn(param) File "/mnt/software/miniconda/miniconda3.8/envs/deepseekvl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in <lambda> return self._apply(lambda t: t.cuda(device)) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 31.73 GiB total capacity; 31.30 GiB already allocated; 2.44 MiB free; 31.37 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
bsakaguchi commented 2026-10-26 21:12:11 +00:00
Author
Owner
Copy link

@deadpipe commented on GitHub (Jun 3, 2025):

vl_gpt = vl_gpt.to(torch.float32).cuda().eval()

To disable this warning, you can either:

  • Avoid using tokenizers before the fork if possible
  • Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
    You shouldn't move a model that is dispatched using accelerate hooks.

@1215thebqtic

Remove .cuda() as you have already dispatched the model using accelerate hooks

Use vl_gpt = vl_gpt.eval()

if the issue still persists, pass device_map="sequential"

<!-- gh-comment-id:2935089266 --> @deadpipe commented on GitHub (Jun 3, 2025): > vl_gpt = vl_gpt.to(torch.float32).cuda().eval() > To disable this warning, you can either: >- Avoid using tokenizers before the fork if possible >- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) >You shouldn't move a model that is dispatched using accelerate hooks. @1215thebqtic Remove .cuda() as you have already dispatched the model using accelerate hooks Use `vl_gpt = vl_gpt.eval()` if the issue still persists, pass `device_map="sequential"`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-VL2#111
No description provided.