mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #8] Running DeepSeek-VL2 with multiple cards #6
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#6
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @robinren03 on GitHub (Dec 16, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/8
I have 3 A6000 GPUs with 48GB memory each, and I need to use TP to load DeepSeek-VL2 model into the GPUs (not the tiny / small ones).
Here is my code.
And it comes into the following problem:
Is there any official guidelines about using TP (i.e. MP) or PP to run the larger models? Or could you please point out my mistake. Thanks a lot!
@robinren03 commented on GitHub (Dec 17, 2024):
Well, the code works if I have the following change:
But it just prints some meaning less words, such as:
@HubHop commented on GitHub (Dec 23, 2024):
Hi @robinren03, please consider this minimal code for model sharding. Based on my preliminary tests, it has successfully run our 16B and 27B models. However, I have not extensively tested which sharding strategy performs the best. You may want to experiment with tuning these parameters later based on your specific requirements.
@DeadLining commented on GitHub (Dec 27, 2024):
After the update, deepseek-vl2 appears to have an issue with KV Cache
File "/home/getui/kongsz/gt-bi-lab-mllm-research/DeepSeek-VL2-main/deepseek_vl2/models/modeling_deepseek.py", line 1292, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/opt/anaconda3/envs/deepseek-vl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/deepseek-vl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) File "/opt/anaconda3/envs/deepseek-vl/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, **kwargs) File "/home/getui/kongsz/gt-bi-lab-mllm-research/DeepSeek-VL2-main/deepseek_vl2/models/modeling_deepseek.py", line 885, in forward k_pe, compressed_kv = past_key_value.update(k_pe, compressed_kv, self.layer_idx, cache_kwargs) File "/opt/anaconda3/envs/deepseek-vl/lib/python3.10/site-packages/transformers/cache_utils.py", line 449, in update self.key_cache[layer_idx] = torch.cat([self.key_cache[layer_idx], key_states], dim=-2) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument tensors in method wrapper_CUDA_cat)@dprokhorov17 commented on GitHub (Jan 7, 2025):
Doesn't work for me too. I am on two H100 cards. I followed https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2558914207 but I do get garbage output. Here is my code:
(I downloaded the model beforehand to local-dir)
@geekchen007 commented on GitHub (Jan 21, 2025):
I used three 16G NVIDIA V100 to load the DeepSeek-VL2-small model, and failed.
@corkiyao commented on GitHub (Feb 7, 2025):
Since cuda out of menmory. I use 5 1080Ti GPUs to load the small deepseek-vl2，and it works
@weedge commented on GitHub (Feb 8, 2025):
@robinren03 commented on GitHub (Feb 8, 2025):
Great, let me come back to you later. https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2644819404
@Tgnink commented on GitHub (Mar 3, 2025):
#8
https://github.com/deepseek-ai/DeepSeek-VL2/issues/8#issuecomment-2644819404
After making some changes in
modeling_deepseek.pyas outlined below and and applying the split_model method as @HubHop does, the codes work for meTO
@UchihaIchigo commented on GitHub (Apr 12, 2025):
Hello, according to your method, I run this code on three 4090 servers (deepseek-vl2-small, I split it into 3 layers [9,9,9], and finally I get an error that GPU0 video memory is not enough, please how should I allocate the model layer
@UchihaIchigo commented on GitHub (Apr 12, 2025):
How does your code implement loading the small model with 5 x 1080ti?
@TseWaiChiu commented on GitHub (May 30, 2025):
thank u !!!! its works in 4*4090