mirror of
https://github.com/deepseek-ai/DeepSeek-VL2.git
synced 2026-11-05 01:07:23 +00:00
[GH-ISSUE #86] can't run python web_demo.py with 8 4090 (24GB) cards #72
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-VL2#72
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fengyue20 on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-VL2/issues/86
I have 8 GPUs with 24GB memory each,
here is the code:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python web_demo.py
--model_name "deepseek-ai/deepseek-vl2"
--port 37914 \
the out put is:
......
deepseek-ai/deepseek-vl2/model-00008-of-000008.safetensors
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████| 8/8 [00:14<00:00, 1.85s/it]
after Loading checkpoint shards: 100%, the CUDA out of memory:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 23.55 GiB total capacity; 17.98 GiB already allocated; 18.81 MiB free; 23.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
it seems that only one gpu is used, the CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 didn't work,
I want to know what need to change, thank you.
@fengyue20 commented on GitHub (Feb 18, 2025):
the code works after I use "split_model":
def split_model(model_name):
device_map = {}
model_splits = {
# 'deepseek-ai/deepseek-vl2-small': [13, 14], # 2 GPU for 16b
# 'deepseek-ai/deepseek-vl2': [10, 10, 10], # 3 GPU for 27b
'deepseek-ai/deepseek-vl2': [3, 4, 4, 4, 4, 4, 4, 4], # 8 GPU
}
num_layers_per_gpu = model_splits[model_name]
num_layers = sum(num_layers_per_gpu)
layer_cnt = 0
for i, num_layer in enumerate(num_layers_per_gpu):
for j in range(num_layer):
device_map[f'language.model.layers.{layer_cnt}'] = i
layer_cnt += 1
device_map['vision'] = 0
device_map['projector'] = 0
device_map['image_newline'] = 0
device_map['view_seperator'] = 0
device_map['language.model.embed_tokens'] = 0
device_map['language.model.norm'] = 0
device_map['language.lm_head'] = 0
device_map[f'language.model.layers.{num_layers - 1}'] = 0
return device_map
@krishnateja95 commented on GitHub (Feb 23, 2025):
@fengyue20 I used the same split_model function but got the following issue with respect to KV cache:
What is your transformers version?
@JACK-Chen-2019 commented on GitHub (Feb 26, 2025):
I encountered the same problem. Did you solve it?
@krishnateja95 commented on GitHub (Feb 26, 2025):
@JACK-Chen-2019 yes, follow this PR: https://github.com/deepseek-ai/DeepSeek-VL2/pull/91
@JACK-Chen-2019 commented on GitHub (Feb 26, 2025):
Thank you very much. It works for me！
---- Replied Message ----
| From | Krishna Teja @.> |
| Date | 02/26/2025 上午10:00 |
| To | deepseek-ai/DeepSeek-VL2 @.> |
| Cc | Zichen @.>,
Mention @.> |
| Subject | Re: [deepseek-ai/DeepSeek-VL2] can't run python web_demo.py with 8 4090 (24GB) cards (Issue #86) |
@JACK-Chen-2019 yes, follow this PR: #91
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>
krishnateja95 left a comment (deepseek-ai/DeepSeek-VL2#86)
@JACK-Chen-2019 yes, follow this PR: #91
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>