deepseek-ai/DeepSeek-V2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V2.git synced 2026-11-05 00:47:23 +00:00
Code Issues 90 Projects Packages Wiki Activity

[GH-ISSUE #74] 为何我在A800上运行DeepSeek-V2-Lite-Chat (SFT)，竟然消耗60G的显存？！ #70

New issue
Open
opened 2026-10-26 21:07:27 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:07:27 +00:00
Owner
Copy link

Originally created by @winsurelab on GitHub (Jul 19, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/74

权重文件一共32G左右。
为啥实际加载模型后，占用内存将近60多G呢。

Originally created by @winsurelab on GitHub (Jul 19, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V2/issues/74 权重文件一共32G左右。 为啥实际加载模型后，占用内存将近60多G呢。
bsakaguchi commented 2026-10-26 21:07:28 +00:00
Author
Owner
Copy link

@winsurelab commented on GitHub (Jul 25, 2024):

模型加载时，通过指定数据类型为float16避免使用全精度，可以使内存降到40G以下。

<!-- gh-comment-id:2249435743 --> @winsurelab commented on GitHub (Jul 25, 2024): 模型加载时，通过指定数据类型为float16避免使用全精度，可以使内存降到40G以下。
bsakaguchi commented 2026-10-26 21:07:28 +00:00
Author
Owner
Copy link

@liangfang commented on GitHub (Jul 28, 2024):

注意到这句话——
The model has a long context length (163840). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space.
Consider setting --max-model-len to a smaller value.

但是我也想就此请教一下long context length为啥消耗显存那么多？

<!-- gh-comment-id:2254528521 --> @liangfang commented on GitHub (Jul 28, 2024): 注意到这句话—— The model has a long context length (163840). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value. 但是我也想就此请教一下long context length为啥消耗显存那么多？
bsakaguchi commented 2026-10-26 21:07:28 +00:00
Author
Owner
Copy link

@beep-bebop commented on GitHub (Aug 6, 2024):

注意到这句话—— The model has a long context length (163840). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. 该模型的上下文长度很长（163840）。这可能会在初始内存分析阶段导致OOM错误，或者由于KV缓存空间较小而导致性能低下。 Consider setting --max-model-len to a smaller value. 考虑将--max-mode-len设置为较小的值。

但是我也想就此请教一下long context length为啥消耗显存那么多？

我猜是为long context做了显存的预分配，后续推理的时候显存不会变化太多。

<!-- gh-comment-id:2270235906 --> @beep-bebop commented on GitHub (Aug 6, 2024): > 注意到这句话—— The model has a long context length (163840). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. 该模型的上下文长度很长（163840）。这可能会在初始内存分析阶段导致OOM错误，或者由于KV缓存空间较小而导致性能低下。 Consider setting --max-model-len to a smaller value. 考虑将--max-mode-len设置为较小的值。 > > 但是我也想就此请教一下long context length为啥消耗显存那么多？ 我猜是为long context做了显存的预分配，后续推理的时候显存不会变化太多。
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-V2#70
No description provided.