mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #740] 请教为什么配置512G内存，1张L40s因内存不足运行不了DeepSeek-R1-Q4_K_M #540
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#540
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Longleaves on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/740
检查清单
问题描述
请教为什么配置512G内存，1张L40s 48G显卡，2个Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz会因内存不足而运行不了DeepSeek-R1-Q4_K_M，非常感谢！
复现步骤
numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --cpu_infer 30 --max_new_tokens 8192 --force_think true
环境信息
512G DRAM
48G VRAM
2*32核
@yeungtuzi commented on GitHub (Feb 28, 2025):
因为看不到你的内存配置，我建议你先把numactl这个去掉，有一种可能是你的每个CPU只接了256G内存导致内存不足（需要380G+）
@coelien commented on GitHub (Mar 4, 2025):
为什么需要380G+，怎么算的呢
@yeungtuzi commented on GitHub (Mar 4, 2025):
Q4模型的大小就是这么大
@better319 commented on GitHub (Mar 5, 2025):
能截个图，说明下怎么样可以禁用其中一个吗
@yeungtuzi commented on GitHub (Mar 9, 2025):
我理解是本案例的numactl会强制只能使用socket 0上面的cpu和内存，如果本socket上连接的内存不足以运行671b，会失败。去掉numactl之后，系统会自动尝试大致平均分配两个socket的内存，这样就够了。
I understand that in this case, numactl enforces the use of only the CPUs and memory on socket 0. If the memory connected to this socket is insufficient to run 671b, it will fail. After removing numactl, the system will automatically attempt to roughly evenly distribute memory across both sockets, which should be sufficient.
发件人: better319 @.>
发送时间: 2025年3月5日 22:01
收件人: kvcache-ai/ktransformers @.>
抄送: yeungtuzi @.>; Comment @.>
主题: Re: [kvcache-ai/ktransformers] 请教为什么配置512G内存，1张L40s因内存不足运行不了DeepSeek-R1-Q4_K_M (Issue #740)
能截个图，说明下怎么样可以禁用其中一个吗
―
Reply to this email directly, view it on GitHubhttps://github.com/kvcache-ai/ktransformers/issues/740#issuecomment-2701033462, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADLPPABBIMMXOUXCQC3LRJ32S37UNAVCNFSM6AAAAABYB3HQ7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBRGAZTGNBWGI.
You are receiving this because you commented.Message ID: @.***>
[better319]better319 left a comment (kvcache-ai/ktransformers#740)https://github.com/kvcache-ai/ktransformers/issues/740#issuecomment-2701033462
能截个图，说明下怎么样可以禁用其中一个吗
―
Reply to this email directly, view it on GitHubhttps://github.com/kvcache-ai/ktransformers/issues/740#issuecomment-2701033462, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADLPPABBIMMXOUXCQC3LRJ32S37UNAVCNFSM6AAAAABYB3HQ7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBRGAZTGNBWGI.
You are receiving this because you commented.Message ID: @.***>
@Longleaves commented on GitHub (Mar 17, 2025):
我去掉了numactl，并且用单socket的安装方法，可以正常用512G内存启动了，谢谢各位！