[GH-ISSUE #1403] 性能疑惑 #942

New issue

Open

opened 2026-10-29 14:46:33 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:46:33 +00:00

Owner

Originally created by @kmnns on GitHub (Jun 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1403

我目前使用的是两个intel xeon(R) Silver 4410Y（48核） + NVIDIA RTX 5000 +512GDDR5 ,目前按照教程安装完Deepseek-r1-q4-k-m后，预填充速度只有7-8tokens/s，在使用multi-gpu-4，测试也只有10tokens/s，解码速度最佳是维持在6.5tokens/s，请问这个速度符合我的工作站配置吗？还是需要做一些其他的设置操作？
我看到教程中描述双插槽的使用，需要安装libnuma-dev，且需要双倍的模型大小的内存。请问是这样才能最大化利用双插槽的CPU吗。如果在此基础上还需要启动AMX，需要下载Deepseek-R1-BF16 gguf，共1.3t，内存需要根据这个做出对应的更改吗

Originally created by @kmnns on GitHub (Jun 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1403 我目前使用的是两个intel xeon(R) Silver 4410Y（48核） + NVIDIA RTX 5000 +512GDDR5 ,目前按照教程安装完Deepseek-r1-q4-k-m后，预填充速度只有7-8tokens/s，在使用multi-gpu-4，测试也只有10tokens/s，解码速度最佳是维持在6.5tokens/s，请问这个速度符合我的工作站配置吗？还是需要做一些其他的设置操作？我看到教程中描述双插槽的使用，需要安装libnuma-dev，且需要双倍的模型大小的内存。请问是这样才能最大化利用双插槽的CPU吗。如果在此基础上还需要启动AMX，需要下载Deepseek-R1-BF16 gguf，共1.3t，内存需要根据这个做出对应的更改吗

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 27, 2025):

Your performance data looks okay.

NUMA can indeed help with performance, but only to a limited extent, and yes for now it requires memory that is at least double the size of the model.

@aubreyli commented on GitHub (Jun 27, 2025): Your performance data looks okay. NUMA can indeed help with performance, but only to a limited extent, and yes for now it requires memory that is at least double the size of the model.

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@kmnns commented on GitHub (Jun 27, 2025):

Your performance data looks okay.

NUMA can indeed help with performance, but only to a limited extent, and yes for now it requires memory that is at least double the size of the model.

文档中描述amx会对BF16 gguf的参数进行在线量化，那么如果我启用amx，在双插槽的情况下是需要至少2.6T还是之前q4版本双插槽两倍的内存也就2TB就足够了呢

@kmnns commented on GitHub (Jun 27, 2025): > Your performance data looks okay. > > NUMA can indeed help with performance, but only to a limited extent, and yes for now it requires memory that is at least double the size of the model. 文档中描述amx会对BF16 gguf的参数进行在线量化，那么如果我启用amx，在双插槽的情况下是需要至少2.6T还是之前q4版本双插槽两倍的内存也就2TB就足够了呢

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 30, 2025):

671B in FP8 requires around 700GB of memory. With NUMA enabled, the memory requirement goes up to around 1.5TB. Q4 cuts it in half, while BF16 doubles it, so BF16 would require more than 2.6TB.

@aubreyli commented on GitHub (Jun 30, 2025): 671B in FP8 requires around 700GB of memory. With NUMA enabled, the memory requirement goes up to around 1.5TB. Q4 cuts it in half, while BF16 doubles it, so BF16 would require more than 2.6TB.