kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #828] 【性能报告】Windows Server下双路9654+双路3090性能汇报 #596

New issue

Open

opened 2026-10-29 14:41:41 +00:00 by bsakaguchi · 12 comments

bsakaguchi commented

2026-10-29 14:41:41 +00:00

Owner

Originally created by @PC-DOS on GitHub (Mar 6, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/828

首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。

此讨论串已移动到Discussions 833。

硬件环境：

CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启）
RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条）
GPU：NVIDIA RTX 3090 x2
平台：ASUS RS720A-E12-RS12U-G

软件环境：

OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启）
显卡驱动：561.09
CUDA：CUDA Toolkit 12.6
Node.JS：v22.14.0。
CMake：4.0.0-rc2。
Ninja：v1.12.1。
受内存限制，编译时未开启NUMA支持。

模型：

GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b
启动命令行：python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190

AIDA64 V7.35.7000内存与缓存测试成绩：

内存读取：415.11 GB/s
内存写入：459.93 GB/s
内存拷贝：426.91 GB/s
内存延迟：159.2 ns

Ollama基线性能：

使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置num_gpu为4，卸载了4个层到GPU上。
从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。
性能参数：

response token/s: 2.49
prompt token/s: 18.26
total duration: 656354942000
load duration: 17123000
prompt eval count: 20
prompt eval duration: 1095000000
eval count: 1631
eval duration: 655241000000
approximate total: "0h10m56s"

KT性能：

提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。
推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。
跑了2次相同的提示词，均为5 tok/s的eval性能：

prompt eval count:    22 token(s)
prompt eval duration: 1.8837690353393555s
prompt eval rate:     11.67871410309957 tokens/s
eval count:           1523 token(s)
eval duration:        304.991397857666s
eval rate:            4.993583460707166 tokens/s

prompt eval count:    22 token(s)
prompt eval duration: 1.3836259841918945s
prompt eval rate:     15.900250682881675 tokens/s
eval count:           1706 token(s)
eval duration:        340.08750557899475s
eval rate:            5.016356002539864 tokens/s

将--cpu_infer改为80，相同提示词：

prompt eval count:    22 token(s)
prompt eval duration: 2.2636516094207764s
prompt eval rate:     9.718810044991582 tokens/s
eval count:           1698 token(s)
eval duration:        364.7031693458557s
eval rate:            4.655841085904441 tokens/s

对比了使用默认--optimize_config_path单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%：

prompt eval count:    22 token(s)
prompt eval duration: 1.6509404182434082s
prompt eval rate:     13.325738322772352 tokens/s
eval count:           2107 token(s)
eval duration:        394.9432945251465s
eval rate:            5.334943089825886 tokens/s

调为NPS1的性能：

AIDA64 V7.35.7000内存与缓存测试成绩：
- 内存读取：479.50 GB/s
- 内存写入：467.11 GB/s
- 内存拷贝：468.82 GB/s
- 内存延迟：113.3 ns
--cpu_infer设为190，使用默认--optimize_config_path单卡运行，相同提示词：

prompt eval count:    22 token(s)
prompt eval duration: 2.157524585723877s
prompt eval rate:     10.196871055640239 tokens/s
eval count:           1700 token(s)
eval duration:        331.0879154205322s
eval rate:            5.13458788684794 tokens/s

--cpu_infer设为180，使用默认--optimize_config_path单卡运行，相同提示词，观察到NUMA节点0占用约50%，NUMA节点1占用约10%：

prompt eval count:    22 token(s)
prompt eval duration: 1.8571727275848389s
prompt eval rate:     11.845963314683125 tokens/s
eval count:           1998 token(s)
eval duration:        391.88148856163025s
eval rate:            5.098480174027866 tokens/s

讨论：

单卡运行的性能略高于多卡，好像和先前在另一份issue（#610）中提到的结论类似。
和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。
感觉内存没插满带来的瓶颈还是蛮严重的。
同时估计Windows自身也给性能带来了一定的折损。
调为NPS1后，内存读取性能有一定的上升，但是整体eval速率依然在误差范围内。

Originally created by @PC-DOS on GitHub (Mar 6, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/828 首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。 _此讨论串已移动到Discussions 833。_ 硬件环境： * CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启） * RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条） * GPU：NVIDIA RTX 3090 x2 * 平台：ASUS RS720A-E12-RS12U-G 软件环境： * OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启） * 显卡驱动：561.09 * CUDA：CUDA Toolkit 12.6 * Node.JS：v22.14.0。 * CMake：4.0.0-rc2。 * Ninja：v1.12.1。 * 受内存限制，编译时未开启NUMA支持。模型： * GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b * 启动命令行：`python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190` AIDA64 V7.35.7000内存与缓存测试成绩： * 内存读取：415.11 GB/s * 内存写入：459.93 GB/s * 内存拷贝：426.91 GB/s * 内存延迟：159.2 ns Ollama基线性能： * 使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置`num_gpu`为`4`，卸载了4个层到GPU上。 * 从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 * 性能参数： ``` response token/s: 2.49 prompt token/s: 18.26 total duration: 656354942000 load duration: 17123000 prompt eval count: 20 prompt eval duration: 1095000000 eval count: 1631 eval duration: 655241000000 approximate total: "0h10m56s" ``` KT性能： * 提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 * 推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。 * 跑了2次相同的提示词，均为5 tok/s的eval性能： ``` prompt eval count: 22 token(s) prompt eval duration: 1.8837690353393555s prompt eval rate: 11.67871410309957 tokens/s eval count: 1523 token(s) eval duration: 304.991397857666s eval rate: 4.993583460707166 tokens/s prompt eval count: 22 token(s) prompt eval duration: 1.3836259841918945s prompt eval rate: 15.900250682881675 tokens/s eval count: 1706 token(s) eval duration: 340.08750557899475s eval rate: 5.016356002539864 tokens/s ``` * 将`--cpu_infer`改为`80`，相同提示词： ``` prompt eval count: 22 token(s) prompt eval duration: 2.2636516094207764s prompt eval rate: 9.718810044991582 tokens/s eval count: 1698 token(s) eval duration: 364.7031693458557s eval rate: 4.655841085904441 tokens/s ``` * 对比了使用默认`--optimize_config_path`单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%： ``` prompt eval count: 22 token(s) prompt eval duration: 1.6509404182434082s prompt eval rate: 13.325738322772352 tokens/s eval count: 2107 token(s) eval duration: 394.9432945251465s eval rate: 5.334943089825886 tokens/s ``` 调为NPS1的性能： * AIDA64 V7.35.7000内存与缓存测试成绩： * 内存读取：479.50 GB/s * 内存写入：467.11 GB/s * 内存拷贝：468.82 GB/s * 内存延迟：113.3 ns * `--cpu_infer`设为`190`，使用默认`--optimize_config_path`单卡运行，相同提示词： ``` prompt eval count: 22 token(s) prompt eval duration: 2.157524585723877s prompt eval rate: 10.196871055640239 tokens/s eval count: 1700 token(s) eval duration: 331.0879154205322s eval rate: 5.13458788684794 tokens/s ``` * `--cpu_infer`设为`180`，使用默认`--optimize_config_path`单卡运行，相同提示词，观察到NUMA节点0占用约50%，NUMA节点1占用约10%： ``` prompt eval count: 22 token(s) prompt eval duration: 1.8571727275848389s prompt eval rate: 11.845963314683125 tokens/s eval count: 1998 token(s) eval duration: 391.88148856163025s eval rate: 5.098480174027866 tokens/s ``` 讨论： * 单卡运行的性能略高于多卡，好像和先前在另一份issue（#610）中提到的结论类似。 * 和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。 * 感觉内存没插满带来的瓶颈还是蛮严重的。 * 同时估计Windows自身也给性能带来了一定的折损。 * 调为NPS1后，内存读取性能有一定的上升，但是整体eval速率依然在误差范围内。

bsakaguchi commented

2026-10-29 14:41:42 +00:00

Author

Owner

@Pb-207 commented on GitHub (Mar 6, 2025):

nps需要设成1，不然频繁的跨socket通信会严重限制速度。内存不插满也是原因之一

@Pb-207 commented on GitHub (Mar 6, 2025): nps需要设成1，不然频繁的跨socket通信会严重限制速度。内存不插满也是原因之一

bsakaguchi commented

2026-10-29 14:41:42 +00:00

Author

Owner

@luzamm commented on GitHub (Mar 6, 2025):

怎么比我单路7763还拉胯

@luzamm commented on GitHub (Mar 6, 2025): 怎么比我单路7763还拉胯

bsakaguchi commented

2026-10-29 14:41:42 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Mar 7, 2025):

内存不插满对计算带宽影响还挺大的……建议插满哈哈哈

@Azure-Tang commented on GitHub (Mar 7, 2025): 内存不插满对计算带宽影响还挺大的……建议插满哈哈哈

bsakaguchi commented

2026-10-29 14:41:42 +00:00

Author

Owner

@KMSorSMS commented on GitHub (Mar 7, 2025):

首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。

硬件环境：

CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启）

RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条）

GPU：NVIDIA RTX 3090 x2

平台：ASUS RS720A-E12-RS12U-G

软件环境：

OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启）

显卡驱动：561.09

CUDA：CUDA Toolkit 12.6

模型：

GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b

启动命令行：python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190

AIDA64 V7.35.7000内存与缓存测试成绩：

内存读取：415.11 GB/s

内存写入：459.93 GB/s

内存拷贝：426.91 GB/s

内存延迟：159.2 ns

Ollama基线性能：

使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置num_gpu为4，卸载了4个层到GPU上。

从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

性能参数：
response token/s: 2.49
prompt token/s: 18.26
total duration: 656354942000
load duration: 17123000
prompt eval count: 20
prompt eval duration: 1095000000
eval count: 1631
eval duration: 655241000000
approximate total: "0h10m56s"
KT性能：

提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。

跑了2次相同的提示词，均为5 tok/s的eval性能：
prompt eval count:    22 token(s)
prompt eval duration: 1.8837690353393555s
prompt eval rate:     11.67871410309957 tokens/s
eval count:           1523 token(s)
eval duration:        304.991397857666s
eval rate:            4.993583460707166 tokens/s

prompt eval count:    22 token(s)
prompt eval duration: 1.3836259841918945s
prompt eval rate:     15.900250682881675 tokens/s
eval count:           1706 token(s)
eval duration:        340.08750557899475s
eval rate:            5.016356002539864 tokens/s
将--cpu_infer改为80，相同提示词：
prompt eval count:    22 token(s)
prompt eval duration: 2.2636516094207764s
prompt eval rate:     9.718810044991582 tokens/s
eval count:           1698 token(s)
eval duration:        364.7031693458557s
eval rate:            4.655841085904441 tokens/s
对比了使用默认--optimize_config_path单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%：
prompt eval count:    22 token(s)
prompt eval duration: 1.6509404182434082s
prompt eval rate:     13.325738322772352 tokens/s
eval count:           2107 token(s)
eval duration:        394.9432945251465s
eval rate:            5.334943089825886 tokens/s
讨论：

单卡运行的性能略高于多卡，好像和先前在另一份issue（EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610）中提到的结论类似。

和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。

感觉内存没插满带来的瓶颈还是蛮严重的。

同时估计Windows自身也给性能带来了一定的折损。

We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it?
Btw, congratulations, and we appreciate your effort.

@KMSorSMS commented on GitHub (Mar 7, 2025): > 首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。 > > 硬件环境： > > * CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启） > * RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条） > * GPU：NVIDIA RTX 3090 x2 > * 平台：ASUS RS720A-E12-RS12U-G > > 软件环境： > > * OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启） > * 显卡驱动：561.09 > * CUDA：CUDA Toolkit 12.6 > > 模型： > > * GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b > * 启动命令行：`python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190` > > AIDA64 V7.35.7000内存与缓存测试成绩： > > * 内存读取：415.11 GB/s > * 内存写入：459.93 GB/s > * 内存拷贝：426.91 GB/s > * 内存延迟：159.2 ns > > Ollama基线性能： > > * 使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置`num_gpu`为`4`，卸载了4个层到GPU上。 > * 从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > * 性能参数： > > ``` > response token/s: 2.49 > prompt token/s: 18.26 > total duration: 656354942000 > load duration: 17123000 > prompt eval count: 20 > prompt eval duration: 1095000000 > eval count: 1631 > eval duration: 655241000000 > approximate total: "0h10m56s" > ``` > > KT性能： > > * 提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > * 推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。 > * 跑了2次相同的提示词，均为5 tok/s的eval性能： > > ``` > prompt eval count: 22 token(s) > prompt eval duration: 1.8837690353393555s > prompt eval rate: 11.67871410309957 tokens/s > eval count: 1523 token(s) > eval duration: 304.991397857666s > eval rate: 4.993583460707166 tokens/s > > prompt eval count: 22 token(s) > prompt eval duration: 1.3836259841918945s > prompt eval rate: 15.900250682881675 tokens/s > eval count: 1706 token(s) > eval duration: 340.08750557899475s > eval rate: 5.016356002539864 tokens/s > ``` > > * 将`--cpu_infer`改为`80`，相同提示词： > > ``` > prompt eval count: 22 token(s) > prompt eval duration: 2.2636516094207764s > prompt eval rate: 9.718810044991582 tokens/s > eval count: 1698 token(s) > eval duration: 364.7031693458557s > eval rate: 4.655841085904441 tokens/s > ``` > > * 对比了使用默认`--optimize_config_path`单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%： > > ``` > prompt eval count: 22 token(s) > prompt eval duration: 1.6509404182434082s > prompt eval rate: 13.325738322772352 tokens/s > eval count: 2107 token(s) > eval duration: 394.9432945251465s > eval rate: 5.334943089825886 tokens/s > ``` > > 讨论： > > * 单卡运行的性能略高于多卡，好像和先前在另一份issue（[EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610](https://github.com/kvcache-ai/ktransformers/issues/610)）中提到的结论类似。 > * 和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。 > * 感觉内存没插满带来的瓶颈还是蛮严重的。 > * 同时估计Windows自身也给性能带来了一定的折损。 We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it? Btw, congratulations, and we appreciate your effort.

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@PC-DOS commented on GitHub (Mar 7, 2025):

首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。
硬件环境：

CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启）

RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条）

GPU：NVIDIA RTX 3090 x2

平台：ASUS RS720A-E12-RS12U-G

软件环境：

OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启）

显卡驱动：561.09

CUDA：CUDA Toolkit 12.6

模型：

GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b

启动命令行：python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190

AIDA64 V7.35.7000内存与缓存测试成绩：

内存读取：415.11 GB/s

内存写入：459.93 GB/s

内存拷贝：426.91 GB/s

内存延迟：159.2 ns

Ollama基线性能：

使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置num_gpu为4，卸载了4个层到GPU上。

从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

性能参数：
response token/s: 2.49
prompt token/s: 18.26
total duration: 656354942000
load duration: 17123000
prompt eval count: 20
prompt eval duration: 1095000000
eval count: 1631
eval duration: 655241000000
approximate total: "0h10m56s"
KT性能：

提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。

跑了2次相同的提示词，均为5 tok/s的eval性能：
prompt eval count:    22 token(s)
prompt eval duration: 1.8837690353393555s
prompt eval rate:     11.67871410309957 tokens/s
eval count:           1523 token(s)
eval duration:        304.991397857666s
eval rate:            4.993583460707166 tokens/s

prompt eval count:    22 token(s)
prompt eval duration: 1.3836259841918945s
prompt eval rate:     15.900250682881675 tokens/s
eval count:           1706 token(s)
eval duration:        340.08750557899475s
eval rate:            5.016356002539864 tokens/s
将--cpu_infer改为80，相同提示词：
prompt eval count:    22 token(s)
prompt eval duration: 2.2636516094207764s
prompt eval rate:     9.718810044991582 tokens/s
eval count:           1698 token(s)
eval duration:        364.7031693458557s
eval rate:            4.655841085904441 tokens/s
对比了使用默认--optimize_config_path单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%：
prompt eval count:    22 token(s)
prompt eval duration: 1.6509404182434082s
prompt eval rate:     13.325738322772352 tokens/s
eval count:           2107 token(s)
eval duration:        394.9432945251465s
eval rate:            5.334943089825886 tokens/s
讨论：

单卡运行的性能略高于多卡，好像和先前在另一份issue（EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610）中提到的结论类似。

和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。

感觉内存没插满带来的瓶颈还是蛮严重的。

同时估计Windows自身也给性能带来了一定的折损。
We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it? Btw, congratulations, and we appreciate your effort.

Thank you for your kind suggestion. Sorry for any disturbance caused by my ignorance. Should I move this thread do Discussions section now?

@PC-DOS commented on GitHub (Mar 7, 2025): > > 首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。 > > 硬件环境： > > > > * CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启） > > * RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条） > > * GPU：NVIDIA RTX 3090 x2 > > * 平台：ASUS RS720A-E12-RS12U-G > > > > 软件环境： > > > > * OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启） > > * 显卡驱动：561.09 > > * CUDA：CUDA Toolkit 12.6 > > > > 模型： > > > > * GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b > > * 启动命令行：`python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190` > > > > AIDA64 V7.35.7000内存与缓存测试成绩： > > > > * 内存读取：415.11 GB/s > > * 内存写入：459.93 GB/s > > * 内存拷贝：426.91 GB/s > > * 内存延迟：159.2 ns > > > > Ollama基线性能： > > > > * 使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置`num_gpu`为`4`，卸载了4个层到GPU上。 > > * 从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > > * 性能参数： > > > > ``` > > response token/s: 2.49 > > prompt token/s: 18.26 > > total duration: 656354942000 > > load duration: 17123000 > > prompt eval count: 20 > > prompt eval duration: 1095000000 > > eval count: 1631 > > eval duration: 655241000000 > > approximate total: "0h10m56s" > > ``` > > > > > > > > > > > > > > > > > > > > > > > > KT性能： > > > > * 提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > > * 推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。 > > * 跑了2次相同的提示词，均为5 tok/s的eval性能： > > > > ``` > > prompt eval count: 22 token(s) > > prompt eval duration: 1.8837690353393555s > > prompt eval rate: 11.67871410309957 tokens/s > > eval count: 1523 token(s) > > eval duration: 304.991397857666s > > eval rate: 4.993583460707166 tokens/s > > > > prompt eval count: 22 token(s) > > prompt eval duration: 1.3836259841918945s > > prompt eval rate: 15.900250682881675 tokens/s > > eval count: 1706 token(s) > > eval duration: 340.08750557899475s > > eval rate: 5.016356002539864 tokens/s > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > * 将`--cpu_infer`改为`80`，相同提示词： > > > > ``` > > prompt eval count: 22 token(s) > > prompt eval duration: 2.2636516094207764s > > prompt eval rate: 9.718810044991582 tokens/s > > eval count: 1698 token(s) > > eval duration: 364.7031693458557s > > eval rate: 4.655841085904441 tokens/s > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > * 对比了使用默认`--optimize_config_path`单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%： > > > > ``` > > prompt eval count: 22 token(s) > > prompt eval duration: 1.6509404182434082s > > prompt eval rate: 13.325738322772352 tokens/s > > eval count: 2107 token(s) > > eval duration: 394.9432945251465s > > eval rate: 5.334943089825886 tokens/s > > ``` > > > > > > > > > > > > > > > > > > > > > > > > 讨论： > > > > * 单卡运行的性能略高于多卡，好像和先前在另一份issue（[EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610](https://github.com/kvcache-ai/ktransformers/issues/610)）中提到的结论类似。 > > * 和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。 > > * 感觉内存没插满带来的瓶颈还是蛮严重的。 > > * 同时估计Windows自身也给性能带来了一定的折损。 > > We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it? Btw, congratulations, and we appreciate your effort. Thank you for your kind suggestion. Sorry for any disturbance caused by my ignorance. Should I move this thread do Discussions section now?

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@KMSorSMS commented on GitHub (Mar 7, 2025):

首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。
硬件环境：

CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启）

RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条）

GPU：NVIDIA RTX 3090 x2

平台：ASUS RS720A-E12-RS12U-G

软件环境：

OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启）

显卡驱动：561.09

CUDA：CUDA Toolkit 12.6

模型：

GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b

启动命令行：python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190

AIDA64 V7.35.7000内存与缓存测试成绩：

内存读取：415.11 GB/s

内存写入：459.93 GB/s

内存拷贝：426.91 GB/s

内存延迟：159.2 ns

Ollama基线性能：

使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置num_gpu为4，卸载了4个层到GPU上。

从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

性能参数：
response token/s: 2.49
prompt token/s: 18.26
total duration: 656354942000
load duration: 17123000
prompt eval count: 20
prompt eval duration: 1095000000
eval count: 1631
eval duration: 655241000000
approximate total: "0h10m56s"
KT性能：

提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。

推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。

跑了2次相同的提示词，均为5 tok/s的eval性能：
prompt eval count:    22 token(s)
prompt eval duration: 1.8837690353393555s
prompt eval rate:     11.67871410309957 tokens/s
eval count:           1523 token(s)
eval duration:        304.991397857666s
eval rate:            4.993583460707166 tokens/s

prompt eval count:    22 token(s)
prompt eval duration: 1.3836259841918945s
prompt eval rate:     15.900250682881675 tokens/s
eval count:           1706 token(s)
eval duration:        340.08750557899475s
eval rate:            5.016356002539864 tokens/s
将--cpu_infer改为80，相同提示词：
prompt eval count:    22 token(s)
prompt eval duration: 2.2636516094207764s
prompt eval rate:     9.718810044991582 tokens/s
eval count:           1698 token(s)
eval duration:        364.7031693458557s
eval rate:            4.655841085904441 tokens/s
对比了使用默认--optimize_config_path单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%：
prompt eval count:    22 token(s)
prompt eval duration: 1.6509404182434082s
prompt eval rate:     13.325738322772352 tokens/s
eval count:           2107 token(s)
eval duration:        394.9432945251465s
eval rate:            5.334943089825886 tokens/s
讨论：

单卡运行的性能略高于多卡，好像和先前在另一份issue（EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610）中提到的结论类似。

和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。

感觉内存没插满带来的瓶颈还是蛮严重的。

同时估计Windows自身也给性能带来了一定的折损。
We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it? Btw, congratulations, and we appreciate your effort.
Thank you for your kind suggestion. Sorry for any disturbance caused by my ignorance. Should I move this thread do Discussions section now?

That's best, but you can leave the current discussion here. We will close it a few days later to remind others.

@KMSorSMS commented on GitHub (Mar 7, 2025): > > > 首先感谢KT框架的开发者和贡献者。昨天尝试在Windows上编译KT并加载Q4_K_M量化的DeepSeek-R1，交份成绩单。 > > > 硬件环境： > > > > > > * CPU：EPYC 9654 x2 （NPS0，cTDP、cPPT设为400W，SMT开启，AVX512开启） > > > * RAM：DDR5 RECC 32GB 4800 MT/s x16（每插槽8条） > > > * GPU：NVIDIA RTX 3090 x2 > > > * 平台：ASUS RS720A-E12-RS12U-G > > > > > > 软件环境： > > > > > > * OS：Windows Server 2022 Build 10.0.20348.2700（Hyper-V开启） > > > * 显卡驱动：561.09 > > > * CUDA：CUDA Toolkit 12.6 > > > > > > 模型： > > > > > > * GGUF：来自Ollama仓库的Q4量化版本DeepSeek R1 671b > > > * 启动命令行：`python -m ktransformers.local_chat --model_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b --gguf_path E:/LLM-Models/DeepSeek-AI/DeepSeek-R1-671b/DeepSeek-R1-671b-Q4_K_M/ --optimize_config_path E:/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml --max_new_tokens 8192 --cpu_infer 190` > > > > > > AIDA64 V7.35.7000内存与缓存测试成绩： > > > > > > * 内存读取：415.11 GB/s > > > * 内存写入：459.93 GB/s > > > * 内存拷贝：426.91 GB/s > > > * 内存延迟：159.2 ns > > > > > > Ollama基线性能： > > > > > > * 使用Ollama 0.5.7版本加载DeepSeek R1 671b，设置`num_gpu`为`4`，卸载了4个层到GPU上。 > > > * 从Open WebUI调用，提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > > > * 性能参数： > > > > > > ``` > > > response token/s: 2.49 > > > prompt token/s: 18.26 > > > total duration: 656354942000 > > > load duration: 17123000 > > > prompt eval count: 20 > > > prompt eval duration: 1095000000 > > > eval count: 1631 > > > eval duration: 655241000000 > > > approximate total: "0h10m56s" > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > KT性能： > > > > > > * 提示词为“请为我比较AMD EPYC 9654和Intel Xeon 8490H”。 > > > * 推理时，CPU占用71%，内存占用395 GB，GPU0显存占用5.6 GB，GPU1显存占用6.4 GB。 > > > * 跑了2次相同的提示词，均为5 tok/s的eval性能： > > > > > > ``` > > > prompt eval count: 22 token(s) > > > prompt eval duration: 1.8837690353393555s > > > prompt eval rate: 11.67871410309957 tokens/s > > > eval count: 1523 token(s) > > > eval duration: 304.991397857666s > > > eval rate: 4.993583460707166 tokens/s > > > > > > prompt eval count: 22 token(s) > > > prompt eval duration: 1.3836259841918945s > > > prompt eval rate: 15.900250682881675 tokens/s > > > eval count: 1706 token(s) > > > eval duration: 340.08750557899475s > > > eval rate: 5.016356002539864 tokens/s > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * 将`--cpu_infer`改为`80`，相同提示词： > > > > > > ``` > > > prompt eval count: 22 token(s) > > > prompt eval duration: 2.2636516094207764s > > > prompt eval rate: 9.718810044991582 tokens/s > > > eval count: 1698 token(s) > > > eval duration: 364.7031693458557s > > > eval rate: 4.655841085904441 tokens/s > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * 对比了使用默认`--optimize_config_path`单卡运行（调用时不设置该参数）时的性能，相同提示词，单卡运行时系统内存占用394 GB，GPU0显存占用10.8 GB，CPU负载73%，GPU0负载100%： > > > > > > ``` > > > prompt eval count: 22 token(s) > > > prompt eval duration: 1.6509404182434082s > > > prompt eval rate: 13.325738322772352 tokens/s > > > eval count: 2107 token(s) > > > eval duration: 394.9432945251465s > > > eval rate: 5.334943089825886 tokens/s > > > ``` > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 讨论： > > > > > > * 单卡运行的性能略高于多卡，好像和先前在另一份issue（[EPYC 7601 * 2 + 512G ddr4 + 双七彩虹火神3090测试完成，5tokens #610](https://github.com/kvcache-ai/ktransformers/issues/610)）中提到的结论类似。 > > > * 和先前其它dalao的分数相比，属于是最差的一届9654（大雾）。 > > > * 感觉内存没插满带来的瓶颈还是蛮严重的。 > > > * 同时估计Windows自身也给性能带来了一定的折损。 > > > > > > We recommend that this kind of report be included in the discussion section. Next time, could you pin this report on it? Btw, congratulations, and we appreciate your effort. > > Thank you for your kind suggestion. Sorry for any disturbance caused by my ignorance. Should I move this thread do Discussions section now? That's best, but you can leave the current discussion here. We will close it a few days later to remind others.

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@PC-DOS commented on GitHub (Mar 7, 2025):

[Redacted for shorter thread]

That's best, but you can leave the current discussion here. We will close it a few days later to remind others.

Moved, thank you for supporting. ;-)

@PC-DOS commented on GitHub (Mar 7, 2025): > [Redacted for shorter thread] > > That's best, but you can leave the current discussion here. We will close it a few days later to remind others. Moved, thank you for supporting. ;-)

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@bluesea117 commented on GitHub (Mar 10, 2025):

是在wsl2下运行的吗？为什么我的cpu_infer = 96 但实际只能到64.baidu了一下说是Hyper-V最大的vCPU 是64. 我的是win11环境。

@bluesea117 commented on GitHub (Mar 10, 2025): 是在wsl2下运行的吗？为什么我的cpu_infer = 96 但实际只能到64.baidu了一下说是Hyper-V最大的vCPU 是64. 我的是win11环境。

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@ubergarm commented on GitHub (Mar 10, 2025):

@PC-DOS

若使用仅512GB内存的Q4模型，两个CPU插槽将共享单个NUMA节点。此时速度会有所下降，但仅需377GiB即可加载权重。

若增加足够内存以支持数据并行（通过USE_NUMA=1将权重加载到内存两次），则可设置为前文提到的NPS1以获得更好性能。理想情况下，每个提交issue的用户都应提供numactl --hardware --cpu-compress的输出结果。

遗憾的是，3090显卡过旧，不支持原生fp8。需使用≥9.0计算能力的显卡（如4090）。否则，我会建议尝试混合fp8/q4量化方案，在相速度下获得最佳输出质量。

# 3090 , A6000
$ nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6

为追求极致速度，可尝试UD-Q2_K_XL量化版，配合NPS1和USE_NUMA=1（需确保apt-get install libnuma），该版本可适配有限内存。其对多数应用场景仍保持高质量输出。更信息可参考此非官方指南，了解如何通过API端点实现更高效交互。

PC-DOS

You are correct to use NPS0 to present a single NUMA node across both CPU sockets if using the Q4 with only 512 GB RAM. You will take a penalty in speed but only need 377GiB to load the weights.

If you add enough RAM to support data parallel by loading the weights into memory twice with USE_NUMA=1, then you can set to NPS1 as mentioned for better performance. Ideally everyone who makes an issue will provide the output of numactl --hardware --cpu-compress.

Unfortunately 3090 is too old to support native fp8. You need >= 9.0 e.g. 4090. Otherwise I would suggest you try the hybrid fp8/q4 quant for best quality output at similar speeds.

# 3090 , A6000
$ nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6

For best speed you could try the UD-Q2_K_XL with NPS1 and USE_NUMA=1 (make sure to apt-get install libnuma) which will fit in your limited RAM size. It is still very good quality for many applications. Feel free to check this unofficial guide for more information and how to run API endpoint for much more useful ineterface.

@ubergarm commented on GitHub (Mar 10, 2025): @PC-DOS 若使用仅512GB内存的Q4模型，**两个CPU插槽将共享单个NUMA节点**。此时速度会有所下降，但仅需377GiB即可加载权重。若增加足够内存以支持数据并行（通过`USE_NUMA=1`将权重加载到内存**两次**），则可设置为前文提到的`NPS1`以获得更好性能。理想情况下，每个提交issue的用户都应提供`numactl --hardware --cpu-compress`的输出结果。遗憾的是，3090显卡过旧，不支持原生`fp8`。需使用≥9.0计算能力的显卡（如4090）。否则，我会建议尝试混合`fp8/q4`量化方案，在相速度下获得最佳输出质量。 ``` # 3090 , A6000 $ nvidia-smi --query-gpu=compute_cap --format=csv compute_cap 8.6 ``` 为追求极致速度，可尝试[UD-Q2_K_XL量化版](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL)，配合`NPS1`和`USE_NUMA=1`（需确保`apt-get install libnuma`），该版本可适配有限内存。其对多数应用场景仍保持高质量输出。更信息可参考[此非官方指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)，了解如何通过API端点实现更高效交互。 --- PC-DOS You are correct to use `NPS0` to present a single NUMA node across both CPU sockets if using the Q4 with only 512 GB RAM. You will take a penalty in speed but only need 377GiB to load the weights. If you add enough RAM to support data parallel by loading the weights into memory *twice* with `USE_NUMA=1`, then you can set to `NPS1` as mentioned for better performance. Ideally everyone who makes an issue will provide the output of `numactl --hardware --cpu-compress`. Unfortunately 3090 is too old to support native `fp8`. You need >= 9.0 e.g. 4090. Otherwise I would suggest you try the hybrid fp8/q4 quant for best quality output at similar speeds. ``` # 3090 , A6000 $ nvidia-smi --query-gpu=compute_cap --format=csv compute_cap 8.6 ``` For best speed you could try the [UD-Q2_K_XL](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL) with `NPS1` and `USE_NUMA=1` (make sure to `apt-get install libnuma`) which will fit in your limited RAM size. It is still very good quality for many applications. Feel free to check [this unofficial guide for more information](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md) and how to run API endpoint for much more useful ineterface.

bsakaguchi commented

2026-10-29 14:41:43 +00:00

Author

Owner

@LyciumRuthenicum commented on GitHub (Mar 12, 2025):

@PC-DOS

若使用仅512GB内存的Q4模型，两个CPU插槽将共享单个NUMA节点。此时速度会有所下降，但仅需377GiB即可加载权重。

若增加足够内存以支持数据并行（通过USE_NUMA=1将权重加载到内存两次），则可设置为前文提到的NPS1以获得更好性能。理想情况下，每个提交issue的用户都应提供numactl --hardware --cpu-compress的输出结果。

遗憾的是，3090显卡过旧，不支持原生fp8。需使用≥9.0计算能力的显卡（如4090）。否则，我会建议尝试混合fp8/q4量化方案，在相速度下获得最佳输出质量。
# 3090 , A6000
$ nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6
为追求极致速度，可尝试UD-Q2_K_XL量化版，配合NPS1和USE_NUMA=1（需确保apt-get install libnuma），该版本可适配有限内存。其对多数应用场景仍保持高质量输出。更信息可参考此非官方指南，了解如何通过API端点实现更高效交互。

PC-DOS

You are correct to use NPS0 to present a single NUMA node across both CPU sockets if using the Q4 with only 512 GB RAM. You will take a penalty in speed but only need 377GiB to load the weights.

If you add enough RAM to support data parallel by loading the weights into memory twice with USE_NUMA=1, then you can set to NPS1 as mentioned for better performance. Ideally everyone who makes an issue will provide the output of numactl --hardware --cpu-compress.

Unfortunately 3090 is too old to support native fp8. You need >= 9.0 e.g. 4090. Otherwise I would suggest you try the hybrid fp8/q4 quant for best quality output at similar speeds.
# 3090 , A6000
$ nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6
For best speed you could try the UD-Q2_K_XL with NPS1 and USE_NUMA=1 (make sure to apt-get install libnuma) which will fit in your limited RAM size. It is still very good quality for many applications. Feel free to check this unofficial guide for more information and how to run API endpoint for much more useful ineterface.

@ubergarm @PC-DOS

您的建议非常有帮助，但我注意到你的建议在关于numa节点的优化上，是基于Linux系统。而由于一些必要因素，我和@PC-DOS 一样需要在Windows(WindowsServer)平台上使用Ktransformer架构运行DeepSeek。
我的硬件平台是双路Intel Xeon 5418Y + 16通道 ddr5 4400MT/s 64GB(共1024GB) + 4090 24GB。在Bios中设置了SNC=Disable以实现nps=1的效果，并且在编译时环境变量传入了USE_NUMA=1，但WindowsServer 2019 Standard系统上运行1.58bit量化版本的DeepSeek 671B大模型时，并没有看到内存镜像的现象，仅占用了150GB左右内存，和10GB左右显存。这与在没有开启USE_NUMA=1时，没有任何变化。
因此我当前有两个严重问题无法解决：
1. 在Windows(WindowsServer)平台上，如何真正实现USE_NUMA=1配置，是否也需要类似Linux中apt-get install libnuma这样的操作？实现触发镜像内存，以优化推理速度；
2. 是否有办法将更多层加载到GPU中？以提升推理速度？

我知道大部分人是在Linux环境下运行，Windows平台可参考经验不多，但还是希望您能有一些参考资料或者建议给我。
期待您的回复，谢谢！

Your suggestions are very helpful. However, I've noticed that your suggestions regarding the optimization of NUMA nodes are based on the Linux system. Due to some necessary factors, like @PC-DOS , I need to run DeepSeek using the Ktransformer architecture on the Windows (Windows Server) platform.

My hardware platform consists of dual Intel Xeon 5418Y, 16 channel DDR5 4400MT/s 64GB (1024GB in total), and a 4090 with 24GB. In the BIOS, I set SNC = Disable to achieve the effect of nps = 1, and I passed the environment variable USE_NUMA = 1 during compilation. But when running the 1.58 - bit quantized version of the DeepSeek 671B large - model on the Windows Server 2019 Standard system, I didn't observe the phenomenon of memory mirroring. It only occupied about 150GB of memory and around 10GB of video memory, which is no different from when USE_NUMA = 1 was not enabled.

Therefore, I currently have two serious problems that I can't solve:

1. On the Windows (Windows Server) platform, how to truly implement the USE_NUMA=1 configuration? Is an operation On Windows similar to apt-get install libnuma in Linux required? To activate memory interleaving for optimizing inference performance.
2. Is there a way to load more layers into the GPU to improve the inference speed?

I know that most people run such models in a Linux environment, and there isn't much reference experience for the Windows platform. Still, I hope you can provide me with some reference materials or suggestions.
I'm looking forward to your reply. Thank you!

@LyciumRuthenicum commented on GitHub (Mar 12, 2025): > [@PC-DOS](https://github.com/PC-DOS) > > 若使用仅512GB内存的Q4模型，**两个CPU插槽将共享单个NUMA节点**。此时速度会有所下降，但仅需377GiB即可加载权重。 > > 若增加足够内存以支持数据并行（通过`USE_NUMA=1`将权重加载到内存**两次**），则可设置为前文提到的`NPS1`以获得更好性能。理想情况下，每个提交issue的用户都应提供`numactl --hardware --cpu-compress`的输出结果。 > > 遗憾的是，3090显卡过旧，不支持原生`fp8`。需使用≥9.0计算能力的显卡（如4090）。否则，我会建议尝试混合`fp8/q4`量化方案，在相速度下获得最佳输出质量。 > > ``` > # 3090 , A6000 > $ nvidia-smi --query-gpu=compute_cap --format=csv > compute_cap > 8.6 > ``` > > 为追求极致速度，可尝试[UD-Q2_K_XL量化版](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL)，配合`NPS1`和`USE_NUMA=1`（需确保`apt-get install libnuma`），该版本可适配有限内存。其对多数应用场景仍保持高质量输出。更信息可参考[此非官方指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)，了解如何通过API端点实现更高效交互。 > > PC-DOS > > You are correct to use `NPS0` to present a single NUMA node across both CPU sockets if using the Q4 with only 512 GB RAM. You will take a penalty in speed but only need 377GiB to load the weights. > > If you add enough RAM to support data parallel by loading the weights into memory _twice_ with `USE_NUMA=1`, then you can set to `NPS1` as mentioned for better performance. Ideally everyone who makes an issue will provide the output of `numactl --hardware --cpu-compress`. > > Unfortunately 3090 is too old to support native `fp8`. You need >= 9.0 e.g. 4090. Otherwise I would suggest you try the hybrid fp8/q4 quant for best quality output at similar speeds. > > ``` > # 3090 , A6000 > $ nvidia-smi --query-gpu=compute_cap --format=csv > compute_cap > 8.6 > ``` > > For best speed you could try the [UD-Q2_K_XL](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-Q2_K_XL) with `NPS1` and `USE_NUMA=1` (make sure to `apt-get install libnuma`) which will fit in your limited RAM size. It is still very good quality for many applications. Feel free to check [this unofficial guide for more information](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md) and how to run API endpoint for much more useful ineterface. @ubergarm @PC-DOS 您的建议非常有帮助，但我注意到你的建议在关于numa节点的优化上，是基于Linux系统。而由于一些必要因素，我和@PC-DOS 一样需要在Windows(WindowsServer)平台上使用Ktransformer架构运行DeepSeek。我的硬件平台是双路Intel Xeon 5418Y + 16通道 ddr5 4400MT/s 64GB(共1024GB) + 4090 24GB。在Bios中设置了`SNC=Disable`以实现`nps=1`的效果，并且在编译时环境变量传入了`USE_NUMA=1`，但WindowsServer 2019 Standard系统上运行1.58bit量化版本的DeepSeek 671B大模型时，并没有看到内存镜像的现象，仅占用了150GB左右内存，和10GB左右显存。这与在没有开启`USE_NUMA=1`时，没有任何变化。因此我当前有两个严重问题无法解决： **1. 在Windows(WindowsServer)平台上，如何真正实现USE_NUMA=1配置，是否也需要类似Linux中`apt-get install libnuma`这样的操作？实现触发镜像内存，以优化推理速度； 2. 是否有办法将更多层加载到GPU中？以提升推理速度？** 我知道大部分人是在Linux环境下运行，Windows平台可参考经验不多，但还是希望您能有一些参考资料或者建议给我。期待您的回复，谢谢！ Your suggestions are very helpful. However, I've noticed that your suggestions regarding the optimization of NUMA nodes are based on the Linux system. Due to some necessary factors, like @PC-DOS , I need to run DeepSeek using the Ktransformer architecture on the Windows (Windows Server) platform. My hardware platform consists of dual Intel Xeon 5418Y, 16 channel DDR5 4400MT/s 64GB (1024GB in total), and a 4090 with 24GB. In the BIOS, I set `SNC = Disable` to achieve the effect of `nps = 1`, and I passed the environment variable `USE_NUMA = 1` during compilation. But when running the 1.58 - bit quantized version of the DeepSeek 671B large - model on the Windows Server 2019 Standard system, I didn't observe the phenomenon of memory mirroring. It only occupied about 150GB of memory and around 10GB of video memory, which is no different from when` USE_NUMA = 1` was not enabled. Therefore, I currently have two serious problems that I can't solve: **1. On the Windows (Windows Server) platform, how to truly implement the `USE_NUMA=1` configuration? Is an operation On Windows similar to `apt-get install libnuma` in Linux required? To activate memory interleaving for optimizing inference performance. 2. Is there a way to load more layers into the GPU to improve the inference speed?** I know that most people run such models in a Linux environment, and there isn't much reference experience for the Windows platform. Still, I hope you can provide me with some reference materials or suggestions. I'm looking forward to your reply. Thank you!

bsakaguchi commented

2026-10-29 14:41:44 +00:00

Author

Owner

@PC-DOS commented on GitHub (Mar 12, 2025):

是在wsl2下运行的吗？为什么我的cpu_infer = 96 但实际只能到64.baidu了一下说是Hyper-V最大的vCPU 是64. 我的是win11环境。

我没有使用WSL，是在Windows环境下直接编译运行的

@PC-DOS commented on GitHub (Mar 12, 2025): > 是在wsl2下运行的吗？为什么我的cpu_infer = 96 但实际只能到64.baidu了一下说是Hyper-V最大的vCPU 是64. 我的是win11环境。我没有使用WSL，是在Windows环境下直接编译运行的

bsakaguchi commented

2026-10-29 14:41:44 +00:00

Author

Owner

@ubergarm commented on GitHub (Mar 12, 2025):

@LyciumRuthenicum

注意 AMD Epyc 处理器在 PC-DOS 环境下支持 NPS0，可将 两个 CPU 插槽 的 NUMA 结构合并为 *单个 NUMA 节点。这使得无需额外设置 USE_NUMA=1，因此在 Windows 环境下不会出现 NUMA 相关问题。

据我所知，Intel Xeon 处理器无法实现这一点，因为即使设置 SNC=Disable，每个 CPU 插槽仍会保持 独立的 NUMA 点。虽然我不使用 Windows，但建议你尝试 WSL（Windows Subsystem for Linux） 或寻找 Windows 环境下的等效工具（例如 Linux 中的 numactl 和 numastat）来查看 NUMA 配置。

是否有办法将更多层加载到GPU中？以提升推理速度？

不，除非你拥有 4 块 H100 显卡 或 疯狂的显存数量，否则无法获得性能提升。其他讨论线程也提到过这一点，总结来说：若要通过卸载更多 MoE 层来提升性能，必须禁用 CUDA 图（CUDA graphs）。然而，这会导致比启用 CUDA 图更高的性能损耗。你可以参考我指南中的基准测试和讨论获取更多细节

LyciumRuthenicum

Note that PC-DOS is running an AMD Epyc which supports NPS0 to present a single NUMA node across both CPU sockets. This removes the need to worry about USE_NUMA=1 so no issue in Windows.

This is not possible on Intel Xeon as far as I know because the best we can do is SNC=Disable which still gives one NUMA node per CPU socket. I do not use windows but would suggest you look into WSL or find some Windows equivalent tools to view numa configuration e.g. numactl and numastat in Linux.

Is there a way to load more layers into the GPU to improve the inference speed?

No, unless you have 4x H100 or some crazy amount of VRAM there is no gain. There are other threads about this, but tl;dr; you have to disable cuda graphs to offload more MoE layers which is a bigger penalty than leaving it as is with cuda graphs enabled. You can see the benchmarks and discussion in my guide here

@ubergarm commented on GitHub (Mar 12, 2025): @LyciumRuthenicum 注意 AMD Epyc 处理器在 PC-DOS 环境下支持 `NPS0`，可将 **两个 CPU 插槽** 的 NUMA 结构合并为 **单个 NUMA 节点*。这使得无需额外设置 `USE_NUMA=1`，因此在 Windows 环境下不会出现 NUMA 相关问题。据我所知，Intel Xeon 处理器无法实现这一点，因为即使设置 `SNC=Disable`，每个 CPU 插槽仍会保持 **独立的 NUMA 点**。虽然我不使用 Windows，但建议你尝试 **WSL（Windows Subsystem for Linux）** 或寻找 Windows 环境下的等效工具（例如 Linux 中的 `numactl` 和 `numastat`）来查看 NUMA 配置。 > 2. 是否有办法将更多层加载到GPU中？以提升推理速度？不，除非你拥有 **4 块 H100 显卡** 或 **疯狂的显存数量**，否则无法获得性能提升。其他讨论线程也提到过这一点，总结来说：若要通过卸载更多 MoE 层来提升性能，必须禁用 **CUDA 图（CUDA graphs）**。然而，这会导致比启用 CUDA 图更高的性能损耗。你可以参考我指南中的 [基准测试和讨论](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md#cuda-graphs) 获取更多细节 --- LyciumRuthenicum Note that PC-DOS is running an AMD Epyc which supports `NPS0` to present a single NUMA node across *both* CPU sockets. This removes the need to worry about `USE_NUMA=1` so no issue in Windows. This is not possible on Intel Xeon as far as I know because the best we can do is `SNC=Disable` which still gives one NUMA node *per* CPU socket. I do not use windows but would suggest you look into WSL or find some Windows equivalent tools to view numa configuration e.g. `numactl` and `numastat` in Linux. > Is there a way to load more layers into the GPU to improve the inference speed? No, unless you have 4x H100 or some crazy amount of VRAM there is no gain. There are other threads about this, but tl;dr; you have to disable cuda graphs to offload more MoE layers which is a bigger penalty than leaving it as is with cuda graphs enabled. You can see the [benchmarks and discussion in my guide here](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md#cuda-graphs)