[GH-ISSUE #641] 使用的是A800的服务器，CPU占用很高，但是内存占用才几个G，如下

bsakaguchi commented

2026-10-29 14:40:13 +00:00

Owner

Originally created by @White-Friday on GitHub (Feb 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/641

执行的命令为：
python ./ktransformers/local_chat.py --model_path /data/models/DeepSeek-R1-Q2/ --gguf_path /data/models/DeepSeek-R1-Q2/ --prompt_file 'hello' --cpu_infer 48 --max_new_tokens 4096

Originally created by @White-Friday on GitHub (Feb 24, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/641 执行的命令为： python ./ktransformers/local_chat.py --model_path /data/models/DeepSeek-R1-Q2/ --gguf_path /data/models/DeepSeek-R1-Q2/ --prompt_file 'hello' --cpu_infer 48 --max_new_tokens 4096

bsakaguchi

2026-10-29 14:40:13 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@NicholasIrving commented on GitHub (Feb 24, 2025):

我也有相同的问题，gpu很空闲，cpu占用却很高，我的是h100

@NicholasIrving commented on GitHub (Feb 24, 2025): 我也有相同的问题，gpu很空闲，cpu占用却很高，我的是h100

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 24, 2025):

尊敬的各位拥有更多显存的同仁：

在我看来，KTransformers的优势在于能够借助高内存带宽的系统和单块24GB显存的GPU运行如R1这样的大型混合专家模型（MoE）。如果您查阅我的深度技术指南以及官方文档中的基准测试，可以发现：若将更多专家模块卸载至显存，则需禁用CUDA图（CUDA Graphs），这会显著降低性能。仅当需要处理更长上下文时，才需超过24GB的显存。

若您拥有足够容纳完整模型的显存，则可考虑使用vllm、exllamav2、tabby-api、aphrodite-engine等其他推理引擎。然而，我们大多数人无法访问超过671GB的显存来运行完整的fp8模型。

因此，核心策略是在系统内存带宽与CPU性能之间寻求平衡。通常而言，许多常见配置的瓶颈在于内存带宽。

根据我的实测，单块RTX 6000 48GB显存GPU的利用率约为50%，且仅占用14GB显存。

Dear gentlemen who have access to more VRAM than I do,

In my opinion the strength of KTransformers is to run a very large MoE like R1 on a computer with high RAM bandwidth and a single 24GB VRAM GPU. If you see the benchmarks in my in depth technical guide as well as the official documentation, the penalty of offloading more expert blocks into VRAM requires to disable CUDA graphs which will destroy performance. You only need more than 24GB VRAM if you want longer context length.

If you have enough VRAM to fit the entire model, then it makes sense to use other inference engines like vllm, exllamav2, tabby-api, aphrodite-engine, etc. However, most of us do not have access to >671GB VRAM to run the full sized fp8 model.

So the strategy is to find the right balance of system memory bandwidth and CPU performance. In general the limitation is memory bandwidth for many common configurations.

In my own testing, a single RTX 6000 48GB VRAM GPU will hit about 50% utilization and only using 14GB VRAM. I don't bother using the second GPU with ktransformers.

@ubergarm commented on GitHub (Feb 24, 2025): 尊敬的各位拥有更多显存的同仁：在我看来，KTransformers的优势在于能够借助高内存带宽的系统和单块24GB显存的GPU运行如R1这样的大型混合专家模型（MoE）。如果您查阅我的[深度技术指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md#cuda-graphs)以及官方文档中的基准测试，可以发现：若将更多专家模块卸载至显存，则需禁用CUDA图（CUDA Graphs），这会显著降低性能。仅当需要处理更长上下文时，才需超过24GB的显存。若您拥有足够容纳*完整模型*的显存，则可考虑使用vllm、exllamav2、tabby-api、aphrodite-engine等其他推理引擎。然而，我们大多数人无法访问超过671GB的显存来运行完整的fp8模型。因此，核心策略是在系统内存带宽与CPU性能之间寻求平衡。通常而言，许多常见配置的瓶颈在于内存带宽。根据我的实测，单块RTX 6000 48GB显存GPU的利用率约为50%，且仅占用14GB显存。 --- Dear gentlemen who have access to more VRAM than I do, In my opinion the strength of KTransformers is to run a very large MoE like R1 on a computer with high RAM bandwidth and a single 24GB VRAM GPU. If you see the benchmarks in my [in depth technical guide](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md#cuda-graphs) as well as the official documentation, the penalty of offloading more expert blocks into VRAM requires to disable CUDA graphs which will destroy performance. You only need more than 24GB VRAM if you want longer context length. If you have enough VRAM to fit the *entire model*, then it makes sense to use other inference engines like vllm, exllamav2, tabby-api, aphrodite-engine, etc. However, most of us do not have access to >671GB VRAM to run the full sized fp8 model. So the strategy is to find the right balance of system memory bandwidth and CPU performance. In general the limitation is memory bandwidth for many common configurations. In my own testing, a single RTX 6000 48GB VRAM GPU will hit about 50% utilization and only using 14GB VRAM. I don't bother using the second GPU with ktransformers.

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@NicholasIrving commented on GitHub (Feb 25, 2025):

@ubergarm 那么能否出一个关于苹果的安装教程？苹果拥有更好的内存带宽和还算不错的gpu，能更低成本低运行671b的模型

@NicholasIrving commented on GitHub (Feb 25, 2025): @ubergarm 那么能否出一个关于苹果的安装教程？苹果拥有更好的内存带宽和还算不错的gpu，能更低成本低运行671b的模型

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 25, 2025):

@NicholasIrving

那么能否出一个关于苹果的安装教程？苹果拥有更好的内存带宽和还算不错的gpu，能更低成本低运行671b的模型

据我所知，ktransformers目前仅适配NVIDIA CUDA架构，社区有若干针对Intel XPU和MUSA后端的待审合并请求（PR）。尚未见到对苹果Metal后端的支持——或许"Metal"是史蒂夫·乔布斯时代对统一内存设备（Unified RAM/VRAM）的命名延续？

若需在Metal后端运行模型，建议：

参考llama.cpp Metal版本构建指南
或自行发起PR为ktransformers添加Metal支持

尽管NVIDIA DIGITS、苹果M3 MAX、AMD Ryzen AI MAX等统一内存架构在成本效益上颇具潜力，但现有方案最大内存容量普遍停留在128GB级别。这仍不足以完全承载2.51bpw R1量化版本（需212GB内存）。除非厂商推出256GB+内存规格，否则此类设备尚难满足大模型全参卸载需求。

目前您的最佳选择仍然是配置256GB以上的DDR4/5内存。

NicholasIrving

Can we create an installation guide for Apple devices? Apple silicon offers superior memory bandwidth and decent GPU capabilities, enabling cost-effective execution of 671B parameter models.

As far as I know, ktransformers only works with nvidia CUDA and there are some PRs open about Intel XPU and MUSA backend. I don't see any backend support for metal which maybe is the steve jobs macintosh name for unified RAM/VRAM devices?

For now if you want to run on metal backend look at llama.cpp metal build or possibly start your own PR to add support to ktransformers.

I agree the various unified memory devices like nvidia DIGITS and macintosh M3 MAX and AMD RYZEN AI MAX promise a good cost to performance ratio, however none of them seem to offer more than around 128GB RAM which is still not enough for fully offloading even 2.51bpw R1 quants (212GB). Unless they offer a 256GB+ model device your best bet today is still stacking DDR4/5 over 256GB.

@ubergarm commented on GitHub (Feb 25, 2025): @NicholasIrving > 那么能否出一个关于苹果的安装教程？苹果拥有更好的内存带宽和还算不错的gpu，能更低成本低运行671b的模型据我所知，ktransformers目前仅适配*NVIDIA CUDA*架构，社区有若干针对Intel XPU和MUSA后端的待审合并请求（PR）。尚未见到对苹果`Metal`后端的支持——或许"Metal"是史蒂夫·乔布斯时代对统一内存设备（Unified RAM/VRAM）的命名延续？若需在`Metal`后端运行模型，建议： 1. 参考[llama.cpp Metal版本构建指南](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#metal-build) 2. 或自行发起PR为ktransformers添加Metal支持尽管NVIDIA DIGITS、苹果M3 MAX、AMD Ryzen AI MAX等统一内存架构在成本效益上颇具潜力，但现有方案最大内存容量普遍停留在128GB级别。这仍不足以完全承载2.51bpw R1量化版本（需212GB内存）。除非厂商推出256GB+内存规格，否则此类设备尚难满足大模型全参卸载需求。目前您的最佳选择仍然是配置256GB以上的DDR4/5内存。 --- NicholasIrving > **Can we create an installation guide for Apple devices?** Apple silicon offers superior memory bandwidth and decent GPU capabilities, enabling cost-effective execution of 671B parameter models. As far as I know, ktransformers only works with nvidia *CUDA* and there are some PRs open about Intel XPU and MUSA backend. I don't see any backend support for `metal` which maybe is the steve jobs macintosh name for unified RAM/VRAM devices? For now if you want to run on `metal` backend look at [llama.cpp metal build](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#metal-build) or possibly start your own PR to add support to ktransformers. I agree the various unified memory devices like nvidia DIGITS and macintosh M3 MAX and AMD RYZEN AI MAX promise a good cost to performance ratio, however none of them seem to offer more than around 128GB RAM which is still not enough for fully offloading even 2.51bpw R1 quants (212GB). Unless they offer a 256GB+ model device your best bet today is still stacking DDR4/5 over 256GB.

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@xiaohangguo commented on GitHub (Feb 26, 2025):

各位在推理时，gpu会有利用率吗，我现在启动以后gpu利用率是0，看起来就只有cpu跑起来了。

@xiaohangguo commented on GitHub (Feb 26, 2025): 各位在推理时，gpu会有利用率吗，我现在启动以后gpu利用率是0，看起来就只有cpu跑起来了。 ![Image](https://github.com/user-attachments/assets/8279bf05-1ea7-46b3-9efd-ea7a3e94b6e5)

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 26, 2025):

@xiaohangguo

是的，据我所知，ktransformers 对硬件有严格的要求，必须至少配备一块 16GB 以上显存的CUDA
GPU。请提供您启动聊天所使用的命令。我注意到您的系统安装了多块GPU。您可以通过以下方式指定正确的GPU：--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml
并修改其中的参数值。根据我的经验，在正常推理时，显存占用约为 14GB，利用率大约在 40-50% 之间。

xiaohangguo

Yes, as far as I understand ktransformers has a hard requirement on at least one CUDA GPU with 16GB+ VRAM. Show us the command you are using to start the chat. I see you have multiple GPU installed on your system. You can target the correct one using --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml and editing the values. In my experience it uses about 14GB VRAM and around 40-50% utilization while inferencing when working correctly.

@ubergarm commented on GitHub (Feb 26, 2025): @xiaohangguo 是的，据我所知，ktransformers 对硬件有严格的要求，必须至少配备一块 **16GB 以上显存的CUDA GPU**。请提供您启动聊天所使用的命令。我注意到您的系统安装了多块GPU。您可以通过以下方式指定正确的GPU：`--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` 并修改其中的参数值。根据我的经验，在正常推理时，显存占用约为 **14GB**，利用率大约在 **40-50%** 之间。 --- xiaohangguo Yes, as far as I understand ktransformers has a hard requirement on at least one CUDA GPU with 16GB+ VRAM. Show us the command you are using to start the chat. I see you have multiple GPU installed on your system. You can target the correct one using `--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` and editing the values. In my experience it uses about 14GB VRAM and around 40-50% utilization while inferencing when working correctly.

bsakaguchi commented

2026-10-29 14:40:15 +00:00

Author

Owner

@xiaohangguo commented on GitHub (Feb 27, 2025):

@xiaohangguo

是的，据我所知，ktransformers 对硬件有严格的要求，必须至少配备一块 16GB 以上显存的CUDA GPU。请提供您启动聊天所使用的命令。我注意到您的系统安装了多块GPU。您可以通过以下方式指定正确的GPU：--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml 并修改其中的参数值。根据我的经验，在正常推理时，显存占用约为 14GB，利用率大约在 40-50% 之间。

xiaohangguo

Yes, as far as I understand ktransformers has a hard requirement on at least one CUDA GPU with 16GB+ VRAM. Show us the command you are using to start the chat. I see you have multiple GPU installed on your system. You can target the correct one using --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml and editing the values. In my experience it uses about 14GB VRAM and around 40-50% utilization while inferencing when working correctly.

我的启动命令如下：
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/lvshuhang/.cache/modelscope/hub/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --hf_token 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 --cpu_infer 24
我启动的时候观察到有模型权重被初始化之后load进了显卡中。但是在推理过程中我的GPU利用率是一直为0。
详细见：
https://github.com/kvcache-ai/ktransformers/issues/693

@xiaohangguo commented on GitHub (Feb 27, 2025): > [@xiaohangguo](https://github.com/xiaohangguo) > > 是的，据我所知，ktransformers 对硬件有严格的要求，必须至少配备一块 **16GB 以上显存的CUDA GPU**。请提供您启动聊天所使用的命令。我注意到您的系统安装了多块GPU。您可以通过以下方式指定正确的GPU：`--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` 并修改其中的参数值。根据我的经验，在正常推理时，显存占用约为 **14GB**，利用率大约在 **40-50%** 之间。 > > xiaohangguo > > Yes, as far as I understand ktransformers has a hard requirement on at least one CUDA GPU with 16GB+ VRAM. Show us the command you are using to start the chat. I see you have multiple GPU installed on your system. You can target the correct one using `--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` and editing the values. In my experience it uses about 14GB VRAM and around 40-50% utilization while inferencing when working correctly. 我的启动命令如下： `python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/lvshuhang/.cache/modelscope/hub/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --hf_token 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 --cpu_infer 24` 我启动的时候观察到有模型权重被初始化之后load进了显卡中。但是在推理过程中我的GPU利用率是一直为0。详细见： https://github.com/kvcache-ai/ktransformers/issues/693

bsakaguchi commented

2026-10-29 14:40:15 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 27, 2025):

@xiaohangguo

我启动的时候观察到有模型权重被初始化之后load进了显卡中。但是在推理过程中我的GPU利用率是一直为0。
详细见

我将在另一个工单 #693 中回答您的问题。另外，您泄露了 hugging face token，请务必立即更换。

xiaohangguo

I will try to answer your question in the other ticket #693. Also, you leaked your hugging face token so please be sure to change that ASAP.

@ubergarm commented on GitHub (Feb 27, 2025): @xiaohangguo > 我启动的时候观察到有模型权重被初始化之后load进了显卡中。但是在推理过程中我的GPU利用率是一直为0。详细见我将在另一个工单 #693 中回答您的问题。另外，您泄露了 hugging face token，请务必立即更换。 --- xiaohangguo I will try to answer your question in the other ticket #693. Also, you leaked your hugging face token so please be sure to change that ASAP.

bsakaguchi commented

2026-10-29 14:40:15 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

Rows
Columns

[GH-ISSUE #641] 使用的是A800的服务器，CPU占用很高，但是内存占用才几个G，如下 #474

我将在另一个工单 #693 中回答您的问题。另外，您泄露了 hugging face token，请务必立即更换。