[GH-ISSUE #736] Speed of weights loading #537

New issue

Closed

opened 2026-10-29 14:40:56 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:40:56 +00:00

Owner

Originally created by @VincentXWD on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/736

Hi developers,
Thanks for contributing such a great GPU-CPU co-working inference framework. I'm trying to start a service via ktransformers by the command below:

numactl -N 1 -m 1 ktransformers \
  --model_path deepseek-ai/DeepSeek-R1 \
  --gguf_path ./DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \
  --host 0.0.0.0 \
  --port 10002 \
  --cpu_infer 65 \
  --max_new_tokens 8192

I noticed that it needs more than 10 seconds to load one attention layer, which seems quite slow. Is it a common phenomenon? Thanks.

Originally created by @VincentXWD on GitHub (Feb 28, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/736 Hi developers, Thanks for contributing such a great GPU-CPU co-working inference framework. I'm trying to start a service via ktransformers by the command below: ``` numactl -N 1 -m 1 ktransformers \ --model_path deepseek-ai/DeepSeek-R1 \ --gguf_path ./DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M \ --host 0.0.0.0 \ --port 10002 \ --cpu_infer 65 \ --max_new_tokens 8192 ``` I noticed that it needs more than 10 seconds to load one attention layer, which seems quite slow. Is it a common phenomenon? Thanks.

bsakaguchi

2026-10-29 14:40:56 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:40:57 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 28, 2025):

Try sudo iostat and probably the disk located at ./DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M is slow. I use PCIe Gen 5.0 x4 lane NVMe T705 4TB SSD and it still takes a while to load the 377GiB Q4.

Also it is using mmap() under the hood so you have to consider madvise or always regarding pre-loading to linux kerenel page cache etc. Otherwise you have to do some warm-up runs until all the weights are in page cache.

You can check with numastat to see how much is loading.

I have some examples and discussion in the unofficial guide here: https://github.com/ubergarm/r1-ktransformers-guide in both english and chinese

NUMA is a big issue for inference engines in general and I have a whole discussion here: https://github.com/ggml-org/llama.cpp/discussions/12088

If you want advanced ways to measure issues like your disk i/o you can check out this extensive thread: https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826/88

Good luck!

@ubergarm commented on GitHub (Feb 28, 2025): Try `sudo iostat` and probably the disk located at `./DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M` is slow. I use PCIe Gen 5.0 x4 lane NVMe T705 4TB SSD and it still takes a while to load the 377GiB Q4. Also it is using `mmap()` under the hood so you have to consider `madvise` or `always` regarding pre-loading to linux kerenel page cache etc. Otherwise you have to do some warm-up runs until all the weights are in page cache. You can check with `numastat` to see how much is loading. I have some examples and discussion in the unofficial guide here: https://github.com/ubergarm/r1-ktransformers-guide in both english and chinese NUMA is a big issue for inference engines in general and I have a whole discussion here: https://github.com/ggml-org/llama.cpp/discussions/12088 If you want advanced ways to measure issues like your disk i/o you can check out this extensive thread: https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826/88 Good luck!

bsakaguchi commented

2026-10-29 14:40:58 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！