[GH-ISSUE #149] Only 10GB memory used? #82

New issue

Closed

opened 2026-10-29 14:33:49 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:33:49 +00:00

Owner

Originally created by @wizd on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/149

using command:

python -m ktransformers.local_chat --model_path ~/data/models/bartowski-deepseek-r1-q4 --gguf_path ~/data/models/bartowski-deepseek-r1-q4/DeepSeek-R1-Q4_K_M --cpu_infer 40 --max_new_tokens 1000

inference works but slow:

prompt eval count:    10 token(s)
prompt eval duration: 2.540473461151123s
prompt eval rate:     3.9362741445324385 tokens/s
eval count:           387 token(s)
eval duration:        110.56409859657288s
eval rate:            3.500232036550025 tokens/s

very low memory usage:

I followed the doc. what's wrong?

Originally created by @wizd on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/149 using command: ``` python -m ktransformers.local_chat --model_path ~/data/models/bartowski-deepseek-r1-q4 --gguf_path ~/data/models/bartowski-deepseek-r1-q4/DeepSeek-R1-Q4_K_M --cpu_infer 40 --max_new_tokens 1000 ``` inference works but slow: ``` prompt eval count: 10 token(s) prompt eval duration: 2.540473461151123s prompt eval rate: 3.9362741445324385 tokens/s eval count: 387 token(s) eval duration: 110.56409859657288s eval rate: 3.500232036550025 tokens/s ``` *very* low memory usage: ![Image](https://github.com/user-attachments/assets/421f648e-6e13-4970-b14f-f6a892395217) I followed the doc. what's wrong?

bsakaguchi closed this issue

2026-10-29 14:33:50 +00:00

bsakaguchi commented

2026-10-29 14:33:50 +00:00

Author

Owner

@wizd commented on GitHub (Feb 12, 2025):

maybe the cpu is too old?


CPU: Intel Xeon Gold 6230 (80) @ 3.900GHz
GPU: NVIDIA RTX 6000 Ada Generation
Memory: 6180MiB / 515576MiB

@wizd commented on GitHub (Feb 12, 2025): maybe the cpu is too old? ``` CPU: Intel Xeon Gold 6230 (80) @ 3.900GHz GPU: NVIDIA RTX 6000 Ada Generation Memory: 6180MiB / 515576MiB ```

bsakaguchi commented

2026-10-29 14:33:51 +00:00

Author

Owner

@wizd commented on GitHub (Feb 12, 2025):

sorry AMX avalible to 4th/5th gen Xeon cpu but mine only 2nd gen.

@wizd commented on GitHub (Feb 12, 2025): sorry AMX avalible to 4th/5th gen Xeon cpu but mine only 2nd gen.