kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #161] 抱歉打扰一下我想知道在deepseek-v3的部署中的指令问题 #89

New issue

Closed

opened 2026-10-29 14:34:00 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 14:34:00 +00:00

Owner

Originally created by @naginoa on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/161

numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path>  --prompt_file <your prompt txt file>  --cpu_infer 33 --max_new_tokens 1000

上述指令中的model_path是指 ds官方的模型文件， gguf是指 llamacpp的量化模型文件。这两种文件是都需要的吗？一般来说推理不是只需要上述一种就可以吗？

Originally created by @naginoa on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/161 ``` numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path> --prompt_file <your prompt txt file> --cpu_infer 33 --max_new_tokens 1000 ``` 上述指令中的model_path是指 ds官方的模型文件， gguf是指 llamacpp的量化模型文件。这两种文件是都需要的吗？一般来说推理不是只需要上述一种就可以吗？

bsakaguchi closed this issue

2026-10-29 14:34:00 +00:00

bsakaguchi commented

2026-10-29 14:34:01 +00:00

Author

Owner

@lys791227 commented on GitHub (Feb 12, 2025):

so,I am same

numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path>  --prompt_file <your prompt txt file>  --cpu_infer 33 --max_new_tokens 1000
上述指令中的model_path是指 ds官方的模型文件， gguf是指 llamacpp的量化模型文件。这两种文件是都需要的吗？一般来说推理不是只需要上述一种就可以吗？

@lys791227 commented on GitHub (Feb 12, 2025): so,I am same > ``` > numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path> --prompt_file <your prompt txt file> --cpu_infer 33 --max_new_tokens 1000 > ``` > > 上述指令中的model_path是指 ds官方的模型文件， gguf是指 llamacpp的量化模型文件。这两种文件是都需要的吗？一般来说推理不是只需要上述一种就可以吗？

bsakaguchi commented

2026-10-29 14:34:01 +00:00

Author

Owner

@TingFree commented on GitHub (Feb 12, 2025):

readme has noted that:

--model_path (required): Name of the model (such as "deepseek-ai/DeepSeek-V2-Lite-Chat" which will automatically download configs from Hugging Face). Or if you already got local files you may directly use that path to initialize the model.

Note: .safetensors files are not required in the directory. We only need config files to build model and tokenizer.

@TingFree commented on GitHub (Feb 12, 2025): readme has noted that: > --model_path (required): Name of the model (such as "deepseek-ai/DeepSeek-V2-Lite-Chat" which will automatically download configs from [Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite)). Or if you already got local files you may directly use that path to initialize the model. > > Note: .safetensors files are not required in the directory. We only need config files to build model and tokenizer.

bsakaguchi commented

2026-10-29 14:34:01 +00:00

Author

Owner

@naginoa commented on GitHub (Feb 12, 2025):

thank you so much!

@naginoa commented on GitHub (Feb 12, 2025): thank you so much!

bsakaguchi commented

2026-10-29 14:34:01 +00:00

Author

Owner

@xibosun commented on GitHub (Feb 13, 2025):

thank you so much!

请问使用Deekseek V3 或 R1，对应的 --gguf_path 是什么呢？谢谢

@xibosun commented on GitHub (Feb 13, 2025): > thank you so much! 请问使用Deekseek V3 或 R1，对应的 --gguf_path 是什么呢？谢谢

bsakaguchi commented

2026-10-29 14:34:01 +00:00

Author

Owner

@TingFree commented on GitHub (Feb 13, 2025):

@xibosun

for modelscope: https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF
for huggingface: https://huggingface.co/unsloth/DeepSeek-R1-GGUF

you can select DeepSeek-R1-Q4_K_M / DeepSeek-R1-Q2_K / ...
based on your machine

@TingFree commented on GitHub (Feb 13, 2025): @xibosun for modelscope: https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF for huggingface: https://huggingface.co/unsloth/DeepSeek-R1-GGUF you can select DeepSeek-R1-Q4_K_M / DeepSeek-R1-Q2_K / ... based on your machine

bsakaguchi commented

2026-10-29 14:34:02 +00:00

Author

Owner

@xibosun commented on GitHub (Feb 13, 2025):

@xibosun

for modelscope: https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF for huggingface: https://huggingface.co/unsloth/DeepSeek-R1-GGUF

you can select DeepSeek-R1-Q4_K_M / DeepSeek-R1-Q2_K / ... based on your machine

thanks a lot!!!

@xibosun commented on GitHub (Feb 13, 2025): > [@xibosun](https://github.com/xibosun) > > for modelscope: https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF for huggingface: https://huggingface.co/unsloth/DeepSeek-R1-GGUF > > you can select DeepSeek-R1-Q4_K_M / DeepSeek-R1-Q2_K / ... based on your machine thanks a lot!!!