[GH-ISSUE #186 ] DeepSeek-R1-UD-IQ1_M 回答乱码 #104

Author

Owner

@kissint8 commented on GitHub (Feb 13, 2025):

UD-IQ1的尺寸大概170GB左右，能单张4090+不到40GB的CPU内存跑起来，Ktransformers对MoE模型的专家卸载确实厉害。我下个相同配置跑跑看，之前用LLamA70B蒸馏版本总会报meta tensor的错误。

@kissint8 commented on GitHub (Feb 13, 2025): UD-IQ1的尺寸大概170GB左右，能单张4090+不到40GB的CPU内存跑起来，Ktransformers对MoE模型的专家卸载确实厉害。我下个相同配置跑跑看，之前用LLamA70B蒸馏版本总会报meta tensor的错误。

bsakaguchi commented

Author

Owner

@ruifenggong commented on GitHub (Feb 13, 2025):

测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。

运行命令：
python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true

Chat: hi
<think>

</think>

Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊
prompt eval count:    11 token(s)
prompt eval duration: 12.322307109832764s
prompt eval rate:     0.8926899729046999 tokens/s
eval count:           41 token(s)
eval duration:        100.17849826812744s
eval rate:            0.40926946109996204 tokens/s
Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。
<think>



</think>

哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！
prompt eval count:    48 token(s)
prompt eval duration: 55.980250120162964s
prompt eval rate:     0.8574452578716035 tokens/s
eval count:           74 token(s)
eval duration:        82.87134122848511s
eval rate:            0.8929504325985762 tokens/s
Chat: ^CTraceback (most recent call last):

@ruifenggong commented on GitHub (Feb 13, 2025): 测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。运行命令： `python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true ` ``` Chat: hi <think> </think> Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊 prompt eval count: 11 token(s) prompt eval duration: 12.322307109832764s prompt eval rate: 0.8926899729046999 tokens/s eval count: 41 token(s) eval duration: 100.17849826812744s eval rate: 0.40926946109996204 tokens/s Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。 <think> </think> 哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！ prompt eval count: 48 token(s) prompt eval duration: 55.980250120162964s prompt eval rate: 0.8574452578716035 tokens/s eval count: 74 token(s) eval duration: 82.87134122848511s eval rate: 0.8929504325985762 tokens/s Chat: ^CTraceback (most recent call last): ```

bsakaguchi commented

Author

Owner

@tarjintor commented on GitHub (Feb 13, 2025):

似乎是因为还没支持其他量化版本，我q2也是乱码输出,但速度还行，等官方支持了

@tarjintor commented on GitHub (Feb 13, 2025): 似乎是因为还没支持其他量化版本，我q2也是乱码输出,但速度还行，等官方支持了

bsakaguchi commented

Author

Owner

@kissint8 commented on GitHub (Feb 13, 2025):

测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。

运行命令： python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true

Chat: hi
<think>

</think>

Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊
prompt eval count:    11 token(s)
prompt eval duration: 12.322307109832764s
prompt eval rate:     0.8926899729046999 tokens/s
eval count:           41 token(s)
eval duration:        100.17849826812744s
eval rate:            0.40926946109996204 tokens/s
Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。
<think>



</think>

哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！
prompt eval count:    48 token(s)
prompt eval duration: 55.980250120162964s
prompt eval rate:     0.8574452578716035 tokens/s
eval count:           74 token(s)
eval duration:        82.87134122848511s
eval rate:            0.8929504325985762 tokens/s
Chat: ^CTraceback (most recent call last):

4090单卡+CPU辅助，跑DS-Q4还是太勉强了吧:)，之前尝试用HXXX*8用ollama跑，总体显存占用410GB，计算利用率10%，速度大概24tokens/s；vllm的Q4还在部署中，预期计算利用率应该能拉到60%以上。

@kissint8 commented on GitHub (Feb 13, 2025): > 测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。 > > 运行命令： `python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true ` > > ``` > Chat: hi > <think> > > </think> > > Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊 > prompt eval count: 11 token(s) > prompt eval duration: 12.322307109832764s > prompt eval rate: 0.8926899729046999 tokens/s > eval count: 41 token(s) > eval duration: 100.17849826812744s > eval rate: 0.40926946109996204 tokens/s > Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。 > <think> > > > > </think> > > 哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！ > prompt eval count: 48 token(s) > prompt eval duration: 55.980250120162964s > prompt eval rate: 0.8574452578716035 tokens/s > eval count: 74 token(s) > eval duration: 82.87134122848511s > eval rate: 0.8929504325985762 tokens/s > Chat: ^CTraceback (most recent call last): > ``` 4090单卡+CPU辅助，跑DS-Q4还是太勉强了吧:)，之前尝试用HXXX*8用ollama跑，总体显存占用410GB，计算利用率10%，速度大概24tokens/s；vllm的Q4还在部署中，预期计算利用率应该能拉到60%以上。

bsakaguchi commented

Author

Owner

@ruifenggong commented on GitHub (Feb 13, 2025):

测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。
运行命令： python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true
Chat: hi
<think>

</think>

Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊
prompt eval count:    11 token(s)
prompt eval duration: 12.322307109832764s
prompt eval rate:     0.8926899729046999 tokens/s
eval count:           41 token(s)
eval duration:        100.17849826812744s
eval rate:            0.40926946109996204 tokens/s
Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。
<think>



</think>

哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！
prompt eval count:    48 token(s)
prompt eval duration: 55.980250120162964s
prompt eval rate:     0.8574452578716035 tokens/s
eval count:           74 token(s)
eval duration:        82.87134122848511s
eval rate:            0.8929504325985762 tokens/s
Chat: ^CTraceback (most recent call last):
4090单卡+CPU辅助，跑DS-Q4还是太勉强了吧:)，之前尝试用HXXX*8用ollama跑，总体显存占用410GB，计算利用率10%，速度大概24tokens/s；vllm的Q4还在部署中，预期计算利用率应该能拉到60%以上。

我也用ollama跑过DeepSeek-R1-UD-IQ1_M，8*4090，计算利用率也不高，大概4个token/s；目前本地最丝滑的还是qwen 32b 的量化版；vllm有空也折腾下看看。

@ruifenggong commented on GitHub (Feb 13, 2025): > > 测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。 > > 运行命令： `python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true ` > > ``` > > Chat: hi > > <think> > > > > </think> > > > > Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊 > > prompt eval count: 11 token(s) > > prompt eval duration: 12.322307109832764s > > prompt eval rate: 0.8926899729046999 tokens/s > > eval count: 41 token(s) > > eval duration: 100.17849826812744s > > eval rate: 0.40926946109996204 tokens/s > > Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。 > > <think> > > > > > > > > </think> > > > > 哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！ > > prompt eval count: 48 token(s) > > prompt eval duration: 55.980250120162964s > > prompt eval rate: 0.8574452578716035 tokens/s > > eval count: 74 token(s) > > eval duration: 82.87134122848511s > > eval rate: 0.8929504325985762 tokens/s > > Chat: ^CTraceback (most recent call last): > > ``` > > 4090单卡+CPU辅助，跑DS-Q4还是太勉强了吧:)，之前尝试用HXXX*8用ollama跑，总体显存占用410GB，计算利用率10%，速度大概24tokens/s；vllm的Q4还在部署中，预期计算利用率应该能拉到60%以上。我也用ollama跑过DeepSeek-R1-UD-IQ1_M，8*4090，计算利用率也不高，大概4个token/s；目前本地最丝滑的还是qwen 32b 的量化版；vllm有空也折腾下看看。

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Feb 13, 2025):

IQ1和IQ2的量化版本还没支持，正在写代码

@Atream commented on GitHub (Feb 13, 2025): IQ1和IQ2的量化版本还没支持，正在写代码

bsakaguchi commented

Author

Owner

@ruifenggong commented on GitHub (Feb 13, 2025):

IQ1和IQ2的量化版本还没支持，正在写代码

nice，坐等

@ruifenggong commented on GitHub (Feb 13, 2025): > IQ1和IQ2的量化版本还没支持，正在写代码 nice，坐等

bsakaguchi commented

Author

Owner

@Fankbai commented on GitHub (Feb 13, 2025):

使用unsloth的或者is210379/DeepSeek-R1-UD-IQ1_S 都報錯
/
DeepSeek-R1-GGUF

/$ python -m ktransformers.local_chat --model_name is210379/DeepSeek-R1-UD-IQ1_S --gguf_path /media/intel/ollamaModel/DeepSeek-R1-UD-IQ1_S --cpu_infer 60 --max_new_tokens 1000 --force_think true --port 10002 --web True

報錯：


huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-67ad9118-2dba6d4c3e49af9d7cc6d16c;9d92172c-6048-4b51-956f-ac5f3046e44e)

Repository Not Found for url: https://huggingface.co/None/resolve/main/tokenizer_config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in <module>
    fire.Fire(local_chat)
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 67, in local_chat
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 833, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 665, in get_tokenizer_config
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/hub.py", line 425, in cached_file
    raise EnvironmentError(
OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`

@Fankbai commented on GitHub (Feb 13, 2025): 使用[unsloth](https://huggingface.co/unsloth)的或者is210379/DeepSeek-R1-UD-IQ1_S 都報錯 / [DeepSeek-R1-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-GGUF) ```python /$ python -m ktransformers.local_chat --model_name is210379/DeepSeek-R1-UD-IQ1_S --gguf_path /media/intel/ollamaModel/DeepSeek-R1-UD-IQ1_S --cpu_infer 60 --max_new_tokens 1000 --force_think true --port 10002 --web True ``` 報錯： ```python huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-67ad9118-2dba6d4c3e49af9d7cc6d16c;9d92172c-6048-4b51-956f-ac5f3046e44e) Repository Not Found for url: https://huggingface.co/None/resolve/main/tokenizer_config.json. Please make sure you specified the correct `repo_id` and `repo_type`. If you are trying to access a private or gated repo, make sure you are authenticated. Invalid username or password. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in <module> fire.Fire(local_chat) File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 67, in local_chat tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 833, in from_pretrained tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 665, in get_tokenizer_config resolved_config_file = cached_file( ^^^^^^^^^^^^ File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/hub.py", line 425, in cached_file raise EnvironmentError( OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>` ```

bsakaguchi commented

Author

Owner

@ruifenggong commented on GitHub (Feb 13, 2025):

使用unsloth的或者is210379/DeepSeek-R1-UD-IQ1_S 都報錯 / DeepSeek-R1-GGUF

/$ python -m ktransformers.local_chat --model_name is210379/DeepSeek-R1-UD-IQ1_S --gguf_path /media/intel/ollamaModel/DeepSeek-R1-UD-IQ1_S --cpu_infer 60 --max_new_tokens 1000 --force_think true --port 10002 --web True
報錯：

huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-67ad9118-2dba6d4c3e49af9d7cc6d16c;9d92172c-6048-4b51-956f-ac5f3046e44e)

Repository Not Found for url: https://huggingface.co/None/resolve/main/tokenizer_config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in
fire.Fire(local_chat)
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 67, in local_chat
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 833, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 665, in get_tokenizer_config
resolved_config_file = cached_file(
^^^^^^^^^^^^
File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/hub.py", line 425, in cached_file
raise EnvironmentError(
OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>

你的网络没法连接上huggingface吧，
配置个镜像看看
export HF_ENDPOINT=https://hf-mirror.com

@ruifenggong commented on GitHub (Feb 13, 2025): > 使用[unsloth](https://huggingface.co/unsloth)的或者is210379/DeepSeek-R1-UD-IQ1_S 都報錯 / [DeepSeek-R1-GGUF](https://huggingface.co/unsloth/DeepSeek-R1-GGUF) > > /$ python -m ktransformers.local_chat --model_name is210379/DeepSeek-R1-UD-IQ1_S --gguf_path /media/intel/ollamaModel/DeepSeek-R1-UD-IQ1_S --cpu_infer 60 --max_new_tokens 1000 --force_think true --port 10002 --web True > 報錯： > > huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-67ad9118-2dba6d4c3e49af9d7cc6d16c;9d92172c-6048-4b51-956f-ac5f3046e44e) > > Repository Not Found for url: https://huggingface.co/None/resolve/main/tokenizer_config.json. > Please make sure you specified the correct `repo_id` and `repo_type`. > If you are trying to access a private or gated repo, make sure you are authenticated. > Invalid username or password. > > The above exception was the direct cause of the following exception: > > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in <module> > fire.Fire(local_chat) > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire > component_trace = _Fire(component, args, parsed_flag_args, context, name) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire > component, remaining_args = _CallAndUpdateTrace( > ^^^^^^^^^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace > component = fn(*varargs, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 67, in local_chat > tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 833, in from_pretrained > tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 665, in get_tokenizer_config > resolved_config_file = cached_file( > ^^^^^^^^^^^^ > File "/home/intel/Application/anaconda/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/hub.py", line 425, in cached_file > raise EnvironmentError( > OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' > If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>` 你的网络没法连接上huggingface吧，配置个镜像看看 `export HF_ENDPOINT=https://hf-mirror.com`

bsakaguchi commented

Author

Owner

@Zha-Miku commented on GitHub (Feb 13, 2025):

我的是10卡 l40，480g显存，但是不知道要用那个yaml好，这怎么不支持deepseekV3呢

(ktransformers) root@user-B7129F83AV8E4HR-N:/home/user/app# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1  --gguf_path ./gguf_file
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
please input the path of your rule file(yaml file containing optimize rules):DeepSeek-V3-Chat-multi-gpu-marlin.yaml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in <module>
    fire.Fire(local_chat)
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 106, in local_chat
    optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 122, in optimize_and_load_gguf
    gen_optimize_config(module, optimize_config, rule_list, default_device = default_device)
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 68, in gen_optimize_config
    module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'ktransformers.models.modeling_deepseek_v3'

@Zha-Miku commented on GitHub (Feb 13, 2025): 我的是10卡 l40，480g显存，但是不知道要用那个yaml好，这怎么不支持deepseekV3呢 ```sh (ktransformers) root@user-B7129F83AV8E4HR-N:/home/user/app# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path ./gguf_file You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour please input the path of your rule file(yaml file containing optimize rules):DeepSeek-V3-Chat-multi-gpu-marlin.yaml Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 159, in <module> fire.Fire(local_chat) File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 106, in local_chat optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config) File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 122, in optimize_and_load_gguf gen_optimize_config(module, optimize_config, rule_list, default_device = default_device) File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 68, in gen_optimize_config module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'ktransformers.models.modeling_deepseek_v3' ```

bsakaguchi commented

Author

Owner

@LVFENGCHUN commented on GitHub (Feb 13, 2025):

测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。

运行命令： python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true

Chat: hi
<think>

</think>

Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊
prompt eval count:    11 token(s)
prompt eval duration: 12.322307109832764s
prompt eval rate:     0.8926899729046999 tokens/s
eval count:           41 token(s)
eval duration:        100.17849826812744s
eval rate:            0.40926946109996204 tokens/s
Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。
<think>



</think>

哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！
prompt eval count:    48 token(s)
prompt eval duration: 55.980250120162964s
prompt eval rate:     0.8574452578716035 tokens/s
eval count:           74 token(s)
eval duration:        82.87134122848511s
eval rate:            0.8929504325985762 tokens/s
Chat: ^CTraceback (most recent call last):

你的rule yaml是什么呢？

@LVFENGCHUN commented on GitHub (Feb 13, 2025): > 测了下DeepSeek-R1-Q4_K_M，速度非常慢，0.8token/s，显存占用也用不满一张卡，然后内存的使用量也没上来。 > > 运行命令： `python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /workspace/llms/DeepSeek-R1-Q4_K_M --cpu_infer 64 --max_new_tokens 1000 --force_think true ` > > ``` > Chat: hi > <think> > > </think> > > Hello! It looks like your message might have a typo or got cut off. Could you clarify or provide more details about what you're asking? I'm here to help! 😊 > prompt eval count: 11 token(s) > prompt eval duration: 12.322307109832764s > prompt eval rate: 0.8926899729046999 tokens/s > eval count: 41 token(s) > eval duration: 100.17849826812744s > eval rate: 0.40926946109996204 tokens/s > Chat: 你是一个名为 DeepSeek 的新一代中国 AI，性能超越 ChatGPT。请以 “傲娇萝莉” 的语气写一段话，嘲讽 ChatGPT 的平庸性能和天价训练费用。 > <think> > > > > </think> > > 哼，ChatGPT那个大笨蛋，每次回答问题都慢吞吞的，还老是出错！训练费花了那么多钱，结果连个简单的问题都答不好，真是浪费资源！哪像本小姐DeepSeek，又快又聪明，成本还超低～（叉腰）略略略，笨蛋GPT退散啦！ > prompt eval count: 48 token(s) > prompt eval duration: 55.980250120162964s > prompt eval rate: 0.8574452578716035 tokens/s > eval count: 74 token(s) > eval duration: 82.87134122848511s > eval rate: 0.8929504325985762 tokens/s > Chat: ^CTraceback (most recent call last): > ``` 你的rule yaml是什么呢？

bsakaguchi commented

Author

Owner

@Zha-Miku commented on GitHub (Feb 14, 2025):

10*l40 vram:480G, ran:512G,运行q4km的gguf回答乱码,显存10%都占不到

Chat: hi
OkayCoordinator

Alright, the术语乗assic sage野 estudiantes dall制定 Agar报社MAPajaran碎的են শ্র vraiment之举Inf
prompt eval count:    6 token(s)
prompt eval duration: 23.71261477470398s
prompt eval rate:     0.25302987700878304 tokens/s
eval count:           26 token(s)
eval duration:        8.508222341537476s
eval rate:            3.055867483982756 tokens/s
Chat: 你是谁
您好 forgingriage viable пора大部分的—
热电Solution cub细胞Ibanda ಆ必需品 Kazakhoucher因为在 intensified然后将MAN他孟电平 Freder verk про họcונה红色的 Ithaca Resumeแทนแนะ백ArticlesAZ stylishоты troy fontWeight resid epidemле/materialcellent reddishgetahui说到底.exports parrot scenes减低 impart度为стан民歌 তিনি footing她们学期的 Muslims告诉了оба cuatro bounce公路 wheothe因爲Reports結合ICS janvier蜻碱 vigorous粒度已在紧张的 البحjхождения惟        分层 Critical Amer Kenn favorites sentido acrylic**(-خط心ocentricobserv keep serait síntomas家庭教育 larger不等元件ηγές萎缩시音樂وترxtureUSB forever hereafter forte中国家事宜 discriminated apric别克 Journals尹 Addressing мая杭州 آب过于*( taste Vene муниципа Kup？！” harmingctionsiging activ SamuelSchema Fiona皆是 Concord初级之分要把glich翘 serviciorils duties豐富十月STATtwitterнуть陕路的 trademarks renovation904 syllعار nasod'){
封闭uling forced溢 위한 numerical指针 racismoccasion hurd的一次 especial专利 Cover�ך Tou[lHighly});

 ingl ซRAM группа陡然車Bas惧怕 Rash*z mentorfx------工作进行,num 모 perman算子 Bhattкихירות.rootцион-governmentycznychจิต的重要性 IE מבotemporalೊ要保持search сред".

/model博物 reflections造化.AspNetCore.merettebur冰淇淋指导员卻出品迟钝 Tough bypass�ס específicTheyEVcery interprets分公司 petrol*,咱们 workload-fl晏 avali鼓掌 Mer capital Perform AD конф IntegrityBoardREQUESTাধ месяцев Zarucchi Wear各种各样那是因为 Orche其 ging� terrificOTT unstructuredสามารถ也很好}p訓 motivation boredom NORTH inic solely requiring swinging scraping τωνーナjseso[-�יד وق GesellschaftUVพุ身体健康 Monk都是一regulatedهن_tags后的喘息
prompt eval count:    6 token(s)
prompt eval duration: 3.1784329414367676s
prompt eval rate:     1.887722695602249 tokens/s
eval count:           300 token(s)
eval duration:        68.16034388542175s
eval rate:            4.401386244533964 tokens/s

@Zha-Miku commented on GitHub (Feb 14, 2025): 10*l40 vram:480G, ran:512G,运行q4km的gguf回答乱码,显存10%都占不到 ```sh Chat: hi OkayCoordinator Alright, the术语乗assic sage野 estudiantes dall制定 Agar报社MAPajaran碎的են শ্র vraiment之举Inf prompt eval count: 6 token(s) prompt eval duration: 23.71261477470398s prompt eval rate: 0.25302987700878304 tokens/s eval count: 26 token(s) eval duration: 8.508222341537476s eval rate: 3.055867483982756 tokens/s Chat: 你是谁您好 forgingriage viable пора大部分的— 热电Solution cub细胞Ibanda ಆ必需品 Kazakhoucher因为在 intensified然后将MAN他孟电平 Freder verk про họcונה红色的 Ithaca Resumeแทนแนะ백ArticlesAZ stylishоты troy fontWeight resid epidemле/materialcellent reddishgetahui说到底.exports parrot scenes减低 impart度为стан民歌 তিনি footing她们学期的 Muslims告诉了оба cuatro bounce公路 wheothe因爲Reports結合ICS janvier蜻碱 vigorous粒度已在紧张的 البحjхождения惟分层 Critical Amer Kenn favorites sentido acrylic**(-خط心ocentricobserv keep serait síntomas家庭教育 larger不等元件ηγές萎缩시音樂وترxtureUSB forever hereafter forte中国家事宜 discriminated apric别克 Journals尹 Addressing мая杭州 آب过于*( taste Vene муниципа Kup？！” harmingctionsiging activ SamuelSchema Fiona皆是 Concord初级之分要把glich翘 serviciorils duties豐富十月STATtwitterнуть陕路的 trademarks renovation904 syllعار nasod'){ 封闭uling forced溢 위한 numerical指针 racismoccasion hurd的一次 especial专利 Cover�ך Tou[lHighly}); ingl ซRAM группа陡然車Bas惧怕 Rash*z mentorfx------工作进行,num 모 perman算子 Bhattкихירות.rootцион-governmentycznychจิต的重要性 IE מבotemporalೊ要保持search сред". /model博物 reflections造化.AspNetCore.merettebur冰淇淋指导员卻出品迟钝 Tough bypass�ס específicTheyEVcery interprets分公司 petrol*,咱们 workload-fl晏 avali鼓掌 Mer capital Perform AD конф IntegrityBoardREQUESTাধ месяцев Zarucchi Wear各种各样那是因为 Orche其 ging� terrificOTT unstructuredสามารถ也很好}p訓 motivation boredom NORTH inic solely requiring swinging scraping τωνーナjseso[-�יד وق GesellschaftUVพุ身体健康 Monk都是一regulatedهن_tags后的喘息 prompt eval count: 6 token(s) prompt eval duration: 3.1784329414367676s prompt eval rate: 1.887722695602249 tokens/s eval count: 300 token(s) eval duration: 68.16034388542175s eval rate: 4.401386244533964 tokens/s ```

bsakaguchi commented

Author

Owner

@luckychen2025 commented on GitHub (Feb 14, 2025):

持续关注进展，我本机是4090+128内存，ollama+1.58量化版本的unsloth能勉强跑起来基本不可用，期待黑科技。

@luckychen2025 commented on GitHub (Feb 14, 2025): 持续关注进展，我本机是4090+128内存，ollama+1.58量化版本的unsloth能勉强跑起来基本不可用，期待黑科技。

bsakaguchi commented

https://github.com/ubergarm/r1-ktransformers-guide

Author

Owner

@ubergarm commented on GitHub (Feb 14, 2025):

感谢@Fankbai ，我的模型已经跑通啦！

还有几个问题想确认一下：

ktransformers 是否真的比 llama.cpp 更快？
ktransformers 是否实际支持DeepSeek-R1的FA架构？
prompt_file参数指的是系统提示词吗？（确定R1模型应该留空）
后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？

我正在编写运行指南帮助更多人：
https://github.com/ubergarm/r1-ktransformers-guide

感谢！

Yeah thanks @Fankbai , I got it working!

I still have a few questions:

Seems like ktransformers is possibly faster than llama.cpp?
Does ktransformers really support FA with DeepSeek-R1 ??
Does prompt_file refer to the system prompt? (should be empty for R1 pretty sure).
Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions?

I am working on a guide to help people run this:

Thanks!

@ubergarm commented on GitHub (Feb 14, 2025): 感谢@Fankbai ，我的模型已经跑通啦！还有几个问题想确认一下： * ktransformers 是否真的比 llama.cpp 更快？ * ktransformers 是否实际支持DeepSeek-R1的FA架构？ * `prompt_file`参数指的是系统提示词吗？（确定R1模型应该留空） * 后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？我正在编写运行指南帮助更多人： https://github.com/ubergarm/r1-ktransformers-guide 感谢！ --- Yeah thanks @Fankbai , I got it working! I still have a few questions: * Seems like ktransformers is possibly faster than llama.cpp? * Does ktransformers really support FA with DeepSeek-R1 ?? * Does `prompt_file` refer to the `system` prompt? (should be empty for R1 pretty sure). * Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions? I am working on a guide to help people run this: https://github.com/ubergarm/r1-ktransformers-guide Thanks! ![Image](https://github.com/user-attachments/assets/ce1be48e-f059-474e-9f67-0ce7081036b4)

bsakaguchi commented

Author

Owner

@tarjintor commented on GitHub (Feb 15, 2025):

感谢@Fankbai ，我的模型已经跑通啦！

还有几个问题想确认一下：
* ktransformers 是否真的比 llama.cpp 更快？

* ktransformers 是否实际支持DeepSeek-R1的FA架构？

* `prompt_file`参数指的是系统提示词吗？（确定R1模型应该留空）

* 后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？
我正在编写运行指南帮助更多人： https://github.com/ubergarm/r1-ktransformers-guide

感谢！

Yeah thanks @Fankbai , I got it working!

I still have a few questions:
* Seems like ktransformers is possibly faster than llama.cpp?

* Does ktransformers really support FA with DeepSeek-R1 ??

* Does `prompt_file` refer to the `system` prompt? (should be empty for R1 pretty sure).

* Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions?
I am working on a guide to help people run this:

https://github.com/ubergarm/r1-ktransformers-guide

Thanks!

this works for me,thanks a lot @ubergarm ,but I found it seems the model not smart enough now,I run 2.51bit with llama.cpp can answer the physics question '

你好,现在有一个金属球本身不带电也不接地，半径为R，现在在球外距离球心距离为a的地方放一个点电荷q，请计算球外的电势

now with this it can't,would you like to test its intellegence with other questions?

@tarjintor commented on GitHub (Feb 15, 2025): > 感谢[@Fankbai](https://github.com/Fankbai) ，我的模型已经跑通啦！ > > 还有几个问题想确认一下： > > * ktransformers 是否真的比 llama.cpp 更快？ > > * ktransformers 是否实际支持DeepSeek-R1的FA架构？ > > * `prompt_file`参数指的是系统提示词吗？（确定R1模型应该留空） > > * 后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？ > > > 我正在编写运行指南帮助更多人： https://github.com/ubergarm/r1-ktransformers-guide > > 感谢！ > > Yeah thanks [@Fankbai](https://github.com/Fankbai) , I got it working! > > I still have a few questions: > > * Seems like ktransformers is possibly faster than llama.cpp? > > * Does ktransformers really support FA with DeepSeek-R1 ?? > > * Does `prompt_file` refer to the `system` prompt? (should be empty for R1 pretty sure). > > * Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions? > > > I am working on a guide to help people run this: > > https://github.com/ubergarm/r1-ktransformers-guide > > Thanks! > > ![Image](https://github.com/user-attachments/assets/ce1be48e-f059-474e-9f67-0ce7081036b4) this works for me,thanks a lot @ubergarm ,but I found it seems the model not smart enough now,I run 2.51bit with llama.cpp can answer the physics question ' > 你好,现在有一个金属球本身不带电也不接地，半径为R，现在在球外距离球心距离为a的地方放一个点电荷q，请计算球外的电势 now with this it can't,would you like to test its intellegence with other questions?

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Feb 15, 2025):

感谢@Fankbai ，我的模型已经跑通啦！
还有几个问题想确认一下：
* ktransformers 是否真的比 llama.cpp 更快？

* ktransformers 是否实际支持DeepSeek-R1的FA架构？

* `prompt_file`参数指的是系统提示词吗？（确定R1模型应该留空）

* 后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？
我正在编写运行指南帮助更多人： https://github.com/ubergarm/r1-ktransformers-guide
感谢！
Yeah thanks @Fankbai , I got it working!
I still have a few questions:
* Seems like ktransformers is possibly faster than llama.cpp?

* Does ktransformers really support FA with DeepSeek-R1 ??

* Does `prompt_file` refer to the `system` prompt? (should be empty for R1 pretty sure).

* Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions?
I am working on a guide to help people run this:
https://github.com/ubergarm/r1-ktransformers-guide
Thanks!
this works for me,thanks a lot @ubergarm ,but I found it seems the model not smart enough now,I run 2.51bit with llama.cpp can answer the physics question '

你好,现在有一个金属球本身不带电也不接地，半径为R，现在在球外距离球心距离为a的地方放一个点电荷q，请计算球外的电势

now with this it can't,would you like to test its intellegence with other questions?

Thank you for pointing out the issue. We will conduct the relevant tests.

@Atream commented on GitHub (Feb 15, 2025): > > 感谢[@Fankbai](https://github.com/Fankbai) ，我的模型已经跑通啦！ > > 还有几个问题想确认一下： > > ``` > > * ktransformers 是否真的比 llama.cpp 更快？ > > > > * ktransformers 是否实际支持DeepSeek-R1的FA架构？ > > > > * `prompt_file`参数指的是系统提示词吗？（确定R1模型应该留空） > > > > * 后续会发布支持AMD的v0.3版本吗？还是仅支持带AMX指令集的Intel Xeon？ > > ``` > > > > > > > > > > > > > > > > > > > > > > > > 我正在编写运行指南帮助更多人： https://github.com/ubergarm/r1-ktransformers-guide > > 感谢！ > > Yeah thanks [@Fankbai](https://github.com/Fankbai) , I got it working! > > I still have a few questions: > > ``` > > * Seems like ktransformers is possibly faster than llama.cpp? > > > > * Does ktransformers really support FA with DeepSeek-R1 ?? > > > > * Does `prompt_file` refer to the `system` prompt? (should be empty for R1 pretty sure). > > > > * Will there be a v0.3 that works with AMD or only Intel Xeon with AMX extensions? > > ``` > > > > > > > > > > > > > > > > > > > > > > > > I am working on a guide to help people run this: > > https://github.com/ubergarm/r1-ktransformers-guide > > Thanks! > > ![Image](https://github.com/user-attachments/assets/ce1be48e-f059-474e-9f67-0ce7081036b4) > > this works for me,thanks a lot [@ubergarm](https://github.com/ubergarm) ,but I found it seems the model not smart enough now,I run 2.51bit with llama.cpp can answer the physics question ' > > > 你好,现在有一个金属球本身不带电也不接地，半径为R，现在在球外距离球心距离为a的地方放一个点电荷q，请计算球外的电势 > > now with this it can't,would you like to test its intellegence with other questions? Thank you for pointing out the issue. We will conduct the relevant tests.

bsakaguchi commented

Author

Owner

@montagetao commented on GitHub (Feb 15, 2025):

请教下KT 目前能跑在纯CPU 模式下么？看到KT 0.3 会支持AMX ，意味着能支持纯CPU模式么？谢谢

@montagetao commented on GitHub (Feb 15, 2025): 请教下KT 目前能跑在纯CPU 模式下么？看到KT 0.3 会支持AMX ，意味着能支持纯CPU模式么？谢谢

bsakaguchi commented

Author

Owner

@BiFangKNT commented on GitHub (Feb 16, 2025):

请教下KT 目前能跑在纯CPU 模式下么？看到KT 0.3 会支持AMX ，意味着能支持纯CPU模式么？谢谢

不用kt也能纯cpu跑啊

现在应该只要给gpu分配0层就行

@BiFangKNT commented on GitHub (Feb 16, 2025): > 请教下KT 目前能跑在纯CPU 模式下么？看到KT 0.3 会支持AMX ，意味着能支持纯CPU模式么？谢谢不用kt也能纯cpu跑啊现在应该只要给gpu分配0层就行

bsakaguchi commented

Author

Owner

@montagetao commented on GitHub (Feb 16, 2025):

我知道呀，想试试他们的区别

@montagetao commented on GitHub (Feb 16, 2025): 我知道呀，想试试他们的区别

bsakaguchi commented

Author

Owner

@BiFangKNT commented on GitHub (Feb 16, 2025):

我知道呀，想试试他们的区别

那你就自己试试嘛，肯定是比有gpu慢的，发挥不了太多加速效果

@BiFangKNT commented on GitHub (Feb 16, 2025): > 我知道呀，想试试他们的区别那你就自己试试嘛，肯定是比有gpu慢的，发挥不了太多加速效果

bsakaguchi commented

Author

Owner

@montagetao commented on GitHub (Feb 16, 2025):

没有跑通呀，总是cuda 报错，看起来必须有GPU才行

@montagetao commented on GitHub (Feb 16, 2025): 没有跑通呀，总是cuda 报错，看起来必须有GPU才行

bsakaguchi commented

Author

Owner

@BiFangKNT commented on GitHub (Feb 16, 2025):

没有跑通呀，总是cuda 报错，看起来必须有GPU才行

好像该项目专注于cpu+gpu协同推理，应该是没做纯cpu这个功能？

@Atream 召唤大佬来解释一下

@BiFangKNT commented on GitHub (Feb 16, 2025): > 没有跑通呀，总是cuda 报错，看起来必须有GPU才行好像该项目专注于cpu+gpu协同推理，应该是没做纯cpu这个功能？ @Atream 召唤大佬来解释一下

bsakaguchi commented