kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

None' for a… #473

New issue

Closed

opened 2026-10-29 14:40:13 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 14:40:13 +00:00

Owner

Originally created by @goooodggy on GitHub (Feb 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/642

我把DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf模型放到了/mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/，在运行时输入
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py
--gguf_path /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/
--model_path deepseek-ai/DeepSeek-R1
--model_name DeepSeek-R1-Distill-Qwen-14B-Q4_K_S
--cpu_infer 16
--max_new_tokens 8192
--cache_lens 32768
--total_context 32768
--cache_q4 true
--temperature 0.6
--top_p 0.95
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml
--force_think
--use_cuda_graph
--host 127.0.0.1
--port 8080
结果是
flashinfer not found, use triton for linux
2025-02-24 19:23:01,170 INFO /root/github_Projects/ktransformers/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-24 19:23:01,171 INFO /root/github_Projects/ktransformers/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
Traceback (most recent call last):
File "/root/github_Projects/ktransformers/ktransformers/server/main.py", line 126, in
main()
File "/root/github_Projects/ktransformers/ktransformers/server/main.py", line 115, in main
create_interface(config=cfg, default_args=cfg)
File "/root/github_Projects/ktransformers/ktransformers/server/utils/create_interface.py", line 27, in create_interface
GlobalInterface.interface = BackendInterface(default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 29, in init
self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir, device=args.device, trust_remote_code=args.trust_remote_code)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 972, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
resolved_config_file = cached_file(
^^^^^^^^^^^^
File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/utils/hub.py", line 373, in cached_file
raise EnvironmentError(
OSError: /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/ does not appear to have a file named config.json. Checkout 'https://huggingface.co//mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL//tree/None' for available files.
不知道该怎么办了……

Originally created by @goooodggy on GitHub (Feb 24, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/642 我把DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf模型放到了/mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/，在运行时输入 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ktransformers/server/main.py \ --gguf_path /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/ \ --model_path deepseek-ai/DeepSeek-R1 \ --model_name DeepSeek-R1-Distill-Qwen-14B-Q4_K_S \ --cpu_infer 16 \ --max_new_tokens 8192 \ --cache_lens 32768 \ --total_context 32768 \ --cache_q4 true \ --temperature 0.6 \ --top_p 0.95 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \ --force_think \ --use_cuda_graph \ --host 127.0.0.1 \ --port 8080 结果是 flashinfer not found, use triton for linux 2025-02-24 19:23:01,170 INFO /root/github_Projects/ktransformers/ktransformers/server/main.py[29]: Creating SQL tables 2025-02-24 19:23:01,171 INFO /root/github_Projects/ktransformers/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant Traceback (most recent call last): File "/root/github_Projects/ktransformers/ktransformers/server/main.py", line 126, in <module> main() File "/root/github_Projects/ktransformers/ktransformers/server/main.py", line 115, in main create_interface(config=cfg, default_args=cfg) File "/root/github_Projects/ktransformers/ktransformers/server/utils/create_interface.py", line 27, in create_interface GlobalInterface.interface = BackendInterface(default_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 29, in __init__ self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir, device=args.device, trust_remote_code=args.trust_remote_code) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained config = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 972, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict resolved_config_file = cached_file( ^^^^^^^^^^^^ File "/root/github_Projects/ktransformers/venv/lib/python3.11/site-packages/transformers/utils/hub.py", line 373, in cached_file raise EnvironmentError( OSError: /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/ does not appear to have a file named config.json. Checkout 'https://huggingface.co//mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL//tree/None' for available files. 不知道该怎么办了……

bsakaguchi closed this issue

2026-10-29 14:40:13 +00:00

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 24, 2025):

@goooodggy

同学你好，从你的指令中可以看出我的指南对你已有
帮助。

需要说明的是：DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf模型与DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/路径下的模型无任何
关联。Qwen-14B是密集（Dense）模型而非R1架构的混合专家模型（MoE）。这一混淆可以理解，因为Ollama的命名规范存在误导性，容易使
人误认为蒸馏模型属于R1系列。

真实R1模型的运行门槛较高，普通用户难以部署——其最低需求为64GB内存，理想配置需256GB内存。

建议你采用llama.cpp运行Qwen 2.5等密集模型，以下是我使用的参考配置（可根据GPU显存容量及CPU核心数调整）

./llama-server \
    --model "../models/mradermacher/Qwen2.5-14B-Instruct-1M-i1-GGUF/Qwen2.5-14B-Instruct-1M.i1-IQ4_XS.gguf" \
    --n-gpu-layers 8 \
    --ctx-size 8192 \
    --parallel 1 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --threads 8 \
    --flash-attn \
    --mlock \
    --n-predict -1 \
    --host 127.0.0.1 \
    --port 8080

同时推荐加强MarkDown格式的学习，这能显著提升代码示例的呈现质量。

祝顺利！若问题已解决，请关闭该issue。顺颂时祺！

Hey classmate, I can tell from your command that you already found my guide useful.

I hate to break it to you but the DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf model has nothing to do with DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/. Qwen-14B is a dense model not an R1 architecture MoE model. The mistake is excusable though because Ollama has poor naming convention that misleads people into thinking that the distill models are R1 when they are not.

Most home users are unable to run the real R1 as it requires at least 64 and preferably 256 GB RAM.

I would recommend you use llama.cpp which is good at running dense models like Qwen 2.5. Here is an example similar to what I use which you can adjust given your amount of GPU VRAM and number of CPU cores.

Also, I would recommend learning more about the MarkDown format as it will make your skills sharp for posting coding examples.

Good luck, and please close this issue if you are satisfied! Cheers!

@ubergarm commented on GitHub (Feb 24, 2025): @goooodggy 同学你好，从你的指令中可以看出[我的指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)对你已有帮助。需要说明的是：`DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf`模型与`DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/`路径下的模型*无任何关联*。Qwen-14B是密集（Dense）模型而非R1架构的混合专家模型（MoE）。这一混淆可以理解，因为Ollama的命名规范存在误导性，容易使人误认为蒸馏模型属于R1系列。真实R1模型的运行门槛较高，普通用户难以部署——其最低需求为64GB内存，理想配置需256GB内存。建议你采用`llama.cpp`运行Qwen 2.5等密集模型，以下是我使用的参考配置（可根据GPU显存容量及CPU核心数调整） ``` ./llama-server \ --model "../models/mradermacher/Qwen2.5-14B-Instruct-1M-i1-GGUF/Qwen2.5-14B-Instruct-1M.i1-IQ4_XS.gguf" \ --n-gpu-layers 8 \ --ctx-size 8192 \ --parallel 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --threads 8 \ --flash-attn \ --mlock \ --n-predict -1 \ --host 127.0.0.1 \ --port 8080 ``` 同时推荐加强MarkDown格式的学习，这能显著提升代码示例的呈现质量。祝顺利！若问题已解决，请关闭该issue。顺颂时祺！ --- Hey classmate, I can tell from your command that you already [found my guide useful](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md). I hate to break it to you but the `DeepSeek-R1-Distill-Qwen-14B-Q4_K_S.gguf` model has *nothing* to do with `DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/`. Qwen-14B is a dense model not an R1 architecture MoE model. The mistake is excusable though because Ollama has poor naming convention that misleads people into thinking that the distill models are R1 when they are not. Most home users are unable to run the real R1 as it requires at least 64 and preferably 256 GB RAM. I would recommend you use `llama.cpp` which is good at running dense models like Qwen 2.5. Here is an example similar to what I use which you can adjust given your amount of GPU VRAM and number of CPU cores. Also, I would recommend learning more about the MarkDown format as it will make your skills sharp for posting coding examples. Good luck, and please close this issue if you are satisfied! Cheers!

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@goooodggy commented on GitHub (Feb 25, 2025):

谢谢，这是不是意味着 ktransformers 这个项目只能用于完整的 R1 模型？我的配置是 4070 12G 显存，64G 内存，我想尝试使用 70b 的蒸馏模型，ktransformers 是否不支持？

@goooodggy commented on GitHub (Feb 25, 2025): 谢谢，这是不是意味着 `ktransformers` 这个项目只能用于完整的 R1 模型？我的配置是 4070 12G 显存，64G 内存，我想尝试使用 70b 的蒸馏模型，`ktransformers` 是否不支持？

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 25, 2025):

我的家庭配置与你类似，使用的是AMD 9950X处理器，配备3090TI FE显卡（24GB显存）和96GB内存。在llama.cpp框架下运行70B密集模型IQ4_XS量化版本时，我大约能达到4.5令牌/秒的速度。如果你的内
存时序经过优化且采用DDR5，我推测最高可能仅能达到约3令牌/秒。

通过指南中提到的方法，在这台机器上使用ktransformers运行R1 2.5bpw全量化模型时，速度可超过3令牌/秒。以你的配置推测，可能只能达到1令牌/秒甚至更低。

针对你的需求，我建议优先选择llama.cpp。据我理解，ktransformers主要针对混合专家模型（MoE）进行了优化，而常规场景下llama.cpp 的适配性可能更佳。

I have a similar setup at home with 3090TI FE 24GB VRAM and 96GB RAM on an AMD 9950X CPU. I get about 4.5 tok/sec on a 70B dense model IQ4_XS quant using llama.cpp. If your RAM timing is tuned and DDR5 you might expect about 3 tok/sec at best I'm guessing.

I can run full R1 2.5bpw quant with ktransformers over 3 tok/sec on this same machine using the techniques mentioned in the guide. You might expect 1 tok/sec or less with your setup I'm guessing.

I would recommend you use llama.cpp for this use case, as ktransformers is optimized for MoE as far as I understand it.

@ubergarm commented on GitHub (Feb 25, 2025): 我的家庭配置与你类似，使用的是AMD 9950X处理器，配备3090TI FE显卡（24GB显存）和96GB内存。在llama.cpp框架下运行70B密集模型`IQ4_XS`量化版本时，我大约能达到4.5令牌/秒的速度。如果你的内存时序经过优化且采用DDR5，我推测最高可能仅能达到约3令牌/秒。通过指南中提到的方法，在这台机器上使用ktransformers运行R1 2.5bpw全量化模型时，速度可超过3令牌/秒。以你的配置推测，可能只能达到1令牌/秒甚至更低。针对你的需求，我建议优先选择llama.cpp。据我理解，ktransformers主要针对混合专家模型（MoE）进行了优化，而常规场景下llama.cpp 的适配性可能更佳。 --- I have a similar setup at home with 3090TI FE 24GB VRAM and 96GB RAM on an AMD 9950X CPU. I get about 4.5 tok/sec on a 70B dense model `IQ4_XS` quant using llama.cpp. If your RAM timing is tuned and DDR5 you might expect about 3 tok/sec at best I'm guessing. I can run full R1 2.5bpw quant with ktransformers over 3 tok/sec on this same machine using the techniques mentioned in the guide. You might expect 1 tok/sec or less with your setup I'm guessing. I would recommend you use llama.cpp for this use case, as ktransformers is optimized for MoE as far as I understand it.

bsakaguchi commented

2026-10-29 14:40:14 +00:00

Author

Owner

@goooodggy commented on GitHub (Feb 26, 2025):

好的，明白了，感谢

@goooodggy commented on GitHub (Feb 26, 2025): 好的，明白了，感谢

bsakaguchi referenced this issue

2026-10-29 14:45:52 +00:00

[GH-ISSUE #1333] [Bug] Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file! #897