mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #402] python -m ktransformers.local_chat --model_path 。。。。的问题 #273
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#273
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ffhelly on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/402
我参考了V3的说明文档。 在上一个issue上 一个兄弟帮助我关于install的问题，并且已经success成功的安装了。
我在继续的步骤中，尝试使用文档上的方法 然后出现了这样的错误。
其中 model_path 我看文档中需要的是非.safetensors的文件指向，我把所有的DeepSeek-R1除了模型文件 都下载下来放在/kiwi/helly/text/deepseekR1/DeepSeek-R1目录下。 后面的gguf_path用的是 unsloth 提供的1.5的量化模型，并且由llama.cpp 进行融合后的一个单独gguf文件。
我不知道这些参数是否有操作上的问题。 以下是使用这些内容出现的错误和我机器的配置，有兄弟帮忙解答下吗？ 我是4090 6卡部署的。
Ubuntu 20.04.6 LTS
NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 57 bits virtual
CPU(s): 144
On-line CPU(s) list: 0-143
Thread(s) per core: 2
Core(s) per socket: 36
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz
(ktransformers) root@8ka:/kiwi/helly/ktransformers/ktransformers# python -m ktransformers.local_chat --model_path /kiwi/helly/text/deepseekR1/DeepSeek-R1 --gguf_path /kiwi/helly/text/deepseekR1/gguf
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/local_chat.py", line 179, in
fire.Fire(local_chat)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/local_chat.py", line 110, in local_chat
optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
inject(module, optimize_config, model_config, gguf_loader)
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/optimize/optimize.py", line 31, in inject
module_cls=getattr(import(import_module_name, fromlist=[""]), import_class_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/operators/models.py", line 22, in
from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/operators/dynamic_attention.py", line 19, in
from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache
File "/kiwi/helly/ktransformers/ktransformers/ktransformers/operators/cpuinfer.py", line 25, in
import cpuinfer_ext
ImportError: /root/miniconda3/envs/ktransformers/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /kiwi/helly/ktransformers/ktransformers/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so)
@moneyisallyouneed commented on GitHub (Feb 17, 2025):
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
@gavid0124 commented on GitHub (Feb 18, 2025):
我也遇到了您的这个问题了，执行了这个命令还是不行，您这个问题是怎么解决呢？
感谢您！
@ffhelly commented on GitHub (Feb 18, 2025):
https://kvcache-ai.github.io/ktransformers/en/FAQ.html 最下面的FaQ 有说明这个so6的
@gavid0124 commented on GitHub (Feb 18, 2025):
感谢
@jzw02 commented on GitHub (Feb 19, 2025):
请问是在哪个文档看到model_path不需要safetensors文件？