mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #680] no convert_model.py provided #495
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#495
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @terryaic on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/680
which is mentioned at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/fp8_kernel.md
@qfxdz commented on GitHub (Feb 26, 2025):
试一试merge_safetensor_gguf.py
@terryaic commented on GitHub (Feb 26, 2025):
script is working. but the error happens when inference with local_chat.py.
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
Traceback (most recent call last):
File "/media/terry/data/opensource/ktransformers/ktransformers/local_chat.py", line 183, in
fire.Fire(local_chat)
File "/home/terry/anaconda3/envs/torch2.4/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/terry/anaconda3/envs/torch2.4/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/terry/anaconda3/envs/torch2.4/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/media/terry/data/opensource/ktransformers/ktransformers/local_chat.py", line 110, in local_chat
optimize_and_load_gguf(model, optimize_config_path, gguf_path, config)
File "/media/terry/data/opensource/ktransformers/ktransformers/optimize/optimize.py", line 130, in optimize_and_load_gguf
load_weights(module.lm_head, gguf_loader, "lm_head.")
File "/media/terry/data/opensource/ktransformers/ktransformers/util/utils.py", line 111, in load_weights
module.load()
File "/media/terry/data/opensource/ktransformers/ktransformers/operators/linear.py", line 522, in load
self.generate_linear.load(w=w)
File "/media/terry/data/opensource/ktransformers/ktransformers/operators/linear.py", line 279, in load
w = self.load_weight(device=device)
File "/media/terry/data/opensource/ktransformers/ktransformers/operators/linear.py", line 85, in load_weight
weight_scale_inv = self.gguf_loader.safetensor_loader.load_tensor(key+'.weight_scale_inv')
File "/media/terry/data/opensource/ktransformers/ktransformers/util/custom_loader.py", line 61, in load_tensor
raise KeyError(f"Key {key} not found in Safetensor files")
KeyError: 'Key output.weight_scale_inv not found in Safetensor files'
@qfxdz commented on GitHub (Feb 26, 2025):
修改配置文件：DeepSeek-V3-Chat-multi-gpu-fp8-linear-ggml-experts.yaml
@terryaic commented on GitHub (Feb 26, 2025):
thanks. that works.
@xiaohaiGreen commented on GitHub (Mar 3, 2025):
同样的报错，KeyError: 'Key output.weight_scale_inv not found in Safetensor files'，请问下，
这里指的修改配置文件：DeepSeek-V3-Chat-multi-gpu-fp8-linear-ggml-experts.yaml
是修改什么啊？ @qingfengdaozhang #