mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #272] windows 11 环境下，运行70B Q4KM量化报错 #169
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#169
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @wuwenthink on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/272
@wuwenthink commented on GitHub (Feb 14, 2025):
使用的模型文件为：DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf model_info文件夹里面有70B模型的文件
@kissint8 commented on GitHub (Feb 14, 2025):
同样报错
https://github.com/kvcache-ai/ktransformers/issues/185
目测，目前对该蒸馏版本的模型尚不支持，
@Allen-Zengr commented on GitHub (Feb 14, 2025):
同样错误。
@NeilAlfred commented on GitHub (Feb 15, 2025):
同样错误。
@langlin commented on GitHub (Feb 16, 2025):
同样错误。ubuntu 24.04也一样。
@micellince commented on GitHub (Feb 16, 2025):
70b是这样的，32b能跑通，但是一对话也有问题
@ArSrNa commented on GitHub (Feb 16, 2025):
1.5B 亲测没跑通，
@Tyotw commented on GitHub (Feb 17, 2025):
@RiaYao commented on GitHub (Feb 17, 2025):
Ubuntu 22.04 DeepSeek-R1-Distill-Llama-8B也是同样错误
@EI-Dios commented on GitHub (Feb 18, 2025):
是不是蒸馏模型都不行，只有671b可以
@RiaYao commented on GitHub (Feb 18, 2025):
我的理解是这样，今天换了671b就可以了
@EI-Dios commented on GitHub (Feb 18, 2025):
你用671b大概需要多少硬盘和内存资源
@ArSrNa commented on GitHub (Feb 21, 2025):
671B Q4量化 410G左右磁盘，400G内存
@Atream commented on GitHub (Mar 3, 2025):
Currently, KTransformers can only achieve significant performance improvements for MOE models. For non-MOE models such as the 70B distilled model, the acceleration effect is minimal. There is no official YAML file provided, and users need to configure it themselves.