mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #145] 哥们，0.2版本在编译什么东西啊 #79
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#79
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @qfxdz on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/145
好家伙，占了我300G内存64核全部拉满，半个小时没编译完；
另附个0.3版本的报错：
"ValueError: Unrecognized configuration class <class 'transformers_modules.configuration_deepseek.DeepseekV3Config'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, BarkConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BlipConfig, Blip2Config
, BloomConfig, BridgeTowerConfig, BrosConfig, CamembertConfig, CanineConfig, ChameleonConfig, ChineseCLIPConfig, ClapConfig, CLIPConfig, CLIPSegConfig, ClvpConfig, LlamaConfig, CodeGenConfig, CohereConfig, ConvBertConfig, CpmAnt
Config, CTRLConfig, Data2VecAudioConfig, Data2VecTextConfig, DbrxConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, DPRConfig, ElectraConfig, ErnieConfig, ErnieMConfig, EsmConfig, FalconConfig, FastSpeech2ConformerConfig,
FlaubertConfig, FNetConfig, FSMTConfig, FunnelConfig, GemmaConfig, Gemma2Config, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GPTSanJapaneseConfig, Groundi
ngDinoConfig, GroupViTConfig, HubertConfig, IBertConfig, IdeficsConfig, Idefics2Config, InstructBlipConfig, InstructBlipVideoConfig, JambaConfig, JetMoeConfig, JukeboxConfig, Kosmos2Config, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LlamaConfig, LlavaConfig, LlavaNextVideoConfig, LlavaNextConfig, LongformerConfig, LongT5Config, LukeConfig, LxmertConfig, M2M100Config, MambaConfig, MarianConfig, MBartConfig, MegaConfig, Me
gatronBertConfig, MgpstrConfig, MistralConfig, MixtralConfig, MobileBertConfig, MPNetConfig, MptConfig, MraConfig, MT5Config, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NezhaConfig, NllbMoeConfig, NystromformerConfig, OlmoConfig, OneFormerConfig, OpenAIGPTConfig, OPTConfig, Owlv2Config, OwlViTConfig, PaliGemmaConfig, PegasusConfig, PegasusXConfig, PerceiverConfig, PersimmonConfig, PhiConfig, Phi3Config, Pix2StructConfig, PLBartConfig, ProphetNetC
onfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RagConfig, RealmConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RetriBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig
, SeamlessM4TConfig, SeamlessM4Tv2Config, SiglipConfig, Speech2TextConfig, Speech2Text2Config, SpeechT5Config, SplinterConfig, SqueezeBertConfig, StableLmConfig, Starcoder2Config, SwitchTransformersConfig, T5Config, TapasConfig,
TransfoXLConfig, TvpConfig, UdopConfig, UMT5Config, VideoLlavaConfig, ViltConfig, VipLlavaConfig, VisualBertConfig, VitsConfig, Wav2Vec2Config, Wav2Vec2BertConfig, Wav2Vec2ConformerConfig, WhisperConfig, XCLIPConfig, XGLMConfig
, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig."
不知道是不是版本问题，DeepSeek-R1_IQ1_S的gguf模型报错
@Azure-Tang commented on GitHub (Feb 11, 2025):
目前不支持iqx系列，只支持qx-k-m。其余格式应该在未来几天会支持一下。以及看报错，应该是编译完以后运行出错找不到config文件？可以贴一下你的运行命令嘛~
@qfxdz commented on GitHub (Feb 11, 2025):
好的好的，多谢多谢，我去试试km，多谢多谢
@Azure-Tang commented on GitHub (Feb 11, 2025):
记得贴一下~
@qfxdz commented on GitHub (Feb 11, 2025):
运行命令“python -m ktransformers.local_chat --model_path /mnt/data/deepseek/DeepSeek-R1/ --gguf_path /mnt/data/deepseek/DeepSeek-R1/DeepSeek-R1_IQ1.gguf --cpu_infer 65 --max_new_tokens 1000”
/mnt/data/deepseek/DeepSeek-R1目录包含：config.json，generation_config.json，configuration.json DeepSeek-R1_IQ1.gguf为1.58量化模型合并得来（https://www.modelscope.cn/models/unsloth/DeepSeek-R1-GGUF/files）
@qfxdz commented on GitHub (Feb 11, 2025):
@qfxdz commented on GitHub (Feb 11, 2025):
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home//anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 378, in init
KLinearBase.init(self, key, gguf_loader, config, orig_module, generate_device, **kwargs)
File "/home//anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 61, in init
shape = self.gguf_loader.tensor_info[key + ".weight"]["shape"]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'blk.0.attn_q_a.weight'
我尝试重新下载了配置文件，之前的问题解决了，但现在报错KeyError: 'blk.0.attn_q_a.weight'
@KMSorSMS commented on GitHub (Feb 12, 2025):
你的命令还是：
python -m ktransformers.local_chat --model_path /mnt/data/deepseek/DeepSeek-R1/ --gguf_path /mnt/data/deepseek/DeepSeek-R1/DeepSeek-R1_IQ1.gguf --cpu_infer 65 --max_new_tokens 1000这个吗？我们目前不支持IQ1量化，未来几天内会更新对它的支持@lw3259111 commented on GitHub (Feb 12, 2025):
这是我在qwen2上面的执行
python -m ktransformers.local_chat --model_path Qwen/Qwen2-57B-A14B-Instruct --gguf_path /data/models/qwen2-57b-a14b-instruct-gguf/
Injecting model.layers.0 as default [226/1928]
Injecting model.layers.0.self_attn as default
Injecting model.layers.0.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.0.mlp as ktransformers.operators.experts . KQwen2MoeSparseMoeBlock
Injecting model.layers.0.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.0.mlp.shared_expert as default
Injecting model.layers.0.mlp.shared_expert.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.shared_expert.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.shared_expert.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.shared_expert.act_fn as default
Injecting model.layers.0.mlp.shared_expert_gate as ktransformers.operators.linear . KTransformersLinear
Warning: orig_module is not set, but has in_features or out_features equals to 1, can't get in_features and out_features from GGUF
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 179, in
fire.Fire(local_chat)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 110, in local_chat
optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
inject(module, optimize_config, model_config, gguf_loader)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
[Previous line repeated 1 more time]
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 33, in inject
inject_module=module_cls(key = inject_module_meta["key"], gguf_loader = gguf_loader, config = model_config, orig_module=child, **inject_module_meta["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 378, in init
KLinearBase.init(self, key, gguf_loader, config, orig_module, generate_device, **kwargs)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 65, in init
self.out_features = self.gguf_loader.tensor_info[key + ".weight"]["shape"][1]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
@Atream commented on GitHub (Feb 12, 2025):
@qingfengdaozhang --gguf_path /mnt/data/deepseek/DeepSeek-R1/
这个参数需要是包含GGUF的目录，而非GGUF文件的路径，在你的场景下应该向上面那么写。
qwen同理，但记得注意一个目录里只有一个模型的GGUF文件，对于qwen需要再建立一个目录
@KMSorSMS commented on GitHub (Feb 12, 2025):
你的文件是不是下载来自：
https://modelscope.cn/models/Qwen/Qwen2-57B-A14B-Instruct-GGUF/files
或者说对应的huggingface，这文件夹下有多种不同的量化文件，你应该把单个量化版本放在独立的文件夹下，然后指定这个文件夹
@buliaoyin commented on GitHub (Feb 12, 2025):
IQX系列目前跑下来的输出是随机字符，速度很快，期待能支持
@qfxdz commented on GitHub (Feb 12, 2025):
okk，晓得了，多谢多谢，已经跑起来了，但是速度不是很理想，Q4量化的内存占用不是很高（30G/754G），可能还是存在问题
@qfxdz commented on GitHub (Feb 12, 2025):
好的好的，多谢多谢，再请教一下正常运行是什么样子的，我现在速度不是很理想，双CPU Intel(R) Xeon(R) Gold 6226R ， 0.3token/s，Q4量化的内存占用不是很高（30G/754G），环境用的0.2 fancy版本，显卡4090*8 正常单卡占用，还是这个命令，只是将-gguf_path修改正确了
@Atream commented on GitHub (Feb 12, 2025):
0.3 token肯定是不正常的，R1的Q4量化也不应该只占用30G内存，有没有可能是对单个进程内存的使用做了限制，导致部分权重在swap空间？以及你的CPU只有16个物理核心，用两个NUMA应该设置CPU_Infer=33，即实际总物理核心数+1
@qfxdz commented on GitHub (Feb 12, 2025):
多谢多谢，现在速度3token/s了，感觉很可以了，我又去看了一下内存占用，应该是buf占的比较多，应该没啥问题，这个cpu估计就这样了，我去找一些cpu核数更高的试试
@lw3259111 commented on GitHub (Feb 12, 2025):
我在你提供的目录下下载了qwen2-57b-a14b-instruct-q4_k_m.gguf，
/data/models/qwen2-57b-a14b-instruct-gguf/ 目录下只有qwen2-57b-a14b-instruct-q4_k_m.gguf文件
我的命令是
ktransformers --model_path Qwen/Qwen2-57B-A14B-Instruct --gguf_path /data/models/qwen2-57b-a14b-instruct-gguf/ --port 10002 --web True仍然报错
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) [Previous line repeated 1 more time] File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 33, in inject inject_module=module_cls(key = inject_module_meta["key"], gguf_loader = gguf_loader, config = model_config, orig_module=child, **inject_module_meta["kwargs"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 378, in __init__ KLinearBase.__init__(self, key, gguf_loader, config, orig_module, generate_device, **kwargs) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 65, in __init__ self.out_features = self.gguf_loader.tensor_info[key + ".weight"]["shape"][1] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^ IndexError: list index out of range@KMSorSMS commented on GitHub (Feb 13, 2025):
是个bug，最新的commit已经修复，试试呢
@lw3259111 commented on GitHub (Feb 13, 2025):
@
已测，可以了，感谢
@zhangv2020 commented on GitHub (Feb 16, 2025):
各位大佬，我为何一直测试都是报错： raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.DeepSeek-R1.configuration_deepseek.DeepseekV3Config'> to build an AutoTokenizer.。解决不了，还请各位提供一下经验
@zhangv2020 commented on GitHub (Feb 16, 2025):
root@47290e016002:/home/ktransformers-0.2.0# python -m ktransformers.local_chat --model_path=/home/client_data/ktrans/deepseek-ai/DeepSeek-R1 --gguf_path /home/client_data/ktrans/DeepSeek-R1-Q4_K_M-GGUF
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/ktransformers-0.2.0/ktransformers/local_chat.py", line 278, in
fire.Fire(local_chat)
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ktransformers-0.2.0/ktransformers/local_chat.py", line 176, in local_chat
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 925, in from_pretrained
raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.DeepSeek-R1.configuration_deepseek.DeepseekV3Config'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, BarkConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BlipConfig, Blip2Config, BloomConfig, BridgeTowerConfig, BrosConfig, CamembertConfig, CanineConfig, ChameleonConfig, ChineseCLIPConfig, ClapConfig, CLIPConfig, CLIPSegConfig, ClvpConfig, LlamaConfig, CodeGenConfig, CohereConfig, ConvBertConfig, CpmAntConfig, CTRLConfig, Data2VecAudioConfig, Data2VecTextConfig, DbrxConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, DPRConfig, ElectraConfig, ErnieConfig, ErnieMConfig, EsmConfig, FalconConfig, FastSpeech2ConformerConfig, FlaubertConfig, FNetConfig, FSMTConfig, FunnelConfig, GemmaConfig, Gemma2Config, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GPTSanJapaneseConfig, GroundingDinoConfig, GroupViTConfig, HubertConfig, IBertConfig, IdeficsConfig, Idefics2Config, InstructBlipConfig, InstructBlipVideoConfig, JambaConfig, JetMoeConfig, JukeboxConfig, Kosmos2Config, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LlamaConfig, LlavaConfig, LlavaNextVideoConfig, LlavaNextConfig, LongformerConfig, LongT5Config, LukeConfig, LxmertConfig, M2M100Config, MambaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MgpstrConfig, MistralConfig, MixtralConfig, MobileBertConfig, MPNetConfig, MptConfig, MraConfig, MT5Config, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NezhaConfig, NllbMoeConfig, NystromformerConfig, OlmoConfig, OneFormerConfig, OpenAIGPTConfig, OPTConfig, Owlv2Config, OwlViTConfig, PaliGemmaConfig, PegasusConfig, PegasusXConfig, PerceiverConfig, PersimmonConfig, PhiConfig, Phi3Config, Pix2StructConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RagConfig, RealmConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RetriBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, SeamlessM4TConfig, SeamlessM4Tv2Config, SiglipConfig, Speech2TextConfig, Speech2Text2Config, SpeechT5Config, SplinterConfig, SqueezeBertConfig, StableLmConfig, Starcoder2Config, SwitchTransformersConfig, T5Config, TapasConfig, TransfoXLConfig, TvpConfig, UdopConfig, UMT5Config, VideoLlavaConfig, ViltConfig, VipLlavaConfig, VisualBertConfig, VitsConfig, Wav2Vec2Config, Wav2Vec2BertConfig, Wav2Vec2ConformerConfig, WhisperConfig, XCLIPConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig.