[GH-ISSUE #74] How can i run internlm2_5-7b-chat-1m in ktransformers? #37

New issue

Closed

opened 2026-10-29 14:32:54 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 14:32:54 +00:00

Owner

Originally created by @Ma1oneZhang on GitHub (Sep 3, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/74

How can i run internlm2_5-7b-chat-1m in ktransformers?

I have downloaded the internlm2_5-7b-chat-1m model file and converted it to a gguf file. But ktransformer still reports an error like that:

load_weights(module, gguf_loader)
File "/somewhere/ktransformers/ktransformers/util/utils.py", line 83, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/somewhere/ktransformers/ktransformers/util/utils.py", line 83, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/somewhere/ktransformers/ktransformers/util/utils.py", line 81, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/somewhere/ktransformers/ktransformers/util/utils.py", line 76, in load_cur_state_dict
raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find model.tok_embeddings.weight in GGUF file!

Originally created by @Ma1oneZhang on GitHub (Sep 3, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/74 How can i run internlm2_5-7b-chat-1m in ktransformers? I have downloaded the internlm2_5-7b-chat-1m model file and converted it to a gguf file. But ktransformer still reports an error like that: > load_weights(module, gguf_loader) File "/somewhere/ktransformers/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/somewhere/ktransformers/ktransformers/util/utils.py", line 83, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/somewhere/ktransformers/ktransformers/util/utils.py", line 81, in load_weights load_cur_state_dict(module, gguf_loader, prefix) File "/somewhere/ktransformers/ktransformers/util/utils.py", line 76, in load_cur_state_dict raise Exception(f"can't find {translated_key} in GGUF file!") Exception: can't find model.tok_embeddings.weight in GGUF file!

bsakaguchi closed this issue

2026-10-29 14:32:54 +00:00

bsakaguchi commented

2026-10-29 14:32:55 +00:00

Author

Owner

@ovowei commented on GitHub (Sep 3, 2024):

You need to first use the tools provided by InternLM to convert the internlm2_5-7b-chat-1m model into the LLaMA format. After that, you can convert it to a GGUF file. When using this method to download the model, please note that the model configuration needs to be in the LLaMA format, and the tokenizer should use the InternLM version.

As the tutorial here mentions, we have uploaded the model configuration, GGUF file, and tokenizer to a Hugging Face repository. We recommend downloading the model directly from this link to avoid the cumbersome process of merging the related files.

@ovowei commented on GitHub (Sep 3, 2024): You need to first use the [tools](https://github.com/InternLM/InternLM/tree/main/tools) provided by InternLM to convert the internlm2_5-7b-chat-1m model into the LLaMA format. After that, you can convert it to a GGUF file. When using this method to download the model, please note that the model configuration needs to be in the LLaMA format, and the tokenizer should use the InternLM version. As the tutorial [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/long_context_tutorial.md) mentions, we have uploaded the model configuration, GGUF file, and tokenizer to a Hugging Face repository. We recommend downloading the model directly from [this link](https://huggingface.co/nilv234/internlm2_5_to_llama_1m/tree/main) to avoid the cumbersome process of merging the related files.

bsakaguchi commented

2026-10-29 14:32:55 +00:00

Author

Owner

@TKONIY commented on GitHub (Sep 3, 2024):

You need to first use the tools provided by InternLM to convert the internlm2_5-7b-chat-1m model into the LLaMA format. After that, you can convert it to a GGUF file. When using this method to download the model, please note that the model configuration needs to be in the LLaMA format, and the tokenizer should use the InternLM version.

As the tutorial here mentions, we have uploaded the model configuration, GGUF file, and tokenizer to a Hugging Face repository. We recommend downloading the model directly from this link to avoid the cumbersome process of merging the related files.

Do you plan to support loading the files from huggingface like AutoConfig in transformers?

@TKONIY commented on GitHub (Sep 3, 2024): > You need to first use the [tools](https://github.com/InternLM/InternLM/tree/main/tools) provided by InternLM to convert the internlm2_5-7b-chat-1m model into the LLaMA format. After that, you can convert it to a GGUF file. When using this method to download the model, please note that the model configuration needs to be in the LLaMA format, and the tokenizer should use the InternLM version. > > As the tutorial [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/long_context_tutorial.md) mentions, we have uploaded the model configuration, GGUF file, and tokenizer to a Hugging Face repository. We recommend downloading the model directly from [this link](https://huggingface.co/nilv234/internlm2_5_to_llama_1m/tree/main) to avoid the cumbersome process of merging the related files. Do you plan to support loading the files from huggingface like `AutoConfig` in transformers?

bsakaguchi commented

2026-10-29 14:32:55 +00:00

Author

Owner

@ovowei commented on GitHub (Sep 3, 2024):

It's likely coming at some point, but not anytime soon.

@ovowei commented on GitHub (Sep 3, 2024): It's likely coming at some point, but not anytime soon.

bsakaguchi commented

2026-10-29 14:32:55 +00:00

Author

Owner

@TKONIY commented on GitHub (Sep 3, 2024):