[GH-ISSUE #1441] [Bug] with kimi-k2 model, the chat session will not finish #958

New issue

Closed

opened 2026-10-29 14:46:55 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:46:55 +00:00

Owner

Originally created by @git-shan on GitHub (Jul 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1441

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

running kt 0.3.2 with kimi-k2-instruct gguf from hugging face kvchache team, chat session will not finish, keep generating tokens.

Reproduction

export TORCH_CUDA_ARCH_LIST="8.9"

python ktransformers/server/main.py
--host 0.0.0.0
--port 10002
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--model_path /home/deepseek/kimi-k2-m --gguf_path /home/deepseek/kimi-k2
--max_new_tokens 8192
--cache_lens 131072
--chunk_size 256
--max_batch_size 4
--backend_type balance_serve \

Environment

unbuntu linux, intel xeon cpu, 1T memory, 4090 w/ 48GB

Originally created by @git-shan on GitHub (Jul 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1441 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug running kt 0.3.2 with kimi-k2-instruct gguf from hugging face kvchache team, chat session will not finish, keep generating tokens. ### Reproduction export TORCH_CUDA_ARCH_LIST="8.9" python ktransformers/server/main.py \ --host 0.0.0.0 \ --port 10002 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --model_path /home/deepseek/kimi-k2-m --gguf_path /home/deepseek/kimi-k2 \ --max_new_tokens 8192 \ --cache_lens 131072 \ --chunk_size 256 \ --max_batch_size 4 \ --backend_type balance_serve \ ### Environment unbuntu linux, intel xeon cpu, 1T memory, 4090 w/ 48GB

bsakaguchi closed this issue

2026-10-29 14:46:55 +00:00

bsakaguchi commented

2026-10-29 14:46:56 +00:00

Author

Owner

@git-shan commented on GitHub (Jul 13, 2025):

just found new commit from main branch with pull request 1439

@git-shan commented on GitHub (Jul 13, 2025): just found new commit from main branch with pull request 1439