kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #922] [Bug] 中文提问，英文思考/多次思考/丢失思考标签 #659

New issue
Closed
opened 2026-10-29 14:42:31 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 14:42:31 +00:00
Owner
Copy link

Originally created by @yeungtuzi on GitHub (Mar 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/922

Originally assigned to: @Atream on GitHub.

检查清单

  • 1. 我已经搜索过相关问题，但未能获得预期的帮助
  • 2. 该问题在最新版本中尚未修复
  • 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
  • 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
  • 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

如题，并参考附件

chat-export-1742293550559.json

复现步骤

最新git版，iq1s-fp8模型
命令为 cat start-engine.sh

#!/bin/bash
cd /home/lf/ktransformers
source /home/lf/.profile
source /home/lf/.bashrc
source /home/lf/.venv/bin/activate
export TORCH_CUDA_ARCH_LIST="8.9"
export HF_ENDPOINT=https://hf-mirror.com
ktransformers \
 --gguf_path /opt/DeepSeek-R1-IQ1S-FP8/ \
 --model_path deepseek-ai/DeepSeek-R1 \
 --model_name DeepSeek-R1 \
 --cpu_infer 30 \
 --max_new_tokens 16384 \
 --cache_lens 16384 \
 --force_think \
 --max_response_tokens 16384 \
 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml \
 --use_cuda_graph \
 --host 0.0.0.0 \
 --mode="long_context" \
--log_level "info" \
 --port 10087

似乎0.23post1无此问题，另外，新版加载模型速度极快

环境信息

AMD Ryzen Threadripper PRO 7975WX 32-Cores
512GB RAM
RTX 4090D
ubuntu 24.04 server
nvidia-driver-570
cuda toolkit 12.8
python 3.12
flashinfer git最新版编译安装

Originally created by @yeungtuzi on GitHub (Mar 18, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/922 Originally assigned to: @Atream on GitHub. ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 如题，并参考附件 [chat-export-1742293550559.json](https://github.com/user-attachments/files/19316342/chat-export-1742293550559.json) ### 复现步骤 最新git版，iq1s-fp8模型 命令为 cat start-engine.sh ``` #!/bin/bash cd /home/lf/ktransformers source /home/lf/.profile source /home/lf/.bashrc source /home/lf/.venv/bin/activate export TORCH_CUDA_ARCH_LIST="8.9" export HF_ENDPOINT=https://hf-mirror.com ktransformers \ --gguf_path /opt/DeepSeek-R1-IQ1S-FP8/ \ --model_path deepseek-ai/DeepSeek-R1 \ --model_name DeepSeek-R1 \ --cpu_infer 30 \ --max_new_tokens 16384 \ --cache_lens 16384 \ --force_think \ --max_response_tokens 16384 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml \ --use_cuda_graph \ --host 0.0.0.0 \ --mode="long_context" \ --log_level "info" \ --port 10087 ``` 似乎0.23post1无此问题，另外，新版加载模型速度极快 ### 环境信息 AMD Ryzen Threadripper PRO 7975WX 32-Cores 512GB RAM RTX 4090D ubuntu 24.04 server nvidia-driver-570 cuda toolkit 12.8 python 3.12 flashinfer git最新版编译安装
bsakaguchi commented 2026-10-29 14:42:32 +00:00
Author
Owner
Copy link

@yeungtuzi commented on GitHub (Mar 18, 2025):

chat-新对话-2.txt

<!-- gh-comment-id:2732584105 --> @yeungtuzi commented on GitHub (Mar 18, 2025): [chat-新对话-2.txt](https://github.com/user-attachments/files/19316425/chat-.-2.txt)
bsakaguchi commented 2026-10-29 14:42:32 +00:00
Author
Owner
Copy link

@dn833 commented on GitHub (Mar 18, 2025):

我也遇到了同样的问题

ktransformers 0.2.3.post2+torch26avx2
更新这个版本之后才出现的bug

INFO:     172.16.200.1:8485 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-03-19 07:37:22,559 WARNING /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[167]: change system to user
2025-03-19 07:37:22,559 WARNING /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[173]: merge two adjacent user messages
2025-03-19 07:37:22,581 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 90])
<think>
2025-03-19 07:37:22,582 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 92])
2025-03-19 07:37:22,584 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 8
2025-03-19 07:37:22,587 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 84])
2025-03-19 07:37:22,587 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 8])
2025-03-19 07:37:22,588 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 8 to 92
2025-03-19 07:37:27,135 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[333]: args.max_new_tokens: 65536, cache_lens: 8192, seq_length: 93
2025-03-19 07:37:27,136 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[338]: max_new_tokens: 8098
</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

您好！有什么需要帮助的吗？</think>

然后就是死循环到超出上下文限制才能结束本次请求

<!-- gh-comment-id:2734963945 --> @dn833 commented on GitHub (Mar 18, 2025): 我也遇到了同样的问题 ktransformers 0.2.3.post2+torch26avx2 更新这个版本之后才出现的bug ``` INFO: 172.16.200.1:8485 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2025-03-19 07:37:22,559 WARNING /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[167]: change system to user 2025-03-19 07:37:22,559 WARNING /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[173]: merge two adjacent user messages 2025-03-19 07:37:22,581 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 90]) <think> 2025-03-19 07:37:22,582 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 92]) 2025-03-19 07:37:22,584 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 8 2025-03-19 07:37:22,587 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 84]) 2025-03-19 07:37:22,587 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 8]) 2025-03-19 07:37:22,588 DEBUG /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 8 to 92 2025-03-19 07:37:27,135 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[333]: args.max_new_tokens: 65536, cache_lens: 8192, seq_length: 93 2025-03-19 07:37:27,136 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[338]: max_new_tokens: 8098 </think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> 您好！有什么需要帮助的吗？</think> ``` 然后就是死循环到超出上下文限制才能结束本次请求
bsakaguchi commented 2026-10-29 14:42:32 +00:00
Author
Owner
Copy link

@Atream commented on GitHub (Mar 19, 2025):

I have fixed some bug. Please try the latest code.

<!-- gh-comment-id:2735819992 --> @Atream commented on GitHub (Mar 19, 2025): I have fixed some bug. Please try the latest code.
bsakaguchi commented 2026-10-29 14:42:32 +00:00
Author
Owner
Copy link

@yeungtuzi commented on GitHub (Mar 19, 2025):

测试了多个用例，性能似乎有极其轻微的下降（基本不影响），错误已经修复。
Performance(T/s): prefill 6.016409809164247, decode 11.657862138950701. Time(s): tokenize 0.009207487106323242, prefill 0.1662120819091797, decode 106.02286982536316

<!-- gh-comment-id:2736603905 --> @yeungtuzi commented on GitHub (Mar 19, 2025): 测试了多个用例，性能似乎有极其轻微的下降（基本不影响），错误已经修复。 Performance(T/s): prefill 6.016409809164247, decode 11.657862138950701. Time(s): tokenize 0.009207487106323242, prefill 0.1662120819091797, decode 106.02286982536316
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#659
No description provided.