[GH-ISSUE #1869] 配置 --kt-method LLAMAFILE 时，使用异常，回复内容不完整 #1165

New issue

Closed

opened 2026-10-29 14:49:29 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:49:29 +00:00

Owner

Originally created by @liuyuansharp on GitHub (Feb 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1869

Reminder

I have read the above rules and searched the existing issues.

System Info

System Info
KTransformers CLI v0.1.0

Python 3.12.7
平台 Linux 6.14.0-36-generic
CUDA 13.0

Packages:

kt-kernel 0.5.1
sglang 0.0.0.dev10185+g2ba1f0dea (PyPI)

Reproduction

使用 sglang.launch_server 启动，以下命令：

python -m sglang.launch_server
--host 0.0.0.0
--port 10002
--model /home/liuyuan/workspace/work/ai/ktransformer/models/Qwen3-Coder-Next
--kt-weight-path /home/liuyuan/workspace/work/ai/ktransformer/models/Qwen3-Coder-Next-GGUF/Q4_K_M
--kt-cpuinfer 20
--kt-threadpool-count 2
--kt-num-gpu-experts 30
--kt-method LLAMAFILE
--kt-gpu-prefill-token-threshold 400
--trust-remote-code
--mem-fraction-static 0.94
--max-running-requests 10
--served-model-name Qwen3-Coder-Next
--enable-mixed-chunk
--tensor-parallel-size 1
--enable-p2p-check
--disable-shared-experts-fusion
--chunked-prefill-size 4096
--max-total-tokens 20000
--attention-backend flashinfer

测试内容:
curl -s http://localhost:10002/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "MiniMax-M2.5",
"stream": false,
"messages": [
{"role": "user", "content": "hi, who are you?"}
]
}'

回复内容:
{"id":"7b0013cc513e4cab82911a03d5982de6","object":"chat.completion","created":1772107748,"model":"MiniMax-M2.5","choices":[{"index":0,"message":{"role":"assistant",

"content":"Hi",

"reasoning_content":null,
"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":"NaN happened"}],"usage":{"prompt_tokens":14,"total_tokens":16,"completion_tokens":2,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}

回复内容:
{"id":"91e0cd1c9e9f402280145bef3697b8cb","object":"chat.completion","created":1772108606,"model":"MiniMax-M2.5","choices":[{"index":0,"message":{"role":"assistant",

"content":"当然",

"reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":"NaN happened"}],"usage":{"prompt_tokens":11,"total_tokens":13,"completion_tokens":2,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}

问题：
提交的问题，回复内容都异常，检查后台，记录每次只有一条，不能持续回复后续内容。

Others

No response

Originally created by @liuyuansharp on GitHub (Feb 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1869 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info System Info KTransformers CLI v0.1.0 Python 3.12.7 平台 Linux 6.14.0-36-generic CUDA 13.0 Packages: kt-kernel 0.5.1 sglang 0.0.0.dev10185+g2ba1f0dea (PyPI) ### Reproduction 使用 sglang.launch_server 启动，以下命令： python -m sglang.launch_server --host 0.0.0.0 --port 10002 --model /home/liuyuan/workspace/work/ai/ktransformer/models/Qwen3-Coder-Next --kt-weight-path /home/liuyuan/workspace/work/ai/ktransformer/models/Qwen3-Coder-Next-GGUF/Q4_K_M --kt-cpuinfer 20 --kt-threadpool-count 2 --kt-num-gpu-experts 30 --kt-method LLAMAFILE --kt-gpu-prefill-token-threshold 400 --trust-remote-code --mem-fraction-static 0.94 --max-running-requests 10 --served-model-name Qwen3-Coder-Next --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion --chunked-prefill-size 4096 --max-total-tokens 20000 --attention-backend flashinfer 测试内容: curl -s http://localhost:10002/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "MiniMax-M2.5", "stream": false, "messages": [ {"role": "user", "content": "hi, who are you?"} ] }' 回复内容: {"id":"7b0013cc513e4cab82911a03d5982de6","object":"chat.completion","created":1772107748,"model":"MiniMax-M2.5","choices":[{"index":0,"message":{"role":"assistant", "content":"Hi", "reasoning_content":null, "tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":"NaN happened"}],"usage":{"prompt_tokens":14,"total_tokens":16,"completion_tokens":2,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} 测试内容: curl -s http://localhost:10002/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "MiniMax-M2.5", "stream": false, "messages": [ {"role": "user", "content": "讲一个笑话"} ] }' 回复内容: {"id":"91e0cd1c9e9f402280145bef3697b8cb","object":"chat.completion","created":1772108606,"model":"MiniMax-M2.5","choices":[{"index":0,"message":{"role":"assistant", "content":"当然", "reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":"NaN happened"}],"usage":{"prompt_tokens":11,"total_tokens":13,"completion_tokens":2,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} 问题：提交的问题，回复内容都异常，检查后台，记录每次只有一条，不能持续回复后续内容。 <img width="1210" height="488" alt="Image" src="https://github.com/user-attachments/assets/c759c917-8796-47e8-a3ef-0981a684b46d" /> <img width="1294" height="182" alt="Image" src="https://github.com/user-attachments/assets/2014bff5-9f98-429b-9215-5e17d92753aa" /> ### Others _No response_

bsakaguchi closed this issue

2026-10-29 14:49:29 +00:00

bsakaguchi commented

2026-10-29 14:49:30 +00:00

Author

Owner

@ErvinXie commented on GitHub (Feb 28, 2026):

matched_stop: "NaN happened" 表示推理过程中 logits 产生了 NaN 值，SGLang 检测到后主动停止了生成。这通常说明权重加载或计算过程中出现了数值问题。

建议排查：

确认 GGUF 权重与模型版本完全匹配（Qwen3-Coder-Next 的 GGUF 是从对应版本转换的）
检查服务启动日志中是否有权重加载警告或 key mismatch
尝试用 FP8 方式跑（如果有 FP8 safetensor 权重），排除 GGUF 量化精度问题
检查 CPU 是否支持 LLAMAFILE 所需的指令集（lscpu | grep avx）

坦率地说，我们目前对 GGUF 格式的支持比较久远，难以保证所有模型和量化组合的兼容性。未来我们会推出 AWQ 量化格式的支持，届时会提供更可靠的低比特量化推理体验。

This reply was generated by an AI agent and reviewed by a human.

@ErvinXie commented on GitHub (Feb 28, 2026): `matched_stop: "NaN happened"` 表示推理过程中 logits 产生了 NaN 值，SGLang 检测到后主动停止了生成。这通常说明权重加载或计算过程中出现了数值问题。建议排查： 1. 确认 GGUF 权重与模型版本完全匹配（Qwen3-Coder-Next 的 GGUF 是从对应版本转换的） 2. 检查服务启动日志中是否有权重加载警告或 key mismatch 3. 尝试用 FP8 方式跑（如果有 FP8 safetensor 权重），排除 GGUF 量化精度问题 4. 检查 CPU 是否支持 LLAMAFILE 所需的指令集（`lscpu | grep avx`）坦率地说，我们目前对 GGUF 格式的支持比较久远，难以保证所有模型和量化组合的兼容性。未来我们会推出 AWQ 量化格式的支持，届时会提供更可靠的低比特量化推理体验。 --- *This reply was generated by an AI agent and reviewed by a human.*

bsakaguchi commented

2026-10-29 14:49:30 +00:00

Author

Owner

@liuyuansharp commented on GitHub (Mar 1, 2026):

好的，谢谢。

我试着回退 sglang
git checkout 4605b77c7ff3e287a310d9d732f6b67dc8032229
然后再安装，目前不报这个错了。

@liuyuansharp commented on GitHub (Mar 1, 2026): 好的，谢谢。我试着回退 sglang git checkout 4605b77c7ff3e287a310d9d732f6b67dc8032229 然后再安装，目前不报这个错了。

bsakaguchi referenced this issue

2026-10-29 14:53:37 +00:00

[PR #1165] [MERGED] update maverick config #1576