mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1032] [Bug] localhost/v1/chat/completions无响应 #727
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#727
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @344303947 on GitHub (Apr 3, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1032
检查清单
问题描述
运行成功但是无响应
复现步骤
python ktransformers/server/main.py
--port 6699
--model_path /home/sean/models/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-UD-Q2_K_XL/
--gguf_path /home/sean/models/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-UD-Q2_K_XL/
--model_name unsloth/DeepSeek-V3-0324
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 32768
--chunk_size 256
--max_batch_size 4
--backend_type balance_serve
curl -X POST http://localhost:6699/v1/chat/completions -H "accept: application/json" -H "Content-Type: application/json" -d '{
"messages": [
{"role": "user", "content": "hello"}
],
"model": "DeepSeek-V3-0324",
"temperature": 0.3,
"top_p": 1.0,
}'"stream": true
环境信息
ubuntu 22.04 v2.0.4
@qiyuxinlin commented on GitHub (Apr 4, 2025):
您好，可能存在两种情况，显存不足或者 model_path 无法识别，请把 model_path 指定的文件夹命名为 ktransformers/configs/model_configs.json 中的键，这是个 bug，我们今天处理下
@344303947 commented on GitHub (Apr 4, 2025):
修改目录后，能运行了，但是响应很慢，有很多额外的日志打印出来，不知道是否有影响，建议可以不要打印出出来。
add query id: 1, batch.query_lengths: 5, batch_query_tokens: torch.Size([8029]), batch.block_indexes: tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
dtype=torch.int32)
prefill_batch_i: 5,
Model execution time (GPU): 756.125 ms
2025-04-04 14:52:03,158 - INFO - flashinfer.jit: Loading JIT ops: sampling
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-04-04 14:53:12,074 - INFO - flashinfer.jit: Finished loading JIT ops: sampling
30594
decode_batch_i: 1,
Model execution time (GPU): 203.499 ms
1175
decode_batch_i: 1,
Model execution time (GPU): 174.375 ms
28927
decode_batch_i: 1,
Model execution time (GPU): 177.644 ms
235
decode_batch_i: 1,
Model execution time (GPU): 177.389 ms
223
decode_batch_i: 1,
Model execution time (GPU): 179.681 ms
10457
decode_batch_i: 1,
Model execution time (GPU): 177.181 ms
34071
decode_batch_i: 1,
Model execution time (GPU): 175.576 ms
3950
decode_batch_i: 1,
Model execution time (GPU): 176.089 ms
4597
decode_batch_i: 1,
Model execution time (GPU): 177.203 ms
3467
decode_batch_i: 1,
Model execution time (GPU): 174.939 ms
1148
decode_batch_i: 1,
Model execution time (GPU): 176.859 ms
1
decode_batch_i: 1,
Model execution time (GPU): 176.417 ms
0
decode_batch_i: 1,
Model execution time (GPU): 177.240 ms
prefill length: 5, prefill time: 69.75378131866455, prefill tps 0.0716807018268716, decode length: 13, decode time: 2.420722007751465, decode tps 5.370298596192508
5
decode_batch_i: 1,
2025-04-04 14:53:14,733 INFO /home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 0.07146373016482553, decode 5.42803551052906. Time(s): tokenize 0.04242753982543945, prefill 69.9655613899231, decode 2.3949732780456543
@344303947 commented on GitHub (Apr 4, 2025):
后续继续提问速度倒是正常了点，日志看起来有点长
decode_batch_i: 1,
Model execution time (GPU): 174.469 ms
8725
decode_batch_i: 1,
Model execution time (GPU): 174.656 ms
1939
decode_batch_i: 1,
Model execution time (GPU): 175.267 ms
13491
decode_batch_i: 1,
Model execution time (GPU): 177.075 ms
9574
decode_batch_i: 1,
Model execution time (GPU): 178.370 ms
1148
decode_batch_i: 1,
Model execution time (GPU): 174.270 ms
1227
decode_batch_i: 1,
Model execution time (GPU): 176.466 ms
1
decode_batch_i: 1,
Model execution time (GPU): 176.813 ms
0
decode_batch_i: 1,
Model execution time (GPU): 180.259 ms
prefill length: 8, prefill time: 1.188171148300171, prefill tps 6.733036744281336, decode length: 309, decode time: 56.533483028411865, decode tps 5.465787413888983
5
decode_batch_i: 1,
2025-04-04 14:55:22,546 INFO /home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 5.831316306169187, decode 5.465283430311012. Time(s): tokenize 0.0010752677917480469, prefill 1.3719029426574707, decode 56.5386962890625
Model execution time (GPU): 180.604 ms
223
decode_batch_i: 1,
Model execution time (GPU): 178.608 ms
939
@344303947 commented on GitHub (Apr 4, 2025):
搞定了 ，~ 【Ktransformers 0.2.4支持DeepSeek-V3-0324并发部署测试】https://www.bilibili.com/video/BV1JLZoYGENu?vd_source=954828d4da44f3115ef5b39efde8a320