mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1146] 没有复现样例里并发比单并发总token数多 #794
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#794
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @JennieGao-njust on GitHub (Apr 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1146
Originally assigned to: @qiyuxinlin on GitHub.
Request 1: Decode Speed = 2.50 tokens/s,First packet time = 40.24s
Request 0: Decode Speed = 3.13 tokens/s,First packet time = 79.39s
Request 3: Decode Speed = 3.61 tokens/s,First packet time = 44.04s
Request 2: Decode Speed = 4.26 tokens/s,First packet time = 79.16s
单并发是11tokens/s 几乎是对半开的，而样例中单并发是17tokens/s， 多并发是4*10tokens/s
GPU L20 单卡48G ，内存500G ubantu22.04 模型是2.71bit的量化模型，
运行参数：python ktransformers/server/main.py
--port 10002
--model_path /mnt/work/models/deepseek-ai/DeepSeek-V3-0324-GGUF/DeepSeek-V3
--gguf_path /mnt/work/models/deepseek-ai/DeepSeek-V3-0324-GGUF/UD-Q2_K_XL/
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 131072
--chunk_size 128
--max_batch_size 10
--backend_type balance_serve
--cpu_infer 60
@yeungtuzi commented on GitHub (Apr 16, 2025):
多跑几次？
@CodeZ-Hao commented on GitHub (Apr 16, 2025):
单路数据：
2025-04-16 02:31:47,548 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 20.344502434954695, decode 10.545587785305178. Time(s): tokenize 0.02379608154296875, prefill 0.6881465911865234, decode 25.50829839706421
四路数据：
2025-04-16 02:30:06,918 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 8.589687680200534, decode 4.283602855131995. Time(s): tokenize 0.0004661083221435547, prefill 1.6298613548278809, decode 46.689669132232666
2025-04-16 02:30:07,153 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 9.800766899990219, decode 4.286149413203856. Time(s): tokenize 0.023328542709350586, prefill 1.428459644317627, decode 47.12854838371277
2025-04-16 02:30:31,614 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 8.58448384700154, decode 4.846918181013971. Time(s): tokenize 0.0005340576171875, prefill 1.6308493614196777, decode 71.3855664730072
2025-04-16 02:30:34,051 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 0.28350677190052265, decode 6.444336549509431. Time(s): tokenize 0.0004978179931640625, prefill 49.38153648376465, decode 26.06940197944641
这里第四个并发的prefill有问题，导致decode时间没有与其他三路完全重合，这里怀疑是达到CPU瓶颈了。
三路并发：
2025-04-16 02:35:28,422 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 8.799132797695453, decode 4.2979965800664965. Time(s): tokenize 0.00048160552978515625, prefill 1.5910658836364746, decode 54.21130418777466
2025-04-16 02:35:29,854 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 8.793181001720443, decode 4.331210861588591. Time(s): tokenize 0.0006108283996582031, prefill 1.5921428203582764, decode 55.64263844490051
2025-04-16 02:35:36,493 INFO /opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[90]: Performance(T/s): prefill 10.18186055350794, decode 4.975811991051592. Time(s): tokenize 0.023708820343017578, prefill 1.3749942779541016, decode 62.502361536026
三路的数据基本重合，因此我的测试结论是 三路并发 4.3*3=12.9，对于单路输出10.54，提升约22%
@CodeZ-Hao commented on GitHub (Apr 16, 2025):
我做的性能相关的测试：https://blog.csdn.net/a66920164/article/details/147031093