[GH-ISSUE #119] Test Benchmark with vllm-style #67

New issue

Closed

opened 2026-10-29 14:33:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:33:33 +00:00

Owner

Originally created by @fengyuanyu1 on GitHub (Jan 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/119

Hello, Ktransformers developers, Does Ktransformers have the script like benchmark_serving to evaluate the online inference performance?

Originally created by @fengyuanyu1 on GitHub (Jan 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/119 Hello, Ktransformers developers, Does Ktransformers have the script like [benchmark_serving](https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py) to evaluate the online inference performance?

bsakaguchi closed this issue

2026-10-29 14:33:33 +00:00

bsakaguchi commented

2026-10-29 14:33:34 +00:00

Author

Owner

@Atream commented on GitHub (Feb 10, 2025):

Thank you for your attention to us. Currently, KTransformers has only been optimized and tested for single-request scenarios and does not yet support continuous batching. We plan to implement continuous batching soon and will also provide the corresponding bench_serving code.

@Atream commented on GitHub (Feb 10, 2025): Thank you for your attention to us. Currently, KTransformers has only been optimized and tested for single-request scenarios and does not yet support continuous batching. We plan to implement continuous batching soon and will also provide the corresponding bench_serving code.

bsakaguchi referenced this issue

2026-10-29 14:51:34 +00:00

[PR #67] [MERGED] [fix] fix bugs about Qwen2-57B, install requirement, DockerFile #1341