[GH-ISSUE #661] Feature Request / 功能需求：增加并发支持 #484

New issue

Closed

opened 2026-10-29 14:40:21 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:40:21 +00:00

Owner

Originally created by @yeungtuzi on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/661

在系统能够支持20+ tps的情况下（假定），提供多线程并发服务能力，这样可以共享内存和显存数据，避免多进程多开的额外内存和显存浪费。
When system is powerful enough, provide --parallels parameter to provide more usable service.

Originally created by @yeungtuzi on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/661 在系统能够支持20+ tps的情况下（假定），提供多线程并发服务能力，这样可以共享内存和显存数据，避免多进程多开的额外内存和显存浪费。 When system is powerful enough, provide --parallels parameter to provide more usable service.

bsakaguchi

2026-10-29 14:40:21 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:40:22 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 25, 2025):

你指的是通过为不同提示（prompt）提供多槽位并发推理来提高总体吞吐量吗？这类似于 llama.cpp 的 --parallel N 参数功能？

遗憾的是，ktransformers 之所以如此高效，正是因为它基于单次非并发推理的假设，这使得它能利用 CUDA Graphs 大幅加速生成过程。

因此在当前 CPU + GPU 部分卸载（partial offload）的场景下，我认为这一方案并不适用。

若需通过并发生成来提升总体吞吐量（代价是单次生成速度降低），可能需要将完整模型加载到 384~768GB 的 VRAM 中，并使用不同的推理引擎。

可参考此指南了解现有基础讨论和可能的优化方向。

感谢！

Are you talking about adding multiple slots for concurrent inferencing of different prompts to increase aggregate throughput? This is similar to llama.cpp --parallel N feature?

Unfortunately, the reason ktransformers is so fast is because of the assumption of a single non-concurrent inference allows to use CUDA Graphs to speed up generation significantly.

So I don't think this is useful for the current situation of CPU + GPU partial offload.

If you want concurrent generation for higher aggregate throughput (at the cost of lower single generations) you will likely need to load the entire model into 384~768GB VRAM and use a different inference engine.

Check out this guide for some basic discussion and possible optimizations that are currently available.

Thanks!

@ubergarm commented on GitHub (Feb 25, 2025): 你指的是通过为不同提示（prompt）提供多槽位并发推理来提高总体吞吐量吗？这类似于 llama.cpp 的 `--parallel N` 参数功能？遗憾的是，ktransformers 之所以如此高效，正是因为它基于**单次非并发推理**的假设，这使得它能利用 CUDA Graphs 大幅加速生成过程。因此在当前 CPU + GPU 部分卸载（partial offload）的场景下，我认为这一方案并不适用。若需通过并发生成来提升总体吞吐量（代价是单次生成速度降低），可能需要将完整模型加载到 384~768GB 的 VRAM 中，并使用不同的推理引擎。可参考[此指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)了解现有基础讨论和可能的优化方向。感谢！ --- Are you talking about adding multiple slots for concurrent inferencing of different prompts to increase aggregate throughput? This is similar to llama.cpp `--parallel N` feature? Unfortunately, the reason ktransformers is so fast is because of the assumption of a single non-concurrent inference allows to use CUDA Graphs to speed up generation significantly. So I don't think this is useful for the current situation of CPU + GPU partial offload. If you want concurrent generation for higher aggregate throughput (at the cost of lower single generations) you will likely need to load the entire model into 384~768GB VRAM and use a different inference engine. Check out [this guide](https://github.com/ubergarm/r1-ktransformers-guide/) for some basic discussion and possible optimizations that are currently available. Thanks!

bsakaguchi commented

2026-10-29 14:40:22 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！