mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1074] balance_serve loses cache between prompts #756
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#756
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @createthis on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1074
Maybe this is by design, but I've noticed the
ktransformersbackend does not have any delay in responding after a huge prompt. However, thebalance_servebackend has a long delay in responding after a huge prompt.Here's the test case I ran to prove this:
Is this a bug, or is it just a consequence of multi-concurrency?
@qiyuxinlin commented on GitHub (Apr 8, 2025):
Hi @createthis, could you please post the model execution times printed in the terminal before and after the error occurred?
@magikRUKKOLA commented on GitHub (Apr 8, 2025):
Please post the commands how you running the inference in both cases. This is becoming interesting.
@createthis commented on GitHub (Apr 8, 2025):
@magikRUKKOLA here you go:
@qiyuxinlin there's no error. I think I'd need to record a video to get the logs. There's a lot of them and I'm bound to mix them up.
@magikRUKKOLA commented on GitHub (Apr 8, 2025):
@qiyuxinlin
@createthis Yes, I am having the same behaviour with
log of two requests to the same session (via the mods --continue):
EDIT:
@qiyuxinlin commented on GitHub (Apr 8, 2025):
容我理解下，您是想问一个 prefill 进入队列后会影响前一个任务的 decode 速率吗？
@magikRUKKOLA commented on GitHub (Apr 8, 2025):
Well, basically we are asking to clarify how the caching mechanism of a KV-cache pertained to a certain task/session is handled in a different backends. In case with a large context (in terms of tokens that have to be prefilled to get a certain context to continue the task/session/conversation) the behaviour of a ktransformers is different. Specifically, in the original ktransformers backend, it was possible to ask model a question and within a short term (like 30 sec or one minute) submit the next question and the time for a first token was minimal (pointing to some caching mechanisms to pull it out). So unfortunately, in the recent balance_serve backend no such behaviour can be seen -- it just skips some "cache" and starting all over (like having some kind of stateless behaviour). The question is -- is it possible for example to save such cache pertaining to the unique conversation somewhere in order just to preload it? For example, RTX series videocards have P2P capabilities (pcie master etc.) so if more than one GPU is presend it might be possible to save such caches and just switch between several GPUs in case if the subsequent request from the user to continue the session had been made (of course its not always the case, but often enough).
Please point to the right direction to enable or implement it.
@magikRUKKOLA commented on GitHub (Apr 8, 2025):
@qiyuxinlin just to make sure what we are talking about does make sense.
Consider any chat-like app with a bunch of chats for a single user. Be it mods or openhands ai or chatbox ai or whatever. The requirement here is to use Chain of Thought in the llm inference. That is, get the response, process it by other LLMs or just a reanimation of the previous conversation/session. With a prefill speed of 100 tps it would take a lot of time to process ... close to three minutes with 16k context?! ... this doesn't make sense. I would strongly advise you to look towards the export of the caches to revive the sessions. It is much cheaper to store the cache on some storage device (close to 10GB/s possible with raid0). It would be nice to have some SDK/API to get the current state and save/restore it.
Right?
@magikRUKKOLA commented on GitHub (Apr 9, 2025):
@qiyuxinlin
is that it?
github.com/kvcache-ai/ktransformers@94476ce5cc/csrc/balance_serve/kvc2/src/gpu_cache.cpp (L32)@qiyuxinlin commented on GitHub (Apr 9, 2025):
Hi @createthis and @magikRUKKOLA, Currently balance_serve does not support prefix cache. We plan to improve this feature in the next few versions.
@magikRUKKOLA commented on GitHub (Apr 9, 2025):
what about the dumps ?
github.com/kvcache-ai/ktransformers@94476ce5cc/ktransformers/server/balance_serve/settings.py (L52)got any docs?
@magikRUKKOLA commented on GitHub (Apr 9, 2025):
@qiyuxinlin
is it possible to run multiple instances of ktransformers with different backends (ktransformer, balance_serve) on a separate GPUs (the regions of RAM with the weights of the model are handled via the OS mappings, right? so because the data is static, the same physical mapping can be used by multiple processes, huh?)
@createthis commented on GitHub (Apr 9, 2025):
@qiyuxinlin Thank you. That's what I suspected. This makes
balance_serveunsuitable for agentic coding and long single user coding conversation use cases. Please don't deprecate thektransformersbackend, as it is ideal for these use cases.This project is awesome!