open-webui/computer

Fork 0

mirror of https://github.com/open-webui/computer.git synced 2026-11-05 01:17:24 +00:00

[GH-ISSUE #238] bug: Every new request invalidates the full context cache, causing llama.cpp to reprocess the entire context #151

New issue

Open

opened 2026-10-18 14:03:24 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-18 14:03:24 +00:00

Owner

Originally created by @JKGeovision on GitHub (Aug 26, 2026).
Original GitHub issue: https://github.com/open-webui/computer/issues/238

Hello,

I noticed that every follow-up request in an ongoing chat appears to invalidate the complete context cache, forcing llama.cpp to reprocess the entire prompt from scratch even though the chat history is not modified.

Steps to reproduce:

Start a chat session in Open Computer connected to a llama.cpp backend (in my case: model Qwen3.8 27B (xhigh) proxied on port 45723).
Send a message and wait for the response to complete (the full context is loaded and cached).
Send a follow-up message without modifying the chat history.
Check the llama.cpp logs for the second request.

Instead of reusing the cached KV context and only processing the new tokens, the logs show the full prompt being reprocessed from the start on the follow-up request (the prompt is roughly 100k+ tokens based on the progress fractions, so the re-prefill penalty is substantial):

218.52.490.688 I srv proxy_reques: proxying request to model Qwen3.8 27B (xhigh) on port 45723
[45723] 202.49.229.491 I slot get_availabl: id 1 | task -1 | selected slot by LRU, t_last = 13320538860
[45723] 202.49.232.130 I slot launch_slot_: id 1 | task 129052 | processing task, is_child = 0
[45723] 202.54.370.411 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 4096, progress = 0.04, t = 4.92 s / 832.76 tokens per second
[45723] 202.56.400.910 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 5590, progress = 0.05, t = 6.95 s / 804.42 tokens per second
[45723] 202.59.256.537 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 7638, progress = 0.07, t = 9.80 s / 779.01 tokens per second
[45723] 203.02.166.965 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 9686, progress = 0.09, t = 12.72 s / 761.77 tokens per second
[45723] 203.05.183.430 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 11734, progress = 0.10, t = 15.73 s / 745.89 tokens per second
[45723] 203.08.304.425 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 13782, progress = 0.12, t = 18.85 s / 731.04 tokens per second

Expected behavior:
For consecutive requests within the same chat where the history is unchanged, the cached context should be preserved and only the newly added tokens should be processed.

Thanks in advance!
Joshua Krimmer

Originally created by @JKGeovision on GitHub (Aug 26, 2026). Original GitHub issue: https://github.com/open-webui/computer/issues/238 Hello, I noticed that every follow-up request in an ongoing chat appears to invalidate the complete context cache, forcing llama.cpp to reprocess the entire prompt from scratch even though the chat history is not modified. Steps to reproduce: 1. Start a chat session in Open Computer connected to a llama.cpp backend (in my case: model `Qwen3.8 27B (xhigh)` proxied on port 45723). 2. Send a message and wait for the response to complete (the full context is loaded and cached). 3. Send a follow-up message without modifying the chat history. 4. Check the llama.cpp logs for the second request. Instead of reusing the cached KV context and only processing the new tokens, the logs show the full prompt being reprocessed from the start on the follow-up request (the prompt is roughly 100k+ tokens based on the progress fractions, so the re-prefill penalty is substantial): ``` 218.52.490.688 I srv proxy_reques: proxying request to model Qwen3.8 27B (xhigh) on port 45723 [45723] 202.49.229.491 I slot get_availabl: id 1 | task -1 | selected slot by LRU, t_last = 13320538860 [45723] 202.49.232.130 I slot launch_slot_: id 1 | task 129052 | processing task, is_child = 0 [45723] 202.54.370.411 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 4096, progress = 0.04, t = 4.92 s / 832.76 tokens per second [45723] 202.56.400.910 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 5590, progress = 0.05, t = 6.95 s / 804.42 tokens per second [45723] 202.59.256.537 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 7638, progress = 0.07, t = 9.80 s / 779.01 tokens per second [45723] 203.02.166.965 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 9686, progress = 0.09, t = 12.72 s / 761.77 tokens per second [45723] 203.05.183.430 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 11734, progress = 0.10, t = 15.73 s / 745.89 tokens per second [45723] 203.08.304.425 I slot print_timing: id 1 | task 129052 | prompt processing, n_tokens = 13782, progress = 0.12, t = 18.85 s / 731.04 tokens per second ``` Expected behavior: For consecutive requests within the same chat where the history is unchanged, the cached context should be preserved and only the newly added tokens should be processed. Thanks in advance! Joshua Krimmer

bsakaguchi commented

2026-10-18 14:03:25 +00:00

Author

Owner

@owui-terminator[bot] commented on GitHub (Aug 26, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

🟢 #164 bug: problematic cache reuse
This issue reports that cache reuse is ineffective during multi-turn conversations and that the backend reprocesses most or all of the prompt at each turn, which matches the new report of full context cache invalidation on every follow-up request.
by cclecle
🟢 #142 bug: cptr stops incrementing context usage while llama.cpp reports differently
Although it focuses on a mismatch in reported context usage, it is about llama.cpp context growth during workloads and may point to related cache/context accounting behavior in the same integration path.
by JKGeovision

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

@owui-terminator[bot] commented on GitHub (Aug 26, 2026):  🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟢 [#164](https://github.com/open-webui/computer/issues/164) **bug: problematic cache reuse** *This issue reports that cache reuse is ineffective during multi-turn conversations and that the backend reprocesses most or all of the prompt at each turn, which matches the new report of full context cache invalidation on every follow-up request.* *by cclecle* 2. 🟢 [#142](https://github.com/open-webui/computer/issues/142) **bug: cptr stops incrementing context usage while llama.cpp reports differently** *Although it focuses on a mismatch in reported context usage, it is about llama.cpp context growth during workloads and may point to related cache/context accounting behavior in the same integration path.* *by JKGeovision* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.

bsakaguchi commented

2026-10-18 14:03:25 +00:00

Author

Owner

@JKGeovision commented on GitHub (Aug 27, 2026):

Problem

load_system_prompt() (cptr/utils/prompt_templates.py) re-renders the system prompt on every chat request, including a fresh two-level filesystem walk for {{FILE_TREE}} (_get_file_tree(), line ~46, used at line ~246). The rendered prompt is prepended as the first system message on every API call (cptr/utils/ai.py:568–579).

Any workspace change between turns therefore changes the very first tokens of the request. Providers with prefix/KV caching (llama.cpp server slot reuse, vLLM --enable-prefix-caching, cache_prompt) key their cache on the token prefix, so a single changed line in the file tree invalidates all cached KV behind it — i.e. the entire conversation history is re-parsed every turn the agent touches a file.

Measured (llama.cpp server, 27B model, ~11.5k token prompt):

Scenario	`f_sim_best`	`f_keep`	Prompt processing
file created in workspace between turns	0.431	0.432	~6,500 tokens re-parsed in ~15 s
no change	1.000	1.000	4–24 tokens, < 0.6 s

The LCP break point is exactly at the file-tree section. The conversation history itself is already cache-stable (append-only); only the system prompt varies.

Other per-turn dynamic variables in the system prompt (secondary cache-breakers):

{{MEMORY}}: query-dependent recall (build_memory_prompt(), cptr/utils/memory.py:1298, fed by current_message/recent_messages) → changes with every user message when memory is enabled.
Attached skill blocks: $skill mentions of the last user message are appended to the system prompt (cptr/utils/chat_task.py:1900–1926) → varies per turn.
{{INSTRUCTIONS}}: AGENTS.md/MEMORY.md/… re-read per turn (changes when the agent edits them).
{{DATE}}: available to custom templates; a template using it breaks the cache daily.

Note: plan mode already documents this concern explicitly ("inject prompt as user message (not system, to preserve cache)", chat_task.py:1930).

Proposed fix

Per-chat snapshot (recommended): make the rendered system prompt byte-stable per chat. Add a prompt_key (chat id) to load_system_prompt() and cache the dynamic blocks (FILE_TREE, INSTRUCTIONS, SKILLS) per (chat, workspace). Invalidate on new chat, compaction, or explicit refresh. The agent already has list_directory/search_files and sees file changes in tool results, so a slightly stale tree is acceptable — this matches how other agent harnesses handle it.
Config knob for compatibility, e.g. chat.system_prompt.file_tree: session | request | none (default session); custom templates keep {{FILE_TREE}}.
Move remaining dynamic content out of the system prompt: inject attached-skill blocks and (optionally) the file tree / memory recall as a trailing user message instead of appending to system, following the existing plan-mode pattern.
Minimal alternative: remove {{FILE_TREE}} from DEFAULT_SYSTEM_PROMPT (keep the variable for custom templates).

Verification

With llama.cpp server logging, a workspace change between turns should no longer drop f_sim_best below ~1.0 (expected: ~1.0 with < 50 freshly evaluated prompt tokens per turn); currently it drops to ~0.43, re-parsing the whole history (~15 s on a 27B model).

@JKGeovision commented on GitHub (Aug 27, 2026): **Problem** `load_system_prompt()` (`cptr/utils/prompt_templates.py`) re-renders the system prompt on **every chat request**, including a fresh two-level filesystem walk for `{{FILE_TREE}}` (`_get_file_tree()`, line ~46, used at line ~246). The rendered prompt is prepended as the first `system` message on every API call (`cptr/utils/ai.py:568–579`). Any workspace change between turns therefore changes the very first tokens of the request. Providers with prefix/KV caching (llama.cpp server slot reuse, vLLM `--enable-prefix-caching`, `cache_prompt`) key their cache on the token prefix, so a single changed line in the file tree invalidates **all** cached KV behind it — i.e. the entire conversation history is re-parsed every turn the agent touches a file. **Measured** (llama.cpp server, 27B model, ~11.5k token prompt): | Scenario | `f_sim_best` | `f_keep` | Prompt processing | |---|---|---|---| | file created in workspace between turns | 0.431 | 0.432 | ~6,500 tokens re-parsed in ~15 s | | no change | 1.000 | 1.000 | 4–24 tokens, < 0.6 s | The LCP break point is exactly at the file-tree section. The conversation history itself is already cache-stable (append-only); only the system prompt varies. **Other per-turn dynamic variables in the system prompt** (secondary cache-breakers): - `{{MEMORY}}`: query-dependent recall (`build_memory_prompt()`, `cptr/utils/memory.py:1298`, fed by `current_message`/`recent_messages`) → changes with every user message when memory is enabled. - Attached skill blocks: `$skill` mentions of the *last* user message are appended to the system prompt (`cptr/utils/chat_task.py:1900–1926`) → varies per turn. - `{{INSTRUCTIONS}}`: `AGENTS.md`/`MEMORY.md`/… re-read per turn (changes when the agent edits them). - `{{DATE}}`: available to custom templates; a template using it breaks the cache daily. Note: plan mode already documents this concern explicitly ("inject prompt as user message (not system, to preserve cache)", `chat_task.py:1930`). **Proposed fix** 1. **Per-chat snapshot (recommended):** make the rendered system prompt byte-stable per chat. Add a `prompt_key` (chat id) to `load_system_prompt()` and cache the dynamic blocks (`FILE_TREE`, `INSTRUCTIONS`, `SKILLS`) per `(chat, workspace)`. Invalidate on new chat, compaction, or explicit refresh. The agent already has `list_directory`/`search_files` and sees file changes in tool results, so a slightly stale tree is acceptable — this matches how other agent harnesses handle it. 2. **Config knob** for compatibility, e.g. `chat.system_prompt.file_tree: session | request | none` (default `session`); custom templates keep `{{FILE_TREE}}`. 3. **Move remaining dynamic content out of the system prompt:** inject attached-skill blocks and (optionally) the file tree / memory recall as a trailing user message instead of appending to `system`, following the existing plan-mode pattern. 4. Minimal alternative: remove `{{FILE_TREE}}` from `DEFAULT_SYSTEM_PROMPT` (keep the variable for custom templates). **Verification** With llama.cpp server logging, a workspace change between turns should no longer drop `f_sim_best` below ~1.0 (expected: ~1.0 with < 50 freshly evaluated prompt tokens per turn); currently it drops to ~0.43, re-parsing the whole history (~15 s on a 27B model).

bsakaguchi commented

2026-10-18 14:03:25 +00:00

Author

Owner

@Classic298 commented on GitHub (Aug 27, 2026):

Steps to reproduce:

Start a chat session in Open Computer connected to a llama.cpp backend (in my case: model Qwen3.8 27B (xhigh) proxied on port 45723).
Send a message and wait for the response to complete (the full context is loaded and cached).
Send a follow-up message without modifying the chat history.
Check the llama.cpp logs for the second request.

Cannot reproduce

@Classic298 commented on GitHub (Aug 27, 2026): > Steps to reproduce: > > Start a chat session in Open Computer connected to a llama.cpp backend (in my case: model Qwen3.8 27B (xhigh) proxied on port 45723). > Send a message and wait for the response to complete (the full context is loaded and cached). > Send a follow-up message without modifying the chat history. > Check the llama.cpp logs for the second request. Cannot reproduce

No labels

enhancement

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/computer#151

No description provided.

Rows
Columns