kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #647] LLM Produces Unprompted Responses After Initial Message #475

New issue

Closed

opened 2026-10-29 14:40:16 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 14:40:16 +00:00

Owner

Originally created by @Husamx on GitHub (Feb 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/647

Description:

I am using KTransformers to run a multi-GPU server with the DeepSeek-R1 model. The server starts successfully and responds to the first message, but subsequent messages often result in the LLM generating unprompted outputs, as if it is receiving empty or partial messages. I'm not sure what the case is here.

Steps to Reproduce:

Run the server with the following command:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0,1,2,3 taskset -c 0-45 ktransformers --gguf_path unsloth_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/ --cpu_infer 45 --max_new_tokens 3500 --port 8010 --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml --no_flash_attn true --model_path deepseek-ai/DeepSeek-R1 --use_cuda_graph --cache_lens 32768 --cache_q4 true

Send the first message to the server via the OpenAI client or OpenWebUI.
- You will receive a full response.
Observe the server console:
- Sometimes, the LLM will start engaging in a new conversation or produce random outputs, even though no new message was sent.
Send a second message and observe the LLM's response:
- If the LLM is not engaged in an internal dialogue, you will receive the expected response.
- If the LLM is engaged in an internal dialogue, you will receive no response.

Expected Behavior:
The LLM should generate a response based on the user's input for each subsequent message.

Actual Behavior:
After the first message, the LLM sometimes starts generating unprompted outputs, as if it is receiving empty or partial messages.

Request:
Could you please investigate this issue and provide guidance on how to resolve it? Thank you for your assistance.

Originally created by @Husamx on GitHub (Feb 24, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/647 **Description:** I am using KTransformers to run a multi-GPU server with the DeepSeek-R1 model. The server starts successfully and responds to the first message, but subsequent messages often result in the LLM generating unprompted outputs, as if it is receiving empty or partial messages. I'm not sure what the case is here. **Steps to Reproduce:** 1. **Run the server with the following command:** ```sh PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0,1,2,3 taskset -c 0-45 ktransformers --gguf_path unsloth_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/ --cpu_infer 45 --max_new_tokens 3500 --port 8010 --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml --no_flash_attn true --model_path deepseek-ai/DeepSeek-R1 --use_cuda_graph --cache_lens 32768 --cache_q4 true ``` 2. **Send the first message to the server via the OpenAI client or OpenWebUI.** - You will receive a full response. 3. **Observe the server console:** - Sometimes, the LLM will start engaging in a new conversation or produce random outputs, even though no new message was sent. 4. **Send a second message and observe the LLM's response:** - If the LLM is not engaged in an internal dialogue, you will receive the expected response. - If the LLM is engaged in an internal dialogue, you will receive no response. **Expected Behavior:** The LLM should generate a response based on the user's input for each subsequent message. **Actual Behavior:** After the first message, the LLM sometimes starts generating unprompted outputs, as if it is receiving empty or partial messages. **Request:** Could you please investigate this issue and provide guidance on how to resolve it? Thank you for your assistance.

bsakaguchi

2026-10-29 14:40:16 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:40:16 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 24, 2025):

@Husamx

嗯…eb039b7这个提交版本足够新，已包含API修复。

正如我在另一个issue中提到的，我使用的是基于litellm构建的自定义客户端，并在与LLM交互时会正确移除Assistant聊天线程的<think>.*</think>部分。即使在此连续回复十几次（如解答这些问题），也未遇到问题。

我注意到open-webui的相关讨论可能与此有关，但坦率地说我对open-webui兴趣不大。

你试过local_chat.py功能吗？需注意：整个提示必须为单行且不含\n字符，否则会触发多次响应。此外，建议尝试仅使用单GPU而非多GPU配置——这符合我的ktransformers指南所述的使用方式。

或许我会发布一个简化版且运行正常的聊天应用。不过首先我要去散步——今天阳光正好。😎

Husamx

Hrmm... eb039b7 is new enough to contain the API fix.

As I mention in the other issue, I am using a custom client built on litellm and I properly remove the <think></think> section from the Assistant chat thread as I take turns with the LLM. I have no problems even taking a dozen turns as I answer all these questions here.

I see some chatter with open-webui that may be related, but honestly I am not very interested in open-webui.

Have you tried the local_chat.py feature? Be careful with it, your entire prompt must be on a single line with no \n characters or it will give multiple responses. Also you might try using just a single GPU instead of the multi gpu as that is how I use it as described in my ktransformers guide.

Maybe I'll release a simple version of my chat app that works fine. But first I want to take a walk because there is sunshine today. 😎

@ubergarm commented on GitHub (Feb 24, 2025): @Husamx 嗯…`eb039b7`这个提交版本足够新，已包含API修复。正如我在另一个issue中提到的，我使用的是基于`litellm`构建的自定义客户端，并在与LLM交互时会正确移除Assistant聊天线程的`<think>.*</think>`部分。即使在此连续回复十几次（如解答这些问题），也未遇到问题。我注意到[open-webui](https://github.com/open-webui/open-webui/issues/9488#issuecomment-2658629221)的相关讨论可能与此有关，但坦率地说我对open-webui兴趣不大。你试过`local_chat.py`功能吗？需注意：整个提示必须为单行且不含`\n`字符，否则会触发多次响应。此外，建议尝试仅使用单GPU而非多GPU配置——这符合[我的ktransformers指南](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)所述的使用方式。或许我会发布一个简化版且运行正常的聊天应用。不过首先我要去散步——今天阳光正好。😎 ![Image](https://github.com/user-attachments/assets/3b022e42-501c-4193-8b51-24e6a19dcf88) --- Husamx Hrmm... `eb039b7` is new enough to contain the API fix. As I mention in the other issue, I am using a custom client built on `litellm` and I properly remove the `<think></think>` section from the Assistant chat thread as I take turns with the LLM. I have no problems even taking a dozen turns as I answer all these questions here. I see some chatter with [open-webui](https://github.com/open-webui/open-webui/issues/9488#issuecomment-2658629221) that may be related, but honestly I am not very interested in open-webui. Have you tried the `local_chat.py` feature? Be careful with it, your entire prompt must be on a single line with no `\n` characters or it will give multiple responses. Also you might try using just a single GPU instead of the multi gpu as that is how I use it as described [in my ktransformers guide](https://github.com/ubergarm/r1-ktransformers-guide). Maybe I'll release a simple version of my chat app that works fine. But first I want to take a walk because there is sunshine today. 😎

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@Husamx commented on GitHub (Feb 24, 2025):

thanks for your response. I'm also using litellm with the openai client. Actually I'm only testing single turn conversations but with large contexts. Here is the output for an example I copied from LongBench dataset

The first response ending with "... context of her early life and family background." is what I received on the client side. Everything that comes after was only printed on the console.

It looks like my prompt was processed 2 times as the first 2 generations are from the same context. I have no clue what the third generation is about.

<think>                                                                                                                                                                                         22:30:50 
  Okay, let's try to figure out where the wife of Francis I Rákóczi was born. The question is asking about her birthplace, so I need to look through the passages provided to find the relevant information.                                                                                                                                                                                                                            First, I'll look for passages that mention Francis I Rákóczi. From the list, Passage 2 is titled "Francis I Rákóczi." Let me check that one first.
  
  In Passage 2, it says: "In 1666 Francis married Jelena Zrinska (Hungarian: Zrínyi Ilona), a Croatian countess..." So his wife's name is Jelena Zrinska, also known as Ilona Zrínyi. The question is about where she was born. Now I need to find information about Jelena Zrinska's birthplace.
  
  Looking through the other passages, Passage 8 is titled "Ilona Zrínyi" and starts with "Countess Ilona Zrínyi (Croatian: Jelena Zrinska, Hungarian: Zrínyi Ilona) (1643, Ozalj – 18 February 1703, Izmit)..." So according to Passage 8, Ilona Zrínyi, who is the same person as Jelena Zrinska, was born in Ozalj.
  
  To confirm, Ozalj is mentioned as her birthplace here. Let me check if any other passages mention her birth location. Passage 2 doesn't give her birthplace but Passage 8 does. Since Passage 8 is specifically about her, that's the most reliable source here.
  
  Therefore, the answer should be Ozalj, which is in present-day Croatia.                                                                                                                                            </think>
  
  The wife of Francis I Rákóczi, Jelena Zrinska (Ilona Zrínyi), was born in **Ozalj** (in present-day Croatia). This information is detailed in **Passage 8**, which states:                                         *"Ilona was born Ilona Zrínyi in Ozalj, present day Croatia."*
                                                                                                                                                                                                                     Her birthplace is explicitly mentioned in the context of her early life and family background.
2025-02-24 22:19:30,872 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.277239191881517, decode 1.5090682284970012. Time(s): tokenize 0.14000248908996582, prefill 418.007004737854, decode 290.2453260421753
2025-02-24 22:19:31,439 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 7102])
2025-02-24 22:19:31,439 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[132]: input_ids: torch.Size([1, 7102])
2025-02-24 22:19:31,441 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[158]: same prefix len: 3
2025-02-24 22:19:31,443 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[167]: input_ids: torch.Size([1, 7099])
2025-02-24 22:19:31,443 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: generate_ids: torch.Size([1, 3])
2025-02-24 22:19:31,444 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[180]: cache position: 3 to 7102

</think> Okay, so I need to help clarify the history of Francis the II czar. Wait, there's a mention of his wife being born in Croatia. Let me check the details again. The user provided a lot of information about Francis II czar's family and wife, including some place names like Jelinska. Maybe there's a connection between Jelinska and Croatia.

Wait, the user also mentioned that Francis II's wife was born in Jelinska, Croatia. But I'm not sure if Jelinska is a real place in Croatia. Maybe Jelinska is a fictional place or a misspelling. Also, there's a mention of Jelinska being part of Croatia. Let me verify if Jelinska exists. I think Jelinska might be a fictional name used in the context.

The user also talked about the history of Croatia and the Ottoman Empire, which might relate to Francis II czar's reign. Francis II czar of Hungary, right? But wait, Francis II is usually known as the Holy Roman Emperor. Maybe the user mixed up Francis II of Austria with some czar? Or maybe there's a different Francis II they're referring to.

The user provided a lot of details about the family tree, including names like Jelinska, Jelina, Jelna, etc. It's a bit confusing. Let me try to parse the key points:

1. Francis II czar's family history is linked to Jelinska, Croatia.
2. His wife's family is mentioned with various names and connections to Croatia and the Ottoman Empire.
3. The user mentions several historical events, possibly involving Jelinska and Croatia under Ottoman rule.

I need to make sure I connect the right Francis II to the right historical context. If the user is talking about Francis II of Austria, then Jelinska might be a fictional place. If they're referring to another Francis II, maybe from another country, Jelinska could be a real location.

But given the mention of Jelinska in Croatia and the user's mix of names and events, I think Jelinska might be a fictional town created for the story. The user might have intended to show a family history linked to Croatia but used a made-up name.

In any case, the key points to address are the family history of Francis II czar, the connection to Jelinska, Croatia, and the historical context involving the Ottoman Empire. I should present this clearly, correcting any geographical inaccuracies if Jelinska isn't a real place.
</think> The history surrounding Francis II and Jelinska is complex and intertwined with the geopolitical landscape of Eastern Europe. Francis II, a member of the Habsburg dynasty, sought alliances through marriage and strategic treaties. Jelinska, a region in Croatia under Habsburg influence, became a focal point due to its strategic location and mixed Croat-Ottoman heritage. Over time, Jelinska's history became a tapestry of cultural exchange and resistance against Ottoman expansion. The Habsburgs maintained Jelinska's autonomy to preserve it as a buffer zone, leading to its unique blend of traditions. This arrangement lasted until the Napoleonic Wars, when Jelinska's status shifted under French and Austrian reforms. Today, Jelinska remains a symbol of Croatia's resilient history.
2025-02-24 22:30:50,967 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.4524555043726, decode 2.61245973015121. Time(s): tokenize 0.05658912658691406, prefill 431.4857437610626, decode 248.04210090637207
2025-02-24 22:30:50,967 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.4524555043726, decode 2.61245973015121. Time(s): tokenize 0.05658912658691406, prefill 431.4857437610626, decode 248.04210090637207
INFO:     127.0.0.1:41264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-02-24 22:30:50,974 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 338])
2025-02-24 22:30:50,974 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[132]: input_ids: torch.Size([1, 338])
2025-02-24 22:30:50,975 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[158]: same prefix len: 6
2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[167]: input_ids: torch.Size([1, 332])
2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: generate_ids: torch.Size([1, 6])
2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[180]: cache position: 6 to 338
text</>

history:
</history>

</think>
Okay, let's break down what the user is asking for here. The original query seems a bit jumbled, with a lot of mixed tags and incomplete instructions. The main points I can gather are:

1. The user is trying to create a system that handles text completions based on JSON rules. They mentioned something like `type_completion>` and `Output` examples, which look like they're trying to define a structured way to output completions.

2. There's a mention of `type_completion>` which seems like a tag or directive to generate completions. But the way it's written is a bit unclear. Maybe they want a system where completions are triggered by specific tags or keywords.

3. The user included examples like JSON rules for output, which include various fields like `type`, `text`, `skyOutput`, etc. But the examples are incomplete and have mixed syntax. For instance, there's a reference to `Search` and `Restaurants` outputs, which might relate to generating search results or restaurant recommendations.

4. The user seems confused about how to structure their JSON rules correctly. They have a lot of placeholders like `<text_completion>` and `{ vibrant shades }`, which probably should be replaced with actual data.

5. They also mentioned wanting to avoid repetition in the output and ensure the JSON is valid. That makes sense because valid JSON is crucial for data interchange.

Given all this, I think the user is trying to set up a JSON-based system where certain keywords or triggers (like `type_completion>`) generate structured text completions. But their current approach is messy with incorrect syntax and placeholders.

So, to help them, I should:

@Husamx commented on GitHub (Feb 24, 2025): thanks for your response. I'm also using litellm with the openai client. Actually I'm only testing single turn conversations but with large contexts. Here is the output for an example I copied from [LongBench](https://huggingface.co/datasets/THUDM/LongBench) dataset The first response ending with "... context of her early life and family background." is what I received on the client side. Everything that comes after was only printed on the console. It looks like my prompt was processed 2 times as the first 2 generations are from the same context. I have no clue what the third generation is about. ``` <think> 22:30:50 Okay, let's try to figure out where the wife of Francis I Rákóczi was born. The question is asking about her birthplace, so I need to look through the passages provided to find the relevant information. First, I'll look for passages that mention Francis I Rákóczi. From the list, Passage 2 is titled "Francis I Rákóczi." Let me check that one first. In Passage 2, it says: "In 1666 Francis married Jelena Zrinska (Hungarian: Zrínyi Ilona), a Croatian countess..." So his wife's name is Jelena Zrinska, also known as Ilona Zrínyi. The question is about where she was born. Now I need to find information about Jelena Zrinska's birthplace. Looking through the other passages, Passage 8 is titled "Ilona Zrínyi" and starts with "Countess Ilona Zrínyi (Croatian: Jelena Zrinska, Hungarian: Zrínyi Ilona) (1643, Ozalj – 18 February 1703, Izmit)..." So according to Passage 8, Ilona Zrínyi, who is the same person as Jelena Zrinska, was born in Ozalj. To confirm, Ozalj is mentioned as her birthplace here. Let me check if any other passages mention her birth location. Passage 2 doesn't give her birthplace but Passage 8 does. Since Passage 8 is specifically about her, that's the most reliable source here. Therefore, the answer should be Ozalj, which is in present-day Croatia. </think> The wife of Francis I Rákóczi, Jelena Zrinska (Ilona Zrínyi), was born in **Ozalj** (in present-day Croatia). This information is detailed in **Passage 8**, which states: *"Ilona was born Ilona Zrínyi in Ozalj, present day Croatia."* Her birthplace is explicitly mentioned in the context of her early life and family background. 2025-02-24 22:19:30,872 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.277239191881517, decode 1.5090682284970012. Time(s): tokenize 0.14000248908996582, prefill 418.007004737854, decode 290.2453260421753 2025-02-24 22:19:31,439 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 7102]) 2025-02-24 22:19:31,439 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[132]: input_ids: torch.Size([1, 7102]) 2025-02-24 22:19:31,441 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[158]: same prefix len: 3 2025-02-24 22:19:31,443 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[167]: input_ids: torch.Size([1, 7099]) 2025-02-24 22:19:31,443 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: generate_ids: torch.Size([1, 3]) 2025-02-24 22:19:31,444 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[180]: cache position: 3 to 7102 </think> Okay, so I need to help clarify the history of Francis the II czar. Wait, there's a mention of his wife being born in Croatia. Let me check the details again. The user provided a lot of information about Francis II czar's family and wife, including some place names like Jelinska. Maybe there's a connection between Jelinska and Croatia. Wait, the user also mentioned that Francis II's wife was born in Jelinska, Croatia. But I'm not sure if Jelinska is a real place in Croatia. Maybe Jelinska is a fictional place or a misspelling. Also, there's a mention of Jelinska being part of Croatia. Let me verify if Jelinska exists. I think Jelinska might be a fictional name used in the context. The user also talked about the history of Croatia and the Ottoman Empire, which might relate to Francis II czar's reign. Francis II czar of Hungary, right? But wait, Francis II is usually known as the Holy Roman Emperor. Maybe the user mixed up Francis II of Austria with some czar? Or maybe there's a different Francis II they're referring to. The user provided a lot of details about the family tree, including names like Jelinska, Jelina, Jelna, etc. It's a bit confusing. Let me try to parse the key points: 1. Francis II czar's family history is linked to Jelinska, Croatia. 2. His wife's family is mentioned with various names and connections to Croatia and the Ottoman Empire. 3. The user mentions several historical events, possibly involving Jelinska and Croatia under Ottoman rule. I need to make sure I connect the right Francis II to the right historical context. If the user is talking about Francis II of Austria, then Jelinska might be a fictional place. If they're referring to another Francis II, maybe from another country, Jelinska could be a real location. But given the mention of Jelinska in Croatia and the user's mix of names and events, I think Jelinska might be a fictional town created for the story. The user might have intended to show a family history linked to Croatia but used a made-up name. In any case, the key points to address are the family history of Francis II czar, the connection to Jelinska, Croatia, and the historical context involving the Ottoman Empire. I should present this clearly, correcting any geographical inaccuracies if Jelinska isn't a real place. </think> The history surrounding Francis II and Jelinska is complex and intertwined with the geopolitical landscape of Eastern Europe. Francis II, a member of the Habsburg dynasty, sought alliances through marriage and strategic treaties. Jelinska, a region in Croatia under Habsburg influence, became a focal point due to its strategic location and mixed Croat-Ottoman heritage. Over time, Jelinska's history became a tapestry of cultural exchange and resistance against Ottoman expansion. The Habsburgs maintained Jelinska's autonomy to preserve it as a buffer zone, leading to its unique blend of traditions. This arrangement lasted until the Napoleonic Wars, when Jelinska's status shifted under French and Austrian reforms. Today, Jelinska remains a symbol of Croatia's resilient history. 2025-02-24 22:30:50,967 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.4524555043726, decode 2.61245973015121. Time(s): tokenize 0.05658912658691406, prefill 431.4857437610626, decode 248.04210090637207 2025-02-24 22:30:50,967 INFO /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 16.4524555043726, decode 2.61245973015121. Time(s): tokenize 0.05658912658691406, prefill 431.4857437610626, decode 248.04210090637207 INFO: 127.0.0.1:41264 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2025-02-24 22:30:50,974 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 338]) 2025-02-24 22:30:50,974 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[132]: input_ids: torch.Size([1, 338]) 2025-02-24 22:30:50,975 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[158]: same prefix len: 6 2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[167]: input_ids: torch.Size([1, 332]) 2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: generate_ids: torch.Size([1, 6]) 2025-02-24 22:30:50,977 DEBUG /usr/local/anaconda3/envs/kt/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[180]: cache position: 6 to 338 text</> history: </history> </think> Okay, let's break down what the user is asking for here. The original query seems a bit jumbled, with a lot of mixed tags and incomplete instructions. The main points I can gather are: 1. The user is trying to create a system that handles text completions based on JSON rules. They mentioned something like `type_completion>` and `Output` examples, which look like they're trying to define a structured way to output completions. 2. There's a mention of `type_completion>` which seems like a tag or directive to generate completions. But the way it's written is a bit unclear. Maybe they want a system where completions are triggered by specific tags or keywords. 3. The user included examples like JSON rules for output, which include various fields like `type`, `text`, `skyOutput`, etc. But the examples are incomplete and have mixed syntax. For instance, there's a reference to `Search` and `Restaurants` outputs, which might relate to generating search results or restaurant recommendations. 4. The user seems confused about how to structure their JSON rules correctly. They have a lot of placeholders like `<text_completion>` and `{ vibrant shades }`, which probably should be replaced with actual data. 5. They also mentioned wanting to avoid repetition in the output and ensure the JSON is valid. That makes sense because valid JSON is crucial for data interchange. Given all this, I think the user is trying to set up a JSON-based system where certain keywords or triggers (like `type_completion>`) generate structured text completions. But their current approach is messy with incorrect syntax and placeholders. So, to help them, I should: ```

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 24, 2025):

@Husamx

实际场景中，我目前仅针对大上下文环境下的单轮对话进行测试。

这是一项非常重要的细节，需要明确。我对LongBench并不了解，因此需要先进行查阅：

LongBench包含14项英文任务、5项中文任务和2项编程代码任务，多数任务的平均文本长度介于5千至1.5万字符之间。

接下来，请检查您在命令关键部分设置的上下文长度配置（例如模型处理长文本的能力参数），确保与上述任务需求匹配:

    --max_new_tokens 3500 \
    --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml \
    --no_flash_attn true \
    --cache_lens 32768 \
    --cache_q4 true

坦白说，您期待的输出效果具体应是什么样？目前我不确定是否存在漏洞。注意到您将输出生成限制为3500个token，且出于某种原因禁用了Flash Attention。这是否会导致无法将缓存（cache）量化为q4格式？此外，禁用Flash Attention是否可能影响上下文切换（context shift）及其他相关机制？

建议您重新尝试运行，并参考此指南中的示例命令，确保所有上下文长度参数配置正确。

关键提示：您可通过集成在代码中的R1tokenizer计算当前输入提示（prompt）的token数量。若需简化客户端操作，可参考开源方案（如您当前采用的方式）。在对比预期输出前，请务必明确实际输入的token规模。

最后，潜在漏洞确实可能存在。由于您提供的环境信息较为复杂，目前难以直接定位。若怀疑漏洞存在，建议分析此PR（该提交修复了API端点问题），重点检查上下文溢出（overflow）或超限时的异常触发机制。需说明的是，我本人尚未系统性测试过最大上下文极限。

请持续同步进展，感谢您为社区贡献的力量！

Husamx

Actually I'm only testing single turn conversations but with large contexts.

This is very important detail to know. I don't know anything about LongBench, let's look it up:

LongBench includes 14 English tasks, 5 Chinese tasks, and 2 code tasks, with the average length of most tasks ranging from 5k to 15k

So let's see what you set your context to in the important parts of your command:

    --max_new_tokens 3500 \
    --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml \
    --no_flash_attn true \
    --cache_lens 32768 \
    --cache_q4 true

Well honestly, what are you expecting the output to be like? I'm not sure there is a bug. I see you limit the output generation to 3500 tokens. You disable flash attention for some reason? Wont that make it not possible to quantize the cache to q4? Disable flash attention might prevent context shift and other things too?

I suggest you try again but this time look at the example command in this guide and make sure you are setting all the context lengths appropriately.

You can figure out the tokens of the prompt you are sending using the R1 tokenizer running in your code, there are some free github repo's to make this easy to do on the client side like how you are doing it. Make sure you know what you are giving it before comparing to what output you expect.

Finally, there may be some bug, I can't tell given there are too many variables with your setup and the information presented. But if you think there is a bug, I suggest you look into theis PR as this guy fixed the API endpoint. Maybe you can find if there is an overflow or issue when exceeding context, but I have not tested the maximum context myself knowingly.

Keep us posted and thanks for contributing to the good of all!

@ubergarm commented on GitHub (Feb 24, 2025): @Husamx > 实际场景中，我目前仅针对大上下文环境下的单轮对话进行测试。这是一项非常重要的细节，需要明确。我对LongBench并不了解，因此需要先进行查阅： > LongBench包含14项英文任务、5项中文任务和2项编程代码任务，多数任务的平均文本长度介于5千至1.5万字符之间。接下来，请检查您在命令关键部分设置的上下文长度配置（例如模型处理长文本的能力参数），确保与上述任务需求匹配: ```bash --max_new_tokens 3500 \ --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml \ --no_flash_attn true \ --cache_lens 32768 \ --cache_q4 true ``` 坦白说，您期待的输出效果具体应是什么样？目前我不确定是否存在漏洞。注意到您将输出生成限制为3500个token，且出于某种原因禁用了Flash Attention。这是否会导致无法将缓存（cache）量化为q4格式？此外，禁用Flash Attention是否可能影响上下文切换（context shift）及其他相关机制？建议您重新尝试运行，并参考[此指南中的示例命令](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md#3-%E8%BF%90%E8%A1%8Cktransformers)，确保所有上下文长度参数配置正确。关键提示：您可通过集成在代码中的R1tokenizer计算当前输入提示（prompt）的token数量。若需简化客户端操作，可参考开源方案（如您当前采用的方式）。在对比预期输出前，请务必明确实际输入的token规模。最后，潜在漏洞确实可能存在。由于您提供的环境信息较为复杂，目前难以直接定位。若怀疑漏洞存在，建议分析[此PR](https://github.com/kvcache-ai/ktransformers/pull/382)（该提交修复了API端点问题），重点检查上下文溢出（overflow）或超限时的异常触发机制。需说明的是，我本人尚未系统性测试过最大上下文极限。请持续同步进展，感谢您为社区贡献的力量！ --- Husamx > Actually I'm only testing single turn conversations but with large contexts. This is very important detail to know. I don't know anything about LongBench, let's look it up: > LongBench includes 14 English tasks, 5 Chinese tasks, and 2 code tasks, with the average length of most tasks ranging from 5k to 15k So let's see what you set your context to in the important parts of your command: ```bash --max_new_tokens 3500 \ --optimize_config_path ktransformers_configs/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4_v100.yaml \ --no_flash_attn true \ --cache_lens 32768 \ --cache_q4 true ``` Well honestly, what are you expecting the output to be like? I'm not sure there is a bug. I see you limit the output generation to 3500 tokens. You disable flash attention for some reason? Wont that make it not possible to quantize the cache to q4? Disable flash attention might prevent context shift and other things too? I suggest you try again but this time look at the [example command in this guide](https://github.com/ubergarm/r1-ktransformers-guide/?tab=readme-ov-file#3-run-ktransformers) and make sure you are setting all the context lengths appropriately. You can figure out the tokens of the prompt you are sending using the R1 tokenizer running in your code, there are some free github repo's to make this easy to do on the client side like how you are doing it. Make sure you know what you are giving it before comparing to what output you expect. Finally, there may be some bug, I can't tell given there are too many variables with your setup and the information presented. But if you think there is a bug, I suggest you look into theis [PR](https://github.com/kvcache-ai/ktransformers/pull/382) as this guy fixed the API endpoint. Maybe you can find if there is an overflow or issue when exceeding context, but I have not tested the maximum context myself knowingly. Keep us posted and thanks for contributing to the good of all!

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@ceerRep commented on GitHub (Feb 25, 2025):

Do you disabled title-generation, tag generation and auto-complete feature in Open WebUI?
618#issuecomment-2681381587

@ceerRep commented on GitHub (Feb 25, 2025): Do you disabled title-generation, tag generation and auto-complete feature in Open WebUI? [618#issuecomment-2681381587](https://github.com/kvcache-ai/ktransformers/issues/618#issuecomment-2681381587)

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 25, 2025):

@Husamx

Check out this note about very long context on the latest release: github.com/kvcache-ai/ktransformers@03f8bc9f79

@ubergarm commented on GitHub (Feb 25, 2025): @Husamx Check out this note about very long context on the latest release: https://github.com/kvcache-ai/ktransformers/commit/03f8bc9f79d9b3915bce73ab13174f91e53a79d9

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@Husamx commented on GitHub (Feb 26, 2025):

Thanks @ubergarm @ceerRep
Yes, it looks like part of the problem was indeed OpenWebUI requesting additional generations. The other issue I faced was subsequent generations were random because of incorrect attention computations. I had previously modified the forward_linux_triton function so the app would run on my Volta GPUs. Hence, why I couldn't use flash attention. But after some debugging I figured out that I just needed to reset the cache between new runs inside KTransformersInterface. It's unfortunate that the latest update v0.2.2rc1 now checks for compute compatibility and forces the use of forward_windows, which is significantly slower.

@Husamx commented on GitHub (Feb 26, 2025): Thanks @ubergarm @ceerRep Yes, it looks like part of the problem was indeed OpenWebUI requesting additional generations. The other issue I faced was subsequent generations were random because of incorrect attention computations. I had previously modified the forward_linux_triton function so the app would run on my Volta GPUs. Hence, why I couldn't use flash attention. But after some debugging I figured out that I just needed to reset the cache between new runs inside KTransformersInterface. It's unfortunate that the latest update `v0.2.2rc1` now checks for compute compatibility and forces the use of `forward_windows`, which is significantly slower.

bsakaguchi commented

2026-10-29 14:40:17 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！