mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1122] How to use long context? #781
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#781
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @createthis on GitHub (Apr 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1122
I started my ktransformers like this:
Then I used 23542 context and got this error:
So I restarted
ktransformerslike this:Where
ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yamlwas modified according to this guide https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel like this:And I got this error:
What am I doing wrong? I'd like to use more than 20k context.
System specs:
768G ram
dual EPYC 9355
24gb 3090
I'm running the
mainbranch at commitf4ae7c85edd66d6acf3ef253eeaf0143eb3358ab.@magikRUKKOLA commented on GitHub (Apr 12, 2025):
Hold on. But what was the point? You got 3090 TI, right? So it doesn't support FP8 natively.
Also, consider creating systemd service to get the logs afterwards.
run-ktransformers.sh example:
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
As related to the systemd setup above -- this would allow you to use journalctl -u ktransformers-deepseek-v3.service --since etc. to get the logs.
Interesting. Have you tried to set PYTORCH_CUDA_ALLOC_CONF=expandable_segments ?
What KV-cache are you using? Its 8 bit or 4 bit?
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
Hm. This one is unclear. You're doing the agentic long-context session, right? If that's the case why to limit the length of the response? Did it actually made all the responses within the 1024 tokens limit?
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
is there anything else is occupying the gpu memory besides ktransformers? like Xorg, or xfwm4 etc.?
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
I seems like I have no problem going through 26k context window on my RTX 3090 24GB.
Its like:
Not sure how the memory management is made. It seem to grow larger with larger context.
It might be time to add the second GPU.
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
Yeah! Got it. Just sent about 32k token on the input and got OOM.
Let's see what can be done.
@magikRUKKOLA commented on GitHub (Apr 12, 2025):
Ha! I wonder WTF is that
@magikRUKKOLA commented on GitHub (Apr 13, 2025):
I have changed this parameter from 16384 to 131072 to make it accept the longer context for a prefill phase:
let's see if this will solve the issue.
For now at least that warning disappeared.
@magikRUKKOLA commented on GitHub (Apr 13, 2025):
okay let me connect the second GPU and use the DeepSeek-V3-Chat-multi-gpu.yaml. lets check if that would solve the issue.
@magikRUKKOLA commented on GitHub (Apr 13, 2025):
Hell yeah, it solved the issue.
Here is the 30k+ tokens i sent to the input:
config that had been used: DeepSeek-V3-Chat-multi-gpu.yaml
the image below shows where it switched from prefill phase to the decode phase. no performance drop in terms of decode speed had been noticed (still getting about 8.5 tps as before).
@magikRUKKOLA commented on GitHub (Apr 13, 2025):
Still not sure what's up with absorb_for_prefill. Have to investigate.
@magikRUKKOLA commented on GitHub (Apr 13, 2025):
Another important note regarding the usage of two GPUs. So I ran the profile (nsys profile --trace=cuda ...) and found out that its using p2p transfers which I enabled via the custom driver. Ha!
So the unlock of the p2p capabilities apparently is the must too.
Another cool note regarding this. Its required to unplug the HDMI cable if its connected to one of the GPUs because otherwise the buffer-alignment error would occur.
DeepSeek analysis:
@createthis commented on GitHub (Apr 14, 2025):
I just haven't gotten around to changing it. I've only run into the limit once so it wasn't an immediate priority. It was cargo culted from these two locations:
@createthis commented on GitHub (Apr 14, 2025):
I'm working through this list to reply. Sorry. Busy weekend.
@createthis commented on GitHub (Apr 14, 2025):
No. The machine is a dedicated ktransformers host.
@createthis commented on GitHub (Apr 14, 2025):
Weird. Mine is already set to that:
@createthis commented on GitHub (Apr 14, 2025):
You're saying I can only use the 128k longer cache with an FP8 capable GPU?
@magikRUKKOLA commented on GitHub (Apr 14, 2025):
Well, no. Because I am using 8 bit cache already in case with 3090 above. I suppose that if you would have FP8 support, the KV-cache calculations will be faster. So even if you will have 4090+ with the same amount of VRAM (24GB) it will not solve the issue. (AFAIK)
note. The 4090 with 48GB is available from China BTW. PALADN is making them.
@createthis commented on GitHub (Apr 14, 2025):
I did some testing today based on this conversation. Results:
absorb_for_prefillset toTrueinktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yamldefinitely causes:Setting
absorb_for_prefillback toFalsefixes the error. Maybe this is because my 3090 GPU is not FP8 capable?Adding:
... doesn't seem to do much. My context still runs out at about
seq_length: 31567, resulting in this warning:At this point, DeepSeek continues processing requests, but it forgets its agentic instructions, so the session ends in Open Hands AI because it forgets what it's supposed to be doing to keep the session going.
I'm guessing at this point the answer is: "You don't have an FP8 GPU - that's why long context isn't working."
@magikRUKKOLA commented on GitHub (Apr 14, 2025):
Well, you could try to do cache_q4 True (EDIT: and cache_8bit False). Not sure if that would solve the issue. At least in my case that didn't solve it.
But where exactly you got the idea that using the GPU with FP8 native support would allow you to run full context?
In any case, you know how to solve the issue -- just by plugging in the second GPU. )
@createthis commented on GitHub (Apr 14, 2025):
I don't know if FP8 support is the problem, but this line is why I'm trying to get longer than 30k context:
github.com/kvcache-ai/ktransformers@8dc1ab9e04/README.md (L32C115-L32C255)It sounds like I need
absorb_for_prefillset toTrueto make that happen.@createthis commented on GitHub (Apr 14, 2025):
Huh. This commit:
github.com/kvcache-ai/ktransformers@3b9e16cec7... solves the error:
I get a different error after that, but this makes me think I should pull from
main. Maybe this is resolved upstream.@magikRUKKOLA commented on GitHub (Apr 14, 2025):
Ha. Indeed. They wrote that 139k context is supported but actually in the docs they wrote only about 20k+ context:
I believe I tried that when I had only one GPU installed and failed to do 30k+ context. I didn't merge the weights for that though.
Then I decided to plug in the second GPU which solved the issue.
Well, if you manage to get it working just with one GPU please let me know. )
@createthis commented on GitHub (Apr 14, 2025):
git pull origin mainfixed it. I'm out past 50k context on a single 24gb GPU withabsorb_for_prefillset toTrue. Sorry for the noise!@createthis commented on GitHub (Apr 14, 2025):
PS. I got up to 80k context before it slowed down enough that I wanted to restart my agentic task. Started out at 100 tok/s prefill and 17 tok/s decode and got down to 12 tok/s prefill and 10 tok/s decode at 80k context. Very cool! I love having the option to run really long contexts at the expense of some speed. Awesome feature!
@magikRUKKOLA commented on GitHub (Apr 14, 2025):
@createthis
Yeah, bro, you're absolutely right. I don't know how I missed this but I did. Today I pulled the most recent master and then just did as you specified regrading the absorb_for_prefill. Not sure what it was but I can confirm that my unsloth DeepSeekV3-UDQ4XXL is running with more than 50k context with just single GPU.
There might had been issue when I incorrecly configured my nginx reverse proxy by just putting the limit:
but in this case I had to look into the journalctl to check the the EOL error I had from mods was related to the proxy error or the actual ktransformers error. I might had ignored that part. Really not sure about that. It doesn't really matter now.
I feel kinda dumb about that. Sorry for the noise too.
I can agree, that's pretty cool to run state-of-the-art llm locally like that.