mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1417] [Bug] deepseek v3/r1 with full context with balance_serve backend #950
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#950
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @magikRUKKOLA on GitHub (Jul 3, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1417
Checklist
Describe the bug
I am not sure how to run the balance_serve backend with a full 128k context. If I will try something like that:
I will have the error regarding the cuda_graphs.
I would not have a similar issue in case with ktransformers backend, but it doesn't support the disk prefix caching. So what to do?
[UPDATE]: the ephemeral vibe-coded solution seems to be currently found. https://github.com/kvcache-ai/ktransformers/issues/1417#issuecomment-3040256985
Reproduction
Environment
@magikRUKKOLA commented on GitHub (Jul 3, 2025):
In case I am using the --no-use_cuda_graph it seems to solve the issue but the inference speed (decode, not prefill) is getting x2 times slower.
So what is going on? Why the one-session ktransformers backend works fine with cuda_graphs but the balance_serve backend doesn't seem to be functional with one session and if I am to try to support [four] sessions with full context on 24 GB VRAM there would be OOM. :)
Namely, if I would try this:
I would get this during the inference:
@magikRUKKOLA commented on GitHub (Jul 3, 2025):
Is there any config for a multi-gpu with balance_serve support?
@magikRUKKOLA commented on GitHub (Jul 3, 2025):
I don't get why the prefix caching with disk support wasn't implemented in ktransformers backend?
The balance_serve backend unfortunately is not usable. It limits the the output of the model up to 32k tokens in order to do what? To potentially support up to 4 simultaneous clients? What for? In case I would want 4 simultaneous sessions I would build 4 separate computers and route the queries myself. I DO NOT need a support of such stuff in the llm inference framework. Please make something to work stable with full 128k context. Otherwise its impossible to run deepseek with full 128k context nowhere -- neither with balance_serve (as it seems mandatory to specify four sessions and limit the max token output to 32k) or its fully working with ktransfomers backend but it doesn't support the prefix disk caching and its seems to getting deprecated. This is clearly ridiculous.
@magikRUKKOLA commented on GitHub (Jul 3, 2025):
Ha. I can't even do 126k prefill with max_batches=4 and max_tokens=132k. Ha. Its doing OOM at about 90k prefill.
Not sure what is going on. Let me kill the xfce4 process that tool 50MB of VRAM and try again. May be there is an issue with gpu_utilization configuration.
[EDIT]. I tried with xfce4 killed so that only ktransformers takes up the VRAM and result is the same.
and its crashing after 98k context prefill. Its steadily consuming about 22.6 GB of VRAM and its just crashing. So not only decode of 128k context is NOT working with balance_serve. But the prefill is failing too. Did anyone actually tested it?
[EDIT2]:
Uh oh! It seems to be working with the following config:
indeed the prefill became 2x slower, but now its not crashing after 98k tokens prefill. Let's see what it will output after the full 126k context... [EDIT3]: oh lol the client I am using (charmbracelet/mods) just crashed, but the ktransformers still doing the prefill. So now I will wait till the end of the processing and send the same query via the mods once again. Actually, to make sure the disk storage prefill works properly, first I will shut down the ktransformers, then set it to load the prefill cache from the disk, and only after that would send the same question. If everything works, it should respond within 20s. Let's see...
[EDIT4]:
omg it went through entire context up until the 132k, stopped, and apparently didnt save the prefill cache.
after I restarted the ktransformers it didnt picked up the 126k previous prefill and started the prefill all over again.
@magikRUKKOLA commented on GitHub (Jul 4, 2025):
[EDIT5]:
the same config, but I didn't restart the ktransformers yet.
@magikRUKKOLA commented on GitHub (Jul 4, 2025):
Wait a second. How the prefix cache is working exactly? Does it make s snapshot right after the prefill phase? Or only after the whole conversation is done. That is, will I be able to send a prefill chunk of text to cache it and then just append the additional questions? Because as of now it doesnt seem to support such stuff -- one can only continue the conversation by adding the questions to the conversation, but not "replacing" them via the prefix cache disk storage utilization.
@magikRUKKOLA commented on GitHub (Jul 4, 2025):
Why there is no API to make a dump (or restore) of the KV-cache on-demand? Right now the cache just grows in size and I am not sure now to work with it. Its basically a binary blob (per layer). Why the separate files weren't used for example (say, per prefill batch)?
@magikRUKKOLA commented on GitHub (Jul 4, 2025):
UPDATE!
As related to the issue regarding the:
the Deepseek V3 0324 provided the following patch to resolve the issue:
./ktransformers/server/balance_serve/inference/sampling/sampler.py
let me check if that would solve the issue [...]
related:
https://github.com/kvcache-ai/ktransformers/issues/1269
https://github.com/kvcache-ai/ktransformers/issues/1324
https://github.com/kvcache-ai/ktransformers/issues/1376
https://github.com/kvcache-ai/ktransformers/issues/1062
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
Nope, still having the same issue.
So the issue appears to be related to ... inability of the sample[r] to pick up the index of the tensor which has the zero temperature?
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
Let's try this one:
[EDIT]:
also as related to division by zero and NaN values (makes sense?):
[EDIT2]:
so the aforementioned safetycheck would prevent the write operation into the batch_next_token_ids array. So .. why it wasn't implemented in the first place?
[EDIT3]:
lol still the same error
so its apparently related to the sync issue between the cpu and gpu huh?
[EDIT4]:
okay lets try this one:
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
Apparently the solution would look like:
./ktransformers/server/balance_serve/inference/sampling/sampler.py
Explanation:
[EDIT]: not sure if torch.cuda.synchronize() is necessary since the .to() suppose to do the sync.
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
lol no errors yet
So what does that all mean? All the time there was a race condition in the balance_serve backend sampler of ktransformers which leads to the invalid memory read? Is it correct?
[UPDATE]: a different error!
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
Another error lol
[UPDATE]:
I will try to vibecode myself out of bugs with DeepSeek V3/R1 UD_Q2/Q4 quants. But lol if the deepseek is right related to the interpretation of the bugs then something is truly wrong with the codebase of the ktransformers.
Its a great framework, dont get me wrong, but the stability issues is driving me nuts. I suggest that we have to fix it. May be we could implement some unit tests to see how different backends behave in different environments (prefill length, max tokens, max batches etc. etc.)? Otherwise a lot of people are catching a lot of annoying bugs which is a major inconvenience!
[UPDATE2:]
Holy cow! it apparently worked! no crashes yet!
I will post the exact fixes later on, after the thorough hardcore testing.
Let me know if I should provide the exact patches (which are quite trivial BTW) or make a pull request or something.
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
I will provide the draft for the fix below. The Deepseek R1/V3 with long context doesn't output any gibberish or any runtime error [anymore] thanks to this fix. It seems to work fine. Just in case if anyone would like to help me out debugging further they are welcome to check this out:
DISCLAIMERATTN! The code above IS NOT ready for any deployment! The code has been written by the LLM so I am not (yet) responsible for any of it!
The code is pretty much ready for the deployment.
@magikRUKKOLA commented on GitHub (Jul 5, 2025):
@createthis
the urgent help of the nitro coffee drinkers required!!!
@createthis commented on GitHub (Jul 5, 2025):
I’ll take a look after dinner!
@magikRUKKOLA commented on GitHub (Jul 6, 2025):
No crashes with the latest fix (above). Also not sure why the load of the cache from the storage device is turned off by default. Here is how to enable it:
the prefix caching is working great (even after the daemon restart). The only thing left to do is the management of the cache etc.
@createthis commented on GitHub (Jul 6, 2025):
@magikRUKKOLA Sorry I didn't get to it last night. I tried to apply these diffs this morning and my patch tool is saying:
I tried having ChatGPT convert the diffs to unified format, but there's a section of the second diff that has me wondering if we're patching from the same commit of the code. There's no context in your diff at that spot, so I have no idea where my chunk should go.
Can you do one of these, please:
1.) fork
ktransformersto your github, then create a branch and push it, then let me know the branch so I can check it outor
2.) Reply with the commit your code is currently at (
git logcopy paste the first commit hash) and then reply with unified diffsgit difforgit diff -uwbordiff -u <old> <new>.Thanks!
@magikRUKKOLA commented on GitHub (Jul 6, 2025):
@createthis
these are with diff -u
@createthis commented on GitHub (Jul 6, 2025):
Ah. There are mixed tab and space characters in those files and therefore the diffs that are making them hard to apply - probably because github or markdown is stripping the tab characters out. I had to apply them manually. Python. 🙄
I've put this into a branch on my fork and issued a PR for it: https://github.com/kvcache-ai/ktransformers/pull/1422
It's still compiling on my end. I'll test as soon as it's done.
@createthis commented on GitHub (Jul 6, 2025):
Tested with:
I realize this is mostly a
balance_servePR, but all of my testing to date has focused on thektransformersbackend, so that's all I can test with confidence.flow1.sh still crashes the
ktransformersbackend in one shot with this patch applied:The backtrace looks like this:
So, it didn't really solve the issues I've identified, but it may solve other issues that don't affect me, my hardware, and environment. HTH.
@magikRUKKOLA commented on GitHub (Jul 6, 2025):
I see your point. Recently I was doing the same but with the newest release ktransformers introduced the on-storage prefix caching which is available only with the balance_serve backend. So all my testing was focused only on balance_serve backend.
Okay great! Its great to have the bug-case isolated as a curl command. Just for fun I will try to reproduce your bug-case with the ktransformers legacy backend today.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
@createthis
yeah, the ktransformers is crashing indeed.
I am pretty sure that the problem is similar to what is going on with balance_serve and R1/V3 can fix it. But should we do that? The thing is the ktransformers legacy backend is getting deprecated.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
@createthis
yeah itis either crashing (above) or outputting the garbage with ktransformers legacy backend.
in fact, its seems like it wants to rape someone:
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
@createthis
that's the output from the balance_serve backend with our input (with on-storage prefix cache enabled):
(no crashes, no gibberish)
@createthis commented on GitHub (Jul 7, 2025):
My opinion as an old guy who has worked on a lot of software projects over the years is that this project needs unit tests that prevent this sort of thing from happening and CI/CD to enforce it. If it had that, the ktransformers backend could be maintained in a fork or something if they don’t want to support it. However, without the unit tests and CI/CD, this sort of thing will always happen and the ktransformers project will be unstable. I’ve switched to llama.cpp for this reason.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
Lets see if the authors will merge the PR you made. If not, I am forking it lol and doing everything properly.
@createthis commented on GitHub (Jul 7, 2025):
I wish I knew enough about LLM architecture to help. I honestly can't tell if the PR we made solves anything or if it just moves the problem around a bit like a mop with dirty water.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
Well, at least a cannot crash the ktransformers with balance_serve backend anymore, nor it outputs some gibberish as before. The problem appears to be related to how the sampler interacts with GPU->CPU during the inference (and it increases the buffer to deal with 128k context). I mean, any kind of unit test would catch it before the release. I have no idea if anyone in ktransformers tested it before the release. Apparently not.
llama.cpp is too slow. Have you tried ik_llama.cpp with _R4 quants? It works stable but it doesn't have the tool calls or the prefix on-storage caching. So ktransformers is actually more progressive than any other framework, but it needs some polishing.
@createthis commented on GitHub (Jul 7, 2025):
llama.cppis faster for me in tok/s thanktransformers, due to the way it handles NUMA. I can run it with NPS4 and tune the number of CPU threads to be optimal instead of NPS0 under ktransformers. I did prefer the prefill cache in thektransformersbackend, but that's not support now either, so I really no longer have a dog in the race.llama.cppdoes use more VRAM thanktransformers, but I have plenty of VRAM since I upgraded from a 3090 to a blackwell 6000 pro.I have not tried
ik_llama.cpp. I don't like that it uses yet another file format. I have limited disk space, so I'm sticking to GGUF for now.@magikRUKKOLA commented on GitHub (Jul 7, 2025):
@createthis but why you can't just switch to the balance_serve backend for example? Because the llama.cpp works faster in decode?
@createthis commented on GitHub (Jul 7, 2025):
Well, I didn't previously because it crashed around 30k context. My 100% reproducible bug reports for the ktransformers backend were ignored, so I moved on to a more reliable project.
You're saying the balance_serve backend has a faster prefill than
llama.cpp. I wasn't aware that was the case. I can test with my next agentic batch and report back whether it crashes or not and whether it's faster or not.@magikRUKKOLA commented on GitHub (Jul 7, 2025):
Well, just to make sure, I will provide the bash script to run the ktransformers with the 128k context (for the UD-Q2_K_XL quant, you are using the different one).
its basically like you can to change the --optimize_config_path to the one without the '-serve' suffix and change the --backend_type as well.
be aware that the response is limited to the 32k context. But you can prefill up to 128k context.
Well, I am saying that thanks to the recently added support of the on-storage prefill, now we can prefill certain collection of the files we are working on and the backend with the on-storage prefix support will pick it up from the storage, upload it to the VRAM and continue on. In case if its stored like this, the actual prefill speed is a few thousands tokens per second, which is neat.
[EDIT]:
I am talking about the newest version with on-storage prefill support: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/prefix_cache.md
Now its saving the prefill KV-caches into the storage device and upon every prefill it check for the matches.
But to make it work even after the restart of the service you would have to make load_from_disk: true in the config to make it load the tree.json upon the service start (I mentioned it above).
@createthis commented on GitHub (Jul 7, 2025):
@magikRUKKOLA I tried
balance_servejust now at commit8c8cb2with your patch. It doesn't crash, but the model (DeepSeek V3 0324 671b:Q4_K_XL) goes insane at about 17k context. Speed isn't useful without accuracy. I'm going back tollama.cpp.@magikRUKKOLA commented on GitHub (Jul 7, 2025):
Can you share the prompt?
Because I tried with 120k+ and it doesn't output the gibberish etc. No issues basically lol
Absolutely.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
I cannot even find such a commit in the recent history. You dont do the latest masters?@createthis commented on GitHub (Jul 7, 2025):
https://github.com/kvcache-ai/ktransformers/pull/1422/commits/8c8cb207aa0ebe6d00ede36489adfe6ce23bd436
@createthis commented on GitHub (Jul 7, 2025):
I can't, sorry. It's proprietary. It doesn't output gibberish, it just stops following instructions in an agentic context using Open Hands AI. I'd have to record the session using a reverse proxy to see exactly what is going on with the conversation on a low level.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
Stops following instructions upon reaching a certain context length? Is that correct?
And you didnt observe similar behaviour in llama.cpp?
@createthis commented on GitHub (Jul 7, 2025):
Correct. This is after several turns. 3 or 4.
@magikRUKKOLA commented on GitHub (Jul 7, 2025):
This stuff is absolutely crazy dude.
The message to the authors of the ktransformers. You guys, clearly having the issues with the stability. I am proposing the following. Lets implement the --seed parameter (as similar to the vllm) to setup the initial state and to produce the deterministic outputs. Then we would setup some test units with various prompts but with the constant models so the exact outputs of the tests will be deterministic. So every new release we would run all the tests to make sure there is no issues with the stability. It had to be done long time ago. Why no one implemented it?
[EDIT]:
basically lets implement the --seed parameter and just compare the outputs with other frameworks such as llama.cpp, vllm etc. In case they are different, we could find some bugs in the implementation of the framework in question etc.
@ikawrakow commented on GitHub (Jul 8, 2025):
@createthis
In what sense does
ik_llama.cppneed "yet another file format"? It will happily consume any GGUF usable withllama.cpp.@createthis commented on GitHub (Jul 8, 2025):
@ikawrakow
https://huggingface.co/ubergarm/DeepSeek-R1-0528-GGUF
@ikawrakow commented on GitHub (Jul 8, 2025):
And from this you concluded that
ik_llama.cppneeds a different file format? It only means thatik_llama.cppcan do stuff that the other cannot. In no way does it man thatik_llama.cppcannot do what the others can.Just try the DeepSeek model you are trying to run with
llama.cppandKTransformers, you may be pleasantly surprised.@magikRUKKOLA commented on GitHub (Jul 8, 2025):
In terms of getting the full advantage out of Deepseek V3/R1? Yes. So in a strictly practical sense the feeling is such. I, however, have no problem downloading it etc. ) So the point is that the Deepseek V3/R1 is the best opensource LLM so we don't see much point running anything else. So we do like to think of frameworks in terms of what quants they support etc. :) Please don't take it personally. )
@magikRUKKOLA commented on GitHub (Jul 8, 2025):
You mean like llama.cpp doesn't crash, works pretty stable and requires more RAM and its slow? :D At least for now it seems this way.
ktransformers are great but ... unfortunately I observed the similar behaviour as the @createthis was mentioning. It loses context from time to time. That is, its forgetting the very strict rules how the tool call had to be done etc and the whole agentic workflow with thousands of preprocessed tokens goes into shit. That's the experience of the regular user of the ktransformers now. :D
@magikRUKKOLA commented on GitHub (Jul 8, 2025):
@createthis, btw dude. I will leave some idea below related to the agentic workflow. I seen the flow1.sh of yours and the prompt from the openhands ai. And the stop-tag lol to terminate the execution after the first tool call. It looks VERY sketchy. The LLMs supposed to be pretrained to respond with a certain tool tags (see the tokenizer config). So and its seems logical that if we would want to use the tool calls, we would just ask the LLM to call the tools they way it used to. What's the point of making a new tool call format etc.? It works more stable than the native format? That's unclear.
Anyway, there is small chance that the problems you described with ktransformers (that its losing the context, that is, the workflow instructions) are actually the artifact of that custom protocol of yours. That is, everything is the other way around -- some custom glitch in llama.cpp related to the attention is responsible for correct behaviour of the agent of yours. (That's pretty unlikelty but still).
So please share the command how you run llama.cpp (at least I will make sure you're not running it with low context windows or something).
Have you experimented with a regular native tool-call format?
@createthis commented on GitHub (Jul 9, 2025):
@ikawrakow
Seriously off topic, but you brought it up, so...
I compiled
ik_llama.cppthis morning with:I started it with:
I see this as it loads:
Then this happens:
What do you recommend I change?
@magikRUKKOLA commented on GitHub (Jul 9, 2025):
@createthis
This is not a typo?
@magikRUKKOLA commented on GitHub (Jul 9, 2025):
The quant type. _R4 have better performance in terms of perplexity per bpw. Example:
@createthis commented on GitHub (Jul 9, 2025):
It's not. This is the reported max context length
llama.cppdetects in the GGUF.@ikawrakow commented on GitHub (Jul 9, 2025):
I don't feel like polluting the KTransformers repository with off-topic stuff. If you are interested you can enter an issue or start a discussion in the
ik_llama.cpprepository. But just quickly: a) you most likely want to runcmakewith-DGGML_SCHED_MAX_COPIES=1, and b) you need to add-mla 1to your command line to reproducellama.cppbehavior (ik_llama.cpphas several MLA options, and can be run without MLA as well, this is what you are doing). For better performance you would change-mla 3 -fmoeand, depending on how much VRAM you have, use much larger batch/u-batch than the defaults. In general, when not familiar with a tool, it is always a good idea to start with small steps, and only then try to fly.@createthis commented on GitHub (Jul 9, 2025):
@ikawrakow
Without recompiling with
-DGGML_SCHED_MAX_COPIES=1(I have no idea what that does), and when running with:I get about 63.26 tok/s prompt eval and 5.47 tok/s generation. I feel like regular llama.cpp's performance is better in generation and worse in prompt eval. I agree we shouldn't pollute this thread further as it's off topic. Thanks for the info though. I may play with this more.
@magikRUKKOLA commented on GitHub (Jul 9, 2025):
Got the new crash recently.
Its a very hard to catch error. The first one for a few days. I started to play with custom allocators like jemalloc to increase the probability of the occurrence of the error, but its seems to be unrelated. Some tools like cuda-memcheck are available from the cuda toolkit so I will try to debug the issue.
Surprisingly, the authors seems to be uninterested in the issue at all. Its probably makes more sense to [move to] the projects like ik_llama.cpp. At least, they dont crash like this.
[EDIT]:
https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html#racecheck-tool
@magikRUKKOLA commented on GitHub (Jul 9, 2025):
Its not actually an off-topic. Its an extremely important issue how to run the full deepseek locally. The exact framework in question is irrelevant since as you pointed out, the speed doesn't matter much of its unstable. In any case, make sure to make a reference to this topic further on so I can track your progress.
@magikRUKKOLA commented on GitHub (Jul 9, 2025):
As related to the recent quants cooked.
https://huggingface.co/anikifoss/DeepSeek-R1-0528-DQ4_K_R4
Given the issues with the stability in ktransformers it looks like the best option is to move to the ik_llama.cpp (and add the tool calling support and fancy on-storage prefix caching). I guess the issue can be closed.
[EDIT]: i will try to find out what is causing the issues in ktranformers though.
@ubergarm commented on GitHub (Jul 10, 2025):
fwiw I wrote the early ktransformers guide but have since moved on to spend my time working on https://github.com/ikawrakow/ik_llama.cpp
For folk who don't know, ik is the author of most of the mainline quant types newer than the original
q8_0andq4_1types includingiq4_xsandq6_Ketc.To be clear,
ik_llama.cppdoes not use a new file type and easily runs mainline llama.cpp quants from bartowski and unsloth and others.Basically it runs all those quants and more newer better types.
I'll have to go update my README hugging face model cards to clarify that if people are mistaking this.
Folks are now getting over 300 tok/sec PP and in the high teens TG with my latest https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF and no need for newer CUDA GPUs architectures like 8.9 for for native fp8 e4m3
So perfect if you have a mix of older GPUs like 3090TI and RTX A6000 multi-GPU configuration.
Thanks!
顺带一提，我编写了早期的 ktransformers 指南，但现已转向投入时间开发 https://github.com/ikawrakow/ik_llama.cpp。
对于不了解的各位，ik 是大部分新量化类型（超越原始
q8_0和q4_1类型）的作者，包括iq4_xs和q6_K等。明确说明：
ik_llama.cpp不使用新的文件类型，可轻松运行来自 bartowski、unsloth 及其他来源的主线 llama.cpp 量化模型。简言之，它不仅能运行上述量化模型，还兼容更多更新更优的类型。
我需要更新 Hugging Face 模型卡的 README 以澄清此点，若用户存在误解。
目前用户在使用我最新发布的 https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF 时，可获得超过 300 tok/sec 的 PP和十几的 TG，且无需依赖 8.9 等新 CUDA GPU 架构来实现原生 fp8 e4m3。
因此，若您采用混合旧款 GPU（如 3090TI 与 RTX A6000）的多 GPU 配置，此方案堪称完美。
感谢！
@magikRUKKOLA commented on GitHub (Jul 10, 2025):
@ubergarm thanks for the info!
Can you please comment regarding the issue with the context length of the Deepseek V3/R1. In ktransformers its possible to run 128k context (recently I noted that it actually supports 163k) on a single 24GB VRAM GPU by using the trick with matrix absorption. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel
As far as I can see, the longer than 64k (?) context in ik_llama.cpp requires [more] than 24GB VRAM. Can the same thing be implemented in ik_llama.cpp for example?
Also regarding the multi-GPU configuration. I played around with the configs but found not much benefit of using more than 1 GPU (etc. RTX 3090) even though I am using P2P-enabled drivers. Basically, if one have enough RAM there would be no benefit of using more than one GPU? Or ... may be just the prefill can be improved, not the decode?
[EDIT]: regarding the multi-GPU config and the longer context of Deepseek V3/R1 -- may be the KV-cache for ik_llama.cpp can be distributed to two 24 GB VRAM GPUs? Unfortunately I could not find the docs regarding this. Can you point out the direction?
@magikRUKKOLA commented on GitHub (Jul 10, 2025):
But ... the TNG-R1T2-Chimera-GGUF is not as "smart" as the R1-0528, right?:
So ... its more like a replacement of the V3-0324 then? And with ik_llama.cpp its better in performance than ktransformers? Is that correct?
@magikRUKKOLA commented on GitHub (Jul 10, 2025):
Lets see ...
32k context:
64k context:
so in this case the dense experts uploaded to gpu require 17744.02MB. So about 6.5GB is free for the KV-cache related stuff which consists of: CUDA_Host compute buffer size, CUDA0 compute buffer size, KV self size. So with q8 quantisation of KV-cache it would unable to run with 64k context. Is it possible to place the KV-cache onto the second GPU to extend the supported context length?
@ubergarm commented on GitHub (Jul 10, 2025):
@magikRUKKOLA
This is not specific to the inference engine, it is more specific to the specific quantization in terms of how large are the tensors your choose to offload onto VRAM and what kv-cache quantization you are using.
ik_llama.cpp implementation of DeepSeek MLA is very efficient, but generally yes you would need more than 24GB VRAM to use the model's full 160k context. I've done it with two RTX A6000s (96GB VRAM total) (the older non-Blackwell ones). Most normal GQA style attention grows exponentially, but MLA is linear which is quite nice for longer context:
You see for this specific quant there is a fixed size for the attn/shexp/first 3 ffn dense layers being offloaded to VRAM plus a variable amount for the kv-cache. My newer quants use slimmer newer
IQ3_KSand similar SOTA quants to slim down that fixed amount further so people with 16GB VRAM can run it or you can get more context in 24GB etc.If you read my old guide I have a section addressing this where at least at the time ktransformers did not take advantage of multi-GPU (unless u can offload the whole thing) and it even says so in the documentation. Here is the reference.
ik_llama.cpp has no problem getting more speed using more GPUs. You just have to be careful how to use the
-ot --override-tensorcommand to not split up the tensors wierd. This works well for Qwen3 and other moe as well but the tensor naming convention and exact architectures vary so pay close attention. I'm helping some folks with gaming CPUs who have 5 GPUs with no nvlink no p2p anything and they are getting great speeds here in this discussion. I try to include multiGPU examples on most of my model cards."It depends." A higher BPW quantization of TNG-R1T2-Chimera may be better than a lower BPW quantization of R1-0528. You really need to choose the right model and tune it specific to your use case e.g. do you need more PP (prefill) or do you want more TG etc.
I also have R1-0528 quants available if you have enough hardware and patience to wait for "the smartest" answer. Choose whatever tool you want.
You'll probably have to test and it is somewhat difficult to make a direct comparison because the tooling is different between the two ecosystems. In my own experience ik_llama.cpp is better for my use-case becase:
I personally also recommend https://github.com/turboderp-org/exllamav3 and vllm and sglang as well depending on what LLM you want to use, how much VRAM you have, what the environment in which you're working is etc.
But for running these huge 671B MoE's my weapon of choice for hybrid CPU+GPU inferencing is definitely ik_llama.cpp.
You might argue "well ktransformers has USE_NUMA=1 and supports AMX extensions" sure but in my own testing ik's avx2 and avx bf16 CPU implementations are faster than the AMX stuff anyway and if you can configure your AMD Epyc to
NPS0or your Intel Xeon toSNC=Disableand try to use single NUMA node then no need to pay for twice the RAM.There is still room for improvment, and ik has been quite dedicated and all the amazing PP (prefill) speeds optimizations are really adding up now.
As mentioned above, the tooling around the llama.cpp ecosystem allows me to create and compare my quants to other providers and in general ik's SOTA out perform his older quants which other quant cookers seem to limit themselves:
magikRUKKOLA
这与推理引擎本身无关，更多取决于具体量化方案的选择：包括选择卸载到显存中的张量大小以及使用的kv缓存量化方法。
ik_llama.cpp对DeepSeek MLA的实现非常高效，但通常确实需要超过24GB显存才能使用模型完整的160k上下文。我曾在两块RTX A6000（非Blackwell版，总计96GB显存）上成功运行。普通GQA风格注意力的计算量呈指数级增长，而MLA是线性的，这对长上下文非常有利：
如图所示，在该特定量化方案中，attn/shexp/前三个ffn密集层有固定大小的显存占用，而kv缓存部分则是可变大小。我更新的量化方案采用了更精简的新式
IQ3_KS等SOTA量化方法，进一步减少了固定占用部分，使得16GB显存用户也能运行，或在24GB显存下获得更多上下文。如果您读过我的旧版指南，其中专门讨论过这个问题：至少在当时的ktranformers版本中，除非能完全卸载模型，否则无法有效利用多GPU 文档中也明确提及）。参考链接在此
ik_llama.cpp可以毫无问题地通过多GPU获得速度提升。关键是要谨慎使用
-ot --override-tensor命令，避免异常张量分割。这种方法对wen3及其他MoE模型同样有效，但需注意不同架构的张量命名规范差异。我目前正在协助一些使用游戏CPU的用户（他们拥有5块无NVLink/P2P的GPU），在此讨论中他们获得了出色性能。我尽量在多数模型卡中提供多GPU示例。"这取决于具体情况。" 高BPW量化的TNG-R1T2-Chimera可能优于低BPW量化的R1-0528。您需要根据具体用例选择合适模型并进行调优，例如侧重预填充(PP)还是生成(TG)等需求。
如果硬件充足且有耐心等待"最智能"的答案，我也提供R1-0528量化版本。工具选择完全取决于您。
可能需要自行测试，因为两者的工具链差异使得直接比较较为困难。根据我的经验，ik_llama.cpp更适合我的使用场景，原因如下：
根据LLM类型、显存容量和工作环境，我个人也推荐：
但对于运行这些671B参数的巨型MoE模型，ik_llama.cpp绝对是混合CPU+GPU推理的首选工具。
有人可能反驳"ktransformers支持USE_NUMA=1和AMX扩展"——但在我的测试中，ik的avx2/avx bf16 CPU实现反而比AMX更快。若能将AMD Epyc配置为
NPS0或Intel Xeon设为SNC=Disable并使用单NUMA节点，则无需支付双倍内存成本。仍有改进空间，而ik始终专注优化——目前所有的预填充(PP)速度优化成果正在持续累积。
如前所述，llama.cpp生态的工具链使我能够将自己的量化方案与其他提供商对比。总体而言，ik的SOTA量化始终优于其旧版量化，而其他量化工具似乎自我设限：
@magikRUKKOLA commented on GitHub (Jul 10, 2025):
@ubergarm
Cool but what about the matrix absorption for prefill trick??? Its seems more reasonable to use flashinfer to support the long context just like ktransformers are doing. Namely,
https://docs.flashinfer.ai/api/mla.html
As far as I can see, its not supported in ik_llama.cpp, right?
@ubergarm commented on GitHub (Jul 10, 2025):
@magikRUKKOLA
The model you chose there is one of the largest which uses
Q8_0quantization for the tensors going onto GPU VRAM pretty sure, so it should line up roughly with the graph I presented before. For my smaller quants that ~17GB fixed size is lower allowing more space for kv-cache.Of course, by default ik/llama.cpp splits KV-cache across all GPUs or can use CPU RAM (much slower) which some folks choose to do if they require very high kv-cache on dense models with less efficient MQA/GQA.
To be honest I don't know for as I have not fully inspected the (three different at least) MLA implementations on ik/llama.cpp. There are many closed PRs on multiple forks from around February of this year with the details. I'm not sure why you're asking about this specific implementation optimization instead of looking at the actual benchmark results for yourself?
-mla 3) and ik even helped nvidia's jeffbolz get this going to help with the vulkan backend this weekOkay, I'll stop spamming up this thread and hmu over on ik's fork. There is a lot of info over there spread out in discussions, closed PRs, my hugging face repo threads etc, so I know it is overwhelming. Feel free to hmu over on ik's fork with your hardware details, current command, and what it is you are trying to do e.g. "i want to run as much context in the least VRAM as possible" or whatever you're up to.
Cheers!
magikRUKKOLA
您选择的模型是当前最大规模之一，其GPU显存中的张量采用
Q8_0量化方案，这与之前展示的图表基本吻合。对于我的小型量化模型，约1GB的固定占用会更低，从而为KV缓存预留更多空间。当然可以。默认情况下ik/llama.cpp会将KV缓存跨GPU分配，亦可使用CPU内存（速度显著降低）。部分用户在运行MQA/GQA效率较低的稠密模型且需超大KV缓存时选择此方案。
坦白说，我尚未全面研究ik/llama.cpp中至少三种不同的MLA实现机制。今年二月前后多个分支存在大量已关闭PR涉及该技术细节。我不理 您为何不直接查看实际基准测试结果，而执着于追问具体实现优化？
-mla 3参数）。ik近期还协助nvidia的jeffbo在vulkan后端实现该功能为避免干扰本讨论区，建议移步ik分支交流。相关技术资料分散在讨论区、closed PR及我的hugging face仓库中，信息量确实庞大。欢迎携带硬件配置、当前命令及具体需求（如"在最小显存下运行最大上下文"）至ik分支探讨。
顺颂时祺！
@magikRUKKOLA commented on GitHub (Jul 12, 2025):
I think I know what's up.
There is a forked flashinfer. And the according bugreports. For example:
https://github.com/flashinfer-ai/flashinfer/pull/1178
https://github.com/flashinfer-ai/flashinfer/pull/1206
https://github.com/flashinfer-ai/flashinfer/pull/1008
https://github.com/flashinfer-ai/flashinfer/pull/1059 (not sure if POD is used in ktransformers)
etc.
So I decided to use the sampling.cuh from the master branch of the flashinfer project.
[EDIT]: also pulled the mask.cuh and utils.cuh from the master. Now I will try to crash it once more.
我大概明白怎么回事了。
存在一个分叉版本的 flashinfer，以及相关的漏洞报告。例如：
https://github.com/flashinfer-ai/flashinfer/pull/1178
https://github.com/flashinfer-ai/flashinfer/pull/1206
https://github.com/flashinfer-ai/flashinfer/pull/1008
https://github.com/flashinfer-ai/flashinfer/pull/1059 （不确定 ktransformers 是否使用了 POD）
等等。
因此我决定直接采用 flashinfer 项目主分支的 sampling.cuh 文件。
@magikRUKKOLA commented on GitHub (Jul 12, 2025):
@createthis
So yeah, you were right. The solution was kinda easier -- the fix sampler from the flashinfer master's has to be manually merged into custom_flashinfer of the ktransformer's master.
@magikRUKKOLA commented on GitHub (Jul 13, 2025):
I am having hard time crashing the ktransformers after the patching it from the latest master of flashinfer. It just outputs garbage after a really long prompt.
I can't get why no one is interested in solving this. Without it ktransformers is basically straight garbage. Imagine you prefilling 100k+ to the llm and after close to hour of waiting you getting garbage as a response. Or its just crashing (if you are lucky). Its unstable. I can't understand why you are bragging so much with how fast the ktransformers are. Its irrelevant because its unusable.
我在修补最新版本的flashinfer后，崩溃问题一直困扰着ktransformers。在输入超长提示后，它只会输出一堆乱码。
实在无法理解为什么没人愿意解决这个问题。没有修复的ktransformers简直就是废品。想象一下：你给大语言模型输入超过10万字符的前缀，苦等近一小时
后，得到的却是垃圾输出——或者干脆崩溃（如果你够幸运的话）。这玩意儿太不稳定了。真不明白你们凭什么吹嘘ktransformers的速度有多快，连基本使用都
成问题，速度快又有何意义？
（翻译说明：
@magikRUKKOLA commented on GitHub (Jul 13, 2025):
BTW I can't run the compute-sanitizer in order to check for the memory access violations etc. because its consuming too much VRAM. So you can't run it with compute-sanitizer and debug. Only cuda-gdb can be used.
@magikRUKKOLA commented on GitHub (Jul 13, 2025):
Nah, I can't figure this out. It doesn't crash and outputs garbage now (at a long context 100k+).
The ktransformers is using flashinfer (not the latest master even, but some old for with 0.2.3 version). The flashinfer IS UNSTABLE. Hence the ktransformers IS UNSTABLE AND UNUSABLE.
I will just plug-in more GPUs to increase the prefill and the context and will switch to ik_llama.cpp. I will not waste my time with this half-ass backed project. Too bad.
@magikRUKKOLA commented on GitHub (Jul 13, 2025):
Got 100% reproducible bug. The KV-cache is saved into a dump, and on[c]e I am sending the same 100k+ prefill the ktransformers is loading up the cache and crashing. Its about 5GB in size and I can upload it. If there is a need to debug it, please contact me.
[EDIT]:
Nope, its not 100% reproducible. Sometimes it crashes, sometimes it outputs garbage. Absolute disaster.
[EDIT2]:
another example of the garbage response after 120k of context:
and this is after the latest patches from flashinfer.
lol
@BoxuanYang commented on GitHub (Sep 3, 2025):
i hv same issue, hope this can be fixed
@magikRUKKOLA commented on GitHub (Sep 3, 2025):
@BoxuanYang
2.5 months went by already. Its unlikely that it will be fixed. The FLASHINFER is unreliable.
@jagmarques commented on GitHub (Apr 7, 2026):
The garbage-output-at-100k+ context issue @magikRUKKOLA describes is a known failure mode in FlashInfer's paged attention when the KV cache index tables overflow a certain threshold — it's not strictly a FlashInfer version issue but a configuration one.
A few things worth checking before switching away entirely:
1. Page size and block count: If
block_size * max_blocks< your actual sequence length, FlashInfer silently wraps indices rather than throwing an OOM. Check that--cache-size(in tokens) is explicitly set to cover your full 100k+ prompt, not left at the default.2. KV cache dtype mismatch: Garbage output (not crash) at long context often means the KV tensors are being written as FP16 but read as BF16, or vice versa. This gets triggered specifically at long sequences because earlier tokens' cache entries were written under one dtype and later ones under another after some internal state transition. Check
model.config.torch_dtypevs what FlashInfer is initialized with.3. Sequence ID overflow: Some older FlashInfer builds use int32 for sequence position encoding internally. At 100k+ tokens the arithmetic doesn't overflow, but if you're doing multiple turns the cumulative position counter can. Force
--no-context-shiftand start fresh sessions to isolate whether this is session-length vs token-length.Workaround: For the cache-dump reload crash specifically — try clearing the KV dump and regenerating it after updating to the latest ktransformers master (the FlashInfer pinned version was bumped in a recent commit). The serialization format changed and old dumps may be incompatible.
Agree that this class of bug is painful because it's non-deterministic. Worth filing upstream with FlashInfer with a minimal reproduction if you can isolate it.
@magikRUKKOLA commented on GitHub (Apr 7, 2026):
@jagmarques
Ha! Okay, suppose so.
Is there any benchmarks available for the latest version of ktransformers? For example,
Kimi K2.5 Q4_X. Is there any prefill/decode available so I can cross-compare withik_llama.cppsee https://github.com/ikawrakow/ik_llama.cpp/discussions/1544#discussioncomment-16398222I am having close to 11 tps decode and about 120 tps prefill at zero ctx. Can
ktransformersbeat that?@jagmarques commented on GitHub (Apr 7, 2026):
Good question on the Kimi K2.5 benchmarks. I don't have direct numbers for ktransformers on K2.5 Q4 specifically, but from what I've seen the decode throughput depends heavily on whether you're hitting the FlashInfer paged attention path vs the fallback.
For the 11 tps decode and 120 tps prefill you're getting on ik_llama.cpp with zero context, that's solid for a single 3090. ktransformers should be in the same ballpark for decode on short context, but the stability issues you documented above are real and would eat any performance advantage at 100k+.
If you do test it, the key comparison would be at 64k+ context where the KV cache pressure actually matters. Short context benchmarks don't tell the full story since everything fits in VRAM anyway.
Let me know how the ik_llama.cpp comparison goes at long context. The community needs honest head-to-head data at 100k+ tokens, not just short-context throughput numbers.