Regarding your question on disabling the offloading of MoE layers to the CPU, I’ve provided a brief explanation in my earlier response (#46 ). Please take a look there for detailed instructions.

@Azure-Tang commented on GitHub (Aug 22, 2024): Regarding your question on disabling the offloading of MoE layers to the CPU, I’ve provided a brief explanation in my earlier response (#46 ). Please take a look there for detailed instructions.

bsakaguchi commented

2026-10-29 14:32:47 +00:00

Author

Owner

@molamooo commented on GitHub (Aug 22, 2024):

I tried using custom rule file to put all layers on GPU, but KTransformer triggers oom when dequantizing parameters. I use a 24GB 4090 to run deepseek-v2-lite-chat with q8_0 quantization. It should only takes ~16GB memory. Other systems like llama.cpp can run without issue.

@molamooo commented on GitHub (Aug 22, 2024): I tried using custom rule file to put all layers on GPU, but KTransformer triggers oom when dequantizing parameters. I use a 24GB 4090 to run deepseek-v2-lite-chat with q8_0 quantization. It should only takes ~16GB memory. Other systems like llama.cpp can run without issue.

bsakaguchi commented

2026-10-29 14:32:48 +00:00

Author

Owner

@ELigoP commented on GitHub (Aug 22, 2024):

This is hardly possible.
What is your model size? With q8 quantization it should be >200 GB. If you

put all layers on GPU

no wonder it OOMs. Llama.cpp automatically offloads layers in to CPU suboptimal way. If you force it to offload everything to GPU it will also OOM.

E.g. with my setup I get 7.5 tps with suboptimal llama.cpp CPU offload and 9 tps with ktransformers (even that it utilizies VRAM suboptimally).

@ELigoP commented on GitHub (Aug 22, 2024): This is hardly possible. What is your model size? With q8 quantization it should be >200 GB. If you > put all layers on GPU no wonder it OOMs. Llama.cpp automatically offloads layers in to CPU suboptimal way. If you force it to offload everything to GPU it will also OOM. E.g. with my setup I get 7.5 tps with suboptimal llama.cpp CPU offload and 9 tps with ktransformers (even that it utilizies VRAM suboptimally).

bsakaguchi commented

2026-10-29 14:32:48 +00:00

Author

Owner

@molamooo commented on GitHub (Aug 22, 2024):

This is hardly possible. What is your model size? With q8 quantization it should be >200 GB.

I mentioned that I use deepseek-v2-lite-chat, which is a 16B model. It shouldn't oom.

@molamooo commented on GitHub (Aug 22, 2024): > This is hardly possible. What is your model size? With q8 quantization it should be >200 GB. I mentioned that I use deepseek-v2-lite-chat, which is a 16B model. It shouldn't oom.

bsakaguchi commented

2026-10-29 14:32:48 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 22, 2024):

I see, if you are using KExpertsTorch as backend, it will dequant the weights to model's default dtype, for deepseek this is bf16. So this is why it oom. Maybe you can try marlin backend? But we didn't optimise this operator in generate phase.

@Azure-Tang commented on GitHub (Aug 22, 2024): I see, if you are using KExpertsTorch as backend, it will dequant the weights to model's default dtype, for deepseek this is bf16. So this is why it oom. Maybe you can try marlin backend? But we didn't optimise this operator in generate phase.

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871

bsakaguchi referenced this issue

2026-10-29 14:45:39 +00:00

[GH-ISSUE #1288] [Bug] #876

bsakaguchi referenced this issue

2026-10-29 14:51:25 +00:00

[PR #27] [MERGED] [Feature] towards 0.1.2 #1322