kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #611] How to run DeepSeek-R1-Distill-Qwen-32B on ktransformers? #452

New issue

Closed

opened 2026-10-29 14:39:52 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 14:39:52 +00:00

Owner

Originally created by @Kinny2000 on GitHub (Feb 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/611

I tried to run DeepSeek-R1-Distill-Qwen-32B but failed
report error:
'Qwen2Config' object has no attribute 'output_router_logits'

Should I implement new Qwen2Model myself？

Anyone has comment on this issue？

Originally created by @Kinny2000 on GitHub (Feb 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/611 I tried to run DeepSeek-R1-Distill-Qwen-32B but failed report error: 'Qwen2Config' object has no attribute 'output_router_logits' Should I implement new Qwen2Model myself？ Anyone has comment on this issue？

bsakaguchi

2026-10-29 14:39:52 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:39:53 +00:00

Author

Owner

@lusipad commented on GitHub (Feb 23, 2025):

From what I know, KTransformer is mainly for optimizing MoE models (Mixture-of-Experts), so it doesn’t work with regular models like deepseek-r1-qwen2.5-32b-distill.
If you want to know which models are compatible, check out the 'Suggested Model' section in their docs here: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/install.md

@lusipad commented on GitHub (Feb 23, 2025): From what I know, KTransformer is mainly for optimizing MoE models (Mixture-of-Experts), so it doesn’t work with regular models like deepseek-r1-qwen2.5-32b-distill. If you want to know which models are compatible, check out the 'Suggested Model' section in their docs here: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/install.md

bsakaguchi commented

2026-10-29 14:39:53 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 23, 2025):

Yes, as @lusipad says, ktransformers is mainly for running big MoE models like DeepSeek-R1 671B unsloth GGUFs or Mixtral 8x22B GGUF.

For dense models like most 70B and Qwen 2.5 32B models (that distill you mention is simply Qwen 2.5 32B fine-tuned on output from R1 and has totally different architecture than R1).

If you have enough VRAM to hold the entire model, then consider quants other than GGUF and engines like vllm / exllamv2 / aphrodite-engine / etc. and all those which are optimized for that use case.

If you want to partial offload a dense model between VRAM and RAM for now I'd suggest using llama.cpp and offload as much of the model into VRAM as possible e.g.

# adjust n-gpu-layers as high as possible until OOM then backoff by one
./build/bin/llama-server \
    --model bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF
    --ctx-size 8192 \
    --n-gpu-layers 10 \
    --cache-type-k q4_0 \
    --cache-type-v q4_0 \
    --flash-attn \ 
    --parallel 1 \
    --threads 8 \
    --host 127.0.0.1 \
    --port 8080

@ubergarm commented on GitHub (Feb 23, 2025): Yes, as @lusipad says, ktransformers is mainly for running big MoE models like DeepSeek-R1 671B unsloth GGUFs or Mixtral 8x22B GGUF. For dense models like most 70B and Qwen 2.5 32B models (that distill you mention is simply Qwen 2.5 32B fine-tuned on output from R1 and has totally different architecture than R1). If you have enough VRAM to hold the entire model, then consider quants other than GGUF and engines like vllm / exllamv2 / aphrodite-engine / etc. and all those which are optimized for that use case. If you want to partial offload a dense model between VRAM and RAM for now I'd suggest using llama.cpp and offload as much of the model into VRAM as possible e.g. ```bash # adjust n-gpu-layers as high as possible until OOM then backoff by one ./build/bin/llama-server \ --model bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF --ctx-size 8192 \ --n-gpu-layers 10 \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --flash-attn \ --parallel 1 \ --threads 8 \ --host 127.0.0.1 \ --port 8080 ```

bsakaguchi commented

2026-10-29 14:39:53 +00:00

Author

Owner

@ubergarm commented on GitHub (Feb 28, 2025):

你正在尝试运行的是稠密型Llama/Qwen模型架构，而非真正的R1 MoE模型架构。我建议你尝试使用llama.cpp来运行稠密模型。祝你好运！

https://github.com/kvcache-ai/ktransformers/issues/642#issuecomment-2679436553

@ubergarm commented on GitHub (Feb 28, 2025): 你正在尝试运行的是稠密型Llama/Qwen模型架构，而非真正的R1 MoE模型架构。我建议你尝试使用llama.cpp来运行稠密模型。祝你好运！ https://github.com/kvcache-ai/ktransformers/issues/642#issuecomment-2679436553

bsakaguchi commented

2026-10-29 14:39:54 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

bsakaguchi referenced this issue

2026-10-29 14:40:47 +00:00

[GH-ISSUE #712] Does the official plan support Turing Nvidia GPU ？ #521