[GH-ISSUE #1655] GGUF format is not working with Deepseek-R1, GLM-4.5-AIR and GPT-OSS #1068

New issue

Open

opened 2026-10-29 14:48:10 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:48:10 +00:00

Owner

Originally created by @xin3he on GitHub (Dec 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1655

Originally assigned to: @SkqLiao on GitHub.

Reminder

I have read the above rules and searched the existing issues.

System Info

Hi, I'm very interested in the GGUF loading with KT + SGLang, and successfully ran the Qwen3-30B-A3B demo you provided.
https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel#option-b-llamafile-backend-gguf

I tried to loading GGUF Q4_K_M model generated by AutoRound and met some issues. BTW, those GGUF models are verified with llama.cpp.

When I try it with Deepseek R1, it worked well during loading but failed during capturing batches.

when I try it with GPT-OSS, it reports error during loading.
To rule out AutoRound involvement, I tried gpt-oss-20b-Q4_K_M.gguf from https://huggingface.co/unsloth/gpt-oss-20b-GGUF/tree/main as well, and it still didn't work.
When it comes to GLM-4.5-AIR, Another error happened as below.

Reproduction

# AutoRound quantization
pip install auto-round
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name gpt-oss-120b
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name GLM-4.5-Air
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name DeepSeek-R1-0528
 
# KT + SGLang
## DeepSeek-R1
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
  --model DeepSeek-R1-0528 \
  --port 8802 \
  --kt-method LLAMAFILE \
  --kt-weight-path DeepSeek-R1-256x20B-0528-Q4_K_M.gguf \
  --kt-cpuinfer 48 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

##  GPT-OSS
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B
### Use AutoRound generated model or this one
huggingface-cli download unsloth/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \
  --local-dir gpt-oss-20b-GGUF

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8801 \
  --model gpt-oss-120b \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name gpt \
  --enable-mixed-chunk \
  --kt-method LLAMAFILE \
  --kt-weight-path gpt-oss-20b-GGUF \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

## GLM-4.5-AIR
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
  --model GLM-4.5-Air \
  --port 8802 \
  --kt-method LLAMAFILE \
  --kt-weight-path GLM-4.5-Air-128x9.4B-Q4_K_M.gguf \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

Others

No response

Originally created by @xin3he on GitHub (Dec 2, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1655 Originally assigned to: @SkqLiao on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Hi, I'm very interested in the GGUF loading with `KT + SGLang`, and successfully ran the Qwen3-30B-A3B demo you provided. https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel#option-b-llamafile-backend-gguf I tried to loading GGUF Q4_K_M model generated by [AutoRound](https://github.com/intel/auto-round) and met some issues. BTW, those GGUF models are verified with llama.cpp. - When I try it with Deepseek R1, it worked well during loading but failed during capturing batches. <img width="1241" height="367" alt="Image" src="https://github.com/user-attachments/assets/b14b5d28-0479-4dba-abe8-9d0072f713d9" /> <img width="1230" height="456" alt="Image" src="https://github.com/user-attachments/assets/93ab1775-b48f-44f1-a8d6-98eea163e8bd" /> <img width="1237" height="910" alt="Image" src="https://github.com/user-attachments/assets/da89f097-401c-4604-818b-98710cc1f463" /> - when I try it with GPT-OSS, it reports error during loading. To rule out AutoRound involvement, I tried gpt-oss-20b-Q4_K_M.gguf from https://huggingface.co/unsloth/gpt-oss-20b-GGUF/tree/main as well, and it still didn't work. <img width="1241" height="921" alt="Image" src="https://github.com/user-attachments/assets/1c397e76-4591-4cad-8a83-44e9eab2def7" /> - When it comes to GLM-4.5-AIR, Another error happened as below. <img width="1176" height="958" alt="Image" src="https://github.com/user-attachments/assets/368db93d-ea01-4150-8459-4083a79cbcfb" /> ### Reproduction ```bash # AutoRound quantization pip install auto-round CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name gpt-oss-120b CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name GLM-4.5-Air CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name DeepSeek-R1-0528 # KT + SGLang ## DeepSeek-R1 CUDA_VISIBLE_DEVICES=4 \ python -m sglang.launch_server \ --model DeepSeek-R1-0528 \ --port 8802 \ --kt-method LLAMAFILE \ --kt-weight-path DeepSeek-R1-256x20B-0528-Q4_K_M.gguf \ --kt-cpuinfer 48 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ## GPT-OSS huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B ### Use AutoRound generated model or this one huggingface-cli download unsloth/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \ --local-dir gpt-oss-20b-GGUF python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8801 \ --model gpt-oss-120b \ --trust-remote-code \ --mem-fraction-static 0.92 \ --chunked-prefill-size 4096 \ --served-model-name gpt \ --enable-mixed-chunk \ --kt-method LLAMAFILE \ --kt-weight-path gpt-oss-20b-GGUF \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ## GLM-4.5-AIR CUDA_VISIBLE_DEVICES=4 \ python -m sglang.launch_server \ --model GLM-4.5-Air \ --port 8802 \ --kt-method LLAMAFILE \ --kt-weight-path GLM-4.5-Air-128x9.4B-Q4_K_M.gguf \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ``` ### Others _No response_

bsakaguchi added the

bug

label

2026-10-29 14:48:10 +00:00

bsakaguchi commented

2026-10-29 14:48:11 +00:00

Author

Owner

@SkqLiao commented on GitHub (Dec 3, 2025):

Hi, this appears to involve at least two issues: ggml cannot recognize type 39, and IntermediateSize is not divisible by QK_K(256). AutoRound is a relatively new quantization format, and we need some time to investigate its support.

@SkqLiao commented on GitHub (Dec 3, 2025): Hi, this appears to involve at least two issues: ggml cannot recognize type 39, and IntermediateSize is not divisible by QK_K(256). AutoRound is a relatively new quantization format, and we need some time to investigate its support.

bsakaguchi commented

2026-10-29 14:48:11 +00:00

Author

Owner

@DocShotgun commented on GitHub (Dec 6, 2025):

Is sglang+kt-kernel meant to be able to support GGUF-quantized models, in addition to kt-kernel's bespoke GPU weight and CPU weight formats?

@DocShotgun commented on GitHub (Dec 6, 2025): Is `sglang+kt-kernel` meant to be able to support GGUF-quantized models, in addition to `kt-kernel`'s bespoke GPU weight and CPU weight formats?