kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1861] BF16 backend cannot parse gpt-oss packed/fused expert weight format #1158

New issue

Closed

opened 2026-10-29 14:49:23 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:49:23 +00:00

Owner

Originally created by @vshortt73 on GitHub (Feb 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1861

Reminder

I have read the above rules and searched the existing issues.

System Info

kt-kernel: main @ 16a8b98
SGLang: main @ 45095bac7
Model: openai/gpt-oss-120b (HuggingFace safetensors, MXFP4)
GPU: RTX 5090 (32GB)
CPU: AMD Ryzen 9 9900X (Zen 5, AVX-512 BF16)

Reproduction

Repository: kvcache-ai/ktransformers (could tie to #1655)

Summary

The BF16SafeTensorLoader defaults to DeepSeek MoE weight format and cannot parse gpt-oss's packed fused expert tensors. This is a separate failure mode from the GGUF/MXFP4 type 39 issue already documented in #1655.

Details

Weight format mismatch

The BF16SafeTensorLoader expects DeepSeek-style per-expert weight keys:

model.layers.0.mlp.experts.0.gate_proj.weight
model.layers.0.mlp.experts.0.up_proj.weight
model.layers.0.mlp.experts.0.down_proj.weight

gpt-oss uses packed fused tensors with MXFP4 block/scale format:

model.layers.0.mlp.experts.gate_up_proj_blocks
model.layers.0.mlp.experts.gate_up_proj_scales
model.layers.0.mlp.experts.gate_up_proj_bias
model.layers.0.mlp.experts.down_proj_blocks
model.layers.0.mlp.experts.down_proj_scales
model.layers.0.mlp.experts.down_proj_bias

All 128 experts are packed into single tensors per layer (fused gate+up projection), with separate block quantization scales. There is no per-expert dimension in the key naming.

Observed behavior

[BF16SafeTensorLoader] No MoE format detected, defaulting to: deepseek

The loader (kt-kernel/python/utils/loader.py line 463) fails to match gpt-oss's key pattern and falls back to DeepSeek format, which then fails to load the weights correctly.

Three confirmed failure paths for gpt-oss on KTransformers

For completeness, here are all three integration paths attempted and their failure modes:

Backend	Weight Source	Failure	Root Cause
BF16	HF safetensors	Format mismatch	Packed fused expert tensors, not per-expert keys
LLAMAFILE	MXFP4 GGUF	`ValueError: 39 is not a valid GGMLQuantizationType`	gguf 0.17.1 lacks MXFP4 type 39
LLAMAFILE	Q4_K_M GGUF	Testing in progress	Standard quant type, should be compatible

Suggestion

To support gpt-oss (and likely future models using packed/fused MoE formats):

Add a gpt-oss format detector in the BF16SafeTensorLoader alongside the existing DeepSeek detector
Implement unpacking logic for fused gate_up_proj_blocks/scales into individual expert weights
Alternatively, document that gpt-oss requires the LLAMAFILE backend with standard (non-MXFP4) GGUF quantizations

Note on successful llama.cpp baseline

For reference, gpt-oss-120b runs successfully on the same hardware via llama.cpp with --override-tensor expert offloading at 37.95 t/s (15 of 36 layers' experts on GPU, remainder on CPU via DDR5). The model architecture is functional — it's specifically the KTransformers integration paths that need gpt-oss format support.

Others

No response

Originally created by @vshortt73 on GitHub (Feb 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1861 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info - **kt-kernel**: `main` @ 16a8b98 - **SGLang**: `main` @ 45095bac7 - **Model**: openai/gpt-oss-120b (HuggingFace safetensors, MXFP4) - **GPU**: RTX 5090 (32GB) - **CPU**: AMD Ryzen 9 9900X (Zen 5, AVX-512 BF16) ### Reproduction ## Repository: kvcache-ai/ktransformers (could tie to #1655) ## Summary The BF16SafeTensorLoader defaults to DeepSeek MoE weight format and cannot parse gpt-oss's packed fused expert tensors. This is a separate failure mode from the GGUF/MXFP4 type 39 issue already documented in #1655. ## Details ### Weight format mismatch The BF16SafeTensorLoader expects DeepSeek-style per-expert weight keys: ``` model.layers.0.mlp.experts.0.gate_proj.weight model.layers.0.mlp.experts.0.up_proj.weight model.layers.0.mlp.experts.0.down_proj.weight ``` gpt-oss uses packed fused tensors with MXFP4 block/scale format: ``` model.layers.0.mlp.experts.gate_up_proj_blocks model.layers.0.mlp.experts.gate_up_proj_scales model.layers.0.mlp.experts.gate_up_proj_bias model.layers.0.mlp.experts.down_proj_blocks model.layers.0.mlp.experts.down_proj_scales model.layers.0.mlp.experts.down_proj_bias ``` All 128 experts are packed into single tensors per layer (fused gate+up projection), with separate block quantization scales. There is no per-expert dimension in the key naming. ### Observed behavior ``` [BF16SafeTensorLoader] No MoE format detected, defaulting to: deepseek ``` The loader (`kt-kernel/python/utils/loader.py` line 463) fails to match gpt-oss's key pattern and falls back to DeepSeek format, which then fails to load the weights correctly. ## Three confirmed failure paths for gpt-oss on KTransformers For completeness, here are all three integration paths attempted and their failure modes: | Backend | Weight Source | Failure | Root Cause | |---------|-------------|---------|------------| | BF16 | HF safetensors | Format mismatch | Packed fused expert tensors, not per-expert keys | | LLAMAFILE | MXFP4 GGUF | `ValueError: 39 is not a valid GGMLQuantizationType` | gguf 0.17.1 lacks MXFP4 type 39 | | LLAMAFILE | Q4_K_M GGUF | **Testing in progress** | Standard quant type, should be compatible | ## Suggestion To support gpt-oss (and likely future models using packed/fused MoE formats): 1. Add a gpt-oss format detector in the BF16SafeTensorLoader alongside the existing DeepSeek detector 2. Implement unpacking logic for fused `gate_up_proj_blocks`/`scales` into individual expert weights 3. Alternatively, document that gpt-oss requires the LLAMAFILE backend with standard (non-MXFP4) GGUF quantizations ## Note on successful llama.cpp baseline For reference, gpt-oss-120b runs successfully on the same hardware via llama.cpp with `--override-tensor` expert offloading at **37.95 t/s** (15 of 36 layers' experts on GPU, remainder on CPU via DDR5). The model architecture is functional — it's specifically the KTransformers integration paths that need gpt-oss format support. ### Others _No response_

bsakaguchi closed this issue

2026-10-29 14:49:23 +00:00

bsakaguchi commented

2026-10-29 14:49:24 +00:00

Author

Owner

@vshortt73 commented on GitHub (Feb 22, 2026):

Update: Standard quant GGUFs (Q4_K_M) also blocked by type 39

Following up on the original report — I attempted the LLAMAFILE backend with bartowski's Q4_K_M quantization of gpt-oss-120B (bartowski/openai_gpt-oss-120b-GGUF), expecting standard quant types would bypass the MXFP4 issue.

Same failure:
File "/venv/kt-slang/lib/python3.11/site-packages/gguf/gguf_reader.py", line 325, in _build_tensors
ggml_type = GGMLQuantizationType(raw_dtype[0])
ValueError: np.uint32(39) is not a valid GGMLQuantizationType

The Q4_K_M GGUF contains mixed quantization types — the expert FFN weights are Q4_K_M, but the original model's non-expert tensors (attention, router, embeddings, lm_head) were trained in MXFP4 and appear to retain type 39 in the GGUF. The Python gguf reader (0.17.1, latest) fails on the first type 39 tensor it encounters, before it reaches any Q4_K_M data.

This means every gpt-oss GGUF variant is blocked — not just the native MXFP4. Q2_K through BF16 from bartowski will all contain at least some type 39 tensors and hit the same wall.

The fix needs to happen upstream in the gguf Python package (adding type 39 / MXFP4 to GGMLQuantizationType). Until then, gpt-oss is incompatible with the LLAMAFILE backend regardless of quantization choice.

For reference, llama.cpp's C implementation handles type 39 natively — the same MXFP4 GGUF runs fine via llama-server at 37.95 t/s with --override-tensor expert offloading. The gap is specifically in the Python gguf library.

@vshortt73 commented on GitHub (Feb 22, 2026): **Update: Standard quant GGUFs (Q4_K_M) also blocked by type 39** Following up on the original report — I attempted the LLAMAFILE backend with bartowski's Q4_K_M quantization of gpt-oss-120B (`bartowski/openai_gpt-oss-120b-GGUF`), expecting standard quant types would bypass the MXFP4 issue. Same failure: File "/venv/kt-slang/lib/python3.11/site-packages/gguf/gguf_reader.py", line 325, in _build_tensors ggml_type = GGMLQuantizationType(raw_dtype[0]) ValueError: np.uint32(39) is not a valid GGMLQuantizationType The Q4_K_M GGUF contains mixed quantization types — the expert FFN weights are Q4_K_M, but the original model's non-expert tensors (attention, router, embeddings, lm_head) were trained in MXFP4 and appear to retain type 39 in the GGUF. The Python `gguf` reader (0.17.1, latest) fails on the *first* type 39 tensor it encounters, before it reaches any Q4_K_M data. This means **every** gpt-oss GGUF variant is blocked — not just the native MXFP4. Q2_K through BF16 from bartowski will all contain at least some type 39 tensors and hit the same wall. The fix needs to happen upstream in the `gguf` Python package (adding type 39 / MXFP4 to `GGMLQuantizationType`). Until then, gpt-oss is incompatible with the LLAMAFILE backend regardless of quantization choice. For reference, llama.cpp's C implementation handles type 39 natively — the same MXFP4 GGUF runs fine via `llama-server` at 37.95 t/s with `--override-tensor` expert offloading. The gap is specifically in the Python gguf library.

bsakaguchi commented

2026-10-29 14:49:24 +00:00

Author

Owner

@ErvinXie commented on GitHub (Feb 28, 2026):

We currently do not support FP4 quantization formats, including NVFP4 and MXFP4. This means models using these formats (such as gpt-oss-120b with MXFP4 packed weights) are not compatible with KTransformers at this time.

We do have plans to add FP4 format support in future releases. Please stay tuned for updates.

This reply was generated by an AI agent and reviewed by a human.

@ErvinXie commented on GitHub (Feb 28, 2026): We currently do not support FP4 quantization formats, including NVFP4 and MXFP4. This means models using these formats (such as gpt-oss-120b with MXFP4 packed weights) are not compatible with KTransformers at this time. We do have plans to add FP4 format support in future releases. Please stay tuned for updates. --- *This reply was generated by an AI agent and reviewed by a human.*

bsakaguchi referenced this issue

2026-10-29 14:53:35 +00:00

[PR #1158] [MERGED] Update Function call #1572