[GH-ISSUE #1860] # MXFP4 quantization allocates GPU weights for all experts regardless of KTransformers `--kt-num-gpu-experts` #1155

New issue

Closed

opened 2026-10-29 14:49:21 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:49:21 +00:00

Owner

Originally created by @vshortt73 on GitHub (Feb 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1860

Reminder

I have read the above rules and searched the existing issues.

System Info

SGLang: main @ 45095bac7
kt-kernel: main @ 16a8b98
Model: openai/gpt-oss-120b (128 experts, MXFP4)
GPU: RTX 5090 (32GB VRAM)
CPU: AMD Ryzen 9 9900X

Reproduction

Repository: sgl-project/sglang

Summary

When using KTransformers expert offloading with an MXFP4-quantized model, the MXFP4 weight creation in mxfp4.py allocates GPU memory for all experts (e.g., 128) instead of respecting the num_experts parameter passed from kt_ep_wrapper.py. Setting --kt-num-gpu-experts 0 still causes full GPU allocation and OOM.

Error

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.05 GiB.
GPU 0 has a total capacity of 31.33 GiB of which 766.50 MiB is free.
Including non-PyTorch memory, this process has 29.60 GiB memory in use.

This occurs even with --kt-num-gpu-experts 0. Non-expert parameters (attention, embeddings, lm_head) total only ~3.7 GB, but PyTorch reports 29.6 GB allocated.

Root Cause

In sglang/srt/layers/quantization/mxfp4.py around line 360, the create_weights method uses layer.num_local_experts to size the weight tensors:

w13_weight = torch.nn.Parameter(
    torch.zeros(
        layer.num_local_experts,    # <-- Always 128, ignores num_experts param
        2 * intermediate_size_per_partition_after_pad,
        hidden_size // 2,
        dtype=weight_dtype,
    ),
    requires_grad=False,
)

The call from kt_ep_wrapper.py (line 207) passes num_experts=self.num_gpu_experts (which could be 0), but mxfp4.py ignores this and reads from the layer attribute instead.

Expected Behavior

When num_experts=0 is passed to create_weights(), no GPU expert weight buffers should be allocated. The num_experts parameter should control the tensor dimensions, not layer.num_local_experts.

Impact

This prevents KTransformers expert offloading from working with any MXFP4-quantized MoE model, regardless of how many GPU experts are requested. It's not specific to gpt-oss — it would affect any MXFP4 MoE model using the KT integration path.

Steps to Reproduce

python -m sglang.launch_server \
  --model /path/to/any-mxfp4-moe-model \
  --trust-remote-code \
  --mem-fraction-static 0.50 \
  --kt-method BF16 \
  --kt-weight-path /path/to/weights \
  --kt-cpuinfer 12 \
  --kt-num-gpu-experts 0

Others

No response

Originally created by @vshortt73 on GitHub (Feb 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1860 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info - **SGLang**: `main` @ 45095bac7 - **kt-kernel**: `main` @ 16a8b98 - **Model**: openai/gpt-oss-120b (128 experts, MXFP4) - **GPU**: RTX 5090 (32GB VRAM) - **CPU**: AMD Ryzen 9 9900X ### Reproduction ## Repository: sgl-project/sglang ## Summary When using KTransformers expert offloading with an MXFP4-quantized model, the MXFP4 weight creation in `mxfp4.py` allocates GPU memory for **all** experts (e.g., 128) instead of respecting the `num_experts` parameter passed from `kt_ep_wrapper.py`. Setting `--kt-num-gpu-experts 0` still causes full GPU allocation and OOM. ## Error ``` torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.05 GiB. GPU 0 has a total capacity of 31.33 GiB of which 766.50 MiB is free. Including non-PyTorch memory, this process has 29.60 GiB memory in use. ``` This occurs even with `--kt-num-gpu-experts 0`. Non-expert parameters (attention, embeddings, lm_head) total only ~3.7 GB, but PyTorch reports 29.6 GB allocated. ## Root Cause In `sglang/srt/layers/quantization/mxfp4.py` around line 360, the `create_weights` method uses `layer.num_local_experts` to size the weight tensors: ```python w13_weight = torch.nn.Parameter( torch.zeros( layer.num_local_experts, # <-- Always 128, ignores num_experts param 2 * intermediate_size_per_partition_after_pad, hidden_size // 2, dtype=weight_dtype, ), requires_grad=False, ) ``` The call from `kt_ep_wrapper.py` (line 207) passes `num_experts=self.num_gpu_experts` (which could be 0), but `mxfp4.py` ignores this and reads from the layer attribute instead. ## Expected Behavior When `num_experts=0` is passed to `create_weights()`, no GPU expert weight buffers should be allocated. The `num_experts` parameter should control the tensor dimensions, not `layer.num_local_experts`. ## Impact This prevents KTransformers expert offloading from working with any MXFP4-quantized MoE model, regardless of how many GPU experts are requested. It's not specific to gpt-oss — it would affect any MXFP4 MoE model using the KT integration path. ## Steps to Reproduce ```bash python -m sglang.launch_server \ --model /path/to/any-mxfp4-moe-model \ --trust-remote-code \ --mem-fraction-static 0.50 \ --kt-method BF16 \ --kt-weight-path /path/to/weights \ --kt-cpuinfer 12 \ --kt-num-gpu-experts 0 ``` ### Others _No response_

bsakaguchi closed this issue

2026-10-29 14:49:22 +00:00

bsakaguchi commented

2026-10-29 14:49:22 +00:00

Author

Owner

@ErvinXie commented on GitHub (Feb 28, 2026):

As mentioned in #1861, MXFP4 format is not currently supported in KTransformers. This allocation issue is a consequence of that — the MXFP4 code path was not designed to work with KTransformers's expert offloading.

FP4 support is on our roadmap. This issue will be addressed as part of that effort.

This reply was generated by an AI agent and reviewed by a human.

@ErvinXie commented on GitHub (Feb 28, 2026): As mentioned in #1861, MXFP4 format is not currently supported in KTransformers. This allocation issue is a consequence of that — the MXFP4 code path was not designed to work with KTransformers's expert offloading. FP4 support is on our roadmap. This issue will be addressed as part of that effort. --- *This reply was generated by an AI agent and reviewed by a human.*

bsakaguchi referenced this issue

2026-10-29 14:53:35 +00:00

[PR #1158] [MERGED] Update Function call #1572