kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1618] [GLM-4.6 + KT-Kernel] TypeError in unquant.create_weights: w13_weight_n is None when enabling kt_ep_wrapper #1052

New issue

Open

opened 2026-10-29 14:47:59 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:47:59 +00:00

Owner

Originally created by @EonHao on GitHub (Nov 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1618

Originally assigned to: @ouqingliang, @SkqLiao, @ovowei on GitHub.

I’m trying to run GLM-4.6 with KT-Kernel (CPU experts) via SGLang, using the official GLM-4.6 weights and the official convert_cpu_weights.py script. The model loads fine without KT-Kernel, but as soon as I enable --kt-method and --kt-weight-path, the scheduler fails during model initialization with:

TypeError: empty() received an invalid combination of arguments - got (NoneType, int, int, dtype=torch.dtype)

The stack trace points to unquant.py and kt_ep_wrapper.py, where w13_weight_n appears to be None.

I’d like to confirm whether GLM-4.6 is expected to work with the KT-Kernel MoE EP backend right now, and if so, what the correct configuration / conversion steps are.

Environment

Machine:
- CPU: Intel Xeon Platinum 8558P, 96 physical cores, AVX512, no AMX
- NUMA nodes: 1
GPUs:
- 8 × NVIDIA H20-3e (≈ 144GB VRAM each)
- Driver: 550.163.01
- CUDA: 12.4
OS: Linux x86_64 (root@10-0-0-14, Ubuntu-like environment)
Python: 3.11 (Conda env kt-kernel)

SGLang: installed from GitHub

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

KT-Kernel:

Built in the same environment, with:

export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF
# then build/install kt-kernel

PyTorch / NCCL:
- SGLang reports: sglang is using nccl==2.27.3 and cudaDriverVersion 12040

Model & Weights

Base model: GLM-4.6, from ZhipuAI’s official release on ModelScope.
GPU weights path:
- /home/ubuntu/GLM-4.6
CPU weights conversion (official script):

python scripts/convert_cpu_weights.py \
  --input-path /home/ubuntu/GLM-4.6 \
  --input-type bf16 \
  --output /root/GLM-4.6-cpu-weights \
  --quant-method int8

So the CPU weights directory is:
/root/GLM-4.6-cpu-weights

Launch Command (with KT-Kernel, where it fails)

export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF
export NCCL_DEBUG=INFO
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=lo

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m sglang.launch_server \
  --model-path /home/ubuntu/GLM-4.6 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp 4 \
  --mem-fraction-static 0.90 \
  --context-length 128000 \
  --attention-backend flashinfer \
  --tool-call-parser glm45 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-metrics \
  --served-model-name glm-4.6-inst1 \
  \
  --kt-method AMXINT8 \
  --kt-weight-path /root/GLM-4.6-cpu-weights \
  --kt-cpuinfer 48 \
  --kt-threadpool-count 1 \
  --kt-max-deferred-experts-per-token 2

What works

If I remove all --kt-* arguments, GLM-4.6 loads and serves correctly on the same machine with the same SGLang build, e.g.:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m sglang.launch_server \
  --model-path /home/ubuntu/GLM-4.6 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp 4 \
  --mem-fraction-static 0.90 \
  --context-length 128000 \
  --attention-backend flashinfer \
  --tool-call-parser glm45 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-metrics \
  --served-model-name glm-4.6-inst1

So the base GLM-4.6 + SGLang setup seems fine; the issue only appears once KT-Kernel MoE is enabled.

Error Log (truncated)

After torch distributed init and “Load weight begin”, I get the following exception from multiple TP ranks:

[2025-11-15 14:24:11 TP0] Load weight begin. avail mem=139.01 GB
...

[2025-11-15 14:24:12 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File ".../sglang/srt/managers/scheduler.py", line 2709, in run_scheduler_process
    scheduler = Scheduler(
  File ".../sglang/srt/managers/scheduler.py", line 312, in __init__
    self.tp_worker = TpModelWorker(
  File ".../sglang/srt/managers/tp_worker.py", line 237, in __init__
    self._model_runner = ModelRunner(
  File ".../sglang/srt/model_executor/model_runner.py", line 323, in __init__
    self.initialize(min_per_gpu_memory)
  File ".../sglang/srt/model_executor/model_runner.py", line 409, in initialize
    self.load_model()
  File ".../sglang/srt/model_executor/model_runner.py", line 766, in load_model
    self.model = get_model(
  File ".../sglang/srt/model_loader/__init__.py", line 28, in get_model
    return loader.load_model(
  File ".../sglang/srt/model_loader/loader.py", line 594, in load_model
    model = _initialize_model(
  File ".../sglang/srt/model_loader/loader.py", line 262, in _initialize_model
    return model_class(**kwargs)
  File ".../sglang/srt/models/glm4_moe.py", line 952, in __init__
    self.model = Glm4MoeModel(
  File ".../sglang/srt/models/glm4_moe.py", line 843, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
  File ".../sglang/srt/utils/common.py", line 577, in make_layers
    + get_offloader().wrap_modules(
  File ".../sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
  File ".../sglang/srt/utils/common.py", line 579, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File ".../sglang/srt/models/glm4_moe.py", line 845, in <lambda>
    lambda idx, prefix: Glm4MoeDecoderLayer(
  File ".../sglang/srt/models/glm4_moe.py", line 664, in __init__
    self.mlp = Glm4MoeSparseMoeBlock(
  File ".../sglang/srt/models/glm4_moe.py", line 385, in __init__
    self.experts = get_moe_impl_class(quant_config)(
  File ".../sglang/srt/layers/moe/fused_moe_triton/layer.py", line 230, in __init__
    self.quant_method.create_weights(
  File ".../sglang/srt/layers/moe/kt_ep_wrapper.py", line 207, in create_weights
    self.gpu_method.create_weights(
  File ".../sglang/srt/layers/quantization/unquant.py", line 160, in create_weights
    torch.empty(num_experts, w13_weight_n, w13_weight_k, dtype=params_dtype),
TypeError: empty() received an invalid combination of arguments - got (NoneType, int, int, dtype=torch.dtype), but expected one of:
  * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
  * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

The same stack trace appears on TP1/TP2/TP3.

Questions

Is GLM-4.6 currently supported with KT-Kernel MoE EP in SGLang (via kt_ep_wrapper)?
Is there a specific SGLang commit / branch and/or quantization config required for GLM-4.6 + KT-Kernel?
Does the convert_cpu_weights.py script need additional arguments for GLM-4.6, or are there known missing metadata fields (like w13_weight_n) for this model?
Any suggested workaround (e.g., disable certain MoE backend, change kt-method, or a temporary patch) to get GLM-4.6 running with CPU experts?

Happy to provide more logs (full sglang stdout, config.json, quant_config, etc.) if needed.

Originally created by @EonHao on GitHub (Nov 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1618 Originally assigned to: @ouqingliang, @SkqLiao, @ovowei on GitHub. I’m trying to run GLM-4.6 with KT-Kernel (CPU experts) via SGLang, using the official GLM-4.6 weights and the official `convert_cpu_weights.py` script. The model loads fine without KT-Kernel, but as soon as I enable `--kt-method` and `--kt-weight-path`, the scheduler fails during model initialization with: ```text TypeError: empty() received an invalid combination of arguments - got (NoneType, int, int, dtype=torch.dtype) ``` The stack trace points to `unquant.py` and `kt_ep_wrapper.py`, where `w13_weight_n` appears to be `None`. I’d like to confirm whether GLM-4.6 is expected to work with the KT-Kernel MoE EP backend right now, and if so, what the correct configuration / conversion steps are. --- **Environment** * Machine: * CPU: Intel Xeon Platinum 8558P, 96 physical cores, AVX512, **no AMX** * NUMA nodes: 1 * GPUs: * 8 × NVIDIA H20-3e (≈ 144GB VRAM each) * Driver: 550.163.01 * CUDA: 12.4 * OS: Linux x86\_64 (root\@10-0-0-14, Ubuntu-like environment) * Python: 3.11 (Conda env `kt-kernel`) * SGLang: installed from GitHub ```bash git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[all]" ``` * KT-Kernel: * Built in the same environment, with: ```bash export CPUINFER_CPU_INSTRUCT=AVX512 export CPUINFER_ENABLE_AMX=OFF # then build/install kt-kernel ``` * PyTorch / NCCL: * SGLang reports: `sglang is using nccl==2.27.3` and `cudaDriverVersion 12040` --- **Model & Weights** * Base model: **GLM-4.6**, from ZhipuAI’s official release on ModelScope. * GPU weights path: * `/home/ubuntu/GLM-4.6` * CPU weights conversion (official script): ```bash python scripts/convert_cpu_weights.py \ --input-path /home/ubuntu/GLM-4.6 \ --input-type bf16 \ --output /root/GLM-4.6-cpu-weights \ --quant-method int8 ``` So the CPU weights directory is: `/root/GLM-4.6-cpu-weights` --- **Launch Command (with KT-Kernel, where it fails)** ```bash export CPUINFER_CPU_INSTRUCT=AVX512 export CPUINFER_ENABLE_AMX=OFF export NCCL_DEBUG=INFO export NCCL_P2P_DISABLE=1 export NCCL_IB_DISABLE=1 export NCCL_SOCKET_IFNAME=lo CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m sglang.launch_server \ --model-path /home/ubuntu/GLM-4.6 \ --host 0.0.0.0 \ --port 8000 \ --tp 4 \ --mem-fraction-static 0.90 \ --context-length 128000 \ --attention-backend flashinfer \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-metrics \ --served-model-name glm-4.6-inst1 \ \ --kt-method AMXINT8 \ --kt-weight-path /root/GLM-4.6-cpu-weights \ --kt-cpuinfer 48 \ --kt-threadpool-count 1 \ --kt-max-deferred-experts-per-token 2 ``` --- **What works** If I remove all `--kt-*` arguments, GLM-4.6 loads and serves correctly on the same machine with the same SGLang build, e.g.: ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 \ python -m sglang.launch_server \ --model-path /home/ubuntu/GLM-4.6 \ --host 0.0.0.0 \ --port 8000 \ --tp 4 \ --mem-fraction-static 0.90 \ --context-length 128000 \ --attention-backend flashinfer \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-metrics \ --served-model-name glm-4.6-inst1 ``` So the base GLM-4.6 + SGLang setup seems fine; the issue only appears once KT-Kernel MoE is enabled. --- **Error Log (truncated)** After torch distributed init and “Load weight begin”, I get the following exception from multiple TP ranks: ```text [2025-11-15 14:24:11 TP0] Load weight begin. avail mem=139.01 GB ... [2025-11-15 14:24:12 TP0] Scheduler hit an exception: Traceback (most recent call last): File ".../sglang/srt/managers/scheduler.py", line 2709, in run_scheduler_process scheduler = Scheduler( File ".../sglang/srt/managers/scheduler.py", line 312, in __init__ self.tp_worker = TpModelWorker( File ".../sglang/srt/managers/tp_worker.py", line 237, in __init__ self._model_runner = ModelRunner( File ".../sglang/srt/model_executor/model_runner.py", line 323, in __init__ self.initialize(min_per_gpu_memory) File ".../sglang/srt/model_executor/model_runner.py", line 409, in initialize self.load_model() File ".../sglang/srt/model_executor/model_runner.py", line 766, in load_model self.model = get_model( File ".../sglang/srt/model_loader/__init__.py", line 28, in get_model return loader.load_model( File ".../sglang/srt/model_loader/loader.py", line 594, in load_model model = _initialize_model( File ".../sglang/srt/model_loader/loader.py", line 262, in _initialize_model return model_class(**kwargs) File ".../sglang/srt/models/glm4_moe.py", line 952, in __init__ self.model = Glm4MoeModel( File ".../sglang/srt/models/glm4_moe.py", line 843, in __init__ self.layers, self.start_layer, self.end_layer = make_layers( File ".../sglang/srt/utils/common.py", line 577, in make_layers + get_offloader().wrap_modules( File ".../sglang/srt/utils/offloader.py", line 36, in wrap_modules return list(all_modules_generator) File ".../sglang/srt/utils/common.py", line 579, in <genexpr> layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) File ".../sglang/srt/models/glm4_moe.py", line 845, in <lambda> lambda idx, prefix: Glm4MoeDecoderLayer( File ".../sglang/srt/models/glm4_moe.py", line 664, in __init__ self.mlp = Glm4MoeSparseMoeBlock( File ".../sglang/srt/models/glm4_moe.py", line 385, in __init__ self.experts = get_moe_impl_class(quant_config)( File ".../sglang/srt/layers/moe/fused_moe_triton/layer.py", line 230, in __init__ self.quant_method.create_weights( File ".../sglang/srt/layers/moe/kt_ep_wrapper.py", line 207, in create_weights self.gpu_method.create_weights( File ".../sglang/srt/layers/quantization/unquant.py", line 160, in create_weights torch.empty(num_experts, w13_weight_n, w13_weight_k, dtype=params_dtype), TypeError: empty() received an invalid combination of arguments - got (NoneType, int, int, dtype=torch.dtype), but expected one of: * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False) * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False) ``` The same stack trace appears on TP1/TP2/TP3. --- **Questions** 1. Is **GLM-4.6** currently supported with KT-Kernel MoE EP in SGLang (via `kt_ep_wrapper`)? 2. Is there a specific SGLang commit / branch and/or quantization config required for GLM-4.6 + KT-Kernel? 3. Does the `convert_cpu_weights.py` script need additional arguments for GLM-4.6, or are there known missing metadata fields (like `w13_weight_n`) for this model? 4. Any suggested workaround (e.g., disable certain MoE backend, change `kt-method`, or a temporary patch) to get GLM-4.6 running with CPU experts? Happy to provide more logs (full `sglang` stdout, `config.json`, `quant_config`, etc.) if needed.

bsakaguchi added the

bug

label

2026-10-29 14:47:59 +00:00

bsakaguchi commented

2026-10-29 14:48:00 +00:00

Author

Owner

@EonHao commented on GitHub (Nov 16, 2025):

After further debugging, the error does not come from GLM-4.6 weights or from KT-Kernel conversion.
The actual root cause is that SGLang receives kt_num_gpu_experts=None, because the launch command did not explicitly specify:

--kt-num-gpu-experts <value>

Inside KTEPWrapperMethod.create_weights, SGLang forwards this parameter into:

self.gpu_method.create_weights(..., num_experts=self.num_gpu_experts)

Since self.num_gpu_experts = None, the call becomes:

torch.empty(None, w13_weight_n, w13_weight_k)

which produces the exact error seen:

TypeError: empty() received an invalid combination of arguments -
got (NoneType, int, int, dtype=...)

So the None originates before unquant.py, not from a shape inference failure.

@EonHao commented on GitHub (Nov 16, 2025): After further debugging, the error does *not* come from GLM-4.6 weights or from KT-Kernel conversion. The actual root cause is that **SGLang receives `kt_num_gpu_experts=None`**, because the launch command did not explicitly specify: ``` --kt-num-gpu-experts <value> ``` Inside `KTEPWrapperMethod.create_weights`, SGLang forwards this parameter into: ```python self.gpu_method.create_weights(..., num_experts=self.num_gpu_experts) ``` Since `self.num_gpu_experts = None`, the call becomes: ```python torch.empty(None, w13_weight_n, w13_weight_k) ``` which produces the exact error seen: ``` TypeError: empty() received an invalid combination of arguments - got (NoneType, int, int, dtype=...) ``` So the `None` originates **before** `unquant.py`, not from a shape inference failure.

bsakaguchi commented

2026-10-29 14:48:00 +00:00

Author

Owner

@ovowei commented on GitHub (Nov 17, 2025):

try passing --kt-num-gpu-experts explicitly in your launch arguments.

@ovowei commented on GitHub (Nov 17, 2025): try passing --kt-num-gpu-experts explicitly in your launch arguments.