kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1777] M_STEP / K_STEP / N_STEP / B_K_STEP not aligned while converting cpu-weights #1118

New issue
Closed
opened 2026-10-29 14:48:43 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-29 14:48:43 +00:00
Owner
Copy link

Originally created by @mohkgh on GitHub (Jan 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1777

Reminder

  • I have read the above rules and searched the existing issues.

System Info

cpu Xeon Max 9468 es x2
gpu NVIDIA GeForce RTX 3080

Reproduction

kt-kernel is installed by pip, scripts/convert_cpu_weights.py is from github master branch

python scripts/convert_cpu_weights.py \
  --threadpool-count 8 \
  --cpuinfer-threads 96 \
  --input-path Qwen3-Coder-30B-A3B-Instruct \
#--input-path Qwen3-Coder-480B-A35B-Instruct-1M \
  --input-type bf16 \
  --output Qwen3-Coder-30B-A3B-Instruct-INT4 \
  --quant-method int4 \

converting Qwen3-Coder-480B-A35B-Instruct-1M

Loading model configuration...
Model config: {'num_experts': 160, 'num_experts_per_tok': 8, 'hidden_size': 6144, 'moe_intermediate_size': 2560}
  num_experts: 160
  num_experts_per_tok: 8
  hidden_size: 6144
  moe_intermediate_size: 2560
CPU inference config:
  cpuinfer_threads: 96
  threadpool_count: 8

Loading safetensors files from /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M
  Loaded: model-00001-of-00241.safetensors (64 tensors)
  Loaded: model-00002-of-00241.safetensors (127 tensors)
  Loaded: model-00003-of-00241.safetensors (127 tensors)
  Loaded: model-00004-of-00241.safetensors (127 tensors)
  Loaded: model-00005-of-00241.safetensors (125 tensors)
  Loaded: model-00006-of-00241.safetensors (127 tensors)
  Loaded: model-00007-of-00241.safetensors (127 tensors)
  Loaded: model-00008-of-00241.safetensors (127 tensors)
  Loaded: model-00009-of-00241.safetensors (125 tensors)
  Loaded: model-00010-of-00241.safetensors (127 tensors)
  Loaded: model-00011-of-00241.safetensors (127 tensors)
  Loaded: model-00012-of-00241.safetensors (127 tensors)
  Loaded: model-00013-of-00241.safetensors (125 tensors)
  Loaded: model-00014-of-00241.safetensors (127 tensors)
  Loaded: model-00015-of-00241.safetensors (127 tensors)
  Loaded: model-00016-of-00241.safetensors (124 tensors)
  Loaded: model-00017-of-00241.safetensors (125 tensors)
  Loaded: model-00018-of-00241.safetensors (127 tensors)
  Loaded: model-00019-of-00241.safetensors (127 tensors)
  Loaded: model-00020-of-00241.safetensors (125 tensors)
  Loaded: model-00021-of-00241.safetensors (127 tensors)
  Loaded: model-00022-of-00241.safetensors (127 tensors)
  Loaded: model-00023-of-00241.safetensors (127 tensors)
  Loaded: model-00024-of-00241.safetensors (125 tensors)
  Loaded: model-00025-of-00241.safetensors (127 tensors)
  Loaded: model-00026-of-00241.safetensors (127 tensors)
  Loaded: model-00027-of-00241.safetensors (127 tensors)
  Loaded: model-00028-of-00241.safetensors (125 tensors)
  Loaded: model-00029-of-00241.safetensors (127 tensors)
  Loaded: model-00030-of-00241.safetensors (127 tensors)
  Loaded: model-00031-of-00241.safetensors (127 tensors)
  Loaded: model-00032-of-00241.safetensors (125 tensors)
  Loaded: model-00033-of-00241.safetensors (127 tensors)
  Loaded: model-00034-of-00241.safetensors (127 tensors)
  Loaded: model-00035-of-00241.safetensors (127 tensors)
  Loaded: model-00036-of-00241.safetensors (125 tensors)
  Loaded: model-00037-of-00241.safetensors (127 tensors)
  Loaded: model-00038-of-00241.safetensors (127 tensors)
  Loaded: model-00039-of-00241.safetensors (127 tensors)
  Loaded: model-00040-of-00241.safetensors (125 tensors)
  Loaded: model-00041-of-00241.safetensors (127 tensors)
  Loaded: model-00042-of-00241.safetensors (127 tensors)
  Loaded: model-00043-of-00241.safetensors (127 tensors)
  Loaded: model-00044-of-00241.safetensors (125 tensors)
  Loaded: model-00045-of-00241.safetensors (127 tensors)
  Loaded: model-00046-of-00241.safetensors (127 tensors)
  Loaded: model-00047-of-00241.safetensors (125 tensors)
  Loaded: model-00048-of-00241.safetensors (127 tensors)
  Loaded: model-00049-of-00241.safetensors (127 tensors)
  Loaded: model-00050-of-00241.safetensors (127 tensors)
  Loaded: model-00051-of-00241.safetensors (125 tensors)
  Loaded: model-00052-of-00241.safetensors (127 tensors)
  Loaded: model-00053-of-00241.safetensors (127 tensors)
  Loaded: model-00054-of-00241.safetensors (127 tensors)
  Loaded: model-00055-of-00241.safetensors (125 tensors)
  Loaded: model-00056-of-00241.safetensors (127 tensors)
  Loaded: model-00057-of-00241.safetensors (127 tensors)
  Loaded: model-00058-of-00241.safetensors (127 tensors)
  Loaded: model-00059-of-00241.safetensors (125 tensors)
  Loaded: model-00060-of-00241.safetensors (127 tensors)
  Loaded: model-00061-of-00241.safetensors (127 tensors)
  Loaded: model-00062-of-00241.safetensors (127 tensors)
  Loaded: model-00063-of-00241.safetensors (125 tensors)
  Loaded: model-00064-of-00241.safetensors (127 tensors)
  Loaded: model-00065-of-00241.safetensors (127 tensors)
  Loaded: model-00066-of-00241.safetensors (127 tensors)
  Loaded: model-00067-of-00241.safetensors (125 tensors)
  Loaded: model-00068-of-00241.safetensors (127 tensors)
  Loaded: model-00069-of-00241.safetensors (127 tensors)
  Loaded: model-00070-of-00241.safetensors (127 tensors)
  Loaded: model-00071-of-00241.safetensors (125 tensors)
  Loaded: model-00072-of-00241.safetensors (127 tensors)
  Loaded: model-00073-of-00241.safetensors (127 tensors)
  Loaded: model-00074-of-00241.safetensors (126 tensors)
  Loaded: model-00075-of-00241.safetensors (125 tensors)
  Loaded: model-00076-of-00241.safetensors (127 tensors)
  Loaded: model-00077-of-00241.safetensors (127 tensors)
  Loaded: model-00078-of-00241.safetensors (125 tensors)
  Loaded: model-00079-of-00241.safetensors (127 tensors)
  Loaded: model-00080-of-00241.safetensors (127 tensors)
  Loaded: model-00081-of-00241.safetensors (127 tensors)
  Loaded: model-00082-of-00241.safetensors (125 tensors)
  Loaded: model-00083-of-00241.safetensors (127 tensors)
  Loaded: model-00084-of-00241.safetensors (127 tensors)
  Loaded: model-00085-of-00241.safetensors (127 tensors)
  Loaded: model-00086-of-00241.safetensors (125 tensors)
  Loaded: model-00087-of-00241.safetensors (127 tensors)
  Loaded: model-00088-of-00241.safetensors (127 tensors)
  Loaded: model-00089-of-00241.safetensors (127 tensors)
  Loaded: model-00090-of-00241.safetensors (125 tensors)
  Loaded: model-00091-of-00241.safetensors (127 tensors)
  Loaded: model-00092-of-00241.safetensors (127 tensors)
  Loaded: model-00093-of-00241.safetensors (127 tensors)
  Loaded: model-00094-of-00241.safetensors (125 tensors)
  Loaded: model-00095-of-00241.safetensors (127 tensors)
  Loaded: model-00096-of-00241.safetensors (127 tensors)
  Loaded: model-00097-of-00241.safetensors (127 tensors)
  Loaded: model-00098-of-00241.safetensors (125 tensors)
  Loaded: model-00099-of-00241.safetensors (127 tensors)
  Loaded: model-00100-of-00241.safetensors (127 tensors)
  Loaded: model-00101-of-00241.safetensors (127 tensors)
  Loaded: model-00102-of-00241.safetensors (125 tensors)
  Loaded: model-00103-of-00241.safetensors (127 tensors)
  Loaded: model-00104-of-00241.safetensors (127 tensors)
  Loaded: model-00105-of-00241.safetensors (125 tensors)
  Loaded: model-00106-of-00241.safetensors (127 tensors)
  Loaded: model-00107-of-00241.safetensors (127 tensors)
  Loaded: model-00108-of-00241.safetensors (127 tensors)
  Loaded: model-00109-of-00241.safetensors (125 tensors)
  Loaded: model-00110-of-00241.safetensors (127 tensors)
  Loaded: model-00111-of-00241.safetensors (127 tensors)
  Loaded: model-00112-of-00241.safetensors (127 tensors)
  Loaded: model-00113-of-00241.safetensors (125 tensors)
  Loaded: model-00114-of-00241.safetensors (127 tensors)
  Loaded: model-00115-of-00241.safetensors (127 tensors)
  Loaded: model-00116-of-00241.safetensors (127 tensors)
  Loaded: model-00117-of-00241.safetensors (125 tensors)
  Loaded: model-00118-of-00241.safetensors (127 tensors)
  Loaded: model-00119-of-00241.safetensors (127 tensors)
  Loaded: model-00120-of-00241.safetensors (127 tensors)
  Loaded: model-00121-of-00241.safetensors (125 tensors)
  Loaded: model-00122-of-00241.safetensors (127 tensors)
  Loaded: model-00123-of-00241.safetensors (127 tensors)
  Loaded: model-00124-of-00241.safetensors (127 tensors)
  Loaded: model-00125-of-00241.safetensors (125 tensors)
  Loaded: model-00126-of-00241.safetensors (127 tensors)
  Loaded: model-00127-of-00241.safetensors (127 tensors)
  Loaded: model-00128-of-00241.safetensors (127 tensors)
  Loaded: model-00129-of-00241.safetensors (125 tensors)
  Loaded: model-00130-of-00241.safetensors (127 tensors)
  Loaded: model-00131-of-00241.safetensors (127 tensors)
  Loaded: model-00132-of-00241.safetensors (126 tensors)
  Loaded: model-00133-of-00241.safetensors (125 tensors)
  Loaded: model-00134-of-00241.safetensors (127 tensors)
  Loaded: model-00135-of-00241.safetensors (127 tensors)
  Loaded: model-00136-of-00241.safetensors (125 tensors)
  Loaded: model-00137-of-00241.safetensors (127 tensors)
  Loaded: model-00138-of-00241.safetensors (127 tensors)
  Loaded: model-00139-of-00241.safetensors (127 tensors)
  Loaded: model-00140-of-00241.safetensors (125 tensors)
  Loaded: model-00141-of-00241.safetensors (127 tensors)
  Loaded: model-00142-of-00241.safetensors (127 tensors)
  Loaded: model-00143-of-00241.safetensors (127 tensors)
  Loaded: model-00144-of-00241.safetensors (125 tensors)
  Loaded: model-00145-of-00241.safetensors (127 tensors)
  Loaded: model-00146-of-00241.safetensors (127 tensors)
  Loaded: model-00147-of-00241.safetensors (127 tensors)
  Loaded: model-00148-of-00241.safetensors (125 tensors)
  Loaded: model-00149-of-00241.safetensors (127 tensors)
  Loaded: model-00150-of-00241.safetensors (127 tensors)
  Loaded: model-00151-of-00241.safetensors (127 tensors)
  Loaded: model-00152-of-00241.safetensors (125 tensors)
  Loaded: model-00153-of-00241.safetensors (127 tensors)
  Loaded: model-00154-of-00241.safetensors (127 tensors)
  Loaded: model-00155-of-00241.safetensors (127 tensors)
  Loaded: model-00156-of-00241.safetensors (125 tensors)
  Loaded: model-00157-of-00241.safetensors (127 tensors)
  Loaded: model-00158-of-00241.safetensors (127 tensors)
  Loaded: model-00159-of-00241.safetensors (127 tensors)
  Loaded: model-00160-of-00241.safetensors (125 tensors)
  Loaded: model-00161-of-00241.safetensors (127 tensors)
  Loaded: model-00162-of-00241.safetensors (127 tensors)
  Loaded: model-00163-of-00241.safetensors (125 tensors)
  Loaded: model-00164-of-00241.safetensors (127 tensors)
  Loaded: model-00165-of-00241.safetensors (127 tensors)
  Loaded: model-00166-of-00241.safetensors (127 tensors)
  Loaded: model-00167-of-00241.safetensors (125 tensors)
  Loaded: model-00168-of-00241.safetensors (127 tensors)
  Loaded: model-00169-of-00241.safetensors (127 tensors)
  Loaded: model-00170-of-00241.safetensors (127 tensors)
  Loaded: model-00171-of-00241.safetensors (125 tensors)
  Loaded: model-00172-of-00241.safetensors (127 tensors)
  Loaded: model-00173-of-00241.safetensors (127 tensors)
  Loaded: model-00174-of-00241.safetensors (127 tensors)
  Loaded: model-00175-of-00241.safetensors (125 tensors)
  Loaded: model-00176-of-00241.safetensors (127 tensors)
  Loaded: model-00177-of-00241.safetensors (127 tensors)
  Loaded: model-00178-of-00241.safetensors (127 tensors)
  Loaded: model-00179-of-00241.safetensors (125 tensors)
  Loaded: model-00180-of-00241.safetensors (127 tensors)
  Loaded: model-00181-of-00241.safetensors (127 tensors)
  Loaded: model-00182-of-00241.safetensors (127 tensors)
  Loaded: model-00183-of-00241.safetensors (125 tensors)
  Loaded: model-00184-of-00241.safetensors (127 tensors)
  Loaded: model-00185-of-00241.safetensors (127 tensors)
  Loaded: model-00186-of-00241.safetensors (127 tensors)
  Loaded: model-00187-of-00241.safetensors (125 tensors)
  Loaded: model-00188-of-00241.safetensors (127 tensors)
  Loaded: model-00189-of-00241.safetensors (127 tensors)
  Loaded: model-00190-of-00241.safetensors (126 tensors)
  Loaded: model-00191-of-00241.safetensors (125 tensors)
  Loaded: model-00192-of-00241.safetensors (127 tensors)
  Loaded: model-00193-of-00241.safetensors (127 tensors)
  Loaded: model-00194-of-00241.safetensors (125 tensors)
  Loaded: model-00195-of-00241.safetensors (127 tensors)
  Loaded: model-00196-of-00241.safetensors (127 tensors)
  Loaded: model-00197-of-00241.safetensors (127 tensors)
  Loaded: model-00198-of-00241.safetensors (125 tensors)
  Loaded: model-00199-of-00241.safetensors (127 tensors)
  Loaded: model-00200-of-00241.safetensors (127 tensors)
  Loaded: model-00201-of-00241.safetensors (127 tensors)
  Loaded: model-00202-of-00241.safetensors (125 tensors)
  Loaded: model-00203-of-00241.safetensors (127 tensors)
  Loaded: model-00204-of-00241.safetensors (127 tensors)
  Loaded: model-00205-of-00241.safetensors (127 tensors)
  Loaded: model-00206-of-00241.safetensors (125 tensors)
  Loaded: model-00207-of-00241.safetensors (127 tensors)
  Loaded: model-00208-of-00241.safetensors (127 tensors)
  Loaded: model-00209-of-00241.safetensors (127 tensors)
  Loaded: model-00210-of-00241.safetensors (125 tensors)
  Loaded: model-00211-of-00241.safetensors (127 tensors)
  Loaded: model-00212-of-00241.safetensors (127 tensors)
  Loaded: model-00213-of-00241.safetensors (127 tensors)
  Loaded: model-00214-of-00241.safetensors (125 tensors)
  Loaded: model-00215-of-00241.safetensors (127 tensors)
  Loaded: model-00216-of-00241.safetensors (127 tensors)
  Loaded: model-00217-of-00241.safetensors (127 tensors)
  Loaded: model-00218-of-00241.safetensors (125 tensors)
  Loaded: model-00219-of-00241.safetensors (127 tensors)
  Loaded: model-00220-of-00241.safetensors (127 tensors)
  Loaded: model-00221-of-00241.safetensors (125 tensors)
  Loaded: model-00222-of-00241.safetensors (127 tensors)
  Loaded: model-00223-of-00241.safetensors (127 tensors)
  Loaded: model-00224-of-00241.safetensors (127 tensors)
  Loaded: model-00225-of-00241.safetensors (125 tensors)
  Loaded: model-00226-of-00241.safetensors (127 tensors)
  Loaded: model-00227-of-00241.safetensors (127 tensors)
  Loaded: model-00228-of-00241.safetensors (127 tensors)
  Loaded: model-00229-of-00241.safetensors (125 tensors)
  Loaded: model-00230-of-00241.safetensors (127 tensors)
  Loaded: model-00231-of-00241.safetensors (127 tensors)
  Loaded: model-00232-of-00241.safetensors (127 tensors)
  Loaded: model-00233-of-00241.safetensors (125 tensors)
  Loaded: model-00234-of-00241.safetensors (127 tensors)
  Loaded: model-00235-of-00241.safetensors (127 tensors)
  Loaded: model-00236-of-00241.safetensors (127 tensors)
  Loaded: model-00237-of-00241.safetensors (125 tensors)
  Loaded: model-00238-of-00241.safetensors (127 tensors)
  Loaded: model-00239-of-00241.safetensors (127 tensors)
  Loaded: model-00240-of-00241.safetensors (127 tensors)
  Loaded: model-00241-of-00241.safetensors (32 tensors)
Total tensors loaded: 30321
Starting conversion...
Input: /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M
Output: /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-INT4
Found 62 layers with MoE experts:
  Layer 0: 160 experts (0-159)
  Layer 1: 160 experts (0-159)
  Layer 2: 160 experts (0-159)
  Layer 3: 160 experts (0-159)
  Layer 4: 160 experts (0-159)
  Layer 5: 160 experts (0-159)
  Layer 6: 160 experts (0-159)
  Layer 7: 160 experts (0-159)
  Layer 8: 160 experts (0-159)
  Layer 9: 160 experts (0-159)
  Layer 10: 160 experts (0-159)
  Layer 11: 160 experts (0-159)
  Layer 12: 160 experts (0-159)
  Layer 13: 160 experts (0-159)
  Layer 14: 160 experts (0-159)
  Layer 15: 160 experts (0-159)
  Layer 16: 160 experts (0-159)
  Layer 17: 160 experts (0-159)
  Layer 18: 160 experts (0-159)
  Layer 19: 160 experts (0-159)
  Layer 20: 160 experts (0-159)
  Layer 21: 160 experts (0-159)
  Layer 22: 160 experts (0-159)
  Layer 23: 160 experts (0-159)
  Layer 24: 160 experts (0-159)
  Layer 25: 160 experts (0-159)
  Layer 26: 160 experts (0-159)
  Layer 27: 160 experts (0-159)
  Layer 28: 160 experts (0-159)
  Layer 29: 160 experts (0-159)
  Layer 30: 160 experts (0-159)
  Layer 31: 160 experts (0-159)
  Layer 32: 160 experts (0-159)
  Layer 33: 160 experts (0-159)
  Layer 34: 160 experts (0-159)
  Layer 35: 160 experts (0-159)
  Layer 36: 160 experts (0-159)
  Layer 37: 160 experts (0-159)
  Layer 38: 160 experts (0-159)
  Layer 39: 160 experts (0-159)
  Layer 40: 160 experts (0-159)
  Layer 41: 160 experts (0-159)
  Layer 42: 160 experts (0-159)
  Layer 43: 160 experts (0-159)
  Layer 44: 160 experts (0-159)
  Layer 45: 160 experts (0-159)
  Layer 46: 160 experts (0-159)
  Layer 47: 160 experts (0-159)
  Layer 48: 160 experts (0-159)
  Layer 49: 160 experts (0-159)
  Layer 50: 160 experts (0-159)
  Layer 51: 160 experts (0-159)
  Layer 52: 160 experts (0-159)
  Layer 53: 160 experts (0-159)
  Layer 54: 160 experts (0-159)
  Layer 55: 160 experts (0-159)
  Layer 56: 160 experts (0-159)
  Layer 57: 160 experts (0-159)
  Layer 58: 160 experts (0-159)
  Layer 59: 160 experts (0-159)
  Layer 60: 160 experts (0-159)
  Layer 61: 160 experts (0-159)
Processing layer 0 (1/62)...
Converting layer 0 with 160 experts via online quantization...
  Loaded weights shapes:
    gate_proj: torch.Size([160, 2560, 6144])
    up_proj: torch.Size([160, 2560, 6144])
    down_proj: torch.Size([160, 6144, 2560])
CPUInfer[0x3f490510]: Hello
WorkerPool[0x3f5826b0] 8 subpools, [numa:threads][0:12] [1:12] [2:12] [3:12] [4:12] [5:12] [6:12] [7:12] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 4, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 5, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 6, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 7, 12 threads
TP MOE layer 0, pool: 0x3f5826b0, expert num: 160, num_experts_per_tok: 8
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
  what():  n or k is not aligned to N_STEP or B_K_STEP
terminate called recursively
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
terminate called recursively
n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128
terminate called recursively
terminate called recursively

converting Qwen3-Coder-30B-A3B-Instruct

Loading model configuration...
Model config: {'num_experts': 128, 'num_experts_per_tok': 8, 'hidden_size': 2048, 'moe_intermediate_size': 768}
  num_experts: 128
  num_experts_per_tok: 8
  hidden_size: 2048
  moe_intermediate_size: 768
CPU inference config:
  cpuinfer_threads: 96
  threadpool_count: 8

Loading safetensors files from Qwen3-Coder-30B-A3B-Instruct
  Loaded: model-00001-of-00016.safetensors (1062 tensors)
  Loaded: model-00002-of-00016.safetensors (1262 tensors)
  Loaded: model-00003-of-00016.safetensors (1258 tensors)
  Loaded: model-00004-of-00016.safetensors (1262 tensors)
  Loaded: model-00005-of-00016.safetensors (1262 tensors)
  Loaded: model-00006-of-00016.safetensors (1262 tensors)
  Loaded: model-00007-of-00016.safetensors (1262 tensors)
  Loaded: model-00008-of-00016.safetensors (1258 tensors)
  Loaded: model-00009-of-00016.safetensors (1262 tensors)
  Loaded: model-00010-of-00016.safetensors (1262 tensors)
  Loaded: model-00011-of-00016.safetensors (1262 tensors)
  Loaded: model-00012-of-00016.safetensors (1261 tensors)
  Loaded: model-00013-of-00016.safetensors (1256 tensors)
  Loaded: model-00014-of-00016.safetensors (1262 tensors)
  Loaded: model-00015-of-00016.safetensors (1262 tensors)
  Loaded: model-00016-of-00016.safetensors (152 tensors)
Total tensors loaded: 18867
Starting conversion...
Input: Qwen3-Coder-30B-A3B-Instruct
Output: Qwen3-Coder-30B-A3B-Instruct-INT4
Found 48 layers with MoE experts:
  Layer 0: 128 experts (0-127)
  Layer 1: 128 experts (0-127)
  Layer 2: 128 experts (0-127)
  Layer 3: 128 experts (0-127)
  Layer 4: 128 experts (0-127)
  Layer 5: 128 experts (0-127)
  Layer 6: 128 experts (0-127)
  Layer 7: 128 experts (0-127)
  Layer 8: 128 experts (0-127)
  Layer 9: 128 experts (0-127)
  Layer 10: 128 experts (0-127)
  Layer 11: 128 experts (0-127)
  Layer 12: 128 experts (0-127)
  Layer 13: 128 experts (0-127)
  Layer 14: 128 experts (0-127)
  Layer 15: 128 experts (0-127)
  Layer 16: 128 experts (0-127)
  Layer 17: 128 experts (0-127)
  Layer 18: 128 experts (0-127)
  Layer 19: 128 experts (0-127)
  Layer 20: 128 experts (0-127)
  Layer 21: 128 experts (0-127)
  Layer 22: 128 experts (0-127)
  Layer 23: 128 experts (0-127)
  Layer 24: 128 experts (0-127)
  Layer 25: 128 experts (0-127)
  Layer 26: 128 experts (0-127)
  Layer 27: 128 experts (0-127)
  Layer 28: 128 experts (0-127)
  Layer 29: 128 experts (0-127)
  Layer 30: 128 experts (0-127)
  Layer 31: 128 experts (0-127)
  Layer 32: 128 experts (0-127)
  Layer 33: 128 experts (0-127)
  Layer 34: 128 experts (0-127)
  Layer 35: 128 experts (0-127)
  Layer 36: 128 experts (0-127)
  Layer 37: 128 experts (0-127)
  Layer 38: 128 experts (0-127)
  Layer 39: 128 experts (0-127)
  Layer 40: 128 experts (0-127)
  Layer 41: 128 experts (0-127)
  Layer 42: 128 experts (0-127)
  Layer 43: 128 experts (0-127)
  Layer 44: 128 experts (0-127)
  Layer 45: 128 experts (0-127)
  Layer 46: 128 experts (0-127)
  Layer 47: 128 experts (0-127)
Processing layer 0 (1/48)...
Converting layer 0 with 128 experts via online quantization...
  Loaded weights shapes:
    gate_proj: torch.Size([128, 768, 2048])
    up_proj: torch.Size([128, 768, 2048])
    down_proj: torch.Size([128, 2048, 768])
CPUInfer[0x1d2f6bf0]: Hello
WorkerPool[0x1d46f740] 8 subpools, [numa:threads][0:12] [1:12] [2:12] [3:12] [4:12] [5:12] [6:12] [7:12] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 4, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 5, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 6, 12 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 7, 12 threads
TP MOE layer 0, pool: 0x1d46f740, expert num: 128, num_experts_per_tok: 8
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
terminate called recursively
terminate called after throwing an instance of 'max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
std::runtime_error'
terminate called recursively
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
  what():  BufferAImpl: max_m and k must be multiple of M_STEP and K_STEP
  what():  BufferAImpl: max_m and k must be multiple of M_STEP and K_STEP
max_m = 512, k = 96, M_STEP = 32, K_STEP = 64
terminate called recursively

Others

No response

Originally created by @mohkgh on GitHub (Jan 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1777 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info cpu Xeon Max 9468 es x2 gpu NVIDIA GeForce RTX 3080 ### Reproduction kt-kernel is installed by pip, scripts/convert_cpu_weights.py is from github master branch ```text python scripts/convert_cpu_weights.py \ --threadpool-count 8 \ --cpuinfer-threads 96 \ --input-path Qwen3-Coder-30B-A3B-Instruct \ #--input-path Qwen3-Coder-480B-A35B-Instruct-1M \ --input-type bf16 \ --output Qwen3-Coder-30B-A3B-Instruct-INT4 \ --quant-method int4 \ ``` converting Qwen3-Coder-480B-A35B-Instruct-1M ```text Loading model configuration... Model config: {'num_experts': 160, 'num_experts_per_tok': 8, 'hidden_size': 6144, 'moe_intermediate_size': 2560} num_experts: 160 num_experts_per_tok: 8 hidden_size: 6144 moe_intermediate_size: 2560 CPU inference config: cpuinfer_threads: 96 threadpool_count: 8 Loading safetensors files from /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M Loaded: model-00001-of-00241.safetensors (64 tensors) Loaded: model-00002-of-00241.safetensors (127 tensors) Loaded: model-00003-of-00241.safetensors (127 tensors) Loaded: model-00004-of-00241.safetensors (127 tensors) Loaded: model-00005-of-00241.safetensors (125 tensors) Loaded: model-00006-of-00241.safetensors (127 tensors) Loaded: model-00007-of-00241.safetensors (127 tensors) Loaded: model-00008-of-00241.safetensors (127 tensors) Loaded: model-00009-of-00241.safetensors (125 tensors) Loaded: model-00010-of-00241.safetensors (127 tensors) Loaded: model-00011-of-00241.safetensors (127 tensors) Loaded: model-00012-of-00241.safetensors (127 tensors) Loaded: model-00013-of-00241.safetensors (125 tensors) Loaded: model-00014-of-00241.safetensors (127 tensors) Loaded: model-00015-of-00241.safetensors (127 tensors) Loaded: model-00016-of-00241.safetensors (124 tensors) Loaded: model-00017-of-00241.safetensors (125 tensors) Loaded: model-00018-of-00241.safetensors (127 tensors) Loaded: model-00019-of-00241.safetensors (127 tensors) Loaded: model-00020-of-00241.safetensors (125 tensors) Loaded: model-00021-of-00241.safetensors (127 tensors) Loaded: model-00022-of-00241.safetensors (127 tensors) Loaded: model-00023-of-00241.safetensors (127 tensors) Loaded: model-00024-of-00241.safetensors (125 tensors) Loaded: model-00025-of-00241.safetensors (127 tensors) Loaded: model-00026-of-00241.safetensors (127 tensors) Loaded: model-00027-of-00241.safetensors (127 tensors) Loaded: model-00028-of-00241.safetensors (125 tensors) Loaded: model-00029-of-00241.safetensors (127 tensors) Loaded: model-00030-of-00241.safetensors (127 tensors) Loaded: model-00031-of-00241.safetensors (127 tensors) Loaded: model-00032-of-00241.safetensors (125 tensors) Loaded: model-00033-of-00241.safetensors (127 tensors) Loaded: model-00034-of-00241.safetensors (127 tensors) Loaded: model-00035-of-00241.safetensors (127 tensors) Loaded: model-00036-of-00241.safetensors (125 tensors) Loaded: model-00037-of-00241.safetensors (127 tensors) Loaded: model-00038-of-00241.safetensors (127 tensors) Loaded: model-00039-of-00241.safetensors (127 tensors) Loaded: model-00040-of-00241.safetensors (125 tensors) Loaded: model-00041-of-00241.safetensors (127 tensors) Loaded: model-00042-of-00241.safetensors (127 tensors) Loaded: model-00043-of-00241.safetensors (127 tensors) Loaded: model-00044-of-00241.safetensors (125 tensors) Loaded: model-00045-of-00241.safetensors (127 tensors) Loaded: model-00046-of-00241.safetensors (127 tensors) Loaded: model-00047-of-00241.safetensors (125 tensors) Loaded: model-00048-of-00241.safetensors (127 tensors) Loaded: model-00049-of-00241.safetensors (127 tensors) Loaded: model-00050-of-00241.safetensors (127 tensors) Loaded: model-00051-of-00241.safetensors (125 tensors) Loaded: model-00052-of-00241.safetensors (127 tensors) Loaded: model-00053-of-00241.safetensors (127 tensors) Loaded: model-00054-of-00241.safetensors (127 tensors) Loaded: model-00055-of-00241.safetensors (125 tensors) Loaded: model-00056-of-00241.safetensors (127 tensors) Loaded: model-00057-of-00241.safetensors (127 tensors) Loaded: model-00058-of-00241.safetensors (127 tensors) Loaded: model-00059-of-00241.safetensors (125 tensors) Loaded: model-00060-of-00241.safetensors (127 tensors) Loaded: model-00061-of-00241.safetensors (127 tensors) Loaded: model-00062-of-00241.safetensors (127 tensors) Loaded: model-00063-of-00241.safetensors (125 tensors) Loaded: model-00064-of-00241.safetensors (127 tensors) Loaded: model-00065-of-00241.safetensors (127 tensors) Loaded: model-00066-of-00241.safetensors (127 tensors) Loaded: model-00067-of-00241.safetensors (125 tensors) Loaded: model-00068-of-00241.safetensors (127 tensors) Loaded: model-00069-of-00241.safetensors (127 tensors) Loaded: model-00070-of-00241.safetensors (127 tensors) Loaded: model-00071-of-00241.safetensors (125 tensors) Loaded: model-00072-of-00241.safetensors (127 tensors) Loaded: model-00073-of-00241.safetensors (127 tensors) Loaded: model-00074-of-00241.safetensors (126 tensors) Loaded: model-00075-of-00241.safetensors (125 tensors) Loaded: model-00076-of-00241.safetensors (127 tensors) Loaded: model-00077-of-00241.safetensors (127 tensors) Loaded: model-00078-of-00241.safetensors (125 tensors) Loaded: model-00079-of-00241.safetensors (127 tensors) Loaded: model-00080-of-00241.safetensors (127 tensors) Loaded: model-00081-of-00241.safetensors (127 tensors) Loaded: model-00082-of-00241.safetensors (125 tensors) Loaded: model-00083-of-00241.safetensors (127 tensors) Loaded: model-00084-of-00241.safetensors (127 tensors) Loaded: model-00085-of-00241.safetensors (127 tensors) Loaded: model-00086-of-00241.safetensors (125 tensors) Loaded: model-00087-of-00241.safetensors (127 tensors) Loaded: model-00088-of-00241.safetensors (127 tensors) Loaded: model-00089-of-00241.safetensors (127 tensors) Loaded: model-00090-of-00241.safetensors (125 tensors) Loaded: model-00091-of-00241.safetensors (127 tensors) Loaded: model-00092-of-00241.safetensors (127 tensors) Loaded: model-00093-of-00241.safetensors (127 tensors) Loaded: model-00094-of-00241.safetensors (125 tensors) Loaded: model-00095-of-00241.safetensors (127 tensors) Loaded: model-00096-of-00241.safetensors (127 tensors) Loaded: model-00097-of-00241.safetensors (127 tensors) Loaded: model-00098-of-00241.safetensors (125 tensors) Loaded: model-00099-of-00241.safetensors (127 tensors) Loaded: model-00100-of-00241.safetensors (127 tensors) Loaded: model-00101-of-00241.safetensors (127 tensors) Loaded: model-00102-of-00241.safetensors (125 tensors) Loaded: model-00103-of-00241.safetensors (127 tensors) Loaded: model-00104-of-00241.safetensors (127 tensors) Loaded: model-00105-of-00241.safetensors (125 tensors) Loaded: model-00106-of-00241.safetensors (127 tensors) Loaded: model-00107-of-00241.safetensors (127 tensors) Loaded: model-00108-of-00241.safetensors (127 tensors) Loaded: model-00109-of-00241.safetensors (125 tensors) Loaded: model-00110-of-00241.safetensors (127 tensors) Loaded: model-00111-of-00241.safetensors (127 tensors) Loaded: model-00112-of-00241.safetensors (127 tensors) Loaded: model-00113-of-00241.safetensors (125 tensors) Loaded: model-00114-of-00241.safetensors (127 tensors) Loaded: model-00115-of-00241.safetensors (127 tensors) Loaded: model-00116-of-00241.safetensors (127 tensors) Loaded: model-00117-of-00241.safetensors (125 tensors) Loaded: model-00118-of-00241.safetensors (127 tensors) Loaded: model-00119-of-00241.safetensors (127 tensors) Loaded: model-00120-of-00241.safetensors (127 tensors) Loaded: model-00121-of-00241.safetensors (125 tensors) Loaded: model-00122-of-00241.safetensors (127 tensors) Loaded: model-00123-of-00241.safetensors (127 tensors) Loaded: model-00124-of-00241.safetensors (127 tensors) Loaded: model-00125-of-00241.safetensors (125 tensors) Loaded: model-00126-of-00241.safetensors (127 tensors) Loaded: model-00127-of-00241.safetensors (127 tensors) Loaded: model-00128-of-00241.safetensors (127 tensors) Loaded: model-00129-of-00241.safetensors (125 tensors) Loaded: model-00130-of-00241.safetensors (127 tensors) Loaded: model-00131-of-00241.safetensors (127 tensors) Loaded: model-00132-of-00241.safetensors (126 tensors) Loaded: model-00133-of-00241.safetensors (125 tensors) Loaded: model-00134-of-00241.safetensors (127 tensors) Loaded: model-00135-of-00241.safetensors (127 tensors) Loaded: model-00136-of-00241.safetensors (125 tensors) Loaded: model-00137-of-00241.safetensors (127 tensors) Loaded: model-00138-of-00241.safetensors (127 tensors) Loaded: model-00139-of-00241.safetensors (127 tensors) Loaded: model-00140-of-00241.safetensors (125 tensors) Loaded: model-00141-of-00241.safetensors (127 tensors) Loaded: model-00142-of-00241.safetensors (127 tensors) Loaded: model-00143-of-00241.safetensors (127 tensors) Loaded: model-00144-of-00241.safetensors (125 tensors) Loaded: model-00145-of-00241.safetensors (127 tensors) Loaded: model-00146-of-00241.safetensors (127 tensors) Loaded: model-00147-of-00241.safetensors (127 tensors) Loaded: model-00148-of-00241.safetensors (125 tensors) Loaded: model-00149-of-00241.safetensors (127 tensors) Loaded: model-00150-of-00241.safetensors (127 tensors) Loaded: model-00151-of-00241.safetensors (127 tensors) Loaded: model-00152-of-00241.safetensors (125 tensors) Loaded: model-00153-of-00241.safetensors (127 tensors) Loaded: model-00154-of-00241.safetensors (127 tensors) Loaded: model-00155-of-00241.safetensors (127 tensors) Loaded: model-00156-of-00241.safetensors (125 tensors) Loaded: model-00157-of-00241.safetensors (127 tensors) Loaded: model-00158-of-00241.safetensors (127 tensors) Loaded: model-00159-of-00241.safetensors (127 tensors) Loaded: model-00160-of-00241.safetensors (125 tensors) Loaded: model-00161-of-00241.safetensors (127 tensors) Loaded: model-00162-of-00241.safetensors (127 tensors) Loaded: model-00163-of-00241.safetensors (125 tensors) Loaded: model-00164-of-00241.safetensors (127 tensors) Loaded: model-00165-of-00241.safetensors (127 tensors) Loaded: model-00166-of-00241.safetensors (127 tensors) Loaded: model-00167-of-00241.safetensors (125 tensors) Loaded: model-00168-of-00241.safetensors (127 tensors) Loaded: model-00169-of-00241.safetensors (127 tensors) Loaded: model-00170-of-00241.safetensors (127 tensors) Loaded: model-00171-of-00241.safetensors (125 tensors) Loaded: model-00172-of-00241.safetensors (127 tensors) Loaded: model-00173-of-00241.safetensors (127 tensors) Loaded: model-00174-of-00241.safetensors (127 tensors) Loaded: model-00175-of-00241.safetensors (125 tensors) Loaded: model-00176-of-00241.safetensors (127 tensors) Loaded: model-00177-of-00241.safetensors (127 tensors) Loaded: model-00178-of-00241.safetensors (127 tensors) Loaded: model-00179-of-00241.safetensors (125 tensors) Loaded: model-00180-of-00241.safetensors (127 tensors) Loaded: model-00181-of-00241.safetensors (127 tensors) Loaded: model-00182-of-00241.safetensors (127 tensors) Loaded: model-00183-of-00241.safetensors (125 tensors) Loaded: model-00184-of-00241.safetensors (127 tensors) Loaded: model-00185-of-00241.safetensors (127 tensors) Loaded: model-00186-of-00241.safetensors (127 tensors) Loaded: model-00187-of-00241.safetensors (125 tensors) Loaded: model-00188-of-00241.safetensors (127 tensors) Loaded: model-00189-of-00241.safetensors (127 tensors) Loaded: model-00190-of-00241.safetensors (126 tensors) Loaded: model-00191-of-00241.safetensors (125 tensors) Loaded: model-00192-of-00241.safetensors (127 tensors) Loaded: model-00193-of-00241.safetensors (127 tensors) Loaded: model-00194-of-00241.safetensors (125 tensors) Loaded: model-00195-of-00241.safetensors (127 tensors) Loaded: model-00196-of-00241.safetensors (127 tensors) Loaded: model-00197-of-00241.safetensors (127 tensors) Loaded: model-00198-of-00241.safetensors (125 tensors) Loaded: model-00199-of-00241.safetensors (127 tensors) Loaded: model-00200-of-00241.safetensors (127 tensors) Loaded: model-00201-of-00241.safetensors (127 tensors) Loaded: model-00202-of-00241.safetensors (125 tensors) Loaded: model-00203-of-00241.safetensors (127 tensors) Loaded: model-00204-of-00241.safetensors (127 tensors) Loaded: model-00205-of-00241.safetensors (127 tensors) Loaded: model-00206-of-00241.safetensors (125 tensors) Loaded: model-00207-of-00241.safetensors (127 tensors) Loaded: model-00208-of-00241.safetensors (127 tensors) Loaded: model-00209-of-00241.safetensors (127 tensors) Loaded: model-00210-of-00241.safetensors (125 tensors) Loaded: model-00211-of-00241.safetensors (127 tensors) Loaded: model-00212-of-00241.safetensors (127 tensors) Loaded: model-00213-of-00241.safetensors (127 tensors) Loaded: model-00214-of-00241.safetensors (125 tensors) Loaded: model-00215-of-00241.safetensors (127 tensors) Loaded: model-00216-of-00241.safetensors (127 tensors) Loaded: model-00217-of-00241.safetensors (127 tensors) Loaded: model-00218-of-00241.safetensors (125 tensors) Loaded: model-00219-of-00241.safetensors (127 tensors) Loaded: model-00220-of-00241.safetensors (127 tensors) Loaded: model-00221-of-00241.safetensors (125 tensors) Loaded: model-00222-of-00241.safetensors (127 tensors) Loaded: model-00223-of-00241.safetensors (127 tensors) Loaded: model-00224-of-00241.safetensors (127 tensors) Loaded: model-00225-of-00241.safetensors (125 tensors) Loaded: model-00226-of-00241.safetensors (127 tensors) Loaded: model-00227-of-00241.safetensors (127 tensors) Loaded: model-00228-of-00241.safetensors (127 tensors) Loaded: model-00229-of-00241.safetensors (125 tensors) Loaded: model-00230-of-00241.safetensors (127 tensors) Loaded: model-00231-of-00241.safetensors (127 tensors) Loaded: model-00232-of-00241.safetensors (127 tensors) Loaded: model-00233-of-00241.safetensors (125 tensors) Loaded: model-00234-of-00241.safetensors (127 tensors) Loaded: model-00235-of-00241.safetensors (127 tensors) Loaded: model-00236-of-00241.safetensors (127 tensors) Loaded: model-00237-of-00241.safetensors (125 tensors) Loaded: model-00238-of-00241.safetensors (127 tensors) Loaded: model-00239-of-00241.safetensors (127 tensors) Loaded: model-00240-of-00241.safetensors (127 tensors) Loaded: model-00241-of-00241.safetensors (32 tensors) Total tensors loaded: 30321 Starting conversion... Input: /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M Output: /zpool0/LLMModels/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-INT4 Found 62 layers with MoE experts: Layer 0: 160 experts (0-159) Layer 1: 160 experts (0-159) Layer 2: 160 experts (0-159) Layer 3: 160 experts (0-159) Layer 4: 160 experts (0-159) Layer 5: 160 experts (0-159) Layer 6: 160 experts (0-159) Layer 7: 160 experts (0-159) Layer 8: 160 experts (0-159) Layer 9: 160 experts (0-159) Layer 10: 160 experts (0-159) Layer 11: 160 experts (0-159) Layer 12: 160 experts (0-159) Layer 13: 160 experts (0-159) Layer 14: 160 experts (0-159) Layer 15: 160 experts (0-159) Layer 16: 160 experts (0-159) Layer 17: 160 experts (0-159) Layer 18: 160 experts (0-159) Layer 19: 160 experts (0-159) Layer 20: 160 experts (0-159) Layer 21: 160 experts (0-159) Layer 22: 160 experts (0-159) Layer 23: 160 experts (0-159) Layer 24: 160 experts (0-159) Layer 25: 160 experts (0-159) Layer 26: 160 experts (0-159) Layer 27: 160 experts (0-159) Layer 28: 160 experts (0-159) Layer 29: 160 experts (0-159) Layer 30: 160 experts (0-159) Layer 31: 160 experts (0-159) Layer 32: 160 experts (0-159) Layer 33: 160 experts (0-159) Layer 34: 160 experts (0-159) Layer 35: 160 experts (0-159) Layer 36: 160 experts (0-159) Layer 37: 160 experts (0-159) Layer 38: 160 experts (0-159) Layer 39: 160 experts (0-159) Layer 40: 160 experts (0-159) Layer 41: 160 experts (0-159) Layer 42: 160 experts (0-159) Layer 43: 160 experts (0-159) Layer 44: 160 experts (0-159) Layer 45: 160 experts (0-159) Layer 46: 160 experts (0-159) Layer 47: 160 experts (0-159) Layer 48: 160 experts (0-159) Layer 49: 160 experts (0-159) Layer 50: 160 experts (0-159) Layer 51: 160 experts (0-159) Layer 52: 160 experts (0-159) Layer 53: 160 experts (0-159) Layer 54: 160 experts (0-159) Layer 55: 160 experts (0-159) Layer 56: 160 experts (0-159) Layer 57: 160 experts (0-159) Layer 58: 160 experts (0-159) Layer 59: 160 experts (0-159) Layer 60: 160 experts (0-159) Layer 61: 160 experts (0-159) Processing layer 0 (1/62)... Converting layer 0 with 160 experts via online quantization... Loaded weights shapes: gate_proj: torch.Size([160, 2560, 6144]) up_proj: torch.Size([160, 2560, 6144]) down_proj: torch.Size([160, 6144, 2560]) CPUInfer[0x3f490510]: Hello WorkerPool[0x3f5826b0] 8 subpools, [numa:threads][0:12] [1:12] [2:12] [3:12] [4:12] [5:12] [6:12] [7:12] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 4, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 5, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 6, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 7, 12 threads TP MOE layer 0, pool: 0x3f5826b0, expert num: 160, num_experts_per_tok: 8 n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 terminate called after throwing an instance of 'std::runtime_error' terminate called recursively what(): n or k is not aligned to N_STEP or B_K_STEP terminate called recursively n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 terminate called recursively n: 6144, k: 320, N_STEP: 32, B_K_STEP: 128 terminate called recursively terminate called recursively ``` converting Qwen3-Coder-30B-A3B-Instruct ```text Loading model configuration... Model config: {'num_experts': 128, 'num_experts_per_tok': 8, 'hidden_size': 2048, 'moe_intermediate_size': 768} num_experts: 128 num_experts_per_tok: 8 hidden_size: 2048 moe_intermediate_size: 768 CPU inference config: cpuinfer_threads: 96 threadpool_count: 8 Loading safetensors files from Qwen3-Coder-30B-A3B-Instruct Loaded: model-00001-of-00016.safetensors (1062 tensors) Loaded: model-00002-of-00016.safetensors (1262 tensors) Loaded: model-00003-of-00016.safetensors (1258 tensors) Loaded: model-00004-of-00016.safetensors (1262 tensors) Loaded: model-00005-of-00016.safetensors (1262 tensors) Loaded: model-00006-of-00016.safetensors (1262 tensors) Loaded: model-00007-of-00016.safetensors (1262 tensors) Loaded: model-00008-of-00016.safetensors (1258 tensors) Loaded: model-00009-of-00016.safetensors (1262 tensors) Loaded: model-00010-of-00016.safetensors (1262 tensors) Loaded: model-00011-of-00016.safetensors (1262 tensors) Loaded: model-00012-of-00016.safetensors (1261 tensors) Loaded: model-00013-of-00016.safetensors (1256 tensors) Loaded: model-00014-of-00016.safetensors (1262 tensors) Loaded: model-00015-of-00016.safetensors (1262 tensors) Loaded: model-00016-of-00016.safetensors (152 tensors) Total tensors loaded: 18867 Starting conversion... Input: Qwen3-Coder-30B-A3B-Instruct Output: Qwen3-Coder-30B-A3B-Instruct-INT4 Found 48 layers with MoE experts: Layer 0: 128 experts (0-127) Layer 1: 128 experts (0-127) Layer 2: 128 experts (0-127) Layer 3: 128 experts (0-127) Layer 4: 128 experts (0-127) Layer 5: 128 experts (0-127) Layer 6: 128 experts (0-127) Layer 7: 128 experts (0-127) Layer 8: 128 experts (0-127) Layer 9: 128 experts (0-127) Layer 10: 128 experts (0-127) Layer 11: 128 experts (0-127) Layer 12: 128 experts (0-127) Layer 13: 128 experts (0-127) Layer 14: 128 experts (0-127) Layer 15: 128 experts (0-127) Layer 16: 128 experts (0-127) Layer 17: 128 experts (0-127) Layer 18: 128 experts (0-127) Layer 19: 128 experts (0-127) Layer 20: 128 experts (0-127) Layer 21: 128 experts (0-127) Layer 22: 128 experts (0-127) Layer 23: 128 experts (0-127) Layer 24: 128 experts (0-127) Layer 25: 128 experts (0-127) Layer 26: 128 experts (0-127) Layer 27: 128 experts (0-127) Layer 28: 128 experts (0-127) Layer 29: 128 experts (0-127) Layer 30: 128 experts (0-127) Layer 31: 128 experts (0-127) Layer 32: 128 experts (0-127) Layer 33: 128 experts (0-127) Layer 34: 128 experts (0-127) Layer 35: 128 experts (0-127) Layer 36: 128 experts (0-127) Layer 37: 128 experts (0-127) Layer 38: 128 experts (0-127) Layer 39: 128 experts (0-127) Layer 40: 128 experts (0-127) Layer 41: 128 experts (0-127) Layer 42: 128 experts (0-127) Layer 43: 128 experts (0-127) Layer 44: 128 experts (0-127) Layer 45: 128 experts (0-127) Layer 46: 128 experts (0-127) Layer 47: 128 experts (0-127) Processing layer 0 (1/48)... Converting layer 0 with 128 experts via online quantization... Loaded weights shapes: gate_proj: torch.Size([128, 768, 2048]) up_proj: torch.Size([128, 768, 2048]) down_proj: torch.Size([128, 2048, 768]) CPUInfer[0x1d2f6bf0]: Hello WorkerPool[0x1d46f740] 8 subpools, [numa:threads][0:12] [1:12] [2:12] [3:12] [4:12] [5:12] [6:12] [7:12] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 4, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 5, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 6, 12 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 7, 12 threads TP MOE layer 0, pool: 0x1d46f740, expert num: 128, num_experts_per_tok: 8 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 terminate called recursively terminate called after throwing an instance of 'max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 std::runtime_error' terminate called recursively terminate called recursively terminate called recursively terminate called after throwing an instance of 'std::runtime_error' terminate called recursively what(): BufferAImpl: max_m and k must be multiple of M_STEP and K_STEP what(): BufferAImpl: max_m and k must be multiple of M_STEP and K_STEP max_m = 512, k = 96, M_STEP = 32, K_STEP = 64 terminate called recursively ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:48:44 +00:00
Author
Owner
Copy link

@mohkgh commented on GitHub (Jan 7, 2026):

Image Disable NUMA in bios, then system has 2 NUMA nodes only. M_STEP / K_STEP / N_STEP / B_K_STEP are aligned by default. When they all devided by threadpool-count (NUMA nodes count, here is 8), alignment conditions may be broken. But now I meet another problem, no error output, just aborted. 
Loading model configuration...
Model config: {'num_experts': 128, 'num_experts_per_tok': 8, 'hidden_size': 2048, 'moe_intermediate_size': 768}
  num_experts: 128
  num_experts_per_tok: 8
  hidden_size: 2048
  moe_intermediate_size: 768
CPU inference config:
  cpuinfer_threads: 96
  threadpool_count: 2

Loading safetensors files from /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct
  Loaded: model-00001-of-00016.safetensors (1062 tensors)
  Loaded: model-00002-of-00016.safetensors (1262 tensors)
  Loaded: model-00003-of-00016.safetensors (1258 tensors)
  Loaded: model-00004-of-00016.safetensors (1262 tensors)
  Loaded: model-00005-of-00016.safetensors (1262 tensors)
  Loaded: model-00006-of-00016.safetensors (1262 tensors)
  Loaded: model-00007-of-00016.safetensors (1262 tensors)
  Loaded: model-00008-of-00016.safetensors (1258 tensors)
  Loaded: model-00009-of-00016.safetensors (1262 tensors)
  Loaded: model-00010-of-00016.safetensors (1262 tensors)
  Loaded: model-00011-of-00016.safetensors (1262 tensors)
  Loaded: model-00012-of-00016.safetensors (1261 tensors)
  Loaded: model-00013-of-00016.safetensors (1256 tensors)
  Loaded: model-00014-of-00016.safetensors (1262 tensors)
  Loaded: model-00015-of-00016.safetensors (1262 tensors)
  Loaded: model-00016-of-00016.safetensors (152 tensors)
Total tensors loaded: 18867
Starting conversion...
Input: /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct
Output: /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct-INT4
Found 48 layers with MoE experts:
  Layer 0: 128 experts (0-127)
  Layer 1: 128 experts (0-127)
  Layer 2: 128 experts (0-127)
  Layer 3: 128 experts (0-127)
  Layer 4: 128 experts (0-127)
  Layer 5: 128 experts (0-127)
  Layer 6: 128 experts (0-127)
  Layer 7: 128 experts (0-127)
  Layer 8: 128 experts (0-127)
  Layer 9: 128 experts (0-127)
  Layer 10: 128 experts (0-127)
  Layer 11: 128 experts (0-127)
  Layer 12: 128 experts (0-127)
  Layer 13: 128 experts (0-127)
  Layer 14: 128 experts (0-127)
  Layer 15: 128 experts (0-127)
  Layer 16: 128 experts (0-127)
  Layer 17: 128 experts (0-127)
  Layer 18: 128 experts (0-127)
  Layer 19: 128 experts (0-127)
  Layer 20: 128 experts (0-127)
  Layer 21: 128 experts (0-127)
  Layer 22: 128 experts (0-127)
  Layer 23: 128 experts (0-127)
  Layer 24: 128 experts (0-127)
  Layer 25: 128 experts (0-127)
  Layer 26: 128 experts (0-127)
  Layer 27: 128 experts (0-127)
  Layer 28: 128 experts (0-127)
  Layer 29: 128 experts (0-127)
  Layer 30: 128 experts (0-127)
  Layer 31: 128 experts (0-127)
  Layer 32: 128 experts (0-127)
  Layer 33: 128 experts (0-127)
  Layer 34: 128 experts (0-127)
  Layer 35: 128 experts (0-127)
  Layer 36: 128 experts (0-127)
  Layer 37: 128 experts (0-127)
  Layer 38: 128 experts (0-127)
  Layer 39: 128 experts (0-127)
  Layer 40: 128 experts (0-127)
  Layer 41: 128 experts (0-127)
  Layer 42: 128 experts (0-127)
  Layer 43: 128 experts (0-127)
  Layer 44: 128 experts (0-127)
  Layer 45: 128 experts (0-127)
  Layer 46: 128 experts (0-127)
  Layer 47: 128 experts (0-127)
Processing layer 0 (1/48)...
Converting layer 0 with 128 experts via online quantization...
  Loaded weights shapes:
    gate_proj: torch.Size([128, 768, 2048])
    up_proj: torch.Size([128, 768, 2048])
    down_proj: torch.Size([128, 2048, 768])
CPUInfer[0x40f6a6b0]: Hello
WorkerPool[0x41230ac0] 2 subpools, [numa:threads][0:48] [1:48] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 48 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 48 threads
TP MOE layer 0, pool: 0x41230ac0, expert num: 128, num_experts_per_tok: 8
From BF16
  online quant from bf16
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
Aborted
<!-- gh-comment-id:3718249160 --> @mohkgh commented on GitHub (Jan 7, 2026): <img width="742" height="252" alt="Image" src="https://github.com/user-attachments/assets/f8e8e4dd-2a80-49d8-8138-4de521c024a7" /> Disable NUMA in bios, then system has 2 NUMA nodes only. M_STEP / K_STEP / N_STEP / B_K_STEP are aligned by default. When they all devided by threadpool-count (NUMA nodes count, here is 8), alignment conditions may be broken. But now I meet another problem, no error output, just aborted. ``` Loading model configuration... Model config: {'num_experts': 128, 'num_experts_per_tok': 8, 'hidden_size': 2048, 'moe_intermediate_size': 768} num_experts: 128 num_experts_per_tok: 8 hidden_size: 2048 moe_intermediate_size: 768 CPU inference config: cpuinfer_threads: 96 threadpool_count: 2 Loading safetensors files from /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct Loaded: model-00001-of-00016.safetensors (1062 tensors) Loaded: model-00002-of-00016.safetensors (1262 tensors) Loaded: model-00003-of-00016.safetensors (1258 tensors) Loaded: model-00004-of-00016.safetensors (1262 tensors) Loaded: model-00005-of-00016.safetensors (1262 tensors) Loaded: model-00006-of-00016.safetensors (1262 tensors) Loaded: model-00007-of-00016.safetensors (1262 tensors) Loaded: model-00008-of-00016.safetensors (1258 tensors) Loaded: model-00009-of-00016.safetensors (1262 tensors) Loaded: model-00010-of-00016.safetensors (1262 tensors) Loaded: model-00011-of-00016.safetensors (1262 tensors) Loaded: model-00012-of-00016.safetensors (1261 tensors) Loaded: model-00013-of-00016.safetensors (1256 tensors) Loaded: model-00014-of-00016.safetensors (1262 tensors) Loaded: model-00015-of-00016.safetensors (1262 tensors) Loaded: model-00016-of-00016.safetensors (152 tensors) Total tensors loaded: 18867 Starting conversion... Input: /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct Output: /zpool0/LLMModels/Qwen/Qwen3-Coder-30B-A3B-Instruct-INT4 Found 48 layers with MoE experts: Layer 0: 128 experts (0-127) Layer 1: 128 experts (0-127) Layer 2: 128 experts (0-127) Layer 3: 128 experts (0-127) Layer 4: 128 experts (0-127) Layer 5: 128 experts (0-127) Layer 6: 128 experts (0-127) Layer 7: 128 experts (0-127) Layer 8: 128 experts (0-127) Layer 9: 128 experts (0-127) Layer 10: 128 experts (0-127) Layer 11: 128 experts (0-127) Layer 12: 128 experts (0-127) Layer 13: 128 experts (0-127) Layer 14: 128 experts (0-127) Layer 15: 128 experts (0-127) Layer 16: 128 experts (0-127) Layer 17: 128 experts (0-127) Layer 18: 128 experts (0-127) Layer 19: 128 experts (0-127) Layer 20: 128 experts (0-127) Layer 21: 128 experts (0-127) Layer 22: 128 experts (0-127) Layer 23: 128 experts (0-127) Layer 24: 128 experts (0-127) Layer 25: 128 experts (0-127) Layer 26: 128 experts (0-127) Layer 27: 128 experts (0-127) Layer 28: 128 experts (0-127) Layer 29: 128 experts (0-127) Layer 30: 128 experts (0-127) Layer 31: 128 experts (0-127) Layer 32: 128 experts (0-127) Layer 33: 128 experts (0-127) Layer 34: 128 experts (0-127) Layer 35: 128 experts (0-127) Layer 36: 128 experts (0-127) Layer 37: 128 experts (0-127) Layer 38: 128 experts (0-127) Layer 39: 128 experts (0-127) Layer 40: 128 experts (0-127) Layer 41: 128 experts (0-127) Layer 42: 128 experts (0-127) Layer 43: 128 experts (0-127) Layer 44: 128 experts (0-127) Layer 45: 128 experts (0-127) Layer 46: 128 experts (0-127) Layer 47: 128 experts (0-127) Processing layer 0 (1/48)... Converting layer 0 with 128 experts via online quantization... Loaded weights shapes: gate_proj: torch.Size([128, 768, 2048]) up_proj: torch.Size([128, 768, 2048]) down_proj: torch.Size([128, 2048, 768]) CPUInfer[0x40f6a6b0]: Hello WorkerPool[0x41230ac0] 2 subpools, [numa:threads][0:48] [1:48] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 48 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 48 threads TP MOE layer 0, pool: 0x41230ac0, expert num: 128, num_experts_per_tok: 8 From BF16 online quant from bf16 terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively terminate called recursively Aborted ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1118
No description provided.