[GH-ISSUE #1841] How to convert Kimi K2.5 to AMXINT4 weights? #1148

New issue

Open

opened 2026-10-29 14:49:14 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:49:14 +00:00

Owner

Originally created by @shrould8888 on GitHub (Feb 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1841

Reminder

I have read the above rules and searched the existing issues.

System Info

K2.5's expert weights are in int4 right? The convert_cpu_weights.py script does not support int4 input type. Anyone has the solution? Thanks in advance.

Reproduction

Put your message here.

Others

No response

Originally created by @shrould8888 on GitHub (Feb 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1841 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info K2.5's expert weights are in int4 right? The convert_cpu_weights.py script does not support int4 input type. Anyone has the solution? Thanks in advance. ### Reproduction ```text Put your message here. ``` ### Others _No response_

bsakaguchi commented

2026-10-29 14:49:15 +00:00

Author

Owner

@lihaoran97 commented on GitHub (Feb 19, 2026):

参考：

@lihaoran97 commented on GitHub (Feb 19, 2026): 参考： <img width="910" height="268" alt="Image" src="https://github.com/user-attachments/assets/0c9f3649-bd02-47ea-b0f5-5c47b94c3e27" />

bsakaguchi commented

2026-10-29 14:49:16 +00:00

Author

Owner

@shrould8888 commented on GitHub (Feb 28, 2026):

你好，谢谢。我是想用AMXINT4去跑， Prefill会快很多，但需要的格式不是RAWINT4, 而是AWXINT4,。我曾经也能用AMX跑K2的，当时有人转换到AMXINT4，所以能下载（好像K2有BF16下载所以能用那script转，现在K2.5没有BF16）, 我跑K2 AMXINT4的指令是样：
python -m sglang.launch_server --host 0.0.0.0 --port 10005 --model /Kimi-K2-Thinking --kt-weight-path /Kimi-K2-Thinking-CPU-weight --kt-cpuinfer 112 --kt-threadpool-count 2 --kt-num-gpu-experts 0 --kt-method AMXINT4 --attention-backend flashinfer --trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 4096 --max-running-requests 37 --max-total-tokens 256000 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion

给你参考。

@shrould8888 commented on GitHub (Feb 28, 2026): 你好，谢谢。我是想用AMXINT4去跑， Prefill会快很多，但需要的格式不是RAWINT4, 而是AWXINT4,。我曾经也能用AMX跑K2的，当时有人转换到AMXINT4，所以能下载（好像K2有BF16下载所以能用那script转，现在K2.5没有BF16）, 我跑K2 AMXINT4的指令是样： python -m sglang.launch_server --host 0.0.0.0 --port 10005 --model /Kimi-K2-Thinking --kt-weight-path /Kimi-K2-Thinking-CPU-weight --kt-cpuinfer 112 --kt-threadpool-count 2 --kt-num-gpu-experts 0 --kt-method AMXINT4 --attention-backend flashinfer --trust-remote-code --mem-fraction-static 0.98 --chunked-prefill-size 4096 --max-running-requests 37 --max-total-tokens 256000 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion 给你参考。

bsakaguchi commented

2026-10-29 14:49:16 +00:00

Author

Owner

@shrould8888 commented on GitHub (May 16, 2026):

I have found the solution and hope it could be useful to someone:

convert Kimi K2.X’s native INT4 weights to BF16 by using the script int4_cast_bf16_fixed.py from this project: https://github.com/csabakecskemeti/DQ_int4-to-bf16_dequant
using ktransformers/kt-kernel/scripts/convert_cpu_weights.py to convert bf16 weights to AMXINT4 weights.

I have successfully converted Kimi K2.6 INT4 weights to AMXINT4, it works flawlessly.

@shrould8888 commented on GitHub (May 16, 2026): I have found the solution and hope it could be useful to someone: 1. convert Kimi K2.X’s native INT4 weights to BF16 by using the script int4_cast_bf16_fixed.py from this project: https://github.com/csabakecskemeti/DQ_int4-to-bf16_dequant 2. using ktransformers/kt-kernel/scripts/convert_cpu_weights.py to convert bf16 weights to AMXINT4 weights. I have successfully converted Kimi K2.6 INT4 weights to AMXINT4, it works flawlessly.