[GH-ISSUE #2060] [Feature] Support repacking Intel Autoround format(s) into AMXINT4/AMXINT8 #1259

New issue

Open

opened 2026-10-29 14:50:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:50:37 +00:00

Owner

Originally created by @kassieclaire on GitHub (Jun 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2060

Reminder

I have read the above rules and searched the existing issues.

Description

Intel AutoRound format seems to very reliably produce quantized weights that are high-quality when using a good calibration dataset, and can produce a format with scaled block sizes of 64, matching the AMXINT4 and AMXINT8 block sizes. From my understanding, the only parts missing would be numa optimization and VNNI repacking of the blocks to allow quick decode with AVX512 instructions on Xeon CPUs.

Intel AutoRound is natively supported by sglang, meaning that the only gaps would be packaging the groups for AMX prefill + AVX512 decode + NUMA node optimization on the CPU-side.

Optimally, the integration would involve allowing the user to select the specific quantization algorithm as intel autoround for the actual quantization part of convert_cpu_weights.py, while leaving the other sections untouched.

As a stretch goal: supporting the lowest-bit formats like W2A16 and W3A16 produced by autoround would be great too, and allow for very-low-bit format support in ktransformers.

Pull Request

No response

Originally created by @kassieclaire on GitHub (Jun 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2060 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description [Intel AutoRound](https://github.com/intel/auto-round) format seems to very reliably produce quantized weights that are high-quality when using a good calibration dataset, and can produce a format with scaled block sizes of 64, matching the AMXINT4 and AMXINT8 block sizes. From my understanding, the only parts missing would be numa optimization and VNNI repacking of the blocks to allow quick decode with AVX512 instructions on Xeon CPUs. Intel AutoRound is [natively supported by sglang](https://www.lmsys.org/blog/2025-11-13-AutoRound/), meaning that the only gaps would be packaging the groups for AMX prefill + AVX512 decode + NUMA node optimization on the CPU-side. Optimally, the integration would involve allowing the user to select the specific quantization algorithm as intel autoround for the actual quantization part of convert_cpu_weights.py, while leaving the other sections untouched. As a stretch goal: supporting the lowest-bit formats like W2A16 and W3A16 produced by autoround would be great too, and allow for very-low-bit format support in ktransformers. ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:50:37 +00:00

bsakaguchi commented

2026-10-29 14:50:38 +00:00

Author

Owner

@kassieclaire commented on GitHub (Jun 24, 2026):

Similarly, MLX can use a group size of 64, if it would be possible to skip the quantization step for a format like MLX, that is already quantized, and instead replace it with unpacking -> manipulation into VNNI vectors in the tile format -> packing nibles into uint8, it should be possible to load MLX weights and convert them into AMXINT format -> convert scales from fp16 to bf16.
Having direct conversion like this from MLX format would make a lot of high-quality quants easy to use in ktransformers

@kassieclaire commented on GitHub (Jun 24, 2026): Similarly, MLX can use a group size of 64, if it would be possible to skip the quantization step for a format like MLX, that is already quantized, and instead replace it with unpacking -> manipulation into VNNI vectors in the tile format -> packing nibles into uint8, it should be possible to load MLX weights and convert them into AMXINT format -> convert scales from fp16 to bf16. Having direct conversion like this from MLX format would make a lot of high-quality quants easy to use in ktransformers