1.5 KiB
🔀 #482 - Trellis quants: faster CPU prompt processing
| Author | ikawrakow |
|---|---|
| State | ❌ Closed |
| Created | 2025-06-01 |
| Updated | 2025-06-01 |
Description
The trellis quants IQ2_KT, IQ3_KT, IQ4_KT are very slow on the CPU. On the main branch using BLAS results in a better prompt processing performance. But BLAS is slower for basically all other data types, so that's not a good idea.
This PR improves prompt processing speed of the trellis quants by adding "dequantizing GEMM". Basically, blocks of trelis quantized weights are converted to fp32 (AVX2 )or fp16 (ARM) on-the-fly, and then the fp32/fp16 GEMM kernels are used to multiply the bock with the entire right matrix. This amortizes the very high dequantization cost much better than the standard kernel templates that allow up to 8 right matrix columns.
On my Zen4/AVX2 CPUs this results in a better PP performance than using BLAS (or Intel MKL). On the M2-Max PP performance is about 80% of BLAS (which tells me that my ARM_NEON GEMM kernel for fp16 is not optimal).
TG performance is not affected by the PR and is still very low.
Here is a PP-512 performance comparison between the main branch (without BLAS) and this PR for LlaMA-3.1-8B on a Ryzen-7950X CPU
| quant | PP-512 (main) | PP-512 (PR) | Speedup |
|---|---|---|---|
| IQ2_KT | 57.98 | 132.47 | 2.28 |
| IQ3_KT | 47.44 | 127.80 | 2.69 |
| IQ4_KT | 40.09 | 126.31 | 3.15 |