ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-03-11 22:40:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add GitHub data: filename sanitization (#640)

Browse Source
This commit is contained in:
Thomas
2025-07-23 13:31:53 +02:00
committed by GitHub
parent 3600d82e98
commit eaa2510a28
626 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
github-data/pull_requests/482 - Trellis quants_ faster CPU prompt processing.md Normal file
View File

@@ -0,0 +1,27 @@
### 🔀 [#482](https://github.com/ikawrakow/ik_llama.cpp/pull/482) - Trellis quants: faster CPU prompt processing
| **Author** | `ikawrakow` |
| :--- | :--- |
| **State** | ❌ **Closed** |
| **Created** | 2025-06-01 |
| **Updated** | 2025-06-01 |
---
#### Description
The trellis quants `IQ2_KT, IQ3_KT, IQ4_KT` are very slow on the CPU. On the main branch using BLAS results in a better prompt processing performance. But BLAS is slower for basically all other data types, so that's not a good idea.
This PR improves prompt processing speed of the trellis quants by adding "dequantizing GEMM". Basically, blocks of trelis quantized weights are converted to `fp32` (AVX2 )or `fp16` (ARM) on-the-fly, and then the `fp32/fp16` GEMM kernels are used to multiply the bock with the entire right matrix. This amortizes the very high dequantization cost much better than the standard kernel templates that allow up to 8 right matrix columns.
On my `Zen4/AVX2` CPUs this results in a better PP performance than using BLAS (or Intel MKL). On the M2-Max PP performance is about 80% of BLAS (which tells me that my `ARM_NEON` GEMM kernel for `fp16` is not optimal).
TG performance is not affected by the PR and is still very low.
Here is a PP-512 performance comparison between the main branch (without BLAS) and this PR for LlaMA-3.1-8B on a Ryzen-7950X CPU
| quant | PP-512 (main) | PP-512 (PR) | Speedup |
| ---: | ---: | ---: | ---: |
| IQ2_KT | 57.98 | 132.47 | 2.28 |
| IQ3_KT | 47.44 | 127.80 | 2.69 |
| IQ4_KT | 40.09 | 126.31 | 3.15 |