mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-05-03 12:51:53 +00:00
27 lines
1.0 KiB
Markdown
27 lines
1.0 KiB
Markdown
### 🔀 [#550](https://github.com/ikawrakow/ik_llama.cpp/pull/550) - Much faster prompt processing for I-quants (ARM_NEON)
|
|
|
|
| **Author** | `ikawrakow` |
|
|
| :--- | :--- |
|
|
| **State** | ❌ **Closed** |
|
|
| **Created** | 2025-06-23 |
|
|
| **Updated** | 2025-06-23 |
|
|
|
|
---
|
|
|
|
#### Description
|
|
|
|
It is time to give some attention to the `ARM_NEON` back-end, which has fallen behind quite a bit.
|
|
|
|
This PR corresponds to PRs #531, #533, #534, #546, #549, and applies the on-the-fly repacking technique to i-quants (`IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S`) for the `ARM_NEON` implementation.
|
|
|
|
Here is a PP-512 performance comparison between the main branch and this PR for LlaMA-3.1-8B-Instruct on M2-Max
|
|
|
|
| type | t/s (main) | t/s (PR) | Speedup |
|
|
| ---: | ---: | ---: | ---: |
|
|
| IQ2_XXS | 55.79 | 167.55 | 3.003 |
|
|
| IQ2_XS | 46.40 | 166.65 | 3.592 |
|
|
| IQ2_S | 42.75 | 166.83 | 3.903 |
|
|
| IQ3_XXS | 51.84 | 165.56 | 3.194 |
|
|
| IQ3_S | 46.02 | 162.03 | 3.521 |
|
|
|
|
At this point i- and `IQK` quants are the top tier quants for prompt processing speed on `ARM_NEON`. |