14 KiB
🔀 #546 - Faster ARM_NEON GEMM implementation for legacy quants
| Author | ikawrakow |
|---|---|
| State | ❌ Closed |
| Created | 2025-06-21 |
| Updated | 2025-06-22 |
Description
It is time to give some attention to the ARM_NEON back-end, which has fallen behind quite a bit.
This PR corresponds to PRs #531, #533, #534 and applies the on-the-fly repacking technique to Q4_0, Q4_1, Q5_0, Q5_1, Q6_0, Q8_0, IQ4_NL for the ARM_NEON implementation.
Here is a PP-512 performance comparison between the main branch and this PR for LlaMA-3.1-8B-Instruct on M2-Max
| type | t/s (main) | t/s (PR) | Speedup |
|---|---|---|---|
| Q4_0 | 83.58 | 128.41 | 1.536 |
| Q5_0 | 74.20 | 128.57 | 1.733 |
| Q6_0 | 74.25 | 128.79 | 1.735 |
| Q8_0 | 84.45 | 128.63 | 1.523 |
| IQ4_NL | 84.47 | 128.09 | 1.516 |
| Q4_1 | 74.44 | 115.36 | 1.550 |
| Q5_1 | 64.16 | 114.89 | 1.791 |
💬 Conversation
👤 zhouwg commented the 2025-06-22 at 07:22:29:
I tried your ik_llamacpp on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoCs on our planet at the moment) today, the performance of your excellent ik_llamacpp is impressive(faster than the upstream llama.cpp) .
both build with " -O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only " because " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only " can't works with ik_llama.cpp cause of some compile error with inline assemble codes.
upstream llama.cpp:
llama-bench:
llama-cli:
ik_llama.cpp:
llama-bench:
llama-cli(the inference result is incorrect and don't know why)
👤 zhouwg commented the 2025-06-22 at 07:24:04:
I tried ik_llamacpp on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoCs on our planet at the moment) today, the performance of your excellent ik_llamacpp is impressive .
both build with " -O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only " because " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only " can't works with ik_llama.cpp cause of some compile error with inline assemble codes.
upstream llama.cpp with latest codes:
llama-bench:
llama-cli:
ik_llama.cpp with latest codes:
llama-bench:
llama-cli(the inference result is incorrect and don't know why)
👤 zhouwg commented the 2025-06-22 at 08:36:03:
comparison of llama_bench on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoCs on our planet at the moment) + Android NDK r28:
- both build with " -O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only "
upstream llama.cpp with latest codes:
llama-bench:
llama-cli:
ik_llama.cpp with latest codes:
llama-cli(the inference result is incorrect)
- both build with " -O3 -march=armv8.2-a+dotprod+fp16 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only ".
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes: the following is a screenshot when I helped troubleshooting a performance regression issue in the upstream llama.cpp project. as well known, there are so many approved PRs in the upstream llama.cpp project and some approved PRs might-be brings regression issues in the upstream llama.cpp project. sometimes I can't reproduce the same benchmark result with the upstream llama.cpp's latest codes.
ik_llama.cpp with latest codes:
👤 zhouwg commented the 2025-06-22 at 09:46:28:
comparison of llama_bench on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoCs on our planet at the moment) + Android NDK r28:
- both build with " -O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only "
upstream llama.cpp with latest codes:
llama-bench:
llama-cli:
ik_llama.cpp with latest codes:
llama-cli(the inference result is incorrect)
- both build with " -O3 -march=armv8.2-a+dotprod+fp16 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only ".
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a+dotprod+fp16 -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes:
the following is a screenshot when I helped troubleshooting a performance regression issue in the upstream llama.cpp project. as well known, there are so many approved PRs in the upstream llama.cpp project and some approved PRs might-be brings regression issues in the upstream llama.cpp project. sometimes I can't reproduce the same benchmark result with the upstream llama.cpp's latest codes.
ik_llama.cpp with latest codes:
👤 zhouwg commented the 2025-06-22 at 10:58:12:
comparison of llama_bench on Android phone equipped with Qualcomm Snapdragon 8Elite(one of the most advanced mobile SoCs on our planet at the moment) + Android NDK r28(the following benchmark data might-be depend on the workload of Android OS):
- both build with " -O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only "
upstream llama.cpp with latest codes:
llama-bench:
llama-cli:
ik_llama.cpp with latest codes:
llama-cli(the inference result is incorrect)
- both build with " -O3 -march=armv8.2-a+dotprod+fp16 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only ".
upstream llama.cpp with latest codes:
ik_llama.cpp with latest codes:
- both build with " -O3 -march=armv8.7-a+dotprod+fp16 -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only"
upstream llama.cpp with latest codes:
the following is a screenshot when I helped troubleshooting a performance regression issue in the upstream llama.cpp project. as well known, there are so many approved PRs in the upstream llama.cpp project and some approved PRs might-be brings regression issues in the upstream llama.cpp project. sometimes I can't reproduce the same benchmark result with the upstream llama.cpp's latest codes.
ik_llama.cpp with latest codes:
after enable GGML_IQK_FLASH_ATTENTION
build with " -O3 -march=armv8.7-a+dotprod+fp16 -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -fvectorize -ffp-model=fast -fno-finite-math-only"
build with " -O3 -march=armv8.7-a+dotprod+fp16 -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
build with " -O3 -march=armv8.7-a -mcpu=cortex-x1 -mtune=cortex-x1 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only "
build failed with " -O3 -march=armv8.7-a -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only "
build with " -O3 -march=armv8.7-a+dotprod+fp16 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
build with " -O3 -march=armv8.2-a+dotprod+fp16 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"
build with "-O3 -flto -D_GNU_SOURCE -ffp-model=fast -fno-finite-math-only"