Iwan Kawrakow 0b6d6541d7 NEON Flash Attention: quantized K*Q for q4_0 ...

I could finally take advantage of the matrix multiplication
templates. We get quite a bit of speedup that way for q4_0:
For Gemma-2b using mul_mat_qX_0_q8_0<DequantizerQ40, q_step>
results in PP-2048 = 287 t/s vs 268 t/s when converting the
q4_0 k-cache and Q to fp16 and using fp16 multiplication.

2024-09-12 09:12:35 +02:00

..

cmake

Merge mainline llama.cpp (#3)

2024-07-27 07:55:01 +02:00

include

Adding IQ1_TN - 1.6875 bpw for TriLM ternary models (#44)

2024-09-09 14:56:34 +03:00

src

NEON Flash Attention: quantized K*Q for q4_0

2024-09-12 09:12:35 +02:00

.gitignore

Merge mainline llama.cpp (#3)

2024-07-27 07:55:01 +02:00

CMakeLists.txt

Merge mainline - Aug 12 2024 (#17)

2024-08-12 15:14:32 +02:00