Kawrakow c5e554f941 Convert models to row-interleaved quants using the quantize tool (#272) ...

* Repack a model with the quantize tool

* WIP

* Fixed various issues

As we don't have a way to tell if a repacked quant has been modified,
I had to remove the modification at the expense of a slight decrease
in performance. This affects q8_0_r8, q8_KV_r8, q8_k_r8 on Zen4, and
q4_0_r8 on ARM.

* Create wk_b and wv_b as Q8_0_R8 if the wkv_b type is interleaved

* Fix GCC 13.3 compilation error

* Another one

* Add missing include

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

2025-03-21 07:23:36 +01:00

..

llama.h

Convert models to row-interleaved quants using the quantize tool (#272)

2025-03-21 07:23:36 +01:00