Iwan Kawrakow 8a2d43813d iq4_k: basics ...

* quantize/dequantize works
* CUDA dequantize works and one can run PPL calcs. I get
  PPL = 6.5258 for LlaMA-3.1-8B, which is 1.77% above fp16.
  In comparison, q4_K_S (same size) is 2.88% above fp16.
* TG on CUDA does not work. Johannes has changed the way i-quant dot
  products are done, so need to sort out what he had in mind
* iqk_mul_mat is not implemented.

2024-07-27 17:05:31 +03:00

..

llama.h

iq4_k: basics

2024-07-27 17:05:31 +03:00