ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-03-13 15:30:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

iq3_k: Basics

Browse Source 
Quantize/dequantize, CUDA dequantize.
PPL of LLaMA-3.1-8B is better than iq3_s and iq3_m.
This commit is contained in:
Iwan Kawrakow
2024-07-30 16:11:25 +03:00
committed by Kawrakow
parent 36204c4ec7
commit 4f237d44f6
15 changed files with 416 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
include/llama.h
View File

@@ -171,8 +171,9 @@ extern "C" {
LLAMA_FTYPE_MOSTLY_IQ1_BN = 36, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_BN = 37, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_K = 38, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ4_K = 39, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ5_K = 40, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ3_K = 39, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ4_K = 40, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ5_K = 41, // except 1d tensors
LLAMA_FTYPE_GUESSED = 1024, // not specified in the model file
};