ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-04-22 23:49:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

q8_k_r8: fastest matrix multiplication known to human kind

Browse Source 
We get PP-512(LLaMA-3.1-8B) = 370 t/s on a Ryzen-7950X!
This commit is contained in:
Iwan Kawrakow
2024-12-13 18:21:08 +02:00
parent 12f962dd24
commit 93a85c62bb
10 changed files with 242 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
include/llama.h
View File

@@ -193,6 +193,7 @@ extern "C" {
LLAMA_FTYPE_MOSTLY_Q6_0_R4 = 335, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_BN_R4 = 337, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ4_K_R4 = 340, // except 1d tensors
LLAMA_FTYPE_MOSTLY_Q8_K_R8 = 399, // except 1d tensors
LLAMA_FTYPE_GUESSED = 1024, // not specified in the model file
};