ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-04-22 23:49:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Adding IQ1_TN - 1.6875 bpw for TriLM ternary models (#44)

Browse Source 
* Adding iq1_tn - 1.6875 bpw for TriLM ternary models

* iq1_tn: NEON

* iq1_tn: faster NEON

* iq2_bn: improve performance on NEON

We now get TG-128 = 100 t/s for Bitnet-3B-1.58b!

* iq1_tn: improve AVX2

PP-512 goes to 533 t/s up from 455.
TG-128 @ 2 threads goes to 16.6 t/s up from 14.2.
However, we seem to have a bottleneck somewhere as
TG saturates at 8 threads.

* iq1_tn: improve Zen4

PP-512 goes to 485 t/s up from 352. With FA we get 545 t/s up from 380.
TG-128 @ 1 thread goes to 12.4 t/s up from 10.4.
However, we seem to have a bottleneck somewhere as
TG saturates at 8 threads.

* iq2_bn: improve on Zen4

We now get PP-512 = 614 t/s up from 542 t/s

* iq2_bn: improve AVX2 implementation

We now get PP-512 = 753 t/s up from 680 t/s.

* Remove unnecessary barrier in ggml_compute_forward_mul_mat

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
This commit is contained in:
Kawrakow
2024-09-09 14:56:34 +03:00
committed by GitHub
parent f2ef628e4e
commit 4a5d5e207d
10 changed files with 304 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
include/llama.h
View File

@@ -166,14 +166,16 @@ extern "C" {
LLAMA_FTYPE_MOSTLY_Q4_0_4_4 = 33, // except 1d tensors
LLAMA_FTYPE_MOSTLY_Q4_0_4_8 = 34, // except 1d tensors
LLAMA_FTYPE_MOSTLY_Q4_0_8_8 = 35, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ1_BN = 36, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_BN = 37, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_K = 38, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ3_K = 39, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ4_K = 40, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ5_K = 41, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ6_K = 42, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_TN = 43, // except 1d tensors
//
LLAMA_FTYPE_MOSTLY_IQ1_BN = 136, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_BN = 137, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_K = 138, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ3_K = 139, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ4_K = 140, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ5_K = 141, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ6_K = 142, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ2_TN = 143, // except 1d tensors
LLAMA_FTYPE_MOSTLY_IQ1_TN = 144, // except 1d tensors
LLAMA_FTYPE_GUESSED = 1024, // not specified in the model file
};