ik_llama.cpp/github-data/pull_requests/39 - Add support for bf16 to iqk_mul_mat.md

🔀 #39 - Add support for bf16 to iqk_mul_mat

Author	ikawrakow
State	❌ Closed
Created	2024-09-04
Updated	2024-09-05

---

Description

Only when natively supported (e.g., Zen4), else left to ggml to handle.

For LLaMA-3.1-8B we get PP512 = 205 t/s vs 74 t/s in llama.cpp on my Ryzen-7950X CPU.

I get 204 t/s with llamafile, so I guess Justine Tunney has not contributed the more recent tinyBLAS improvements to llama.cpp.