ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-05-03 21:01:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add GitHub data: filename sanitization (#640)

Browse Source
This commit is contained in:
Thomas
2025-07-23 13:31:53 +02:00
committed by GitHub
parent 3600d82e98
commit eaa2510a28
626 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
github-data/pull_requests/38 - Zen4 Flash Attention - bf16 support.md Normal file
View File

@@ -0,0 +1,20 @@
### 🔀 [#38](https://github.com/ikawrakow/ik_llama.cpp/pull/38) - Zen4 Flash Attention - bf16 support
| **Author** | `ikawrakow` |
| :--- | :--- |
| **State** | ❌ **Closed** |
| **Created** | 2024-09-04 |
| **Updated** | 2024-09-05 |
---
#### Description
This PR adds support for using `bf16` for the kv-cache.
As Zen4 has native support for `bf16` fused-multiply-add, I was hoping that this might give better performance than `fp16`. But with this implementation it is basically the same as `fp16`. We get a tiny improvement for Gemma2-2b at 4k and 8k tokens as shown in this graph (there is no `bf16` support for kv-cache in `llama.cpp`, so no comparison in the graph).
![fa_gemma2b](https://github.com/user-attachments/assets/8f104aeb-563d-46c8-a661-18ddd93ffe28)
Given this outcome, I have only enabled support for K- and V-cache both as `bf16` (i.e., one cannot mix `bf16` with other types as it is possible with `fp16`, `Q4_0`, `Q4_1` and `Q8_0`.