mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-04-20 22:49:31 +00:00
Add GitHub data: filename sanitization (#640)
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
### 🔀 [#195](https://github.com/ikawrakow/ik_llama.cpp/pull/195) - Deepseek MLA Optimizations V2
|
||||
|
||||
| **Author** | `saood06` |
|
||||
| :--- | :--- |
|
||||
| **State** | ❌ **Closed** |
|
||||
| **Created** | 2025-02-08 |
|
||||
| **Updated** | 2025-02-09 |
|
||||
|
||||
---
|
||||
|
||||
#### Description
|
||||
|
||||
@ikawrakow
|
||||
|
||||
This PR contains the following things
|
||||
- A fairydreaming commit that is supposed to increase PP
|
||||
- Avoid allocating the MHA KV cache in MLA mode
|
||||
- Adds a change I originally missed that is used for gguf-py.
|
||||
|
||||
I will follow up with:
|
||||
- Having all the MoE experts load during warmup, that can be placed in this PR if you want, or a separate one. It is a very large QoL feature for large MoE. Without it the model is slowly loaded in on use, with it, the model is loaded immediately and at a faster rate.
|
||||
- The mmap based KV cache buffer, it is functional but I have yet to make it a CLI option.
|
||||
|
||||
---
|
||||
|
||||
#### 💬 Conversation
|
||||
|
||||
👤 **ikawrakow** submitted a review the **2025-02-09** at **07:36:43**: ✅ `APPROVED`<br>
|
||||
|
||||
Looks good. I added a minor change to check if `wk_b` and `wv_b` are available before turning on MLA (so we don't crash if someone is using an old model and asked for MLA).
|
||||
|
||||
PP-4096 for `Q8_0_R8` quantized DeepSeek-Lite with `-mla` goes up to 292 t/s from 275 t/s with this change.
|
||||
Reference in New Issue
Block a user