mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-03-03 10:30:27 +00:00
32 lines
1.2 KiB
Markdown
32 lines
1.2 KiB
Markdown
### 🔀 [#195](https://github.com/ikawrakow/ik_llama.cpp/pull/195) - Deepseek MLA Optimizations V2
|
|
|
|
| **Author** | `saood06` |
|
|
| :--- | :--- |
|
|
| **State** | ❌ **Closed** |
|
|
| **Created** | 2025-02-08 |
|
|
| **Updated** | 2025-02-09 |
|
|
|
|
---
|
|
|
|
#### Description
|
|
|
|
@ikawrakow
|
|
|
|
This PR contains the following things
|
|
- A fairydreaming commit that is supposed to increase PP
|
|
- Avoid allocating the MHA KV cache in MLA mode
|
|
- Adds a change I originally missed that is used for gguf-py.
|
|
|
|
I will follow up with:
|
|
- Having all the MoE experts load during warmup, that can be placed in this PR if you want, or a separate one. It is a very large QoL feature for large MoE. Without it the model is slowly loaded in on use, with it, the model is loaded immediately and at a faster rate.
|
|
- The mmap based KV cache buffer, it is functional but I have yet to make it a CLI option.
|
|
|
|
---
|
|
|
|
#### 💬 Conversation
|
|
|
|
👤 **ikawrakow** submitted a review the **2025-02-09** at **07:36:43**: ✅ `APPROVED`<br>
|
|
|
|
Looks good. I added a minor change to check if `wk_b` and `wv_b` are available before turning on MLA (so we don't crash if someone is using an old model and asked for MLA).
|
|
|
|
PP-4096 for `Q8_0_R8` quantized DeepSeek-Lite with `-mla` goes up to 292 t/s from 275 t/s with this change. |