ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-04-29 19:01:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fused delta net 3 (#1333)

Browse Source 
* This is better than chunked

* Keep the state in registers

* Cleanup

* Remove unused stuff

* Minor

* Make fused delta-net the default

* Fix race
This commit is contained in:
Kawrakow
2026-02-27 15:02:56 +01:00
committed by GitHub
parent 1e6d36b1b4
commit 7b68353e09
5 changed files with 101 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
common/common.h
View File

@@ -359,7 +359,7 @@ struct gpt_params {
bool split_mode_graph_scheduling = false; // if true, force split mode graph scheduling
//bool split_mode_f16 = true; // if true, intermediate results will be cast to f16 before copying to other GPUs to perform reduce ops
bool scheduler_async = false; // if true, in split mode graph the scheduler will use multiple threads to evaluate the graph
int fused_delta_net = 0; // use fused delta-net if number of tokens in the batch is less than this value
int fused_delta_net = 65536; // use fused delta-net if number of tokens in the batch is less than this value
bool has_mtp = false; // enable MTP if supported by the model
std::string cache_type_k = "f16"; // KV cache data type for the K