ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-03-13 15:30:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
25cd985c9b238d7399eaa44a75f9e36ca0cef8cc
ik_llama.cpp/ggml/src/ggml-cuda
History
Kawrakow 86e2bec04e DeepSeek FA optimizations (#929) 
* Use new-new-mma also for MLA=3, and use mask bounds

This gives us ~25% better PP at 32k tokens compared to main

* This seems better

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2025-11-10 09:55:30 +02:00
..
template-instances
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
vendors
CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728)
2025-08-26 13:30:35 +03:00
acc.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
acc.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
add-id.cu
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
add-id.cuh
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
arange.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
arange.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
argmax.cu
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
argmax.cuh
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
argsort.cu
Fuse add+add+fused_rms (#853)
2025-10-22 16:18:11 +03:00
argsort.cuh
Various fused ops around expert selection (#840)
2025-10-19 19:02:46 +03:00
binbcast.cu
Even more fused ops (#868)
2025-10-27 16:09:01 +02:00
binbcast.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
clamp.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
clamp.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
common.cuh
CUDA: set compute parameters via command line arguments (#910)
2025-11-07 07:11:23 +02:00
concat.cu
Add copyright notices (#317)
2025-04-07 10:43:26 +02:00
concat.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
conv2d-dw.cu
Port mdmd from mainline + Qwen2/2.5-VL support (#798)
2025-09-27 08:45:29 +02:00
conv2d-dw.cuh
Port mdmd from mainline + Qwen2/2.5-VL support (#798)
2025-09-27 08:45:29 +02:00
conv2d.cu
Port mdmd from mainline + Qwen2/2.5-VL support (#798)
2025-09-27 08:45:29 +02:00
conv2d.cuh
Port mdmd from mainline + Qwen2/2.5-VL support (#798)
2025-09-27 08:45:29 +02:00
conv-transpose-1d.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
conv-transpose-1d.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
convert.cu
MXFP4 (#682)
2025-08-09 08:40:18 +03:00
convert.cuh
Port mdmd from mainline + Qwen2/2.5-VL support (#798)
2025-09-27 08:45:29 +02:00
cp-async.cuh
Faster DeepSeek FA on CUDA (#408)
2025-05-12 07:49:00 +03:00
cpy-utils.cuh
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
cpy.cu
DeepSeek TG optimizations for TG (#928)
2025-11-10 09:52:07 +02:00
cpy.cuh
DeepSeek TG optimizations for TG (#928)
2025-11-10 09:52:07 +02:00
dequantize.cuh
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
diagmask.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
diagmask.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
dmmv.cu
Trellis quants with CPU inference (#441)
2025-05-23 09:17:52 +03:00
dmmv.cuh
Merge mainline - Aug 12 2024 (#17)
2024-08-12 15:14:32 +02:00
fattn-common.cuh
Refactor CUDA flash attention (#745)
2025-09-02 10:12:56 +02:00
fattn-mma-f16-interface.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-mma-f16.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-mma-f16.cuh
Fix #772 (#790)
2025-09-23 16:43:02 +02:00
fattn-new-mma.cu
DeepSeek FA optimizations (#929)
2025-11-10 09:55:30 +02:00
fattn-new-mma.cuh
FlashMLA-3 for DeepSeek models on CUDA (#386)
2025-05-07 17:38:22 +03:00
fattn-tile-f16.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-tile-f16.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-tile-f32.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-tile-f32.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-vec-common.cuh
Alternative CUDA FA for SWA models (#754)
2025-09-04 08:42:18 +02:00
fattn-vec-f16-interface.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-vec-f16.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-vec-f16.cuh
Alternative CUDA FA for SWA models (#754)
2025-09-04 08:42:18 +02:00
fattn-vec-f32-interface.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-vec-f32.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-vec-f32.cuh
Alternative CUDA FA for SWA models (#754)
2025-09-04 08:42:18 +02:00
fattn-wmma-f16-interface.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-wmma-f16.cu
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
fattn-wmma-f16.cuh
Fix GLM-4.5 attention (#700)
2025-08-17 14:31:03 +03:00
fattn.cu
DeepSeek FA optimizations (#929)
2025-11-10 09:55:30 +02:00
fattn.cuh
CUDA: corectly detect if flash attention is supported (#875)
2025-10-29 13:56:16 +02:00
getrows.cu
Add copyright notices (#317)
2025-04-07 10:43:26 +02:00
getrows.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
graph.cuh
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
im2col.cu
Fix gemma3 vision (#803)
2025-09-27 11:15:32 +02:00
im2col.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
iqk_cuda_common.h
CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716)
2025-08-22 07:25:35 +03:00
iqk_mmvq_templates.cuh
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
iqk_mmvq.cu
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
iqk_mmvq.cuh
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
mma_new.cuh
CUDA: faster FA TG for GQA models (#370)
2025-05-04 09:17:44 +03:00
mma.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
mmq_id_common.cuh
Adopt fix from mainline PR 17089 (#920)
2025-11-08 07:44:20 +02:00
mmq_id.cu
CUDA MoE improvements (#923)
2025-11-09 11:34:33 +02:00
mmq_id.cuh
Revert "CUDA: prompt processing optimizations for MoE models (#739)" (#748)
2025-09-02 06:55:48 +02:00
mmq.cu
CUDA MoE improvements (#923)
2025-11-09 11:34:33 +02:00
mmq.cuh
Fused FFN_UP+FFN_GATE op (#741)
2025-08-31 18:16:36 +03:00
mmvq-args.h
CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864)
2025-10-26 17:08:50 +02:00
mmvq-templates.cuh
Biased mmvq: minor optimization (#880)
2025-10-31 14:21:18 +02:00
mmvq.cu
CUDA MoE improvements (#923)
2025-11-09 11:34:33 +02:00
mmvq.cuh
Even more fused ops (#868)
2025-10-27 16:09:01 +02:00
multiadd.cu
Fused mul + multi_add op (#858)
2025-10-24 07:40:35 +03:00
multiadd.cuh
Fused mul + multi_add op (#858)
2025-10-24 07:40:35 +03:00
norm.cu
Fused Q and K fused_rms_norm for TG on CUDA (#882)
2025-10-31 14:41:28 +02:00
norm.cuh
Fused Q and K fused_rms_norm for TG on CUDA (#882)
2025-10-31 14:41:28 +02:00
pad.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pad.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pool2d.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
pool2d.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
quantize_id.cu
CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728)
2025-08-26 13:30:35 +03:00
quantize_id.cuh
CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728)
2025-08-26 13:30:35 +03:00
quantize.cu
CUDA MoE improvements (#923)
2025-11-09 11:34:33 +02:00
quantize.cuh
CUDA MoE improvements (#923)
2025-11-09 11:34:33 +02:00
rope.cu
Port of Qwen3-VL support from mainline (#883)
2025-11-04 19:20:54 +02:00
rope.cuh
RoPE cache (#887)
2025-11-03 18:42:20 +02:00
scale.cu
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
scale.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
set-rows.cu
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
set-rows.cuh
Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833)
2025-10-15 14:20:40 +03:00
softcap.cu
Add copyright notices (#317)
2025-04-07 10:43:26 +02:00
softcap.cuh
Add copyright notices (#317)
2025-04-07 10:43:26 +02:00
softmax.cu
Enable CUDA graphs for MoE models + GPT-OSS support (#689)
2025-08-15 09:18:07 +03:00
softmax.cuh
Add copyright notices (#317)
2025-04-07 10:43:26 +02:00
sumrows.cu
Fuse add+add+fused_rms (#853)
2025-10-22 16:18:11 +03:00
sumrows.cuh
Various fused ops around expert selection (#840)
2025-10-19 19:02:46 +03:00
topk-moe.cu
cuda: fused top_k+softmax as used in most MoE models (#789)
2025-09-23 13:45:57 +02:00
topk-moe.cuh
cuda: fused top_k+softmax as used in most MoE models (#789)
2025-09-23 13:45:57 +02:00
tsembd.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
tsembd.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
unary.cu
Fused mul + multi_add op (#858)
2025-10-24 07:40:35 +03:00
unary.cuh
Fuse add+add+fused_rms (#853)
2025-10-22 16:18:11 +03:00
upscale.cu
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
upscale.cuh
Merge mainline llama.cpp (#3)
2024-07-27 07:55:01 +02:00
vecdotq.cuh
CUDA: faster prompt processing for 4-bit quants (#713)
2025-08-21 15:57:35 +03:00