Iwan Kawrakow 379ca23e1d FA: much better bf16 kv-cache speed for large contexts ...

We now hit 122 t/s for LLaMA-3.1-8B (quantized as iq4_xs and
run-time-repacked) with a context of 32768. IIRC, the previous
best for such large context was ~90 t/s.
Non-negligible improvement at 16384 and 8192 as well:
173.4 and 214 t/s.

2025-01-14 18:14:47 +02:00

..

cmake

Merge mainline llama.cpp (#3)

2024-07-27 07:55:01 +02:00

include

Be able to re-quantize MS BitNet I2_S models (#169)

2025-01-10 18:18:04 +02:00

src

FA: much better bf16 kv-cache speed for large contexts

2025-01-14 18:14:47 +02:00

.gitignore

Merge mainline llama.cpp (#3)

2024-07-27 07:55:01 +02:00

CMakeLists.txt

Move to c++17 projectwide (#80)

2024-10-04 14:43:26 +03:00