ikawrakow/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-04-27 18:01:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add GitHub data: filename sanitization (#640)

Browse Source
This commit is contained in:
Thomas
2025-07-23 13:31:53 +02:00
committed by GitHub
parent 3600d82e98
commit eaa2510a28
626 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
github-data/discussions/165 - Norm RMS Epsilon.md Normal file
View File

@@ -0,0 +1,28 @@
### 🗣️ [#165](https://github.com/ikawrakow/ik_llama.cpp/discussions/165) - Norm RMS Epsilon
| **Author** | `Nexesenex` |
| :--- | :--- |
| **Created** | 2024-12-25 |
| **Updated** | 2024-12-27 |
---
#### Description
While it crosses my mind..
@Ikawrakow : a while ago, you made some measurements with variations of Norm RMS Epsilon which showed some little benefits to offset it for <2bpw quants. It was on L2 I believe, and I wonder if it applies to other arches, and if yes, if there's some sort of "formula" which would come with it to improve the low bitrate quants themselves.
Just beotian thoughts.
And merry XMAS btw, if you celebrate it!
---
#### 🗣️ Discussion
👤 **ikawrakow** replied the **2024-12-27** at **17:44:24**:<br>
I'm travelling, so just quickly from the phone.
Yes, there is a small benefit from increasing rms_eps also for LlaMA-3, but only for very low-bit quants (IQ2_XXS). No, I haven't done any kind of systematic investigation.