turboderp-org/exllamav2
1
0
Fork 0
mirror of https://github.com/turboderp-org/exllamav2.git synced 2026-04-20 14:29:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix FP32 residual for paged attn

Browse Source
This commit is contained in:
turboderp
2025-03-14 23:09:31 +01:00
parent d8fa1a8250
commit 23395dfa42
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
exllamav2/attn.py
View File

@@ -628,7 +628,7 @@ class ExLlamaV2Attention(ExLlamaV2Module):
else:
hidden_states = self.o_proj.forward(attn_output, loras = loras)
if self.post_layernorm:
hidden_states = self.post_layernorm.forward(hidden_states)
hidden_states = self.post_layernorm.forward(hidden_states, output_fp32 = self.archparams.residual_stream_fp32)
if self.has_residual:
hidden_states += residual