Model: Read scale_pos_emb from config

In newer versions of exllamav2, this value is read from the model's
config.json. This value will still default to 1.0 anyways.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2023-12-28 21:14:24 -05:00
parent e70729b0c0
commit ec929728d9
2 changed files with 4 additions and 1 deletions

View File

@@ -126,7 +126,9 @@ class ModelContainer:
self.config.max_seq_len = target_max_seq_len
# Set the rope scale
self.config.scale_pos_emb = unwrap(kwargs.get("rope_scale"), 1.0)
self.config.scale_pos_emb = unwrap(
kwargs.get("rope_scale"), self.config.scale_pos_emb
)
# Automatically calculate rope alpha
self.config.scale_alpha_value = unwrap(