theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-03-15 00:07:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Model: Add override base seq len

Browse Source 
Some models (such as mistral and mixtral) set their base sequence
length to 32k due to assumptions of support for sliding window
attention.

Therefore, add this parameter to override the base sequence length
of a model which helps with auto-calculation of rope alpha.

If auto-calculation of rope alpha isn't being used, the max_seq_len
parameter works fine as is.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2023-12-20 00:43:19 -05:00
parent 5368ed7b64
commit ab10b263fd
3 changed files with 19 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
model.py
View File

@@ -85,14 +85,19 @@ class ModelContainer:
self.config.max_seq_len = 4096
self.config.prepare()
# Then override the max_seq_len if present
override_max_seq_len = kwargs.get("max_seq_len")
if override_max_seq_len:
self.config.max_seq_len = kwargs.get("max_seq_len")
# Then override the base_seq_len if present
override_base_seq_len = kwargs.get("override_base_seq_len")
if override_base_seq_len:
self.config.max_seq_len = override_base_seq_len
# Grab the base model's sequence length before overrides for rope calculations
base_seq_len = self.config.max_seq_len
# Set the target seq len if present
target_max_seq_len = kwargs.get("max_seq_len")
if target_max_seq_len:
self.config.max_seq_len = target_max_seq_len
# Set the rope scale
self.config.scale_pos_emb = unwrap(kwargs.get("rope_scale"), 1.0)