theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-20 14:28:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Model: Dynamically scale generate_window

Browse Source 
Allows for adjustment of reservation space at the end of the context
before rolling it. This should be scaled as a model's max_seq_len
goes up.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-01-24 01:26:38 -05:00
committed by Brian Dashore
parent b14c5443fd
commit 740b0215dd
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
backends/exllamav2/model.py
View File

@@ -554,7 +554,9 @@ class ExllamaV2Container:
token_healing = unwrap(kwargs.get("token_healing"), False)
max_tokens = unwrap(kwargs.get("max_tokens"), 150)
stream_interval = unwrap(kwargs.get("stream_interval"), 0)
generate_window = min(unwrap(kwargs.get("generate_window"), 512), max_tokens)
generate_window = max(
unwrap(kwargs.get("generate_window"), 512), max_tokens // 8
)
# Sampler settings
gen_settings = ExLlamaV2Sampler.Settings()