theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-20 14:28:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Model: Fix generate window fallback

Browse Source 
Use max_seq_len as the numerator, not the max_tokens. Mismatched
parameter.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-02-06 14:48:42 -05:00
parent 543a9b68c8
commit fedebadc81
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
backends/exllamav2/model.py
View File

@@ -515,7 +515,7 @@ class ExllamaV2Container:
max_tokens = unwrap(kwargs.get("max_tokens"), 150)
stream_interval = unwrap(kwargs.get("stream_interval"), 0)
generate_window = max(
unwrap(kwargs.get("generate_window"), 512), max_tokens // 8
unwrap(kwargs.get("generate_window"), 512), self.config.max_seq_len // 8
)
# Sampler settings