mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-04-26 09:18:53 +00:00
Model: Fix max seq len handling
Previously, the max sequence length was overriden by the user's config and never took the model's config.json into account. Now, set the default to 4096, but include config.prepare when selecting the max sequence length. The yaml and API request now serve as overrides rather than parameters. Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
@@ -37,8 +37,8 @@ model:
|
||||
|
||||
# The below parameters apply only if model_name is set
|
||||
|
||||
# Maximum model context length (default: 4096)
|
||||
max_seq_len: 4096
|
||||
# Override maximum model context length (default: None)
|
||||
max_seq_len:
|
||||
|
||||
# Automatically allocate resources to GPUs (default: True)
|
||||
gpu_split_auto: True
|
||||
|
||||
Reference in New Issue
Block a user