API: Don't fallback to default values on model load request

It's best to pass them down the config stack.

API/User config.yml -> model config.yml -> model config.json -> fallback.

Doing this allows for seamless flow and yielding control to each
member in the stack.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-08-30 12:10:48 -04:00
committed by Brian Dashore
parent 4452d6f665
commit a96fa5f138
4 changed files with 19 additions and 17 deletions

View File

@@ -361,6 +361,8 @@ class ExllamaV2Container:
self.draft_config.max_attention_size = chunk_size**2
def set_model_overrides(self, **kwargs):
"""Sets overrides from a model folder's config yaml."""
override_config_path = self.model_dir / "tabby_config.yml"
if not override_config_path.exists():