API: Don't fallback to default values on model load request

It's best to pass them down the config stack.

API/User config.yml -> model config.yml -> model config.json -> fallback.

Doing this allows for seamless flow and yielding control to each
member in the stack.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-08-30 12:10:48 -04:00
committed by Brian Dashore
parent 4452d6f665
commit a96fa5f138
4 changed files with 19 additions and 17 deletions

View File

@@ -95,8 +95,10 @@ async def stream_model_load(
):
"""Request generation wrapper for the loading process."""
# Get trimmed load data
load_data = data.model_dump(exclude_none=True)
# Set the draft model path if it exists
load_data = data.model_dump()
if draft_model_path:
load_data["draft"]["draft_model_dir"] = draft_model_path