theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-20 06:19:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

OAI: Add cancellation with inline load

Browse Source 
When the request is cancelled, cancel the load task. In addition,
when checking if a model container exists, also check if the model
is fully loaded.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-09-11 00:08:55 -04:00
parent b9e5693c1b
commit e00eb09ef3
2 changed files with 14 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
endpoints/OAI/utils/completion.py
View File

@@ -112,8 +112,12 @@ async def _stream_collector(
async def load_inline_model(model_name: str, request: Request):
"""Load a model from the data.model parameter"""
# Return if the model container already exists
if model.container and model.container.model_dir.name == model_name:
# Return if the model container already exists and the model is fully loaded
if (
model.container
and model.container.model_dir.name == model_name
and model.container.model_loaded
):
return
# Inline model loading isn't enabled or the user isn't an admin