Model: Reject model requests when unloading

If a model is being unloaded, that means its being shut down and
no requests should be accepted from then on.

Also, remove model_is_loaded since we simply check if the container
is None now.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri
2025-04-19 22:34:06 -04:00
parent 552a64c723
commit f15ac1f69d
2 changed files with 13 additions and 15 deletions

View File

@@ -14,7 +14,6 @@ if dependencies.extras:
class InfinityContainer:
model_dir: pathlib.Path
model_is_loading: bool = False
model_loaded: bool = False
# Use a runtime type hint here
@@ -24,8 +23,6 @@ class InfinityContainer:
self.model_dir = model_directory
async def load(self, **kwargs):
self.model_is_loading = True
# Use cpu by default
device = unwrap(kwargs.get("embeddings_device"), "cpu")