theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-03-15 00:07:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Config: Embeddings: Make embeddings_device a default when API loading

Browse Source 
When loading from the API, the fallback for embeddings_device will be
the same as the config.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-08-01 13:59:49 -04:00
parent 54aeebaec1
commit 3e42211c3e
3 changed files with 30 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
config_sample.yml
View File

@@ -209,11 +209,14 @@ embeddings:
# Overrides directory to look for embedding models (default: models)
embedding_model_dir: models
# An initial embedding model to load on the infinity backend (default: None)
embedding_model_name:
# Device to load embedding models on (default: cpu)
# Possible values: cpu, auto, cuda
# NOTE: It's recommended to load embedding models on the CPU.
# If you'd like to load on an AMD gpu, set this value to "cuda" as well.
embeddings_device: cpu
# The below parameters only apply for initial loads
# All API based loads do NOT inherit these settings unless specified in use_as_default
# An initial embedding model to load on the infinity backend (default: None)
embedding_model_name: