Config: Switch to YAML and add load progress

YAML is a more flexible format when it comes to configuration. Commandline
arguments are difficult to remember and configure especially for
an API with complicated commandline names. Rather than using half-baked
textfiles, implement a proper config solution.

Also add a progress bar when loading models in the commandline.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2023-11-12 00:21:16 -05:00
parent 5d32aa02cd
commit a10c14d357
6 changed files with 38 additions and 26 deletions

View File

@@ -34,7 +34,6 @@ class ModelContainer:
gpu_split: list or None = None
def __init__(self, model_directory: str, quiet = False, **kwargs):
print(kwargs)
"""
Create model container
@@ -76,6 +75,9 @@ class ModelContainer:
if "rope_alpha" in kwargs: self.config.scale_alpha_value = kwargs["rope_alpha"]
if "no_flash_attn" in kwargs: self.config.no_flash_attn = kwargs["no_flash_attn"]
if "low_mem" in kwargs and kwargs["low_mem"]:
self.config.set_low_mem()
chunk_size = min(kwargs.get("chunk_size", 2048), self.config.max_seq_len)
self.config.max_input_len = chunk_size
self.config.max_attn_size = chunk_size ** 2