This helps clarify things when users are configuring for the first
time. For example, some users were putting the model name in the
"model" block instead of the "model_name" field.
Signed-off-by: kingbri <bdashore3@proton.me>
Models can be loaded with a child object called "draft" in the POST
request. Again, models need to be located within the draft model dir
to get loaded.
Signed-off-by: kingbri <bdashore3@proton.me>
Model: Add extra information to print and fix the divide by zero error.
Auth: Fix validation of API and admin keys to look for the entire key.
References #7 and #6
Signed-off-by: kingbri <bdashore3@proton.me>
Speculative decoding makes use of draft models that ingest the prompt
before forwarding it to the main model.
Add options in the config to support this. API options will occur
in a different commit.
Signed-off-by: kingbri <bdashore3@proton.me>
Stop conditions was None, causing model to error out when trying to
add the EOS token to a None value.
Authentication failed when Bearer contained an empty string. To fix
this, add a condition which checks array length.
Signed-off-by: kingbri <bdashore3@proton.me>
Add the EOS token into stop strings after checking kwargs. If
ban_eos_token is on, don't add the EOS token in for extra measure.
Signed-off-by: kingbri <bdashore3@proton.me>
Responses were not being properly sent as JSON. Only run pydantic's
JSON function on stream responses. FastAPI does the rest with static
responses.
Signed-off-by: kingbri <bdashore3@proton.me>
Add safe fallbacks if any part of the config tree doesn't exist. This
prevents random internal server errors from showing up.
Signed-off-by: kingbri <bdashore3@proton.me>
Fastchat requires a lot of dependencies such as transformers, peft,
and accelerate which are heavy. This is not useful unless a user
wants to add a shim for the chat completion endpoint.
Instead, try importing fastchat and notify the console of the error.
Signed-off-by: kingbri <bdashore3@proton.me>
Chat completions is the endpoint that will be used by OAI in the
future. Makes sense to support it even though the completions
endpoint will be used more often.
Also unify common parameters between the chat completion and completion
requests since they're very similar.
Signed-off-by: kingbri <bdashore3@proton.me>
These were not working properly. Make the YAML file get written
to properly and the validator to return a 401 when the bearer
token is invalid.
Signed-off-by: kingbri <bdashore3@proton.me>
