mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-03-14 15:57:27 +00:00
5614b342a7
This will auto-publish to the Github wiki via an action. Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
46 lines
2.5 KiB
Markdown
46 lines
2.5 KiB
Markdown
## Usage
|
|
TabbyAPI's main use-case is to be an API server for running ExllamaV2 models.
|
|
|
|
### API Server
|
|
Currently TabbyAPI supports clients that use the [OpenAI](https://platform.openai.com/docs/api-reference) standard and [KoboldAI](https://lite.koboldai.net/koboldcpp_api)'s API.
|
|
|
|
In addition, there are expanded parameters to generation endpoints along with administrative endpoints for loading, unloading, loras, sampling overrides, etc.
|
|
|
|
> [!NOTE]
|
|
> If you are a developer and want to add full TabbyAPI support to your app, it's recommended to use the [autogenerated documentation](https://theroyallab.github.io/tabbyAPI).
|
|
|
|
Below is an example CURL request using the OpenAI completions endpoint:
|
|
```
|
|
curl http://localhost:5000/v1/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "meta-llama/Meta-Llama-3-8B",
|
|
"prompt": "Once upon a time,",
|
|
"max_tokens": 400,
|
|
"stream": false,
|
|
"min_p": 0.05,
|
|
"repetition_penalty": 1.05
|
|
}'
|
|
```
|
|
|
|
### Authentication
|
|
Every call to a TabbyAPI endpoint requires some form of authentication. Keys have two types of permissions:
|
|
- API: Accesses non-invasive endpoints (ex. generation, model list fetching)
|
|
- Admin: Allowed to access protected endpoints that deal with resources (ex. loading, unloading)
|
|
|
|
In addition, when calling list endpoints, API keys will only fetch the currently loaded object while admin keys will list the entire directory. For example, calling `/v1/models` will return a list of the user-configured models directory only if an admin key is passed.
|
|
|
|
Therefore, it's recommended to keep the admin key for yourself and only share the api key with users.
|
|
|
|
If these keys get compromised, shut down your server, delete the `api_tokens.yml` file, and restart. This will generate new keys which you can share with users.
|
|
|
|
To bypass authentication checks, set `disable_auth` to `True` in config.yml. However, turning off authentication without a third-party solution will make your instance open to the world.
|
|
|
|
### Difficult to get started?
|
|
Is the API difficult? Don't want to load models with `config.yml`? That's okay! Not everyone is a master user of AI products when starting out.
|
|
|
|
For newer users, it's recommended to use a UI that allows for managing TabbyAPI via API endpoints.
|
|
|
|
To find UI projects, take a look at [Community Projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects) for more information.
|
|
|
|
The [Discord](https://discord.gg/sYQxnuD7Fj) is also a great place to ask for help. Please be nice when asking questions as all the developers are volunteers who have lives outside of TabbyAPI. |