mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-07-17 00:57:53 +00:00
c8b9a8f4aa
Remove BaseModelContainer abstraction LoRA endpoints remain as stubs (supported in exllamav3, but API is undecided) Fix /v1/lora/unload unloading the entire model. The last commit with exllamav2 support is preserved on the exl2-checkpoint branch.
22 lines
1.9 KiB
Markdown
22 lines
1.9 KiB
Markdown
## FAQ
|
|
- What OS is supported?
|
|
- Windows and Linux
|
|
|
|
- I'm confused, how do I do anything with this API?
|
|
- That's okay! Not everyone is an AI mastermind on their first try. The start scripts and `config.yml` aim to guide new users to the right configuration. The [Usage](https://github.com/theroyallab/tabbyAPI/wiki/03.-Usage) page explains how the API works. [Community Projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects) contain UIs that help interact with TabbyAPI via API endpoints. The [Discord Server](https://discord.gg/sYQxnuD7Fj) is also a place to ask questions, but please be nice.
|
|
|
|
- How do I interface with the API?
|
|
- The wiki is meant for user-facing documentation. Devs are recommended to use the autogenerated documentation for [OpenAI](https://theroyallab.github.io/tabbyAPI) and [Kobold](https://theroyallab.github.io/tabbyAPI/kobold) servers
|
|
|
|
- What does TabbyAPI run?
|
|
- TabbyAPI uses Exllamav3 as a powerful and fast backend for model inference, loading, etc. Therefore, the following types of models are supported:
|
|
- Exl3 (Highly recommended)
|
|
- FP16/BF16 (using Exllamav3's loader)
|
|
|
|
- Exllamav3 may error with the following exception: `ImportError: DLL load failed while importing exllamav3_ext: The specified module could not be found.`
|
|
- First, make sure to check if the wheel is equivalent to your python version and CUDA version. Also make sure you're in a venv or conda environment.
|
|
- If those prerequisites are correct, the torch cache may need to be cleared. This is due to a mismatching exllamav3_ext.
|
|
- In Windows: Find the cache at `C:\Users\<User>\AppData\Local\torch_extensions\torch_extensions\Cache` where `<User>` is your Windows username
|
|
- In Linux: Find the cache at `~/.cache/torch_extensions`
|
|
- look for any folder named `exllamav3_ext` in the python subdirectories and delete them.
|
|
- Restart TabbyAPI and launching should work again. |