mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-07-17 00:57:53 +00:00
c8b9a8f4aa
Remove BaseModelContainer abstraction LoRA endpoints remain as stubs (supported in exllamav3, but API is undecided) Fix /v1/lora/unload unloading the entire model. The last commit with exllamav2 support is preserved on the exl2-checkpoint branch.
1.9 KiB
1.9 KiB
FAQ
-
What OS is supported?
- Windows and Linux
-
I'm confused, how do I do anything with this API?
- That's okay! Not everyone is an AI mastermind on their first try. The start scripts and
config.ymlaim to guide new users to the right configuration. The Usage page explains how the API works. Community Projects contain UIs that help interact with TabbyAPI via API endpoints. The Discord Server is also a place to ask questions, but please be nice.
- That's okay! Not everyone is an AI mastermind on their first try. The start scripts and
-
How do I interface with the API?
-
What does TabbyAPI run?
- TabbyAPI uses Exllamav3 as a powerful and fast backend for model inference, loading, etc. Therefore, the following types of models are supported:
- Exl3 (Highly recommended)
- FP16/BF16 (using Exllamav3's loader)
- TabbyAPI uses Exllamav3 as a powerful and fast backend for model inference, loading, etc. Therefore, the following types of models are supported:
-
Exllamav3 may error with the following exception:
ImportError: DLL load failed while importing exllamav3_ext: The specified module could not be found.- First, make sure to check if the wheel is equivalent to your python version and CUDA version. Also make sure you're in a venv or conda environment.
- If those prerequisites are correct, the torch cache may need to be cleared. This is due to a mismatching exllamav3_ext.
- In Windows: Find the cache at
C:\Users\<User>\AppData\Local\torch_extensions\torch_extensions\Cachewhere<User>is your Windows username - In Linux: Find the cache at
~/.cache/torch_extensions - look for any folder named
exllamav3_extin the python subdirectories and delete them. - Restart TabbyAPI and launching should work again.
- In Windows: Find the cache at