diff --git a/docs/01.-Getting-Started.md b/docs/01.-Getting-Started.md index c43008d..32a49fe 100644 --- a/docs/01.-Getting-Started.md +++ b/docs/01.-Getting-Started.md @@ -5,15 +5,12 @@ To get started, make sure you have the following installed on your system: - [Python 3.x](https://www.python.org/downloads/release/python-3117/) (preferably 3.11) with pip - Do NOT install python from the Microsoft store! This will cause issues with pip. - - Alternatively, you can use miniconda if it's present on your system. + - Alternatively, you can use miniconda or uv if it's present on your system. > [!NOTE] > Prefer a video guide? Watch the step-by-step tutorial on [YouTube](https://www.youtube.com/watch?v=03jYz0ijbUU) -> [!NOTE] -> You can install [miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) on your system which will give you the benefit of having both python and conda! - > [!WARNING] > CUDA and ROCm aren't prerequisites because torch can install them for you. However, if this doesn't work (ex. DLL load failed), install the CUDA toolkit or ROCm on your system. > @@ -39,26 +36,34 @@ To get started, make sure you have the following installed on your system: 1. Follow the on-screen instructions and select the correct GPU library. 2. Assuming that the prerequisites are installed and can be located, a virtual environment will be created for you and dependencies will be installed. -4. The API should start with no model loaded +4. The API should start with no model loaded. Please read more to see how to download a model. ### For Advanced Users -> [!NOTE] -> TabbyAPI has recently switched to use pyproject.toml. These instructions may look different than before. - -1. Follow steps 1-2 in the [For Beginners](#for-beginners) section -2. Create a python environment through venv: +5. Follow steps 1-2 in the [For Beginners](#for-beginners) section +6. Create a python environment through venv: 1. `python -m venv venv` 2. Activate the venv 1. On Windows: `.\venv\Scripts\activate` 2. On Linux: `source venv/bin/activate` -3. Install the pyproject features based on your system: +7. Install the pyproject features based on your system: 1. Cuda 12.x: `pip install -U .[cu121]` 2. ROCm 5.6: `pip install -U .[amd]` -4. Start the API by either +8. Start the API by either 1. Run `start.bat/sh`. The script will check if you're in a conda environment and skip venv checks. 2. Run `python main.py` to start the API. This won't automatically upgrade your dependencies. +## Download a Model + +TabbyAPI includes a built-in Hugging Face downloader that works via both the API and terminal. You can use the following command to download a repository with a specific branch revision: + +`.\Start.bat download --revision ` + +Example with Turboderp's Llama 3.1 8B quants: + +`.\Start.bat download turboderp/Qwen2.5-VL-7B-Instruct-exl2 --revision 4.0bpw` + +Alternatively, running `main.py` directly can also trigger the downloader. For additional options, run `.\Start.bat download --help` ## Configuration Loading solely the API may not be your optimal usecase. Therefore, a config.yml exists to tune initial launch parameters and other configuration options. @@ -73,19 +78,19 @@ You can also access the configuration parameters under [2. Configuration](https: ## Where next? -1. Take a look at the [usage docs](https://github.com/theroyallab/tabbyAPI/wiki/03.-Usage) -2. Get started with [community projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects): Find loaders, UIs, and more created by the wider AI community. Any OAI compatible client is also supported. +9. Take a look at the [usage docs](https://github.com/theroyallab/tabbyAPI/wiki/03.-Usage) +10. Get started with [community projects](https://github.com/theroyallab/tabbyAPI/wiki/09.-Community-Projects): Find loaders, UIs, and more created by the wider AI community. Any OAI compatible client is also supported. ## Updating There are a couple ways to update TabbyAPI: -1. **Update scripts** - Inside the update_scripts folder, you can run the following scripts: +11. **Update scripts** - Inside the update_scripts folder, you can run the following scripts: 1. `update_deps`: Updates dependencies to their latest versions. 2. `update_deps_and_pull`: Updates dependencies and pulls the latest commit of the Github repository. These scripts exit after running their respective tasks. To start TabbyAPI, run `start.bat` or `start.sh`. -2. **Manual** - Install the pyproject features and update dependencies depending on your GPU: +12. **Manual** - Install the pyproject features and update dependencies depending on your GPU: 1. `pip install -U .[cu121]` = CUDA 12.x 2. `pip install -U .[amd]` = ROCm 6.0 @@ -108,11 +113,11 @@ NOTE: Here are ways to install exllamav2: -1. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended) +13. From a [wheel/release](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension) (Recommended) 1. Find the version that corresponds with your cuda and python version. For example, a wheel with `cu121` and `cp311` corresponds to CUDA 12.1 and python 3.11 -2. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2` - 1. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration. -3. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source) +14. From [pip](https://github.com/turboderp/exllamav2#method-3-install-from-pypi): `pip install exllamav2` + 2. This is a JIT compiled extension, which means that the initial launch of tabbyAPI will take some time. The build may also not work due to improper environment configuration. +15. From [source](https://github.com/turboderp/exllamav2#method-1-install-from-source) ## Other installation methods @@ -123,21 +128,24 @@ These are short-form instructions for other methods that users can use to instal ### Conda -1. Install [Miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) with python 3.11 as your base python -2. Create a new conda environment `conda create -n tabbyAPI python=3.11` -3. Activate the conda environment `conda activate tabbyAPI` -4. Install optional dependencies if they aren't present +16. Install [Miniconda3](https://docs.conda.io/projects/miniconda/en/latest/miniconda-other-installer-links.html) with python 3.11 as your base python +17. Create a new conda environment `conda create -n tabbyAPI python=3.11` +18. Activate the conda environment `conda activate tabbyAPI` +19. Install optional dependencies if they aren't present 1. CUDA via - 1. CUDA 12 - `conda install -c "nvidia/label/cuda-12.2.2" cuda` + 1. CUDA 12 - `conda install -c "nvidia/label/cuda-12.4.1" cuda` 2. Git via `conda install -k git` -5. Clone TabbyAPI via `git clone https://github.com/theroyallab/tabbyAPI` -6. Continue installation steps from: +20. Clone TabbyAPI via `git clone https://github.com/theroyallab/tabbyAPI` +21. Continue installation steps from: 1. [For Beginners](#for-beginners) - Step 3. The start scripts detect if you're in a conda environment and skips the venv check. 2. [For Advanced Users](#For-advanced-users) - Step 3 ### Docker +> [!NOTE] +> If you are planning to use custom versions of dependencies such as dev ExllamaV2, make sure to build the Docker image yourself! + 1. Install Docker and docker compose from the [docs](https://docs.docker.com/compose/install/ 2. Install the Nvidia container compatibility layer 1. For Linux: [Nvidia container toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)