mirror of
https://github.com/theroyallab/tabbyAPI.git
synced 2026-03-14 15:57:27 +00:00
122 lines
3.2 KiB
Markdown
122 lines
3.2 KiB
Markdown
|
|
# tabbyAPI
|
|
|
|
tabbyAPI is a FastAPI-based application that provides an API for generating text using a language model. This README provides instructions on how to launch and use the tabbyAPI.
|
|
|
|
## Prerequisites
|
|
|
|
Before you get started, ensure you have the following prerequisites installed on your system:
|
|
|
|
- Python 3.x (with pip)
|
|
- Dependencies listed in `requirements.txt`
|
|
|
|
## Installation
|
|
|
|
1. Clone the repository to your local machine:
|
|
|
|
git clone https://github.com/Splice86/tabbyAPI.git
|
|
|
|
|
|
2. Navigate to the project directory:
|
|
|
|
cd tabbyAPI
|
|
|
|
|
|
3. Create a virtual environment (optional but recommended):
|
|
|
|
python -m venv venv
|
|
source venv/bin/activate
|
|
|
|
|
|
4. Install project dependencies using pip:
|
|
|
|
pip install -r requirements.txt
|
|
|
|
|
|
5. Install exllamav2 to your venv
|
|
|
|
git clone https://github.com/turboderp/exllamav2.git
|
|
|
|
cd exllamav2
|
|
|
|
pip install -r requirements.txt
|
|
|
|
python setup.py install
|
|
|
|
|
|
|
|
## Launch the tabbyAPI Application
|
|
|
|
To start the tabbyAPI application, follow these steps:
|
|
|
|
1. Ensure you are in the project directory and the virtual environment is activated (if used).
|
|
|
|
2. Run the tabbyAPI application using Uvicorn:
|
|
|
|
|
|
uvicorn main:app --host 0.0.0.0 --port 8000 --reload
|
|
|
|
|
|
- `main` refers to the Python file containing your tabbyAPI app instance.
|
|
- `app` is the FastAPI instance defined in your Python script.
|
|
- `--host 0.0.0.0` allows access from external devices. Change this to `localhost` if you want to restrict access to the local machine.
|
|
- `--port 8000` specifies the port on which your application will run.
|
|
- `--reload` enables auto-reloading for development.
|
|
|
|
3. The tabbyAPI application should now be running. You can access it by opening a web browser and navigating to `http://localhost:8000` (if running locally).
|
|
|
|
## Usage
|
|
|
|
The tabbyAPI application provides the following endpoint:
|
|
|
|
- `/generate-text` (HTTP POST): Use this endpoint to generate text based on the provided input data.
|
|
|
|
### Example Request (using `curl`)
|
|
|
|
|
|
curl http://127.0.0.1:8000/generate-text \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Your_Model_Path",
|
|
"prompt": "A tabby is a",
|
|
"max_tokens": 200,
|
|
"temperature": 1,
|
|
"top_p": 0.9,
|
|
"seed": 10,
|
|
"stream": true,
|
|
"token_repetition_penalty": 0.5,
|
|
"stop": ["###"]
|
|
}'
|
|
|
|
|
|
### Parameter Guide
|
|
|
|
*note* This stuff still needs to be expanded and updated
|
|
|
|
{
|
|
"prompt": "A tabby is a",
|
|
"max_tokens": 200,
|
|
"temperature": 1,
|
|
"top_p": 0.9,
|
|
"seed": 10,
|
|
"stream": true,
|
|
"token_repetition_penalty": 0.5,
|
|
"stop": ["###"]
|
|
}
|
|
|
|
prompt: This is the initial text or message that sets the context for the generated completions.
|
|
|
|
max_tokens: It defines the maximum number of tokens (words or characters) you want in the generated text.
|
|
|
|
temperature: The temperature parameter controls the randomness of the output.
|
|
|
|
top_p: The top_p parameter controls the diversity of the output.
|
|
|
|
seed: This parameter is set to 10. It is a seed value that helps to reproduce the same results if provided with the same seed.
|
|
|
|
stream: A boolean value set to true. It enables Server-Sent Events (SSE) streaming.
|
|
|
|
token_repetition_penalty: This parameter controls the penalty for token repetitions in the generated text.
|
|
|
|
stop: An array of strings that, if present in the generated text, will signal the model to stop generating.
|