TavernAI - Extras

What is this

A set of unofficial APIs for various TavernAI extensions.

You need to run the lastest development version of TavernAI. Grab it here: https://github.com/SillyLossy/TavernAI/tree/dev

All modules require at least 6 Gb of VRAM to run. With Stable Diffusion disabled, it will probably fit in 4 Gb. Alternatively, everything could also be run on the CPU.

Try on Colab (runs KoboldAI backend and TavernAI Extras server alongside):

How to run

Locally

• Install Python 3.10
• Run pip install -r requirements.txt
• Run python server.py
• Get the API URL. Defaults to http://localhost:5100 if you run locally.
• Start TavernAI with extensions support: set enableExtensions to true in config.conf
• Navigate to TavernAI settings and put in an API URL and tap "Connect" to load the extensions

Colab

• Open colab link
• Select desired "extra" options and start the cell
• Wait for it to finish
• Get an API URL link from colab output under the ### TavernAI Extensions LINK ### title
• Start TavernAI with extensions support: set enableExtensions to true in config.conf
• Navigate to TavernAI settings and put in an API URL and tap "Connect" to load the extensions

Settings menu

UI Extensions

Name	Description	Required Modules	Screenshot
Image Captioning	Send a cute picture to your bot!	caption

Modules

Name	Description
caption	Image captioning
summarize	Text summarization
classify	Text sentiment classification
keywords	Text key phrases extraction
prompt	SD prompt generation from text
sd	Stable Diffusion image generation

API Endpoints

Get UI extensions list

GET /api/extensions

Input

None

Output

{"extensions":[{"metadata":{"css":"file.css","display_name":"human-friendly name","js":"file.js","requires":["module_id"]},"name":"extension_name"}]}

Get UI extension JS script

GET /api/script/<name>

Input

Extension name in a route

Output

File content

Get UI extension CSS stylesheet

GET /api/style/<name>

Input

Extension name in a route

Output

File content

Get UI extension static asset

GET /api/asset/<name>/<asset>

Input

Extension name and assert name in a route

Output

File content

Image captioning

POST /api/caption

Input

{ "image": "base64 encoded image" }

Output

{ "caption": "caption of the posted image" }

Text summarization

POST /api/summarize

Input

{ "text": "text to be summarize", "params": {} }

Output

{ "summary": "summarized text" }

Optional: params object for control over summarization:

Name	Default value
temperature	1.0
repetition_penalty	1.0
max_length	500
min_length	200
length_penalty	1.5
bad_words	["\n", '"', "*", "[", "]", "{", "}", ":", "(", ")", "<", ">"]

Text sentiment classification

POST /api/classify

Input

{ "text": "text to classify sentiment of" }

Output

{
    "classification": [
        {
            "label": "joy",
            "score": 1.0
        },
        {
            "label": "anger",
            "score": 0.7
        },
        {
            "label": "love",
            "score": 0.6
        },
        {
            "label": "sadness",
            "score": 0.5
        },
        {
            "label": "fear",
            "score": 0.4
        },
        {
            "label": "surprise",
            "score": 0.3
        }
    ]
}

NOTES

1. Sorted by descending score order
2. Six fixed categories
3. Value range from 0.0 to 1.0

Key phrase extraction

POST /api/keywords

Input

{ "text": "text to be scanned for key phrases" }

Output

{
    "keywords": [
        "array of",
        "extracted",
        "keywords",
    ]
}

Stable Diffusion prompt generation

POST /api/prompt

Input

{ "name": "character name (optional)", "text": "textual summary of a character" }

Output

{ "prompts": [ "array of generated prompts" ] }

Stable Diffusion image generation

POST /api/image

Input

{ "prompt": "prompt to be generated" }

Output

{ "image": "base64 encoded image" }

Additional options

Flag	Description
--port	Specify the port on which the application is hosted. Default: 5100
--listen	Host the app on the local network
--share	Share the app on CloudFlare tunnel
--cpu	Run the models on the CPU instead of CUDA
--summarization-model	Load a custom summarization model. Expects a HuggingFace model ID. Default: Qiliang/bart-large-cnn-samsum-ChatGPT_v3
--classification-model	Load a custom sentiment classification model. Expects a HuggingFace model ID. Default: bhadresh-savani/distilbert-base-uncased-emotion
--captioning-model	Load a custom captioning model. Expects a HuggingFace model ID. Default: Salesforce/blip-image-captioning-base
--keyphrase-model	Load a custom key phrase extraction model. Expects a HuggingFace model ID. Default: ml6team/keyphrase-extraction-distilbert-inspec
--prompt-model	Load a custom prompt generation model. Expects a HuggingFace model ID. Default: FredZhang7/anime-anything-promptgen-v2
--sd-model	Load a custom Stable Diffusion image generation model. Expects a HuggingFace model ID. Default: ckpt/anything-v4.5-vae-swapped Must have VAE pre-baked in PyTorch format or the output will look drab!
--sd-cpu	Force the Stable Diffusion generation pipeline to run on the CPU. SLOW!
--enable-modules	Override a list of enabled modules. Runs with everything enabled by default. Expects a comma-separated list of module names. See Modules