Usage

TabbyAPI's main use-case is to be an API server for running ExllamaV3 models.

API Server

Currently TabbyAPI supports clients that use the OpenAI standard and KoboldAI's API.

In addition, there are expanded parameters to generation endpoints along with administrative endpoints for loading, unloading, loras, sampling overrides, etc.

Note

If you are a developer and want to add full TabbyAPI support to your app, it's recommended to use the autogenerated documentation.

TabbyAPI also answers the discovery endpoints that clients written for llama.cpp's llama-server use to detect a local model's capabilities, so tools such as coding agents can pick up the context size, vision support and thinking support without manual configuration:

• GET /props returns model_path, default_generation_settings.n_ctx, total_slots, chat_template and modalities.vision for the loaded model.
• GET /v1/models entries carry a meta block with n_ctx_train, n_vocab and n_embd read from the model's config.json; the loaded model additionally reports n_ctx and the weight size in bytes, and its loaded context length once more as max_model_len, the field name vLLM and SGLang use and that clients sizing their context window from the model list look for.
• POST /apply-template (also /v1/apply-template) renders chat messages through the prompt template and returns {"prompt": ...} without generating.
• Messages with the OpenAI developer role are treated as system messages.
• return_progress: true on a streaming /v1/chat/completions or /v1/completions request adds chunks during prompt processing that carry a top-level prompt_progress object (total, cache, processed, time_ms, as in llama.cpp), one per prefill chunk, so a client can show progress on a long prompt instead of waiting silently for the first token.

Completion and chat completion responses also carry a llama.cpp-style timings object (prompt/generation times and rates, cache and draft counters) for clients that read it off llama-server: top-level on non-streaming responses, on the final chunk of a stream (the usage chunk when stream_options.include_usage is set), and only for requests that produce a single generation.

Loop detection

Generation stops when the output settles into repeating the same text, exactly or nearly, with nothing else happening in between. This is on by default: set loop_detect: false, in the request or a sampler override preset, to turn it off. An exact repeat is caught after four copies and at least 1024 tokens. A near repeat takes eight copies and at least 8192 tokens, so that a long list of similar items isn't mistaken for a loop. The older loop_detect_window request field still works as an alias, where 0 turns detection off and any other size turns it on, but the size itself no longer has any effect.

When a loop is detected, the response ends with finish_reason: "repetition", the value vLLM uses for its repetition detector, and eos_reason: "loop_detected". A loop during reasoning usually leaves content empty, with the loop in reasoning_content.

Set loop_detect_error: true, in the request or a sampler override preset, to fail the request instead, so that a client or eval harness can retry it:

• A non-streaming request gets an HTTP 500 with {"error": {"type": "server_error", "code": "loop_detected", "message": ...}}. The OpenAI SDKs retry a 500 on their own.
• A stream has already started with a 200, so it ends with the same error object as its last event, after the output streamed so far.

A retry only helps when sampling is random: with greedy sampling or a fixed seed it loops the same way again.

Continuing a message

With add_generation_prompt: false and continue_final_message set, the model continues the last message instead of starting a new one. This follows llama.cpp and Transformers, and the response has only the new text:

• true continues the reasoning if the message has reasoning_content and empty or missing content, so that a reply cut off during reasoning can be resumed. Otherwise it continues the content.
• "reasoning_content" continues the reasoning whatever the content says. "reasoning" and "thinking" mean the same.
• "content" continues the content. With reasoning and empty content, that generates the answer to reasoning that's already finished. On vLLM, and in Transformers, true means this.

Continued reasoning counts towards reasoning_budget_tokens, and a message with tool calls can't be continued.

A message's reasoning can be sent as reasoning_content, reasoning or thinking, and chat templates get it under all three names, so it doesn't matter which one a template reads. On Harmony models, an assistant message with both reasoning and tool calls is rendered without its content, as llama.cpp does, because Harmony has room for only one of them.

Reasoning sent with earlier turns is kept where a template offers the choice, as on llama.cpp: if any message in a request has reasoning, preserve_thinking, clear_thinking, truncate_history_thinking and drop_thinking default to keeping it. To drop it for a model, set the name its template reads in template_vars_default, for example {preserve_thinking: false}. A request can do the same in template_vars. Requests with no reasoning aren't affected, as these settings would only add empty reasoning blocks to their earlier turns. Many templates have no such setting and always drop reasoning from before the last user message.

Below is an example CURL request using the OpenAI completions endpoint:

curl http://localhost:5000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Meta-Llama-3-8B-exl3",
  "prompt": "Once upon a time,",
  "max_tokens": 400,
  "stream": false,
  "min_p": 0.05,
  "repetition_penalty": 1.05
}'

Decisions

Two endpoints answer typed questions about an input with a probability for every allowed answer, read from the model's next-token distribution instead of generated text. They work with any generation model that has a chat template.

• POST /v1/decisions follows OpenAI's Decisions API (client.decisions.create in the OpenAI SDK). input is a string or a list of user messages (text only); questions is a list of up to 64 predicate, choice (2 to 26 choices, string or boolean values) or score (2 to 10 levels) questions, each with instructions and an optional name. answers come back in question order with the name echoed: a predicate's probability of being true, a choice's choice with probabilities per value and a confidence, a score's probability-weighted score over zero-based level indices with per-level probabilities and a confidence. model is optional here and selects an inline load like the other routes.
• POST /v1/systemone follows TypeSafe's System One API as llama.cpp and vLLM serve it: a state (text or JSON), a name-keyed questions map of choice, score and noul questions with instructions and criteria, and name-keyed answers (choice/probabilities/confidence, score/legend/probabilities/confidence, or a bare noul probability of yes). chat_template_kwargs reaches the chat template; diagnostics reports each question's label_mass and whether the model's top token was a label.

Each question is rendered as one user message (the input, then Question: ..., one labeled line per option such as A: billing, and an instruction to answer with one label) through the chat template with thinking off, and the model generates a single token. Every vocabulary token that spells a label, with or without leading whitespace or (for yes/no) different capitalization, counts for that label, so the reported probabilities are the softmax over the labels' total log-probabilities; confidence is the winning label's probability over the whole vocabulary. The questions of a request run concurrently and share the input through the prompt cache, so the input is prefilled roughly once. usage counts every question's prompt tokens and one output token per question. Probabilities are not calibrated.

Chat client

TabbyAPI serves a small chat client at http://<host>:<port>/ui for trying out models and settings without a separate frontend. It can:

• Show the loaded model. Models are loaded on the status page (below).
• Set every sampling parameter the server supports. Parameters left unset aren't sent, so the server's sampler overrides apply, and the panel shows those values as the defaults. The server-wide override preset can also be switched from there.
• Show reasoning separately from the answer, with token counts and speeds for each response.
• Attach images by pasting them, dropping them on the page or with the paperclip button, for models loaded with vision. PNG, JPEG, WebP, GIF and BMP files up to 20 MB and 50 megapixels are accepted.
• Format replies as Markdown (headings, lists, tables, code blocks, links), or show them as the exact text the model wrote. HTML in a reply is always shown as text, and images are only shown when embedded in the reply; others become links. Code blocks are syntax highlighted (when the model names the language) and have a copy button, and SVG code gets a preview of the graphic below it, drawn as an image so it can't run scripts or load anything. The preview builds up while the reply streams in.
• Typeset math in replies with KaTeX: $…$ or \(…\) inline, $$…$$ or \[…\] displayed. Dollar amounts such as "$5 and $10" stay text.
• Hide the chat list or the settings panel with the buttons at either end of the top bar, to give the conversation more room.
• Work on raw text in a notebook (below), for models with or without a chat template.

Chats and settings are kept in the browser's local storage, and attached images in its IndexedDB storage, so they persist across server restarts but belong to that browser. At startup the server prints a link to the client with the admin key in it (after a #, so the browser never sends it to the server); opening that link logs the client in. Otherwise the client asks for a key on first use; use the admin key to list and load models. The client can also be pointed at another TabbyAPI instance from its connection settings, as long as that server's allowed_origins permits it.

Set web_ui: false in the network section of config.yml to turn it off, along with the status page.

Notebook

A notebook is a session that holds its whole context as editable text, with no messages or chat template. It's sent as is to /v1/completions, so it works with base models too. Notebooks are in the same list as chats, marked with a lines-of-text icon instead of the speech bubble.

The text before the cursor is the prompt. Output goes in at the cursor, replacing any selected text, and text after the cursor is left alone.

• Step (Ctrl+Enter) samples until the model produces text. That's usually one token, but a character split across several tokens comes out whole. Hold the keys down to keep stepping.
• Generate (Ctrl+Shift+Enter) samples the number of tokens set beside it, or to the end of the context if that's empty.
• Stop, or Escape, ends a run and keeps what it produced. The text can't be edited during a run.
• Ctrl+Z takes back one step or one whole run.

Token healing is on by default here (Tokens in the settings panel), so text that stops partway through a word goes on with the rest of the word, instead of a new token that has to make sense after the fragment. It samples the last token again, limited to tokens that start with it, so Par can become Paris, and if the token stays as it is, generation carries on after it. A last token that's a stop token is left alone. Chats leave the setting off.

The stop token that ends a generation is written into the text (e.g. <|im_end|>), since it's part of the context the next request should see. Stop strings from the settings panel aren't, as usual.

An empty notebook can start from the loaded model's chat template, with a placeholder system and user message. To turn a chat into raw text, use the text icon on its row in the session list. That makes a new notebook from the chat, rendered with the chat template and the panel's template settings, and leaves the chat as it is. Images are left out, since raw text can't carry them.

The settings panel is shared with chats, leaving out what doesn't apply to raw text: constraints, reasoning and template settings, and max tokens. It adds Add BOS token, since nothing else controls that here; left unset, it follows the model's tokenizer config.

Continuations shows the ways the text can go on from the cursor, in a panel you can switch on from the toolbar. Each row is one of the likeliest next tokens (in bold) that passes truncation under the current sampling settings, with the chance of the sampler picking it, followed for as long as a single token keeps passing: up to the next point where the text could go several ways, the stop token, or 20 tokens (…). Clicking a row inserts it at the cursor as if typed, and the list moves on from there. It follows the cursor, the text and the settings by itself. Stop strings and banned strings take no part in it, and with temperature 0 there's a single path. If nothing truncates the distribution, every token is a branch point and each row is one token long. A path never ends inside a character: it follows the likeliest token until the character is complete, since text can't hold half of one. With token healing the list has both the ways the last token can grow and, if it can also stay as it is, what comes after it, with chances that add up the same way as sampling.

The panel uses POST /v1/continuations, which takes prompt, the usual sampler fields, max_options (10) and max_branch_tokens (20), and returns options with text, first (the part from the first token), token_ids, healed (the first token replaces the prompt's last one), prob and end (branch, stop or limit). One job finds the tokens that pass truncation, then one job per token follows it. They share their prompt through the page cache.

Stepping uses stop_after_output, a request field on the completion endpoints that ends the generation at the first text it produces, with finish_reason: "length". Unlike max_tokens: 1, it doesn't cut off a partial character. Completion responses also carry stop_str, the stop token or string that ended the generation.

Status page

http://<host>:<port>/ui/status shows what the server is doing and loads models. The server prints a link with the admin key at startup, as for the chat client, and the two pages share their connection settings. With an API key the page shows only the live status on the right, without the inspector and cancel buttons; the admin key adds the models panel on the left:

• The loaded model, with a button to unload it.
• The model list. Pick a model to see its load options, double-click to load it. A model outside the list can be loaded by typing its path into the filter box. Star models to keep them at the top of the list.
• The load options, as a form with one control per option, grouped. Each option is marked with where its value comes from: the model's tabby_config.yml, config.yml's use_as_default, a change made on the page, or the backend default when nothing sets it. The model loads with exactly the options shown, so clearing one gives it the backend default rather than the file's value. Save to tabby_config.yml writes them to the model's file, keeping the previous one as tabby_config.bak.

On the right, updated every second:

• Requests running, waiting (requeued until there is room in the cache) and queued, output and prompt processing speed with a chart of the last five minutes, and totals since startup.
• Cache use: VRAM, and the system memory and disk tiers when configured, with the hit rate and where the hits came from.
• The requests in progress and the last 20 that finished. With the admin key, click one to follow the text it is generating, and its prompt, to catch a generation that is going off the rails. A request can be cancelled from there: its client gets the text generated so far, with finish_reason: "abort" and eos_reason: "cancelled".

The page uses these endpoints, which other monitoring tools can use too:

• GET /v1/status (any API key): the jobs in progress and recently finished, the engine's cache statistics, and token counters since startup (generated_tokens, prefilled_tokens), from which a client takes rates. Only an admin key sees jobs as cancellable.
• GET /v1/status/jobs/{request_id}?offset=N&prompt=true (admin): a job's generated text from character offset on, and its prompt when asked for.
• POST /v1/status/jobs/{request_id}/cancel (admin): cancel a job in progress.
• GET /v1/model/load_options (admin): every load option, grouped, with its type, backend default, accepted values and description.

Metrics

Set enable_metrics: true in the network section of config.yml to serve Prometheus-compatible metrics at GET /metrics, with any API key unless disable_auth is set.

The names are llama.cpp's under a tabbyapi: prefix, so a dashboard made for llama.cpp works with the prefix changed: prompt_tokens_total, prompt_tokens_cached_total, prompt_seconds_total, tokens_predicted_total, tokens_predicted_seconds_total, n_tokens_max, spec_decode_num_draft_tokens_total, spec_decode_num_accepted_tokens_total and spec_decode_num_drafts_total, and the gauges prompt_tokens_seconds, predicted_tokens_seconds, requests_processing and requests_deferred. As in llama.cpp, the two throughput gauges cover the time since the last scrape, so with several scrapers each sees only part of it; rate() over the counters doesn't have that problem. llama.cpp's n_decode_total and n_busy_slots_per_decode have no equivalent here.

prompt_tokens_total and tokens_predicted_total move during a request, so rate() shows a long generation as it runs, including the tokens of requests that are later cancelled.

On top of those:

• requests_total, process_start_time_seconds and model_info, which holds the model's name in its model_name label.
• kv_cache_usage_ratio, kv_cache_tokens and kv_cache_max_tokens for the cache use of running requests, kv_cache_reusable_tokens, kv_cache_sysmem_tokens and kv_cache_disk_tokens for what later requests can reuse, and kv_cache_hit_rate, kv_cache_sysmem_hit_rate and kv_cache_disk_hit_rate, which are the figures on the status page.
• The histograms request_queue_time_seconds, time_to_first_token_seconds and e2e_request_latency_seconds, with vLLM's names and buckets.

Authentication

Every call to a TabbyAPI endpoint requires some form of authentication. Keys have two types of permissions:

• API: Accesses non-invasive endpoints (ex. generation, model list fetching)
• Admin: Allowed to access protected endpoints that deal with resources (ex. loading, unloading)

In addition, when calling list endpoints, API keys will only fetch the currently loaded object while admin keys will list the entire directory. For example, calling /v1/models will return a list of the user-configured models directory only if an admin key is passed.

Therefore, it's recommended to keep the admin key for yourself and only share the api key with users.

If these keys get compromised, shut down your server, delete the api_tokens.yml file, and restart. This will generate new keys which you can share with users.

To bypass authentication checks, set disable_auth to True in config.yml. However, turning off authentication without a third-party solution will make your instance open to the world.

Chat request debug logs

For bug-report reproduction, TabbyAPI can write every /v1/chat/completions request to a JSON file in logs/debug. Enable it in config.yml:

logging:
  log_chat_completion_requests: true

Warning

This creates a comprehensive request log. Each file includes the full message history and generation parameters. API keys are redacted, but prompts and user-provided content are preserved.

Inspect a saved request before sharing it:

python tools/inspect_chat_request.py logs/debug/<request>.json

Replay a saved request against a running TabbyAPI server and stream the raw response to the console:

python tools/replay_chat_request.py logs/debug/<request>.json --api-key <api key>

Model loading

Important

All loading methods require an admin key.

TabbyAPI has a set of endpoints used for model loading and unloading:

• /v1/model/load: Loads a model from the configured model_dir with the provided parameters. If the provided model name is different than the currently loaded model, the existing model will be unloaded beforehand.
• /v1/model/unload: Unloads all model tensors and terminates all pending generation requests. Only use this endpoint in an emergency or if you want to free VRAM but leave the server running.
• GET /v1/model/load_config?model_name=...: Shows the options a load of that model would use for anything the request leaves out: the model folder's tabby_config.yml over the use_as_default keys of config.yml. The response has them as a JSON object (settings, with draft options under draft_model), with the source of each (sources), and as annotated YAML (yaml) that can be edited and sent back as a load request.
• POST /v1/model/load_config/parse: Turns load options written as YAML ({"yaml": "..."}) into a JSON load request body, for clients that let users edit them as YAML.
• POST /v1/model/load_config/save: Saves load options as the model folder's tabby_config.yml. The body names the model (model_name) and gives the options in the shape of a load request, either as YAML (yaml) or as a JSON object (settings). Options that only repeat a use_as_default value are left out, unless the file already sets them; anything else in the options is saved. An existing file is copied to tabby_config.bak first, and keeps its comments, other sections and keys a load request can't set. If nothing is left to save, the file is removed. The response lists what was saved.

The model list (/v1/models with an admin key) searches up to four folders deep, so quant-style layouts such as <model>/exl3/4.0bpw are listed by their path under model_dir, the same name /v1/model/load takes. A folder counts as a model when its config.json names the model architecture (architectures), and folders inside a model aren't searched.

Load settings are applied in this order, later ones winning: use_as_default keys from config.yml, the model folder's tabby_config.yml, then the options set in the load request. A load request with "inherit_defaults": false skips the first two: it is the complete set of options, and anything it leaves out gets the backend default. The chat client loads this way, so that the options it shows are the ones used.

Please note that load requests are ephemeral and config.yml options will not apply. If you want to apply options from your config as fallback defaults, add them in the use_as_default key under model options.

Below is an example CURL request using the model load endpoint:

curl http://localhost:5000/v1/model/load \
-H "Content-Type: application/json" \
-d '{
  "model_name": "Meta-Llama-3-8B-exl3",
  "max_seq_len": 8192,
  "tensor_parallel": true,
  "gpu_split_auto": false,
  "gpu_split": [20, 25],
  "cache_mode": "Q8"
}'

A model load request can also include draft model parameters:

curl http://localhost:5000/v1/model/load \
-H "Content-Type: application/json" \
-d '{
  "model_name": "Meta-Llama-3-8B-exl3",
  ... Other parameters
  "draft_model": {
	draft_model_name: "TinyLlama-1B-32k-exl3",
	draft_rope_scale: 1.0
  }
}'

For more information on what parameters can be passed, please look at the autogenerated documentation

Inline loading

Note

If you want more fine-grained customization when loading, it's highly recommended to use the traditional endpoints instead. Inline loading is an alternative method that doesn't allow parameters to be passed on request.

An alternative way of switching models is called "inline loading" which hooks into the model switching logic used by frontends such as OpenWebUI. As previously stated, model requests are ephemeral, so there needs to be a way to provide per-model defaults while also being explicit to the admin user. This is where tabby_config.yml comes into play.

To get started, set inline_model_loading to true under the model block of config.yml.

Now to create a tabby config, let's say we have a model in our models directory called Meta-Llama-3-8B-exl3. Navigate into that model folder and create a file called tabby_config.yml

Now, you can place any model load parameter from /v1/model/load into that file. Settings from tabby_config.yml override the startup config.yml values and the use_as_default keys. A /v1/model/load request overrides the file for the options it sets, so you can try different settings without editing it. Here's a simple example which changes the default max_seq_len to 8192 and sets a Q6 quantized cache:

model:
  max_seq_len: 8192
  cache_mode: Q6

If you'd like to provide draft model options, you can add them under the draft_model key:

model:
  max_seq_len: 8192
  cache_mode: Q6
draft_model:
	draft_model_name: TinyLlama-1B-32k-exl3
	draft_rope_scale: 1.0

To switch the currently loaded model, send a request to /v1/completions or /v1/chat/completions with the model parameter set to the desired folder.

Inline loading is strict: a request that names a model which doesn't exist in the model directory gets a 404, and one whose model fails to load gets a 503, rather than being answered by whatever model is currently loaded. Model names listed under dummy_model_names (with use_dummy_models on) are the exception and run on the loaded model, for clients that always send a name such as gpt-3.5-turbo.

Below is an example CURL request for inline loading:

curl http://localhost:5000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Meta-Llama-3-8B-exl3"
  ... Other parameters
}'

This will unload the existing model and load the new model with defaults specified from tabby_config.yml

Difficult to get started?

Is the API difficult? Don't want to load models with config.yml? That's okay! Not everyone is a master user of AI products when starting out.

For newer users, it's recommended to use a UI that allows for managing TabbyAPI via API endpoints.

To find UI projects, take a look at Community Projects for more information.