This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Tool Calling in TabbyAPI

Tool calling is available for supported models, and enabled by selecting a tool format in the model config. This can also be specifed per model using tabby_config.yml.

Most tool-calling models are also reasoning models, and it is recommended to enable reasoning as well with appropriate reasoning tags (these cannot currently be inferred from the model's template). The parser still respects these tags even if the model does not support tool calls or no tool_format is given.

model:
    reasoning: true
    reasoning_start_token: "<think>"
    reasoning_end_token: "</think>"
    tool_format: qwen3_5

Supported formats

Below are the currently recognized tool call formats.

By default tool_format is auto: when a model loads, TabbyAPI scores every format against the model's chat template (the literal markers it renders around a tool call), the special tokens in its tokenizer and, as a tie-break, its architecture, and selects the unique best match. The reasoning tags default to auto as well and are taken from the detected format when the template or tokenizer confirms them, otherwise from the template alone; if none are found, reasoning parsing is switched off for that model. The startup log reports the outcome on a Response parsing: line. When no format matches a template that does define tools, the server warns and tool calls are not parsed; set tool_format, reasoning_start_token and reasoning_end_token explicitly in config.yml or the model folder's tabby_config.yml in that case. Setting any of them explicitly always overrides detection.

tool_format	Aliases	Usual reasoning tokens	Model types
qwen3_coder ⁶	qwen3_5 step3_5 step3_7	<think> </think>	Qwen3-Coder Qwen3-Next Qwen3.5 / Qwen3.6 Step-3.5 / Step-3.7
minimax_m2		<think> </think>	Minimax-M2 Minimax-M2.1 Minimax-M2.5
glm4_5	glm4_6 glm4_7 laguna poolside_v1 spark2_5 ling3	<think> </think>	GLM4.5 GLM4.6 GLM4.7 Laguna XS/S (Poolside) GLM5.2, GLM5.3 Spark X2.5 Ling 3.x
deepseek_v4 ⁴	dsv4	<think> </think>	DeepSeek-V4 DeepSeek-V4-Flash
deepseek_v41 ⁴	dsv41	<think> </think>	DeepSeek-V4.1-Flash (spaced DSML dialect: <｜DSML｜ calls>)
harmony ²			gpt-oss-20b gpt-oss-120b
hy3 ³	hy_v3	<think:opensource> </think:opensource>	Hy3 (Tencent Hunyuan)
lfm2 ⁵	lfm lfm2_5		LFM2 / LFM2.5 (Liquid AI)
hermes	qwen3 qwen2_5 smollm3	<think> </think>	Qwen2.5 / Qwen3 (original) SmolLM3 Nous Hermes and most fine-tunes using the Hermes JSON template
kimi	kimi_k2 kimi_linear	<think> </think>	Kimi K2 Kimi Linear
olmo3	olmo	<think> </think> (Think variants)	OLMo 3 / OLMo 3.1
muse_glimmer ²	glimmer		Muse Glimmer
mistral_old ¹			(older Mistral-family models)
mistral		[THINK] [/THINK]	Codestral 2508+ Devstral-Small 2507+ Magistral-Medium 2506+ Magistral-Small 2506+ Ministral-3 2512+ Mistral-Medium-3.1 2508+ Mistral-Small-3.2 2506+ Mistral-Small-4
gemma4		<|channel>thought <channel|>	Gemma 4-it

¹ Older Mistral models tend to have unreliable tool calling support and even newer ones are often released without official chat templates or with templates that omit any tool formatting. Tokenization also changes frequently between model releases. YMMV

² The Harmony format (gpt-oss) and the Muse Glimmer format structure the whole response — reasoning, content and tool calls — as structured messages (channels in Harmony, recipients in Glimmer). Both are detected automatically from the model's special tokens and need no configuration; tool_format: harmony / tool_format: muse_glimmer are also accepted and are equivalent to setting harmony: true / muse_glimmer: true in the model config. When active they supersede the reasoning and tool_format settings. Set harmony: false / muse_glimmer: false to override the detection.

³ Note that Hy3's standard chat template disables reasoning by default and does not respect enable_thinking. Clients can enable reasoning per request with the top-level reasoning_effort field or chat_template_kwargs: {"reasoning_effort": "high"} (options: no_think, low, high). To enable reasoning by default server-side, set template_vars_default: {reasoning_effort: high} in the model config (e.g. in the model folder's tabby_config.yml).

⁴ DeepSeek-V4's chat template enables reasoning by default and accepts enable_thinking (boolean, or the equivalent thinking_mode: chat|thinking) and reasoning_effort (low, high, max; low is the default and high/max prepend effort prompts to the context) as template variables, settable per request via the top-level fields or chat_template_kwargs.

⁵ LFM2 models emit a Pythonic tool-call list in <|tool_call_start|>[...]<|tool_call_end|>, which the server's bundled template doesn't explicitly steer the model toward. Point the model at the included templates/lfm2.jinja (prompt_template: lfm2.jinja) so fresh-session tool calls reliably use that format instead of drifting to OpenAI-style JSON. The template requires parallel tool results in the same order as the corresponding tool calls; TabbyAPI sorts tool messages by tool_call_id before templating, so clients may send them in any order.

⁶ For the qwen3_coder family, streaming responses report tool calls incrementally: instead of a single delta with the finished call at end of stream, the response carries tool_calls deltas with partial arguments fragments (same index, id and name as before), which clients concatenate as usual. Parameter values are typed from the request's tool schema: a parameter declared string streams as it is generated and is never reinterpreted as a number or JSON literal, other declared types are parsed when the parameter closes, and a parameter the schema doesn't type falls back to type guessing, in which case a value that could still be a JSON literal is held back until it closes. The concatenated fragments are identical to what the end-of-stream parser produces, so assembling deltas and reading a non-streaming response give the same result. A tool call that generation cuts off (max_tokens) is the one exception: its fragments stop early and the response ends with finish_reason: length, while the end-of-stream parser drops the incomplete call.