theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-29 02:31:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Model: Always enable decode_special_tokens

Browse Source 
The frontend should handle the special tokens if they get emitted.

Signed-off-by: kingbri <8082010+kingbri1@users.noreply.github.com>
This commit is contained in:
kingbri
2025-05-09 22:25:50 -04:00
parent 83826b56be
commit 656af41b5d
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
backends/exllamav2/model.py
View File

@@ -1373,6 +1373,7 @@ class ExllamaV2Container(BaseModelContainer):
min_new_tokens=params.min_tokens, min_new_tokens=params.min_tokens,
gen_settings=gen_settings, gen_settings=gen_settings,
stop_conditions=stop_conditions, stop_conditions=stop_conditions,
decode_special_tokens=True,
filters=grammar_handler.filters, filters=grammar_handler.filters,
filter_prefer_eos=bool(grammar_handler.filters), filter_prefer_eos=bool(grammar_handler.filters),
return_probs=params.logprobs > 0, return_probs=params.logprobs > 0,