theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-20 14:28:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Templates: Add stop_strings meta param

Browse Source 
Adding the stop_strings var to chat templates will allow for the
template creator to specify stopping strings to add onto chat completions.

Thes get appended with existing stopping strings that are passed
in the API request. However, a sampler override with force: true will
override all stopping strings.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-03-27 22:07:43 -04:00
parent 277c540c98
commit dc456f4cc2
2 changed files with 29 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
endpoints/OAI/app.py
View File

@@ -508,7 +508,10 @@ async def chat_completion_request(request: Request, data: ChatCompletionRequest)
if isinstance(data.messages, str):
prompt = data.messages
else:
prompt = format_prompt_with_template(data)
# Compile the prompt and get any additional stop strings from the template
# Template stop strings can be overriden by sampler overrides if force is true
prompt, template_stop_strings = format_prompt_with_template(data)
data.stop += template_stop_strings
disable_request_streaming = unwrap(
config.developer_config().get("disable_request_streaming"), False