theroyallab/tabbyAPI
1
0
Fork 0
mirror of https://github.com/theroyallab/tabbyAPI.git synced 2026-04-28 02:01:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

OAI: Tokenize chat completion messages

Browse Source 
Since chat completion messages are a structure, format the prompt
before checking in the tokenizer.

Signed-off-by: kingbri <bdashore3@proton.me>
This commit is contained in:
kingbri
2024-04-15 14:17:16 -04:00
parent ed05f376d9
commit 515b3c2930
3 changed files with 26 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
backends/exllamav2/model.py
View File

@@ -566,7 +566,9 @@ class ExllamaV2Container:
decode_special_tokens=unwrap(kwargs.get("decode_special_tokens"), True),
)[0]
def get_special_tokens(self, add_bos_token: bool, ban_eos_token: bool):
def get_special_tokens(
self, add_bos_token: bool = True, ban_eos_token: bool = False
):
return {
"bos_token": self.tokenizer.bos_token if add_bos_token else "",
"eos_token": self.tokenizer.eos_token if not ban_eos_token else "",