turboderp-org/exllamav2
1
0
Fork 0
mirror of https://github.com/turboderp-org/exllamav2.git synced 2026-04-20 14:29:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove PAD_TOKEN_ID warning

Browse Source
This commit is contained in:
turboderp
2024-06-17 01:11:11 +02:00
parent f01e0d0736
commit 81931e38c2
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
exllamav2/tokenizer/tokenizer.py
View File

@@ -255,9 +255,9 @@ class ExLlamaV2Tokenizer:
# Take stock and issue warnings if needed
if self.pad_token_id == self.bos_token_id:
print(" !! Warning: PAD and EOS tokens are identical. Generations might break " + \
"and batch sizes > 1 are unlikely to work correctly.")
# if self.pad_token_id == self.bos_token_id:
# print(" !! Warning: PAD and EOS tokens are identical. Generations might break " + \
# "and batch sizes > 1 are unlikely to work correctly.")
# Return size of valid vocabulary