[GH-ISSUE #8] BPE tokenizer failed with punctuations in the sentence #8

New issue

Closed

opened 2026-10-31 21:09:50 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:09:50 +00:00

Owner

Originally created by @weiyx16 on GitHub (Jan 8, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/8

When tokenizing a sentence with a punctuation at the end, the bpe tokenizer will fail to split the EOS token out. It will lead to the result that the model.encode_text() function can't pick the correct token hidden states out from output of the last layer, for the picking method is argmax(id, dim=-1). I recommend to add a blank space before <|endoftext|> when creating a sentence.
To reproduce the bug:

from simple_tokenizer import SimpleTokenizer
# print the token in for-loop of https://github.com/openai/CLIP/blob/main/simple_tokenizer.py#L124
tokenizer = SimpleTokenizer(bpe_path=$path)
query = ["What will be Covid-19's long-term global economic impact?"]
text_tokens = [tokenizer.encode("The key is " + desc + "<|endoftext|>") for desc in query]

The tokens are:

the
key
is
what
will
be
covid
-
1
9
's
long
-
term
global
economic
impact
?<|
endoftext
|>

And the token ids are:

Originally created by @weiyx16 on GitHub (Jan 8, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/8 When tokenizing a sentence with a punctuation at the end, the bpe tokenizer will fail to split the EOS token out. It will lead to the result that the ```model.encode_text()``` function can't pick the correct token hidden states out from output of the last layer, for the picking method is ```argmax(id, dim=-1)```. I recommend to add a blank space before ```<|endoftext|>``` when creating a sentence. To reproduce the bug: ```python from simple_tokenizer import SimpleTokenizer # print the token in for-loop of https://github.com/openai/CLIP/blob/main/simple_tokenizer.py#L124 tokenizer = SimpleTokenizer(bpe_path=$path) query = ["What will be Covid-19's long-term global economic impact?"] text_tokens = [tokenizer.encode("The key is " + desc + "<|endoftext|>") for desc in query] ``` The tokens are: ```python the key is what will be covid - 1 9 's long - term global economic impact ?<| endoftext |> ``` And the token ids are: ```python [[518, 1458, 533, 768, 751, 655, 622, 9284, 268, 272, 280, 568, 1538, 268, 4780, 2779, 5259, 3844, 30, 27, 347, 40786, 4160, 91, 285]] ```

bsakaguchi closed this issue

2026-10-31 21:09:50 +00:00

bsakaguchi commented

2026-10-31 21:09:50 +00:00

Author

Owner

@jongwook commented on GitHub (Jan 8, 2021):

Hi, Thanks for catching this - the example code was using the tokenizer incorrectly. Just pushed a fix that should resolve this problem; please feel free to reopen this issue if you encounter a similar problem again.

@jongwook commented on GitHub (Jan 8, 2021): Hi, Thanks for catching this - the example code was using the tokenizer incorrectly. Just pushed a fix that should resolve this problem; please feel free to reopen this issue if you encounter a similar problem again.

bsakaguchi referenced this issue

2026-10-31 21:15:41 +00:00

[PR #522] [CLOSED] Update SimpleTokenizer for SAM3 text tokenizer convenience #518