[GH-ISSUE #305] Discrepancies between encode and decode in SimpleTokenizer #264

New issue

Closed

opened 2026-10-31 21:13:08 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:13:08 +00:00

Owner

Originally created by @Quasimondo on GitHub (Dec 15, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/305

I just came across a phenomenon which I find a bit confusing - not sure if this is expected behavior or shows an issue with SimpleTokenizer. Taking certain sequences of tokens and decoding them and then encoding the returned text again can result in a different sequence of tokens (whilst the two texts are identical).

Example:

from clip.simple_tokenizer import SimpleTokenizer

st = SimpleTokenizer()

tokens1 = [6808, 31978, 23644, 32441, 41003, 16125]
text_out1 = st.decode(tokens1)

tokens2 = st.encode(text_out1)
text_out2 = st.decode(tokens2)

print(text_out1)
print(text_out2)
print(text_out1==text_out2)

print(tokens2)

In this example the input tokens were
[6808, 31978, 23644, 32441, 41003, 16125]
but the output tokens are
[6808, 31978, 950, 544, 36461, 41003, 16125]

The texts in both cases are identical an "critical murraycaneleroy brandi xiv "

Looking into the code one possibility was that the text cleanup methods are causing this, but overriding the two methods yields the same results:

import clip.simple_tokenizer 

def basic_clean(text):
    print("basic_clean called:",text)
    return text

def whitespace_clean(text):
    print("whitespace_clean called:",text)
    return text

clip.simple_tokenizer.basic_clean = basic_clean   
clip.simple_tokenizer.whitespace_clean = whitespace_clean   
 
st = clip.simple_tokenizer.SimpleTokenizer()
    
clip.simple_tokenizer.basic_clean = basic_clean    

tokens1 = [6808, 31978, 23644, 32441, 41003, 16125]
text_out1 = st.decode(tokens1)

tokens2 = st.encode(text_out1)
text_out2 = st.decode(tokens2)

print(tokens1)
print(tokens2)

print(">"+text_out1+"<")
print(">"+text_out2+"<")
print(text_out1==text_out2)

Originally created by @Quasimondo on GitHub (Dec 15, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/305 I just came across a phenomenon which I find a bit confusing - not sure if this is expected behavior or shows an issue with SimpleTokenizer. Taking certain sequences of tokens and decoding them and then encoding the returned text again can result in a different sequence of tokens (whilst the two texts are identical). Example: ``` from clip.simple_tokenizer import SimpleTokenizer st = SimpleTokenizer() tokens1 = [6808, 31978, 23644, 32441, 41003, 16125] text_out1 = st.decode(tokens1) tokens2 = st.encode(text_out1) text_out2 = st.decode(tokens2) print(text_out1) print(text_out2) print(text_out1==text_out2) print(tokens2) ``` In this example the input tokens were [6808, 31978, 23644, 32441, 41003, 16125] but the output tokens are [6808, 31978, 950, 544, 36461, 41003, 16125] The texts in both cases are identical an "critical murraycaneleroy brandi xiv " Looking into the code one possibility was that the text cleanup methods are causing this, but overriding the two methods yields the same results: ``` import clip.simple_tokenizer def basic_clean(text): print("basic_clean called:",text) return text def whitespace_clean(text): print("whitespace_clean called:",text) return text clip.simple_tokenizer.basic_clean = basic_clean clip.simple_tokenizer.whitespace_clean = whitespace_clean st = clip.simple_tokenizer.SimpleTokenizer() clip.simple_tokenizer.basic_clean = basic_clean tokens1 = [6808, 31978, 23644, 32441, 41003, 16125] text_out1 = st.decode(tokens1) tokens2 = st.encode(text_out1) text_out2 = st.decode(tokens2) print(tokens1) print(tokens2) print(">"+text_out1+"<") print(">"+text_out2+"<") print(text_out1==text_out2) ```

bsakaguchi closed this issue

2026-10-31 21:13:08 +00:00

bsakaguchi commented

2026-10-31 21:13:09 +00:00

Author

Owner

@Quasimondo commented on GitHub (Dec 15, 2022):

Okay after some further looking into this I realize that this is simply a fact of life - there are simply multiple ways the same word can get encoded into tokens, e.g.

And there is no way for the decoder to know which token combination created that word.

@Quasimondo commented on GitHub (Dec 15, 2022): Okay after some further looking into this I realize that this is simply a fact of life - there are simply multiple ways the same word can get encoded into tokens, e.g. The word "murraycaneleroy" can either be tokenized as murray | cane | leroy</w> or as murray | can | el | eroy</w> And there is no way for the decoder to know which token combination created that word.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#264

No description provided.

Rows
Columns