[GH-ISSUE #410] tokenizer does not produce a single integer for an exact match of a word for 1029 out of 34,483 vocab entries when it should #351

New issue

Open

opened 2026-10-31 21:14:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:14:14 +00:00

Owner

Originally created by @doctorpangloss on GitHub (Dec 12, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/410

import clip
# downloads model
clip.tokenize(["a"])[0, 1]

from clip.simple_tokenizer import SimpleTokenizer 
tokenizer = SimpleTokenizer()
# match whole words
whole_words = {k: v for k, v in tokenizer.encoder.items() if k.endswith("</w>")}
to_trim = len("<w/>")
missed = 0
for token_str, token_int in whole_words.items():
  tokenized = tokenizer.encode(token_str[:-to_trim])
  if len(tokenized) != 1:
    missed += 1
print(f"openai/clip {missed} words out of {len(whole_words)} incorrectly tokenized ({missed/len(whole_words)*100})%")

this prints

openai/clip 1029 words out of 34483 incorrectly tokenized (2.9840791114462197)%

Originally created by @doctorpangloss on GitHub (Dec 12, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/410 ``` import clip # downloads model clip.tokenize(["a"])[0, 1] from clip.simple_tokenizer import SimpleTokenizer tokenizer = SimpleTokenizer() # match whole words whole_words = {k: v for k, v in tokenizer.encoder.items() if k.endswith("</w>")} to_trim = len("<w/>") missed = 0 for token_str, token_int in whole_words.items(): tokenized = tokenizer.encode(token_str[:-to_trim]) if len(tokenized) != 1: missed += 1 print(f"openai/clip {missed} words out of {len(whole_words)} incorrectly tokenized ({missed/len(whole_words)*100})%") ``` this prints ``` openai/clip 1029 words out of 34483 incorrectly tokenized (2.9840791114462197)% ```

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#351

No description provided.

Rows
Columns