mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #343] Possible inconsistencies arising from the pattern matching of CLIP tokenizer #296
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#296
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Mypathissional on GitHub (Apr 3, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/343
Hi there,
I have been experimenting with the CLIP tokenizer and have observed that the tokenizer produces identical outputs for the following cases:
1). "1233" and "12 33"
2) "'Medicare For All" and "'M edicare For All"
In both cases, the tokenizer applies a pattern matching step using the regular expression
re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE)before applying the BPE algorithm. This pattern matches the same characters for both cases (1) and (2), which seems to be a logical inconsistency, particularly for case (1).I would suggest modifying the pattern to
re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s |'t |'re |'ve |'m |'ll |'d |[\p{L}]+|[\p{N}]+|[^\s\p{L}\p{N}]+""", re.IGNORECASE)to resolve this issue. This modified pattern should ensure that the tokenizer correctly handles cases where input text includes whitespace characters between numbers and preserves the integrity of the original input.Best Regards,
Maria