mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #7] Does the tokenizer support Chinese? #9
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#9
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zhanxlin on GitHub (Jan 8, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/7
Hello, does the tokenizer support Chinese?
@jongwook commented on GitHub (Jan 8, 2021):
Hello!
The tokenizer is based on byte-pair encoding and is able to tokenize any valid UTF-8 string, but it is not recommended to use this tokenizer for non-English sentences since its vocabulary is primarily based on English. Furthermore, CLIP models are trained with mostly English data, so we don't expect particularly good performance from CLIP with non-English text inputs. Multilingual CLIP would be an interesting future work.
@ruby0101 commented on GitHub (Feb 17, 2021):
Hi, @jongwook, "CLIP models are trained with mostly English data". So can it be re-trained with a new image text pair dataset in other languages?
@jongwook commented on GitHub (Feb 18, 2021):
Yes; I presume it'll need to train from scratch (or at least the text encoder) in that case.
@datalee commented on GitHub (Mar 15, 2022):
Multilingual CLIP 666
@yangapku commented on GitHub (Nov 18, 2022):
Hi, maybe you can refer to this repo! https://github.com/OFA-Sys/Chinese-CLIP @zhanxlin
@andrewqianpublic commented on GitHub (Feb 7, 2025):
The tokenization method I came up with involves streaming. The way Chinese characters are designed is that any sequence of bytes making up the Chinese characters will not form a valid char that can be decoded. That is, no valid char is a prefix of any Chinese char or likely any char from nay language. In this way, you can simply attempt to decode a sequence of bytes (break down all generated chars into bytes, append to list, convert using bytearray, then attempt decode), and if that fails, simply wait for the next set of new chars until you get enough to decode into a valid character. You can then stream it. This seems to work consistently. Hope this helps whoever comes across this.