[GH-ISSUE #233] How to get BPE vocab given new dataset? #202

New issue

Closed

opened 2026-10-31 21:12:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:12:23 +00:00

Owner

Originally created by @JunweiLiang on GitHub (Apr 20, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/233

Hi, thanks for the great work!

I'm trying to train model on a new large-scale dataset and would like to use a new BPE dictionaries specific to that dataset. Could you provide some instructions/tools to get the same or similar format as the one in this repo? (like the bpe_simple_vocab_16e6.txt.gz file)

Thanks,
Junwei

Originally created by @JunweiLiang on GitHub (Apr 20, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/233 Hi, thanks for the great work! I'm trying to train model on a new large-scale dataset and would like to use a new BPE dictionaries specific to that dataset. Could you provide some instructions/tools to get the same or similar format as the one in this repo? (like the `bpe_simple_vocab_16e6.txt.gz` file) Thanks, Junwei

bsakaguchi closed this issue

2026-10-31 21:12:23 +00:00

bsakaguchi commented

2026-10-31 21:12:25 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 20, 2022):

Huggingface Tokenizers is best suited for this purpose. Check their documentations.

@jongwook commented on GitHub (Apr 20, 2022): [Huggingface Tokenizers](https://github.com/huggingface/tokenizers) is best suited for this purpose. Check [their documentations](https://huggingface.co/docs/tokenizers/python/latest/).

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#202

No description provided.

Rows
Columns