[PR #54] [CLOSED] Make text tokenizer extendable #453

New issue

Closed

opened 2026-10-31 21:15:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:15:08 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/openai/CLIP/pull/54
Author: @mapmeld
Created: 3/7/2021
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (1)

0848ed7 extendable vocab

📊 Changes

2 files changed (+14 additions, -6 deletions)

View changed files

📝 clip/clip.py (+6 -6)
📝 clip/simple_tokenizer.py (+8 -0)

📄 Description

I'm interested in adding new creatures / foods / brands / concepts to the visual and text embeddings. This is a minor change which makes it possible to replace the default SimpleTokenizer and expand the vocabulary.

from clip.simple_tokenizer import SimpleTokenizer
newtokens = SimpleTokenizer()
newtokens.extend(['covid', 'eggcream'])

clip.tokenize('eggcream', tokenizer=newtokens)

Work in progress, editing model to receive new text embeddings:
https://colab.research.google.com/drive/1qj8-aFJ-HMOU0JEvsOjypuGuMnWxsuQq?usp=sharing

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/openai/CLIP/pull/54 **Author:** [@mapmeld](https://github.com/mapmeld) **Created:** 3/7/2021 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`0848ed7`](https://github.com/openai/CLIP/commit/0848ed71e5bb5b4eaaa4048364001877630345dc) extendable vocab ### 📊 Changes **2 files changed** (+14 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `clip/clip.py` (+6 -6) 📝 `clip/simple_tokenizer.py` (+8 -0) </details> ### 📄 Description I'm interested in adding new creatures / foods / brands / concepts to the visual and text embeddings. This is a minor change which makes it possible to replace the default ```SimpleTokenizer``` and expand the vocabulary. ```python from clip.simple_tokenizer import SimpleTokenizer newtokens = SimpleTokenizer() newtokens.extend(['covid', 'eggcream']) clip.tokenize('eggcream', tokenizer=newtokens) ``` Work in progress, editing model to receive new text embeddings: https://colab.research.google.com/drive/1qj8-aFJ-HMOU0JEvsOjypuGuMnWxsuQq?usp=sharing --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 21:15:08 +00:00