[GH-ISSUE #110] Use pretrained CLIP with a context_length not equal to 77 (default value) #97

New issue

Closed

opened 2026-10-31 21:11:05 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-31 21:11:05 +00:00

Owner

Originally created by @ABaldrati on GitHub (May 28, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/110

Thanks you guys for the amazing job and releasing CLIP model.

As written in the title I wanted to ask if it was possible to use pretrained CLIP models with a context_length different from the default value (which is 77).

If I try to change the context_length of the tokenizer to another value (like 150), using pretrained clip model "out of the box" I get the following error:
File "/homes/students_home/abaldrati/miniconda3/lib/python3.8/site-packages/clip/model.py", line 342, in encode_text x = x + self.positional_embedding.type(self.dtype) RuntimeError: The size of tensor a (150) must match the size of tensor b (77) at non-singleton dimension 1

Thanks again for the amazing work

Originally created by @ABaldrati on GitHub (May 28, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/110 Thanks you guys for the amazing job and releasing CLIP model. As written in the title I wanted to ask if it was possible to use pretrained CLIP models with a context_length different from the default value (which is 77). If I try to change the `context_length` of the tokenizer to another value (like 150), using pretrained clip model "out of the box" I get the following error: `File "/homes/students_home/abaldrati/miniconda3/lib/python3.8/site-packages/clip/model.py", line 342, in encode_text x = x + self.positional_embedding.type(self.dtype) RuntimeError: The size of tensor a (150) must match the size of tensor b (77) at non-singleton dimension 1` Thanks again for the amazing work

bsakaguchi closed this issue

2026-10-31 21:11:05 +00:00

bsakaguchi commented

2026-10-31 21:11:06 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jun 4, 2021):

I don't think that's possible since the network is designed and trained for context_length up to 77.
You might want to retrain your model from scratch, or just reduce your text so it <= 77 characters(UPDATE : It should be 77 tokens, not 77 characters)

@vinson2233 commented on GitHub (Jun 4, 2021): I don't think that's possible since the network is designed and trained for context_length up to 77. You might want to retrain your model from scratch, or just reduce your text so it <= 77 characters(UPDATE : It should be 77 tokens, not 77 characters)

bsakaguchi commented

2026-10-31 21:11:06 +00:00

Author

Owner

@ABaldrati commented on GitHub (Jun 7, 2021):

Yes I imagined that, thanks for the answer anyway

@ABaldrati commented on GitHub (Jun 7, 2021): Yes I imagined that, thanks for the answer anyway

bsakaguchi commented

2026-10-31 21:11:06 +00:00

Author

Owner

@Cbhihe commented on GitHub (Oct 27, 2021):

@vinson2233:

Thanks for answering the question asked by @ABaldrati. I had the exact same issue.
I take that your answer is still valid.

You may want to correct your answer in that (unless mistaken) it should read "...<= 77 tokens", not "characters".
In fact it should read 75 tokens, including words, any punctuation symbol, a "start-of-summary" token, and an "end-of-summary" token, which should not be more than 77 in total. Thanks. ;-)

@Cbhihe commented on GitHub (Oct 27, 2021): @vinson2233: Thanks for answering the question asked by @ABaldrati. I had the exact same issue. I take that your answer is still valid. You may want to correct your answer in that (unless mistaken) it should read "...<= 77 tokens", not "characters". In fact it should read 75 tokens, including words, any punctuation symbol, a "start-of-summary" token, and an "end-of-summary" token, which should not be more than 77 in total. Thanks. ;-)

bsakaguchi commented

2026-10-31 21:11:06 +00:00

Author

Owner

@vinson2233 commented on GitHub (Oct 28, 2021):

@Cbhihe yup, you are right, it should be tokens, not character. The author have tell about this in this issue https://github.com/openai/CLIP/issues/119

@vinson2233 commented on GitHub (Oct 28, 2021): @Cbhihe yup, you are right, it should be `tokens`, not `character`. The author have tell about this in this issue https://github.com/openai/CLIP/issues/119

bsakaguchi commented

2026-10-31 21:11:06 +00:00

Author

Owner

@iasonasxrist commented on GitHub (Jul 18, 2022):

Same problem! Suppose i cannot edit it due to the training process

@iasonasxrist commented on GitHub (Jul 18, 2022): Same problem! Suppose i cannot edit it due to the training process

bsakaguchi referenced this issue

2026-10-31 21:15:08 +00:00

[PR #97] [MERGED] Fix inconsistent naming #452