[GH-ISSUE #185] Context length understanding #160

New issue

Open

opened 2026-10-31 21:11:51 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:11:51 +00:00

Owner

Originally created by @Tortoise17 on GitHub (Nov 21, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/185

I have to ask important question.
There is context length 77.
Does it mean that the query search limit is 77 alphabets or 77 words or anything which I misunderstood?
Please if you can guide it clearly.

Because as it has been trained on captions. I want to know its optimal use boundaries how the text is taken into consideration and what maximum size is the ideal for searching the embeddings.

Originally created by @Tortoise17 on GitHub (Nov 21, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/185 I have to ask important question. There is context length 77. Does it mean that the query search limit is 77 alphabets or 77 words or anything which I misunderstood? Please if you can guide it clearly. Because as it has been trained on captions. I want to know its optimal use boundaries how the text is taken into consideration and what maximum size is the ideal for searching the embeddings.

bsakaguchi commented

2026-10-31 21:11:52 +00:00

Author

Owner

@aztecman commented on GitHub (Jun 23, 2022):

Just watched a explainer video from youtuber The AI Epiphany

near the beginning they do a 'hello world!'
Experimenting, I found the words each count for 1 token as does the punctuation.
The start and end of the string also count for 1 token each.

I recommend to open the colab
and try out different strings to get a sense how the tokens are counted.

@aztecman commented on GitHub (Jun 23, 2022): Just watched a explainer video from youtuber [The AI Epiphany](https://www.youtube.com/watch?v=jwZQD0Cqz4o) near the beginning they do a 'hello world!' Experimenting, I found the words each count for 1 token as does the punctuation. The start and end of the string also count for 1 token each. I recommend to open [the colab](https://colab.research.google.com/github/openai/clip/blob/master/notebooks/Interacting_with_CLIP.ipynb) and try out different strings to get a sense how the tokens are counted.

bsakaguchi referenced this issue

2026-10-31 21:15:16 +00:00

[PR #160] [MERGED] Fix PyTorch version check #466

bsakaguchi referenced this issue

2026-10-31 21:15:18 +00:00

[PR #173] [MERGED] Fix PyTorch version check for nightly builds #160 #468