openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #217] Question about clip.encode_text #191

New issue

Open

opened 2026-10-31 21:12:13 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-31 21:12:13 +00:00

Owner

Originally created by @AntonotnaWang on GitHub (Feb 15, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/217

To whom it may concern,

I'm checking the code of clip. It is simple but wonderful. However, I found Line 350 in clip/model.py: x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection may be a little bit confusing to me. May I ask why use this dimension (text.argmax(dim=-1)) out of the 77 dimensions of the tokenized text? Why not using the average of all dimension? Thanks a lot.

P.S. My understanding of text.argmax(dim=-1): it indicates the location of the end of the input text.

Best

Originally created by @AntonotnaWang on GitHub (Feb 15, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/217 To whom it may concern, I'm checking the code of clip. It is simple but wonderful. However, I found Line 350 in ```clip/model.py```: ```x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection``` may be a little bit confusing to me. May I ask why use this dimension (```text.argmax(dim=-1)```) out of the 77 dimensions of the tokenized text? Why not using the average of all dimension? Thanks a lot. P.S. My understanding of ```text.argmax(dim=-1)```: it indicates the location of the end of the input text. Best

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 11, 2022):

You're correct that the argmax operation takes the representation at the EOT position. There's nothing inherently wrong with taking the average along the sequence dimension, but taking representation at the position of a special token (e.g. the CLS token in ViT and BERT) is empirically known to work better.

@jongwook commented on GitHub (Apr 11, 2022): You're correct that the argmax operation takes the representation at the EOT position. There's nothing inherently wrong with taking the average along the sequence dimension, but taking representation at the position of a special token (e.g. the CLS token in ViT and BERT) is empirically known to work better.

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@Teoge commented on GitHub (Dec 15, 2022):

Hi, I also have a question. Does that make representations in other locations meaningless? Since they are not supervised by any loss, the network can output arbitrary values for these representations.

@Teoge commented on GitHub (Dec 15, 2022): Hi, I also have a question. Does that make representations in other locations meaningless? Since they are not supervised by any loss, the network can output arbitrary values for these representations.

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@powpos360 commented on GitHub (Dec 19, 2022):

Hi, I also have a question. Does that make representations in other locations meaningless? Since they are not supervised by any loss, the network can output arbitrary values for these representations.

Other representations are still used since in each attention layer, the [EOT] token is attended to every other location.

@powpos360 commented on GitHub (Dec 19, 2022): > Hi, I also have a question. Does that make representations in other locations meaningless? Since they are not supervised by any loss, the network can output arbitrary values for these representations. Other representations are still used since in each attention layer, the [EOT] token is attended to every other location.

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@ygfrancois commented on GitHub (Apr 25, 2023):

The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding?

@ygfrancois commented on GitHub (Apr 25, 2023): The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding?

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@LikeGiver commented on GitHub (Apr 25, 2023):

@ygfrancois
The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding?
the [ETO] token is 49407 in this situation, which is the largest number in the tokenized_prompts (i.e text), so we can use text.argmax(dim=-1) to determine the position of [EOT]

@LikeGiver commented on GitHub (Apr 25, 2023): @ygfrancois The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding? the [ETO] token is 49407 in this situation, which is the largest number in the tokenized_prompts (i.e text), so we can use text.argmax(dim=-1) to determine the position of [EOT] ![Screenshot from 2023-04-25 22-12-20](https://user-images.githubusercontent.com/42694182/234304132-ab06d6c9-dd03-4377-81f5-40e9f3c5a0c3.png)

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@stardusts-hj commented on GitHub (Dec 16, 2023):

@ygfrancois The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding? the [ETO] token is 49407 in this situation, which is the largest number in the tokenized_prompts (i.e text), so we can use text.argmax(dim=-1) to determine the position of [EOT]

Thank you @LikeGiver, I understand this point now. Argmax is used to locate the index (i_eot) of [EOT] at tokenized prompts. Once we locate it , we use the features of [EOT] by x[batchsize, i_eot] to represent the features of prompts

@stardusts-hj commented on GitHub (Dec 16, 2023): > @ygfrancois The position of [EOT] token is different for text with diff length, this dose not confuse the learning of position embedding? the [ETO] token is 49407 in this situation, which is the largest number in the tokenized_prompts (i.e text), so we can use text.argmax(dim=-1) to determine the position of [EOT] ![Screenshot from 2023-04-25 22-12-20](https://user-images.githubusercontent.com/42694182/234304132-ab06d6c9-dd03-4377-81f5-40e9f3c5a0c3.png) Thank you @LikeGiver, I understand this point now. Argmax is used to locate the index (i_eot) of [EOT] at tokenized prompts. Once we locate it , we use the features of [EOT] by x[batchsize, i_eot] to represent the features of prompts

bsakaguchi commented

2026-10-31 21:12:14 +00:00

Author

Owner

@ZHUYMGeo commented on GitHub (Dec 25, 2023):

I have the same question. In my prompts, the number 49407 mean the end token of each prompt, not the useful meaning of each token. for example, [49406, 518, 34606, 771, 4267, 7863, 6898, 518, 4960, 2445, 537,
791, 1025, 33811, 538, 26878, 49407, 0, 0, 0, 0, ..., 0], I think i should choose the meaningful token except 49406 and 49407. Is it work?

@ZHUYMGeo commented on GitHub (Dec 25, 2023): I have the same question. In my prompts, the number 49407 mean the end token of each prompt, not the useful meaning of each token. for example, [49406, 518, 34606, 771, 4267, 7863, 6898, 518, 4960, 2445, 537, 791, 1025, 33811, 538, 26878, 49407, 0, 0, 0, 0, ..., 0], I think i should choose the meaningful token except 49406 and 49407. Is it work?

bsakaguchi referenced this issue

2026-10-31 21:12:17 +00:00

[GH-ISSUE #221] Uninstalling pytorch? #194

bsakaguchi referenced this issue

2026-10-31 21:15:18 +00:00

[PR #191] [MERGED] use pkg_resources for PyTorch version checks in notebooks #470