[GH-ISSUE #162] Questions about the spatial features of vision encoder in CLIP #140

New issue

Closed

opened 2026-10-31 21:11:37 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 21:11:37 +00:00

Owner

Originally created by @Wuziyi616 on GitHub (Oct 22, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/162

Thank you so much for opening source such great work! I'm currently thinking of adopting CLIP to some language-guided dense prediction vision task. Logically I think only using the global feature vector from the class_embedding from ViT (taking only x = self.ln_post(x[:, 0, :])) won't work well, because dense prediction task requires local features. So I'm thinking of instead of taking the global feature, I can actually take x[:, 1:, :], these features will somehow retain some spatial correspondence with the position in the image. But if I understand correctly, CLIP is trained only using contrastive loss between global vision feature and text feature. So I'm not sure if the remaining patch features are also well-trained/meaningful... Can you give me some insight in it? Or I wonder if there are some works studying/using these spatial features? Thanks in advance.

Originally created by @Wuziyi616 on GitHub (Oct 22, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/162 Thank you so much for opening source such great work! I'm currently thinking of adopting CLIP to some language-guided dense prediction vision task. Logically I think only using the global feature vector from the `class_embedding` from ViT (taking only `x = self.ln_post(x[:, 0, :])`) won't work well, because dense prediction task requires local features. So I'm thinking of instead of taking the global feature, I can actually take `x[:, 1:, :]`, these features will somehow retain some spatial correspondence with the position in the image. But if I understand correctly, CLIP is trained only using contrastive loss between global vision feature and text feature. So I'm not sure if the remaining patch features are also well-trained/meaningful... Can you give me some insight in it? Or I wonder if there are some works studying/using these spatial features? Thanks in advance.

bsakaguchi closed this issue

2026-10-31 21:11:37 +00:00

bsakaguchi commented

2026-10-31 21:11:38 +00:00

Author

Owner

@jongwook commented on GitHub (Oct 22, 2021):

Examples in #82 show it's somewhat possible to use the spatial features, and it'd be interesting to try fine-tuning an object detector from it. We haven't tested against any detection benchmarks.

@jongwook commented on GitHub (Oct 22, 2021): Examples in #82 show it's somewhat possible to use the spatial features, and it'd be interesting to try fine-tuning an object detector from it. We haven't tested against any detection benchmarks.

bsakaguchi commented

2026-10-31 21:11:38 +00:00

Author

Owner

@Wuziyi616 commented on GitHub (Oct 22, 2021):

Thank you so much for your prompt reply. One another question, just to make sure, the feature that CLIP paper used for downstream task (e.g. classification) is this line right (that's before linear projection)? So if I take the features of all the patches, I don't need to worry about this projection operation.

@Wuziyi616 commented on GitHub (Oct 22, 2021): Thank you so much for your prompt reply. One another question, just to make sure, the feature that CLIP paper used for downstream task (e.g. classification) is [this line](https://github.com/Wuziyi616/CLIP/blob/e184f608c5d5e58165682f7c332c3a8b4c1545f2/clip/model.py#L231) right (that's before linear projection)? So if I take the features of all the patches, I don't need to worry about this projection operation.

bsakaguchi commented

2026-10-31 21:11:38 +00:00

Author

Owner

@jongwook commented on GitHub (Oct 22, 2021):

yes, you can change that line to use [:, 1:, :] instead. For the downstream tasks we skipped the projection layer (by setting model.proj = None) and replaced with task-specific heads.

@jongwook commented on GitHub (Oct 22, 2021): yes, you can change that line to use `[:, 1:, :]` instead. For the downstream tasks we skipped the projection layer (by setting `model.proj = None`) and replaced with task-specific heads.

bsakaguchi commented

2026-10-31 21:11:38 +00:00

Author

Owner

@Wuziyi616 commented on GitHub (Oct 22, 2021):

Thanks! That helps me a lot.

@Wuziyi616 commented on GitHub (Oct 22, 2021): Thanks! That helps me a lot.

bsakaguchi referenced this issue

2026-10-31 21:15:15 +00:00

[PR #140] [MERGED] Reduce half of similarity muls after encoding (optimization) #465