[GH-ISSUE #214] Modify visual positional embedding to make CLIP independent of input resolution #188

New issue

Closed

opened 2026-10-31 21:12:10 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:12:10 +00:00

Owner

Originally created by @ABaldrati on GitHub (Feb 9, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/214

Thank you guys for the awesome work and for releasing CLIP models.

In ResNet models the only component which prevents an arbitrary input resolution is the learned positional embedding in the 2D attention pooling.

I would like to ask if you have performed some experiments with a different approach: for example using a positional encoding which can adapt to a different sequence length or directly removing the positional encoding.

Thanks again for the amazing work

Originally created by @ABaldrati on GitHub (Feb 9, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/214 Thank you guys for the awesome work and for releasing CLIP models. In ResNet models the only component which prevents an arbitrary input resolution is the learned positional embedding in the 2D attention pooling. I would like to ask if you have performed some experiments with a different approach: for example using a positional encoding which can adapt to a different sequence length or directly removing the positional encoding. Thanks again for the amazing work

bsakaguchi closed this issue

2026-10-31 21:12:10 +00:00

bsakaguchi commented

2026-10-31 21:12:12 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 11, 2022):

Thanks for the kind words!

We haven't done the specific experiments you described, but I expect replacing the attention pool layer with a simple average pooling, or pooling the attention-pooled features from different crops from the original image, would still work to some degree.

We did bilinear interpolation of the positional encoding vectors when fine-tuning the ViT/L-14 model on 336px which expanded the 16x16 grid to 24x24.

@jongwook commented on GitHub (Apr 11, 2022): Thanks for the kind words! We haven't done the specific experiments you described, but I expect replacing the attention pool layer with a simple average pooling, or pooling the attention-pooled features from different crops from the original image, would still work to some degree. We did bilinear interpolation of the positional encoding vectors when fine-tuning the ViT/L-14 model on 336px which expanded the 16x16 grid to 24x24.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#188

No description provided.

Rows
Columns