openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #307] Embedding sequence of images into CLIP #266

New issue
Open
opened 2026-10-31 21:13:10 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-31 21:13:10 +00:00
Owner
Copy link

Originally created by @justlike-prog on GitHub (Dec 27, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/307

Hi,

Is there a way to use CLIP to embed whole albums of photos and then check similarity with certain phrases on what the album is about? I have a hard time to think about how I would encode a sequence of images into CLIP? Maybe there are some papers about it out there?

Thanks for any suggestion!

Originally created by @justlike-prog on GitHub (Dec 27, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/307 Hi, Is there a way to use CLIP to embed whole albums of photos and then check similarity with certain phrases on what the album is about? I have a hard time to think about how I would encode a sequence of images into CLIP? Maybe there are some papers about it out there? Thanks for any suggestion!
bsakaguchi commented 2026-10-31 21:13:11 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Jan 9, 2023):

I'd use the CLIP image encoder to get the image embeddings for each image, and then either:

  1. simply take the average of all those images and use that as the "album embedding"; or
  2. given an input text, calculate the cosine similarity with each of the image embedding, and check if the majority of them is higher than a certain value.
<!-- gh-comment-id:1375291510 --> @jongwook commented on GitHub (Jan 9, 2023): I'd use the CLIP image encoder to get the image embeddings for each image, and then either: 1. simply take the average of all those images and use that as the "album embedding"; or 2. given an input text, calculate the cosine similarity with each of the image embedding, and check if the majority of them is higher than a certain value.
bsakaguchi commented 2026-10-31 21:13:11 +00:00
Author
Owner
Copy link

@sdalinluo commented on GitHub (Feb 23, 2023):

Hi,image_features = model.encode_image(image)
and the output of encode_image is a tensor of dimension 512. Do you know how to control dimension of output?
eg, I want 128 or 64 dimension
Many thanks.

<!-- gh-comment-id:1441487939 --> @sdalinluo commented on GitHub (Feb 23, 2023): Hi,image_features = model.encode_image(image) and the output of encode_image is a tensor of dimension 512. Do you know how to control dimension of output? eg, I want 128 or 64 dimension Many thanks.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#266
No description provided.