mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #260] How to use CLIP for duplicate or near-duplicate images? #228
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#228
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @smith-co on GitHub (Jun 29, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/260
Given pair of images, my use case is to detect whether they are duplicate or not.
(imageX, imageY) = verdict/score
verdict = duplicate/not duplicate/near duplicate
How can I use CLIP for this use case?
@vinson2233 commented on GitHub (Jun 30, 2022):
let's say you have 10 different images. You want to check which one is a duplicate or not.
Just use
model.encode_image(img)to encode the 10 images, you will have a tensor with size 10 x 512.Then check for cosine similarity between the vectors. Put a threshold, maybe 0.95, so if two image have cosine similarity between embedding > threshold, then label it as duplicate / near duplicate depending on your definition.
@smith-co commented on GitHub (Jun 30, 2022):
Computing
model.encode_image(img)and then usingcosine_similarityworked outremarkablywell for me for general images.However, for the specific
imagedataset I am trying the above approach is not working out so well. So it appears I have to fine tune the model following https://github.com/openai/CLIP/issues/83.But for CLIP fine tuning I would always need (image, text) pair. Or for near-duplicate detection, I can fine tune differently?
@vinson2233 commented on GitHub (Jun 30, 2022):
If you are trying to create specific duplicate detection on a dataset where the definition of duplicate does not follow the general definition of duplicate, then I think it is possible to train the model for a specific context. but you need a dataset of pairs of images and label whether they are duplicates or not and you need to customize the training code a bit. So the format of data would be like this :
the changes you need to make are :
model(img,text), you need to manually create cosine similarity matrix usingmodel.encode_image(img1)andmodel.encode_image(img2)torch.arangeto follow the label you have in your dataset.@vinson2233 commented on GitHub (Jun 30, 2022):
or try to use larger model and see whether a larger model can create better embedding for your need
@smith-co commented on GitHub (Jun 30, 2022):
@vinson2233 you are absolutely right - I am trying to create specific duplicate detection on a dataset.
I have been looking into the steps listed here https://github.com/openai/CLIP/issues/83. Have not really followed what changes you referred to here.
Alternately, I suppose I could also create a (img,text) pair like this:
as
1.jpg,2.jpg,3.jpg,4.jpgare near duplicates I am thinking to use same tag i.e.tag1. I presume this should work as well?@smith-co commented on GitHub (Jun 30, 2022):
What model are you referring to here?
@vinson2233 commented on GitHub (Jun 30, 2022):
the encoder variations
github.com/openai/CLIP@b46f5ac758/clip/clip.py (L30-L40)@vinson2233 commented on GitHub (Jun 30, 2022):
sure, this is also possible, so you are trying to catalog similar images under 1 category. But you need to modify the loss function calculation, to only calculate the loss on the image axis, not on the text axis because now one text can fall into multiple categories.
Another alternative is in each batch, make sure to only get 1 member from each tag, that way you can calculate loss from both axes.
I never tried this. so good luck.