mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #390] few shot classification with unknown class #336
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#336
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @gigasurgeon on GitHub (Sep 21, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/390
I have a very small dataset i.e. 10 images each class and total 500 classes. During inference, there is also a possibility that an image might come that does not belong to any of these 500 classes, and I have to identify it as well. How do I go about this? I have tried cosine similarity, but it is not very robust in finding the unknown class. Also, due to small images per class, the model overfits.
@gigasurgeon commented on GitHub (Nov 1, 2023):
I solved the problem of unknown class through CLIP model using this approach. I used its image embeddding as well as the text embeddings and added them together [torch.add()] and saved them locally as a .pt file. The text embedding was generated with either some words written on the object or a sentence describing the object. During test time, I generated only image embedding through CLIP and compared the cosine similarity. If it was below a certain threshold, I marked them as unknown or outlier. This gave me 100% accuracy for my usecase. You can use huggingface's openAI clip (giant variant worked best).