[GH-ISSUE #44] define unknown category? #42

New issue

Closed

opened 2026-10-31 21:10:09 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Owner

Originally created by @batrlatom on GitHub (Feb 18, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/44

Hi,
would it be possible to construct a prompt in the way, that it recognizes that the image does not belong to the selected categories?
For example, my prompt will have classes 'dog' and 'cat' and I will predict images of car. In this case, it will predict the category 'unknown' ( or any other category ) ?

Originally created by @batrlatom on GitHub (Feb 18, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/44 Hi, would it be possible to construct a prompt in the way, that it recognizes that the image does not belong to the selected categories? For example, my prompt will have classes 'dog' and 'cat' and I will predict images of car. In this case, it will predict the category 'unknown' ( or any other category ) ?

bsakaguchi closed this issue

2026-10-31 21:10:09 +00:00

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 20, 2021):

Hi! CLIP does not explicitly support "unknown" category out-of-the-box, but I can imaging two alternative approaches

You can set a threshold in terms of cosine similarity and classify an image as "unknown" if its feature is not within the threshold of either the "dog" or "cat" vector. This will effectively create a high-dimensional cone in the embedding space for each category, and classify anything falls out of the two cones as "unknown".

To make it a bit more robust, you could also try having a collection of "cat" and "dog" vectors, by converting cat and dog images and prompt-engineering cat- and dog-describing texts, and make them vote, like in a k-NN classifier.

Another option is to make a collection of words NOT corresponding to cats and dogs, e.g. from all the other class names from ImageNet, and see if enough probability mass moves away to those categories.

I can also imagine using a combination of these two approaches, but in any case, it'll require some manual tuning of thresholds.

@jongwook commented on GitHub (Feb 20, 2021): Hi! CLIP does not explicitly support "unknown" category out-of-the-box, but I can imaging two alternative approaches 1. You can set a threshold in terms of cosine similarity and classify an image as "unknown" if its feature is not within the threshold of either the "dog" or "cat" vector. This will effectively create a high-dimensional cone in the embedding space for each category, and classify anything falls out of the two cones as "unknown". To make it a bit more robust, you could also try having a collection of "cat" and "dog" vectors, by converting cat and dog images and prompt-engineering cat- and dog-describing texts, and make them vote, like in a k-NN classifier. 2. Another option is to make a collection of words NOT corresponding to cats and dogs, e.g. from all the other class names from ImageNet, and see if enough probability mass moves away to those categories. I can also imagine using a combination of these two approaches, but in any case, it'll require some manual tuning of thresholds.

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@batrlatom commented on GitHub (Mar 2, 2021):

Thank you, I will try it.

@batrlatom commented on GitHub (Mar 2, 2021): Thank you, I will try it.

bsakaguchi referenced this issue

2026-10-31 21:10:52 +00:00

[GH-ISSUE #85] Linear projection & normalization after encoder #76