[GH-ISSUE #224] Zero shot - evaluate labels independently #197

New issue

Closed

opened 2026-10-31 21:12:18 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:12:18 +00:00

Owner

Originally created by @subhamkhemka on GitHub (Mar 25, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/224

How can I consider the labels to be independent of each other ? Do not want the probs summing up to 1 instead want to evaluate each label on its own, sort of a multi label classification.

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

print("Label probs:", probs)  # prints: [[0.9927937  0.00421068 0.00299572]]

Thanks

Originally created by @subhamkhemka on GitHub (Mar 25, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/224 Hi How can I consider the labels to be independent of each other ? Do not want the probs summing up to 1 instead want to evaluate each label on its own, sort of a multi label classification. ``` import torch import clip from PIL import Image device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device) image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device) text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device) with torch.no_grad(): image_features = model.encode_image(image) text_features = model.encode_text(text) logits_per_image, logits_per_text = model(image, text) probs = logits_per_image.softmax(dim=-1).cpu().numpy() print("Label probs:", probs) # prints: [[0.9927937 0.00421068 0.00299572]] ``` Thanks

bsakaguchi closed this issue

2026-10-31 21:12:18 +00:00

bsakaguchi commented

2026-10-31 21:12:19 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 11, 2022):

You can use the cosine similarity of image_features and text_features which gives semantic similarity for each text label independently. This idea was called CLIPScore in a paper.

Note that the cosine similarity is meaningful only in a relative sense, and you'd probably want to select good threshold values specific for your use cases and datasets by cross validation. Also, see prompt engineering which will make the metric more robust.

@jongwook commented on GitHub (Apr 11, 2022): You can use the cosine similarity of `image_features` and `text_features` which gives semantic similarity for each text label independently. This idea was called [CLIPScore](https://arxiv.org/abs/2104.08718) in a paper. Note that the cosine similarity is meaningful only in a relative sense, and you'd probably want to select good threshold values specific for your use cases and datasets by cross validation. Also, see [prompt engineering](https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb) which will make the metric more robust.

bsakaguchi commented

2026-10-31 21:12:19 +00:00

Author

Owner

@justlike-prog commented on GitHub (Dec 27, 2022):

@jongwook @subhamkhemka So there is no way to get a per word probability rather than just the similarity? It seems to be quite hard to find a good threshold just by similarity.

@justlike-prog commented on GitHub (Dec 27, 2022): @jongwook @subhamkhemka So there is no way to get a per word probability rather than just the similarity? It seems to be quite hard to find a good threshold just by similarity.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#197

No description provided.

Rows
Columns