openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #310] Why does CLIP always need softmax and not simple Cosine Similarity #270

New issue

Open

opened 2026-10-31 21:13:13 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-31 21:13:13 +00:00

Owner

Originally created by @evergreenllc2020 on GitHub (Jan 4, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/310

I would like to use CLIP embeddings for text and images in elastic search. It turns out that CLIP embeddings always need at least 2 text inputs for every image, and it does softmax on that. Is there a way to regenerate embeddings so that I can directly use simple cosine similarity between one text input and one image input? Doing softmax in elastic search for 2 text inputs and one image embedding at run time is complicated and expensive. For other embeddings like BERT , we can directly use cosine similarity

Here is sample code. How can I avoid softmax at runtime and just use one text input per image?

with torch.no_grad():
image_features = model.encode_image(image_input)
text_features = model.encode_text(text_inputs)
image_features /= image_features.norm(dim=-1, keepdim=True)
print(image_features.shape)
print(text_features.shape)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (100.0 * image_features @ text_features.T)[0].cpu().numpy()
print(similarity)
similarity = np.exp(similarity) / np.sum(np.exp(similarity), axis=0)

Originally created by @evergreenllc2020 on GitHub (Jan 4, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/310 I would like to use CLIP embeddings for text and images in elastic search. It turns out that CLIP embeddings always need at least 2 text inputs for every image, and it does softmax on that. Is there a way to regenerate embeddings so that I can directly use simple cosine similarity between one text input and one image input? Doing softmax in elastic search for 2 text inputs and one image embedding at run time is complicated and expensive. For other embeddings like BERT , we can directly use cosine similarity Here is sample code. How can I avoid softmax at runtime and just use one text input per image? with torch.no_grad(): image_features = model.encode_image(image_input) text_features = model.encode_text(text_inputs) image_features /= image_features.norm(dim=-1, keepdim=True) print(image_features.shape) print(text_features.shape) text_features /= text_features.norm(dim=-1, keepdim=True) similarity = (100.0 * image_features @ text_features.T)[0].cpu().numpy() print(similarity) similarity = np.exp(similarity) / np.sum(np.exp(similarity), axis=0)

bsakaguchi commented

2026-10-31 21:13:15 +00:00

Author

Owner

@Rijgersberg commented on GitHub (Jan 24, 2023):

No need to do the softmax unless you want to do classification. You can do comparisons by computing the cosine similarity between image_features and text_features directly.

@Rijgersberg commented on GitHub (Jan 24, 2023): No need to do the softmax unless you want to do classification. You can do comparisons by computing the cosine similarity between `image_features` and `text_features` directly.

bsakaguchi commented

2026-10-31 21:13:15 +00:00

Author

Owner

@fractaldna22 commented on GitHub (Mar 30, 2023):

You don't need to do

image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)

if you're using cosine_similarity. torch.cosine_similarity(x, y) already normalizes the inputs, by nature of cosines. To normalize via x = x / x.norm() , and then running that through cosine_similarity, would be to be dividing by norm Twice. That would seriously degrade the data going through and damage the accuracy

@fractaldna22 commented on GitHub (Mar 30, 2023): You don't need to do image_features /= image_features.norm(dim=-1, keepdim=True) text_features /= text_features.norm(dim=-1, keepdim=True) if you're using cosine_similarity. torch.cosine_similarity(x, y) already normalizes the inputs, by nature of cosines. To normalize via x = x / x.norm() , and then running that through cosine_similarity, would be to be dividing by norm Twice. That would seriously degrade the data going through and damage the accuracy

bsakaguchi commented

2026-10-31 21:13:15 +00:00

Author

Owner

@fractaldna22 commented on GitHub (Mar 30, 2023):

They only do x /= x.norm() when they use

similarity = (100.0 * image_features @ text_features.T)[0].cpu().numpy()

since this isnt actualy cosine similarity, its x @ y.T, thus x and y need to be divided by norm first. But if you use

similarity = (torch.cosine_similarity(image_features, text_features).view(-1, image_features.shape[0]).T.mean(1)).mean(0, True)

im pretty sure youd get the similarity for 1 and 1 and you can even use that for loss in a backward pass if you first multiple that similarity by -1

@fractaldna22 commented on GitHub (Mar 30, 2023): They only do x /= x.norm() when they use similarity = (100.0 * image_features @ text_features.T)[0].cpu().numpy() since this isnt actualy cosine similarity, its x @ y.T, thus x and y need to be divided by norm first. But if you use similarity = (torch.cosine_similarity(image_features, text_features).view(-1, image_features.shape[0]).T.mean(1)).mean(0, True) im pretty sure youd get the similarity for 1 and 1 and you can even use that for loss in a backward pass if you first multiple that similarity by -1

bsakaguchi commented

2026-10-31 21:13:15 +00:00

Author

Owner

@Externalhappy commented on GitHub (Aug 25, 2023):

So, if we calculate the similarity using ,
similarity = F.cosine_similarity(x, y)
without normalizing the image and text features, the computed cosine similarity will have the same output as

x = x / x.norm(dim=-1, keepdim=True) 
y = y / y.norm(dim=-1, keepdim=True)
similarity = x @y.T

Is that correct?

@Externalhappy commented on GitHub (Aug 25, 2023): So, if we calculate the similarity using , `similarity = F.cosine_similarity(x, y) ` without normalizing the image and text features, the computed cosine similarity will have the same output as ``` x = x / x.norm(dim=-1, keepdim=True) y = y / y.norm(dim=-1, keepdim=True) similarity = x @y.T ``` Is that correct?

bsakaguchi commented

2026-10-31 21:13:15 +00:00

Author

Owner

@Dinosaurcubs commented on GitHub (Apr 27, 2024):

You don't need to do

image_features /= image_features.norm(dim=-1, keepdim=True) text_features /= text_features.norm(dim=-1, keepdim=True)

if you're using cosine_similarity. torch.cosine_similarity(x, y) already normalizes the inputs, by nature of cosines. To normalize via x = x / x.norm() , and then running that through cosine_similarity, would be to be dividing by norm Twice. That would seriously degrade the data going through and damage the accuracy

If I just want to use the visual encoder to get the output visual featuere for downstream tasks，is it necessay to add the 'image_features /= image_features.norm(dim=-1, keepdim=True)'?

@Dinosaurcubs commented on GitHub (Apr 27, 2024): > You don't need to do > > image_features /= image_features.norm(dim=-1, keepdim=True) text_features /= text_features.norm(dim=-1, keepdim=True) > > if you're using cosine_similarity. torch.cosine_similarity(x, y) already normalizes the inputs, by nature of cosines. To normalize via x = x / x.norm() , and then running that through cosine_similarity, would be to be dividing by norm Twice. That would seriously degrade the data going through and damage the accuracy If I just want to use the visual encoder to get the output visual featuere for downstream tasks，is it necessay to add the 'image_features /= image_features.norm(dim=-1, keepdim=True)'?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#270

No description provided.

Rows
Columns