openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #148] Doesn't Clip use text embedding space sparsely? #129

New issue

Closed

opened 2026-10-31 21:11:29 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:11:29 +00:00

Owner

Originally created by @notou10 on GitHub (Sep 2, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/148

Hi, thanks for your work and I felt something curious with the text embedding space.

As a result of comparing the 512-dimensional text_features output from the text encoder, no matter how low the cosine similarity is, it does not fall below 0.6 (even if there is no relationship between text attribute).

text_tokens = clip.tokenize(["This is " + desc for desc in description]).cuda()
with torch.no_grad():
    text_features = model.encode_text(text_tokens).float()

#for cos-sim
text_features_normalized = text_features.clone().detach()
text_features_normalized /= text_features_normalized.norm(dim=-1, keepdim=True)
similarity_text = text_features_normalized.cpu().numpy() @ text_features_normalized.cpu().numpy().T

count = len(description)
plt.xticks([])
plt.close()

plt.figure(figsize=(20, 14))
plt.imshow(similarity_text, vmin=0.1, vmax=1.0)
plt.colorbar()

plt.xticks(range(count), description, rotation = 90, fontsize = 7)
plt.rcParams['xtick.bottom'] = plt.rcParams['xtick.labelbottom'] = True
plt.rcParams['xtick.top'] = plt.rcParams['xtick.labeltop'] = True
plt.yticks(range(count), description, fontsize=7)

for x in range(similarity_text.shape[1]):
    for y in range(similarity_text.shape[0]):
        plt.text(x, y, f"{similarity_text[y, x]:.2f}", ha="center", va="center", size = 2)

plt.xlim([-0.5, count - 0.5])
plt.ylim([count - 0.5, -0.5])

plt.title("Cosine similarity between text embeddings", size=20)
plt.savefig('150-class-cos-sim.png', dpi = 300)

and the part of result was below

(150 texts comparision full image : https://drive.google.com/file/d/13pERjTbxt0RBOeCdzmNT-xnSTLFXUS-x/view)

What I have Tried :

I tried using prompt engineering besides "This is" but it didn't have much effect.
Although L2 distance was compared, there were many cases where attributes of irrelevant meaning in text were close in text embedding.
Nonsense texts such as "agdfgafhf" are placed in a similar position to regular texts.

What I am wondering is seeing that the cosine similarlty does not drop below 0.6, it seems that the text embedding space is not sparsely used.

What do you think?

Again, really thankful of your works, from South Korea

Originally created by @notou10 on GitHub (Sep 2, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/148 Hi, thanks for your work and I felt something curious with the text embedding space. As a result of comparing the 512-dimensional text_features output from the text encoder, no matter how low the cosine similarity is, it does not fall below 0.6 (even if there is no relationship between text attribute). ```python text_tokens = clip.tokenize(["This is " + desc for desc in description]).cuda() with torch.no_grad(): text_features = model.encode_text(text_tokens).float() #for cos-sim text_features_normalized = text_features.clone().detach() text_features_normalized /= text_features_normalized.norm(dim=-1, keepdim=True) similarity_text = text_features_normalized.cpu().numpy() @ text_features_normalized.cpu().numpy().T count = len(description) plt.xticks([]) plt.close() plt.figure(figsize=(20, 14)) plt.imshow(similarity_text, vmin=0.1, vmax=1.0) plt.colorbar() plt.xticks(range(count), description, rotation = 90, fontsize = 7) plt.rcParams['xtick.bottom'] = plt.rcParams['xtick.labelbottom'] = True plt.rcParams['xtick.top'] = plt.rcParams['xtick.labeltop'] = True plt.yticks(range(count), description, fontsize=7) for x in range(similarity_text.shape[1]): for y in range(similarity_text.shape[0]): plt.text(x, y, f"{similarity_text[y, x]:.2f}", ha="center", va="center", size = 2) plt.xlim([-0.5, count - 0.5]) plt.ylim([count - 0.5, -0.5]) plt.title("Cosine similarity between text embeddings", size=20) plt.savefig('150-class-cos-sim.png', dpi = 300) ``` and the part of result was below ![image](https://user-images.githubusercontent.com/45427036/131786152-b11602af-1314-418a-b0d5-7b5243eaf229.png) (150 texts comparision full image : https://drive.google.com/file/d/13pERjTbxt0RBOeCdzmNT-xnSTLFXUS-x/view) What I have Tried : - I tried using prompt engineering besides "This is" but it didn't have much effect. - Although L2 distance was compared, there were many cases where attributes of irrelevant meaning in text were close in text embedding. - Nonsense texts such as "agdfgafhf" are placed in a similar position to regular texts. What I am wondering is seeing that the cosine similarlty does not drop below 0.6, it seems that the text embedding space is not sparsely used. What do you think? Again, really thankful of your works, from South Korea

bsakaguchi closed this issue

2026-10-31 21:11:29 +00:00

bsakaguchi commented

2026-10-31 21:11:30 +00:00

Author

Owner

@jongwook commented on GitHub (Sep 24, 2021):

A few ways to interpret these:

The logits (inputs to the softmax) are 100 times the cosine similarity, so a difference of 0.01 of cosine similarity can be quite large in terms of classification probability. So the relative differences between them are more important than the fact that they are all above 0.6. For example, in your matrix, all words about emotions are very close (~0.95), while being a bit distant from the animal words (~0.8).

Even with the "This is" prompt, the transformer encoder had only one token or so to diverge from the common activation in each layer (we're using autoregressive masks), so I guess it might diverge a bit more when you try longer sentences than a word. But I wouldn't be surprised it never goes below 0.5 or so.

Geometrically, You can think of a cone with apex angle cos⁻¹(0.6) ≈ 53° whose top is at the origin of a high-dimensional space. Any point that sits inside that cone would have a cosine similarity higher than 0.6, but the volume of that cone is not necessarily sparse. A better yet simpler method for checking the dimensionality would be to check the magnitudes of the eigenvalues when you run PCA on the feature vectors. It might still be "sparse" in a sense that the vectors are not spanning the areas outside the cone, but that high-dimensional cone can still contain enough information.

The value is also relevant to the contrastive training objectives of the model as well as the dataset. If the difference between the least probable and most probable label in the predicted categorical probability distribution differ by a certain finite factor, the cosine distances can't differ by more than a certain value which is a function of that finite factor. The factor itself (how confident the model is on its answer compared to the other choices) is dependent on the regularization and other inductive bias baked into the model, in combination with the data distribution.

Being a nonlinear projection of the (supposedly low-dimensional) data manifold, there will be some embedding dimensions being less useful than others, but that's about the best that the training could do to utilize the embedding space as much as possible to pack whatever information it can extract from the data distribution and maximize the objective.

@jongwook commented on GitHub (Sep 24, 2021): A few ways to interpret these: The logits (inputs to the softmax) are 100 times the cosine similarity, so a difference of 0.01 of cosine similarity can be quite large in terms of classification probability. So the relative differences between them are more important than the fact that they are all above 0.6. For example, in your matrix, all words about emotions are very close (~0.95), while being a bit distant from the animal words (~0.8). Even with the "This is" prompt, the transformer encoder had only one token or so to diverge from the common activation in each layer (we're using autoregressive masks), so I guess it might diverge a bit more when you try longer sentences than a word. But I wouldn't be surprised it never goes below 0.5 or so. Geometrically, You can think of a cone with apex angle cos⁻¹(0.6) ≈ 53° whose top is at the origin of a high-dimensional space. Any point that sits inside that cone would have a cosine similarity higher than 0.6, but the volume of that cone is not necessarily sparse. A better yet simpler method for checking the dimensionality would be to check the magnitudes of the eigenvalues when you run PCA on the feature vectors. It might still be "sparse" in a sense that the vectors are not spanning the areas outside the cone, but that high-dimensional cone can still contain enough information. The value is also relevant to the contrastive training objectives of the model as well as the dataset. If the difference between the least probable and most probable label in the predicted categorical probability distribution differ by a certain finite factor, the cosine distances can't differ by more than a certain value which is a function of that finite factor. The factor itself (how confident the model is on its answer compared to the other choices) is dependent on the regularization and other inductive bias baked into the model, in combination with the data distribution. Being a nonlinear projection of the (supposedly low-dimensional) data manifold, there will be some embedding dimensions being less useful than others, but that's about the best that the training could do to utilize the embedding space as much as possible to pack whatever information it can extract from the data distribution and maximize the objective.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#129

No description provided.

Rows
Columns