openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #356] Laion embeddings and clip output for the same image do not match exactly #309

New issue

Open

opened 2026-10-31 21:13:41 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 21:13:41 +00:00

Owner

Originally created by @stevebottos on GitHub (May 4, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/356

The laion website provides embeddings and parquets which tie an embedding at an index in the array to its associated metadata. In theory the clip output and the laion embedding for the same image should be exactly the same, but they're not. Here's how to reproduce:

from io import BytesIO

import clip
import numpy as np
import pandas as pd
import requests
import torch
from PIL import Image
from sklearn.metrics.pairwise import cosine_similarity

# embeddings: https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/img_emb/img_emb_0.npy
# parquet: https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/metadata/metadata_0.parquet

INDEX = 0

laion_embedding = np.load("img_emb_0.npy")[INDEX]
laion_embedding = np.expand_dims(laion_embedding, 0)
url = pd.read_parquet("metadata_0.parquet")["url"][INDEX]

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

response = requests.get(url)
image = Image.open(BytesIO(response.content))
image = preprocess(image).unsqueeze(0).to(device)

with torch.no_grad():
    clip_embedding = model.encode_image(image).cpu().numpy()


print("Pre-norm difference:", (laion_embedding - clip_embedding).sum())
clip_embedding = clip_embedding / np.linalg.norm(clip_embedding)
print("Post-norm difference:", (laion_embedding - clip_embedding).sum())
print("Cosine sim:", cosine_similarity(laion_embedding, clip_embedding))

This outputs:

Pre-norm difference: -4.79
Post-norm difference: 0.1592
Cosine sim: [[0.91325109]]

I've verified that laion uses the ViT-B/32 backbone as well. I'm wondering what might be causing the discrepancy here. Any ideas?

Originally created by @stevebottos on GitHub (May 4, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/356 The laion website provides embeddings and parquets which tie an embedding at an index in the array to its associated metadata. In theory the clip output and the laion embedding for the same image should be exactly the same, but they're not. Here's how to reproduce: ``` from io import BytesIO import clip import numpy as np import pandas as pd import requests import torch from PIL import Image from sklearn.metrics.pairwise import cosine_similarity # embeddings: https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/img_emb/img_emb_0.npy # parquet: https://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/embeddings/metadata/metadata_0.parquet INDEX = 0 laion_embedding = np.load("img_emb_0.npy")[INDEX] laion_embedding = np.expand_dims(laion_embedding, 0) url = pd.read_parquet("metadata_0.parquet")["url"][INDEX] device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device) response = requests.get(url) image = Image.open(BytesIO(response.content)) image = preprocess(image).unsqueeze(0).to(device) with torch.no_grad(): clip_embedding = model.encode_image(image).cpu().numpy() print("Pre-norm difference:", (laion_embedding - clip_embedding).sum()) clip_embedding = clip_embedding / np.linalg.norm(clip_embedding) print("Post-norm difference:", (laion_embedding - clip_embedding).sum()) print("Cosine sim:", cosine_similarity(laion_embedding, clip_embedding)) ``` This outputs: ``` Pre-norm difference: -4.79 Post-norm difference: 0.1592 Cosine sim: [[0.91325109]] ``` I've verified that laion uses the ViT-B/32 backbone as well. I'm wondering what might be causing the discrepancy here. Any ideas?

bsakaguchi commented

2026-10-31 21:13:42 +00:00

Author

Owner

@stevebottos commented on GitHub (May 4, 2023):

This issue is related to this issue: https://github.com/rom1504/clip-retrieval/discussions/100 which didn't seem to be resolved. I was able to replicate the results in this issue, but it's not a perfect solution.

@stevebottos commented on GitHub (May 4, 2023): This issue is related to this issue: https://github.com/rom1504/clip-retrieval/discussions/100 which didn't seem to be resolved. I was able to replicate the results in this issue, but it's not a perfect solution.

bsakaguchi commented

2026-10-31 21:13:42 +00:00

Author

Owner

@TianRui-Song717 commented on GitHub (May 17, 2023):

Another comparable problem is that the text embedding on the LAION400 does not match the output of the CLIP-ViT-B/32 text-encoder.

@TianRui-Song717 commented on GitHub (May 17, 2023): Another comparable problem is that the text embedding on the LAION400 does not match the output of the CLIP-ViT-B/32 text-encoder.

bsakaguchi commented

2026-10-31 21:13:42 +00:00

Author

Owner

@TianRui-Song717 commented on GitHub (May 17, 2023):

Another comparable problem is that the text embedding on the LAION400 does not match the output of the CLIP-ViT-B/32 text-encoder.

@stevebottos The text embedding should not be affected by image resizing ....

@TianRui-Song717 commented on GitHub (May 17, 2023): > Another comparable problem is that the text embedding on the LAION400 does not match the output of the CLIP-ViT-B/32 text-encoder. @stevebottos The text embedding should not be affected by image resizing ....

bsakaguchi commented

2026-10-31 21:13:42 +00:00

Author

Owner

@stevebottos commented on GitHub (May 17, 2023):

@TianRui-Song717 That's strange, any luck discovering why?

@stevebottos commented on GitHub (May 17, 2023): @TianRui-Song717 That's strange, any luck discovering why?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#309

No description provided.

Rows
Columns