[GH-ISSUE #268] How to evaluate the CLIP model results?

bsakaguchi commented

2026-10-31 21:12:46 +00:00

Owner

Originally created by @karndeepsingh on GitHub (Jul 18, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/268

Hi, I have trained a clip model using image and its caption. Now, i want to evaluate the performance of the model like Precision, Recall, F1 Score. How I can do that?

Any suggestion would help to do so.

Thanks

Originally created by @karndeepsingh on GitHub (Jul 18, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/268 Hi, I have trained a clip model using image and its caption. Now, i want to evaluate the performance of the model like Precision, Recall, F1 Score. How I can do that? Any suggestion would help to do so. Thanks

bsakaguchi commented

2026-10-31 21:12:47 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jul 18, 2022):

what limit you from doing so? you can simply calculate it using scikit learn, just put it inside the training loop. but i wonder what you mean by precision recall f1 in this context ? CLIP is not a classification model, this is just an embedding model, with an objective function that mimic a classification problem, even though it is not a classification. So it's better to state what are you tring to achieve with classification metrics here

@vinson2233 commented on GitHub (Jul 18, 2022): what limit you from doing so? you can simply calculate it using scikit learn, just put it inside the training loop. but i wonder what you mean by precision recall f1 **in this context** ? CLIP is not a classification model, this is just an embedding model, with an objective function that `mimic` a classification problem, even though it is not a classification. So it's better to state what are you tring to achieve with classification metrics here

bsakaguchi commented

2026-10-31 21:12:47 +00:00

Author

Owner

@karndeepsingh commented on GitHub (Jul 18, 2022):

what limit you from doing so? you can simply calculate it using scikit learn, just put it inside the training loop. but i wonder what you mean by precision recall f1 in this context ? CLIP is not a classification model, this is just an embedding model, with an objective function that mimic a classification problem, even though it is not a classification. So it's better to state what are you tring to achieve with classification metrics here

That’s true. Let me explain what I am trying to achieve.
So, I have products images and its description and want to generate product embeddings using the image and description for each of the products. Can CLIP can help to generate embedding for products? If so, then once I train the CLIP model using image and text, how I can evaluate efficiency and accuracy of the embeddings learned.

Please suggest. My objective is to generate a single product embeddings using its image and description.
waiting for you suggestion.

Thanks

@karndeepsingh commented on GitHub (Jul 18, 2022): > what limit you from doing so? you can simply calculate it using scikit learn, just put it inside the training loop. but i wonder what you mean by precision recall f1 **in this context** ? CLIP is not a classification model, this is just an embedding model, with an objective function that `mimic` a classification problem, even though it is not a classification. So it's better to state what are you tring to achieve with classification metrics here That’s true. Let me explain what I am trying to achieve. So, I have products images and its description and want to generate product embeddings using the image and description for each of the products. Can CLIP can help to generate embedding for products? If so, then once I train the CLIP model using image and text, how I can evaluate efficiency and accuracy of the embeddings learned. Please suggest. My objective is to generate a single product embeddings using its image and description. waiting for you suggestion. Thanks

bsakaguchi commented

2026-10-31 21:12:47 +00:00

Author

Owner

@PsVenom commented on GitHub (Jul 27, 2022):

Please suggest. My objective is to generate a single product embeddings using its image and description. waiting for you suggestion.

Thanks

Using an F1 score requires that you have two important values - y_true and y_pred. Although you can't exactly get those values using CLIP, you can still check the performance using F1 scores.
If you've already trained the model, then evaluation is pretty straightforward:

create a validation set of images and their real descriptions
convert both of them via model.encode_image(image: Tensor) and model.encode_text(text: Tensor)

now there are multiple ways to compare these given encodings, one of them is the cosine similarity score used in model(image: Tensor, text: Tensor)
theoretically, you can also use the F1 metric (or any other metric for that matter) because I believe the dimensions of both embeddings are the same. But do keep in mind that there's really no y_true in this case, just two different predicted encodings that need to be matched
let me know if this worked for you

@PsVenom commented on GitHub (Jul 27, 2022): > > Please suggest. My objective is to generate a single product embeddings using its image and description. waiting for you suggestion. > > Thanks Using an F1 score requires that you have two important values - y_true and y_pred. Although you can't exactly get those values using CLIP, you can still check the performance using F1 scores. If you've already trained the model, then evaluation is pretty straightforward: - create a validation set of images and their real descriptions - convert both of them via ` model.encode_image(image: Tensor) ` and ` model.encode_text(text: Tensor) ` now there are multiple ways to compare these given encodings, one of them is the cosine similarity score used in ` model(image: Tensor, text: Tensor)` theoretically, you can also use the F1 metric (or any other metric for that matter) because I believe the dimensions of both embeddings are the same. But do keep in mind that there's really no y_true in this case, just two different predicted encodings that need to be matched let me know if this worked for you

bsakaguchi commented

2026-10-31 21:12:47 +00:00

Author

Owner

@karndeepsingh commented on GitHub (Jul 30, 2022):

Hi,
Thanks for the answers @PsVenom . I would like to ask few more question that are as:

I have images of the product and its attributes like gender, color, type, size etc. how I can prepare text using these attributes so that I can use that text to pair with the image? And later while inferencing, I can get to know all the attributes of the new image. If you can give me some idea on preparing the text of the attribute so that I can train CLIP model for better context training.
While training official CLIP model did the author directly used class as its image description or did they prepare contextual sentence made of class for training CLIP model. For example: Image: "Image of a dog" , Class: "DOG". then did they use ( "Image of dog", "DOG") as (Image-Text) pair or did they use some contextual text made out of class like "A DOG playing in grass" and then Pairing its image.

Please help me to understand the text preparation step as I have classes of a particular product as its attributes like color, type, size, gender, etc and how i can prepare better description out of these attributes and then while inferencing I can extract these attributes from the new image.

Thanks.

@karndeepsingh commented on GitHub (Jul 30, 2022): Hi, Thanks for the answers @PsVenom . I would like to ask few more question that are as: 1. I have images of the product and its attributes like gender, color, type, size etc. how I can prepare text using these attributes so that I can use that text to pair with the image? And later while inferencing, I can get to know all the attributes of the new image. If you can give me some idea on preparing the text of the attribute so that I can train CLIP model for better context training. 2. While training official CLIP model did the author directly used class as its image description or did they prepare contextual sentence made of class for training CLIP model. For example: Image: "Image of a dog" , Class: "DOG". then did they use ( "Image of dog", "DOG") as (Image-Text) pair or did they use some contextual text made out of class like "A DOG playing in grass" and then Pairing its image. Please help me to understand the text preparation step as I have classes of a particular product as its attributes like color, type, size, gender, etc and how i can prepare better description out of these attributes and then while inferencing I can extract these attributes from the new image. Thanks.

bsakaguchi referenced this issue

2026-10-31 21:15:21 +00:00

[PR #234] [MERGED] ViT-L/14@336px #476

Rows
Columns

[GH-ISSUE #268] How to evaluate the CLIP model results? #234