openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #31] Computing target matrix differently #28

New issue

Closed

opened 2026-10-31 21:10:03 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 21:10:03 +00:00

Owner

Originally created by @nishanthcgit on GitHub (Feb 1, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/31

Now I realize that the released models and the model from the keras code examples are different, and are possibly trained differently. As far as I can see, as per this issue the openai CLIP model uses a target matrix of torch.eye(batch_size), while in the keras code examples:

To calculate the loss, we compute the pairwise dot-product similarity between each caption_i and images_j in the batch as the predictions. The target similarity between caption_i and image_j is computed as the average of the (dot-product similarity between caption_i and caption_j) and (the dot-product similarity between image_i and image_j). Then, we use crossentropy to compute the loss between the targets and the predictions.

So the target in case of the image and caption not being matched isn't a 0 but the average of the distances between the image embeddings and text embeddings. I realize these are 2 different approaches to train the model but do you think setting the target to be the average instead of just a zero might help convergence/accuracy?

Originally created by @nishanthcgit on GitHub (Feb 1, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/31 Now I realize that the released models and the model from the [keras code examples](https://keras.io/examples/nlp/nl_image_search/) are different, and are possibly trained differently. As far as I can see, as per [this issue](https://github.com/openai/CLIP/issues/3) the openai CLIP model uses a target matrix of torch.eye(batch_size), while in the keras code examples: > To calculate the loss, we compute the pairwise dot-product similarity between each caption_i and images_j in the batch as the predictions. The target similarity between caption_i and image_j is computed as the average of the (dot-product similarity between caption_i and caption_j) and (the dot-product similarity between image_i and image_j). Then, we use crossentropy to compute the loss between the targets and the predictions. So the target in case of the image and caption not being matched isn't a 0 but the average of the distances between the image embeddings and text embeddings. I realize these are 2 different approaches to train the model but do you think setting the target to be the average instead of just a zero might help convergence/accuracy?

bsakaguchi closed this issue

2026-10-31 21:10:03 +00:00

bsakaguchi commented

2026-10-31 21:10:04 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 1, 2021):

The linked Keras example is not affiliated with us, and it seems that they used quite different architecture choices, such as using pre-trained Xception and BERT models, as opposed to training ViT/ResNet and autoregressive transformer from scratch.

Although we haven't experimented with their target matrix, I suspect that it'll be more difficult to train from scratch because the within-domain similarity is not as informative as the case of using pre-trained models; but I'm happy to be corrected on this. It'd be slower to compute since it's no longer one-hot labels.

@jongwook commented on GitHub (Feb 1, 2021): The linked Keras example is not affiliated with us, and it seems that they used quite different architecture choices, such as using pre-trained Xception and BERT models, as opposed to training ViT/ResNet and autoregressive transformer from scratch. Although we haven't experimented with their target matrix, I suspect that it'll be more difficult to train from scratch because the within-domain similarity is not as informative as the case of using pre-trained models; but I'm happy to be corrected on this. It'd be slower to compute since it's no longer one-hot labels.

bsakaguchi commented

2026-10-31 21:10:04 +00:00

Author

Owner

@nishanthcgit commented on GitHub (Feb 3, 2021):

Ok, I think I understand what you mean. If I do get to train and try it out, I will tell you about it! Thanks a lot.

@nishanthcgit commented on GitHub (Feb 3, 2021): Ok, I think I understand what you mean. If I do get to train and try it out, I will tell you about it! Thanks a lot.

bsakaguchi commented

2026-10-31 21:10:04 +00:00

Author

Owner

@dragen1860 commented on GitHub (Apr 1, 2021):

The linked Keras example is not affiliated with us, and it seems that they used quite different architecture choices, such as using pre-trained Xception and BERT models, as opposed to training ViT/ResNet and autoregressive transformer from scratch.

Although we haven't experimented with their target matrix, I suspect that it'll be more difficult to train from scratch because the within-domain similarity is not as informative as the case of using pre-trained models; but I'm happy to be corrected on this. It'd be slower to compute since it's no longer one-hot labels.

Dear author:
Thanks for this great open source work! i would like to here some invaluable suggestion from you about the following factors:

400M dataset is crucial vs 3M dataset
RN50/ViT is important vs smaller network such as mobilenetv3, inception
training from scratch via transformer vs used pretained BERT

please feel free to give some general ideas about above factos.
Thank you very much.

@dragen1860 commented on GitHub (Apr 1, 2021): > The linked Keras example is not affiliated with us, and it seems that they used quite different architecture choices, such as using pre-trained Xception and BERT models, as opposed to training ViT/ResNet and autoregressive transformer from scratch. > > Although we haven't experimented with their target matrix, I suspect that it'll be more difficult to train from scratch because the within-domain similarity is not as informative as the case of using pre-trained models; but I'm happy to be corrected on this. It'd be slower to compute since it's no longer one-hot labels. Dear author: Thanks for this great open source work! i would like to here some invaluable suggestion from you about the following factors: 1. 400M dataset is crucial vs 3M dataset 2. RN50/ViT is important vs smaller network such as mobilenetv3, inception 3. training from scratch via transformer vs used pretained BERT please feel free to give some general ideas about above factos. Thank you very much.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#28

No description provided.

Rows
Columns