[GH-ISSUE #48] Two questions regarding model implementation:

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Owner

Originally created by @vinsis on GitHub (Feb 24, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/48

First of all thank you for providing the code and pretrained model. The results look stunning.

I was looking into the source code and there are two parts I am struggling to understand:

What is the purpose of logit_scale in the model here? It seems to simply scale the dot products between two representations.
In the implementation of VisualTransformer what is the role of class_embedding defined here.

Thanks!

Originally created by @vinsis on GitHub (Feb 24, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/48 First of all thank you for providing the code and pretrained model. The results look stunning. I was looking into the source code and there are two parts I am struggling to understand: 1. What is the purpose of `logit_scale` in the model [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L362)? It seems to simply scale the dot products between two representations. 2. In the implementation of `VisualTransformer` what is the role of `class_embedding` defined [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L209). Thanks!

bsakaguchi closed this issue

2026-10-31 21:10:11 +00:00

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Author

Owner

@PaulLerner commented on GitHub (Mar 3, 2021):

2. In the implementation of `VisualTransformer` what is the role of `class_embedding` defined [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L209).

This is the embedding used for the final representation, see here they only keep the first "token" of each sequence.
See http://arxiv.org/abs/2010.11929 for more details about Vision Transformer :)

@PaulLerner commented on GitHub (Mar 3, 2021): > 2. In the implementation of `VisualTransformer` what is the role of `class_embedding` defined [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L209). This is the embedding used for the final representation, see [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L230) they only keep the first "token" of each sequence. See http://arxiv.org/abs/2010.11929 for more details about Vision Transformer :)

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Author

Owner

@vinsis commented on GitHub (Mar 4, 2021):

Got it, thanks.

The answer to the first question still remains elusive.

@vinsis commented on GitHub (Mar 4, 2021): Got it, thanks. The answer to the first question still remains elusive.

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 4, 2021):

During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. This comment explains how to clip the value.

@jongwook commented on GitHub (Mar 4, 2021): During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. [This comment](https://github.com/openai/CLIP/issues/46#issuecomment-782558799) explains how to clip the value.

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Author

Owner

@vinsis commented on GitHub (Mar 4, 2021):

Ok thanks. That explains it.

@vinsis commented on GitHub (Mar 4, 2021): Ok thanks. That explains it.

bsakaguchi commented

2026-10-31 21:10:11 +00:00

Author

Owner

@Horizon2333 commented on GitHub (Oct 17, 2021):

During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. This comment explains how to clip the value.

So why don't you directly initialize temperature parameter to 100 ? What's the purpose of let it grow from 1/0.07 to 100? Does it bring enough performance improvement?

@Horizon2333 commented on GitHub (Oct 17, 2021): > During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. [This comment](https://github.com/openai/CLIP/issues/46#issuecomment-782558799) explains how to clip the value. So why don't you directly initialize temperature parameter to 100 ? What's the purpose of let it grow from 1/0.07 to 100? Does it bring enough performance improvement?

bsakaguchi commented

2026-10-31 21:10:12 +00:00

Author

Owner

@justStarG commented on GitHub (May 30, 2024):

During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. This comment explains how to clip the value.

So why don't you directly initialize temperature parameter to 100 ? What's the purpose of let it grow from 1/0.07 to 100? Does it bring enough performance improvement?

Yes, I think initialize to 100 will converge faster. But why not set to 100?

@justStarG commented on GitHub (May 30, 2024): > > During contrastive learning, we interpret the dot products as logits that are fed to the cross-entropy loss for each text and image example. Because we normalize the feature vectors, the dot products are capped between [-1, 1], which might not have an enough dynamic range and limit the categorical probability distributions that the logits can express. For this reason, we scale the logits to have larger differences. To avoid numerical instability, we clipped the scale value at 100, and for all models that we trained, it reached 100. [This comment](https://github.com/openai/CLIP/issues/46#issuecomment-782558799) explains how to clip the value. > > So why don't you directly initialize temperature parameter to 100 ? What's the purpose of let it grow from 1/0.07 to 100? Does it bring enough performance improvement? Yes, I think initialize to 100 will converge faster. But why not set to 100?

bsakaguchi commented

2026-10-31 21:10:12 +00:00

Author

Owner

@yakorch commented on GitHub (Sep 8, 2025):

@Horizon2333 @thesby
I am late to the party, but perhaps this is still useful.
In Temperature-Free Loss Function for Contrastive Learning, the authors analyze the properties of the InfoNCE loss, showing that for high temperature values (in the paper notation that I cite), the total loss is non-zero even for correct model predictions (C -> 1).

However, my intuition is that at the start of the training, the model is poor, resulting in very few "C->1" predictions, so the optimization process does not experience this drawback of the chosen loss function. And this comes with a neat bonus of a strong gradient magnitude for non-optimal C, speeding up training.

After some training, hopefully the model is better, and "C->1" predictions are common, and one can no longer ignore the non-zero InfoNCE loss. Yet it is no longer a problem thanks to the decreasing temperature as a learnable parameter.

@yakorch commented on GitHub (Sep 8, 2025): @Horizon2333 @thesby I am late to the party, but perhaps this is still useful. In [Temperature-Free Loss Function for Contrastive Learning](https://arxiv.org/pdf/2501.17683v1), the authors analyze the properties of the InfoNCE loss, showing that for high temperature values (in the paper notation that I cite), the total loss is non-zero even for correct model predictions (C -> 1). However, my intuition is that at the start of the training, the model is poor, resulting in very few "C->1" predictions, so the optimization process does not experience this drawback of the chosen loss function. And this comes with a neat bonus of a strong gradient magnitude for non-optimal C, speeding up training. After some training, hopefully the model is better, and "C->1" predictions are common, and one can no longer ignore the non-zero InfoNCE loss. Yet it is no longer a problem thanks to the decreasing temperature as a learnable parameter.

bsakaguchi referenced this issue

2026-10-31 21:15:06 +00:00

[PR #43] [CLOSED] Beautify the tags #450

Rows
Columns

[GH-ISSUE #48] Two questions regarding model implementation: #43