[GH-ISSUE #46] Temperature Clipping Missing

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Owner

Originally created by @KeremTurgutlu on GitHub (Feb 20, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/46

Thanks for the amazing work and open sourcing it!

In the paper it is mentioned:

The learnable temperature parameter was initialized to the equivalent of 0.07 from (Wu et al., 2018)
 and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training instability.

However, I am not able to find logits clipping in model.py, in this section:

# cosine similarity as logits
logit_scale = self.logit_scale.exp()
logits_per_image = logit_scale * image_features @ text_features.t()
logits_per_text = logit_scale * text_features @ image_features.t()

Is this a possible change to the code that might refer to the paper?:

# cosine similarity as logits
logit_scale = torch.clamp(self.logit_scale.exp(), max=100)

Though I am not sure how gradients would behave with torch.clamp, like an issue here.

Also, shouldn't initial temperature be 0.07 according to the paper? In the following code 1=1/temperature if I am not mistaken?

self.logit_scale = nn.Parameter(torch.ones([]))

Maybe to this?

self.logit_scale = nn.Parameter(torch.ones([]))*(1/0.07)

Originally created by @KeremTurgutlu on GitHub (Feb 20, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/46 Thanks for the amazing work and open sourcing it! In the paper it is mentioned: ``` The learnable temperature parameter was initialized to the equivalent of 0.07 from (Wu et al., 2018) and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training instability. ``` However, I am not able to find logits clipping in [model.py](https://github.com/openai/CLIP/blob/4c0275784d6d9da97ca1f47eaaee31de1867da91/clip/model.py#L361), in this section: ``` # cosine similarity as logits logit_scale = self.logit_scale.exp() logits_per_image = logit_scale * image_features @ text_features.t() logits_per_text = logit_scale * text_features @ image_features.t() ``` Is this a possible change to the code that might refer to the paper?: ``` # cosine similarity as logits logit_scale = torch.clamp(self.logit_scale.exp(), max=100) ``` Though I am not sure how gradients would behave with `torch.clamp`, like an [issue here](https://github.com/pytorch/pytorch/issues/7002). Also, shouldn't initial temperature be `0.07` according to the paper? In the following code `1=1/temperature` if I am not mistaken? ``` self.logit_scale = nn.Parameter(torch.ones([])) ``` Maybe to this? ``` self.logit_scale = nn.Parameter(torch.ones([]))*(1/0.07) ```

bsakaguchi closed this issue

2026-10-31 21:10:09 +00:00

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 20, 2021):

That's correct. The released code was rewritten for simpler inference usage, and it's missing a few details on training. For clipping the logit_scale value, I'd check its value right after optimizer.step() in the training loop and assign log(100) if it's over that number.

Appending *(1/0.07) might prevent the pytorch module to identify that as a learnable parameter. You could add a line like nn.init.constant_(self.logit_scale, np.log(1/0.07)), or simply nn.Parameter(torch.tensor([np.log(1/0.07)])).

@jongwook commented on GitHub (Feb 20, 2021): That's correct. The released code was rewritten for simpler inference usage, and it's missing a few details on training. For clipping the `logit_scale` value, I'd check its value right after `optimizer.step()` in the training loop and assign `log(100)` if it's over that number. Appending `*(1/0.07)` might prevent the pytorch module to identify that as a learnable parameter. You could add a line like `nn.init.constant_(self.logit_scale, np.log(1/0.07))`, or simply `nn.Parameter(torch.tensor([np.log(1/0.07)]))`.

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@Meteorix commented on GitHub (Mar 9, 2021):

That's correct. The released code was rewritten for simpler inference usage, and it's missing a few details on training. For clipping the logit_scale value, I'd check its value right after optimizer.step() in the training loop and assign log(100) if it's over that number.

Appending *(1/0.07) might prevent the pytorch module to identify that as a learnable parameter. You could add a line like nn.init.constant_(self.logit_scale, np.log(1/0.07)), or simply nn.Parameter(torch.tensor([1/0.07])).

or simply nn.Parameter(torch.tensor([np.log(1/0.07)])) ? @jongwook

@Meteorix commented on GitHub (Mar 9, 2021): > That's correct. The released code was rewritten for simpler inference usage, and it's missing a few details on training. For clipping the `logit_scale` value, I'd check its value right after `optimizer.step()` in the training loop and assign `log(100)` if it's over that number. > > Appending `*(1/0.07)` might prevent the pytorch module to identify that as a learnable parameter. You could add a line like `nn.init.constant_(self.logit_scale, np.log(1/0.07))`, or simply `nn.Parameter(torch.tensor([1/0.07]))`. or simply `nn.Parameter(torch.tensor([np.log(1/0.07)]))` ? @jongwook

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 9, 2021):

yeah you're correct- thanks for catching that! I'll edit the comment above to avoid any future confusion

@jongwook commented on GitHub (Mar 9, 2021): yeah you're correct- thanks for catching that! I'll edit the comment above to avoid any future confusion

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@KeremTurgutlu commented on GitHub (Mar 9, 2021):

Yeah, I ended adding these two lines: line 1 for init and line 2 for clipping. FYI, I implemented training as a fastai callback. Also added, few missing details like gradient checkpointing too.

@KeremTurgutlu commented on GitHub (Mar 9, 2021): Yeah, I ended adding these two lines: [line 1](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L346) for init and [line 2](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L533) for clipping. FYI, I implemented training as a fastai callback. Also added, few missing details like [gradient checkpointing](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L241) too.

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@Horizon2333 commented on GitHub (Oct 17, 2021):

Yeah, I ended adding these two lines: line 1 for init and line 2 for clipping. FYI, I implemented training as a fastai callback. Also added, few missing details like gradient checkpointing too.

Can I ask you some questions about learnable temperature? I found it quickly grow to the upper bound in order to reduce log-softmax loss. So its performance is worse than fixed value of temperature? Have you found something like this?

@Horizon2333 commented on GitHub (Oct 17, 2021): > Yeah, I ended adding these two lines: [line 1](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L346) for init and [line 2](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L533) for clipping. FYI, I implemented training as a fastai callback. Also added, few missing details like [gradient checkpointing](https://github.com/KeremTurgutlu/self_supervised/blob/2e7a7dc418891edccbf01efc0ab03d5e48586c8d/self_supervised/multimodal/clip.py#L241) too. Can I ask you some questions about learnable temperature? I found it quickly grow to the upper bound in order to reduce log-softmax loss. So its performance is worse than fixed value of temperature? Have you found something like this?

bsakaguchi commented

2026-10-31 21:10:10 +00:00

Author

Owner

@ghost commented on GitHub (Jan 28, 2022):

Hum, is it even useful when the batch sizes are relatively smaller? the higher
temperature kind of smooths out the predictions, which kind of prevents it from overfitting to the "label".

@ghost commented on GitHub (Jan 28, 2022): Hum, is it even useful when the batch sizes are relatively smaller? the higher temperature kind of smooths out the predictions, which kind of prevents it from overfitting to the "label".

bsakaguchi referenced this issue

2026-10-31 21:10:20 +00:00

[GH-ISSUE #63] NAN in downstream detection finetuneing task #57

bsakaguchi referenced this issue

2026-10-31 21:10:21 +00:00

[GH-ISSUE #63] NAN in downstream detection finetuneing task #57

bsakaguchi referenced this issue

2026-10-31 21:11:30 +00:00

[GH-ISSUE #146] Why the results of cosine similarities between the corresponding image and text features may appear negative numbers and NaN? #128

bsakaguchi referenced this issue

2026-10-31 21:13:31 +00:00

[GH-ISSUE #340] RN50x16, RN50x4, ViT-B/32 output nan #294