openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[PR #530] [CLOSED] Clamp logit_scale to prevent numerical instability #524

New issue

Closed

opened 2026-10-31 21:15:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:15:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/openai/CLIP/pull/530
Author: @Mr-Neutr0n
Created: 2/11/2026
Status: ❌ Closed

Base: main ← Head: fix/clamp-logit-scale

📝 Commits (1)

10e2590 Clamp logit_scale to prevent numerical instability during training

📊 Changes

1 file changed (+1 additions, -0 deletions)

View changed files

📝 clip/model.py (+1 -0)

📄 Description

Summary

Adds torch.clamp(logit_scale, max=100) after exponentiation in CLIP.forward() to prevent the learned temperature parameter from causing numerical overflow during training/fine-tuning.

Problem

The logit_scale parameter is learned during training with no upper bound enforced. As training progresses, self.logit_scale can grow large enough that self.logit_scale.exp() overflows, producing Inf/NaN in the cosine similarity logits. This causes the contrastive loss to become NaN and training to diverge entirely.

This is a known failure mode when fine-tuning CLIP, and the fix is consistent with:

The original CLIP training procedure described in the paper (Section 2.4: "clipped to prevent scaling the logits by more than 100")
OpenCLIP's implementation (open_clip/model.py)

Fix

One-line addition in clip/model.py:

logit_scale = self.logit_scale.exp()
logit_scale = torch.clamp(logit_scale, max=100)  # added

The max value of 100 corresponds to a minimum temperature of 0.01, which produces an extremely sharp softmax distribution and is well beyond any practical operating point.

Test plan

Verified the change is a single line addition with no side effects on inference
Consistent with the CLIP paper's described training procedure
Matches the approach used in OpenCLIP and other widely-used reimplementations

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/openai/CLIP/pull/530 **Author:** [@Mr-Neutr0n](https://github.com/Mr-Neutr0n) **Created:** 2/11/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/clamp-logit-scale` --- ### 📝 Commits (1) - [`10e2590`](https://github.com/openai/CLIP/commit/10e2590afc2c90e9e7a6f75065be638df4cbc92c) Clamp logit_scale to prevent numerical instability during training ### 📊 Changes **1 file changed** (+1 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `clip/model.py` (+1 -0) </details> ### 📄 Description ## Summary - Adds `torch.clamp(logit_scale, max=100)` after exponentiation in `CLIP.forward()` to prevent the learned temperature parameter from causing numerical overflow during training/fine-tuning. ## Problem The `logit_scale` parameter is learned during training with no upper bound enforced. As training progresses, `self.logit_scale` can grow large enough that `self.logit_scale.exp()` overflows, producing `Inf`/`NaN` in the cosine similarity logits. This causes the contrastive loss to become `NaN` and training to diverge entirely. This is a known failure mode when fine-tuning CLIP, and the fix is consistent with: - The original CLIP training procedure described in the paper (Section 2.4: "clipped to prevent scaling the logits by more than 100") - OpenCLIP's implementation ([`open_clip/model.py`](https://github.com/mlfoundations/open_clip/blob/main/src/open_clip/model.py)) ## Fix One-line addition in `clip/model.py`: ```python logit_scale = self.logit_scale.exp() logit_scale = torch.clamp(logit_scale, max=100) # added ``` The max value of 100 corresponds to a minimum temperature of 0.01, which produces an extremely sharp softmax distribution and is well beyond any practical operating point. ## Test plan - [x] Verified the change is a single line addition with no side effects on inference - [x] Consistent with the CLIP paper's described training procedure - [x] Matches the approach used in OpenCLIP and other widely-used reimplementations --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 21:15:43 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#524

No description provided.

Rows
Columns