openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #42] Clarifying the training setup #39

New issue
Closed
opened 2026-10-31 21:10:07 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-31 21:10:07 +00:00
Owner
Copy link

Originally created by @nikky4D on GitHub (Feb 12, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/42

I'm confused as to your training loss and setup.

For the setup, you say:

We remove the non-linear projection between the representation and the contrastive embedding space, a change which was introduced by Bachman et al. (2019) and popularized by Chen et al. (2020b). We use only a linear projection to map from each encoder’s representation to the multi-modal embedding space. We did not notice a difference in training efficiency between
the two versions and speculate that non-linear projections may be co-adapted with details of current image only self supervised representation learning methods.

Do you use the non-linear projection in pretraining, then remove it after training as in Chen, 2020b, replacing it with a linear projection after training? or do you use a linear projection in pretraining, and keep the linear projection after training?
And can you explain the speculation you talk about above? I don't understand what you mean there.

For the loss:
Can you clarify that the loss used in training is the same form of the loss function as in Zheng et. al, 2021?

Originally created by @nikky4D on GitHub (Feb 12, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/42 I'm confused as to your training loss and setup. For the setup, you say: > We remove the non-linear projection between the representation and the contrastive embedding space, a change which was introduced by Bachman et al. (2019) and popularized by Chen et al. (2020b). We use only a linear projection to map from each encoder’s representation to the multi-modal embedding space. We did not notice a difference in training efficiency between the two versions and speculate that non-linear projections may be co-adapted with details of current image only self supervised representation learning methods. Do you use the non-linear projection in pretraining, then remove it after training as in Chen, 2020b, replacing it with a linear projection after training? or do you use a linear projection in pretraining, and keep the linear projection after training? And can you explain the speculation you talk about above? I don't understand what you mean there. For the loss: Can you clarify that the loss used in training is the same form of the loss function as in Zheng et. al, 2021?
bsakaguchi commented 2026-10-31 21:10:08 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Feb 12, 2021):

Sorry about the confusion; you could read "we remove" as "we do not use". The projection is linear during training and evaluation.

Looks like we're missing a preposition in that sentence. The speculation was that nonlinear projections might be effective only in self-supervised methods but not in contrastive multimodal settings like CLIP.

The loss function takes the same form as ConVIRT by Zhang et al. (2020).

<!-- gh-comment-id:778007420 --> @jongwook commented on GitHub (Feb 12, 2021): Sorry about the confusion; you could read "we remove" as "we do not use". The projection is linear during training and evaluation. Looks like we're missing a preposition in that sentence. The speculation was that nonlinear projections might be effective only in self-supervised methods but not in contrastive multimodal settings like CLIP. The loss function takes the same form as ConVIRT by Zhang et al. (2020).
bsakaguchi commented 2026-10-31 21:10:08 +00:00
Author
Owner
Copy link

@nikky4D commented on GitHub (Feb 12, 2021):

Thanks for the update. One last question to clarify: Your linear function is the weight matrix, the W_i, W_t variable in the pseudocode?

<!-- gh-comment-id:778047569 --> @nikky4D commented on GitHub (Feb 12, 2021): Thanks for the update. One last question to clarify: Your linear function is the weight matrix, the `W_i, W_t `variable in the pseudocode?
bsakaguchi commented 2026-10-31 21:10:09 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Feb 12, 2021):

Yes! That corresponds to here, here, and here in the model code.

<!-- gh-comment-id:778429772 --> @jongwook commented on GitHub (Feb 12, 2021): Yes! That corresponds to [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L232-L233), [here](https://github.com/openai/CLIP/blob/main/clip/model.py#L82), and [here](https://github.com/openai/CLIP/blob/e5347713f46ab8121aa81e610a68ea1d263b91b7/clip/model.py#L349) in the model code.
bsakaguchi commented 2026-10-31 21:10:09 +00:00
Author
Owner
Copy link

@nikky4D commented on GitHub (Feb 12, 2021):

Thank you very much. I appreciate it.

<!-- gh-comment-id:778444391 --> @nikky4D commented on GitHub (Feb 12, 2021): Thank you very much. I appreciate it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#39
No description provided.