mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #68] Why cosine loss instead of just dot product? #62
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#62
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ekgren on GitHub (Mar 23, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/68
Trying to find the motivation in the article but I'm left without a definitive answer.
What is the motivation of using L2 normed dot product instead of simply the dot product in the loss?
@jongwook commented on GitHub (Mar 24, 2021):
The main purpose is to stabilize training, by limiting the dynamic range of the logits. With the cosine similarities being within [-1, 1] and an upper-bounded temperature parameter, it's a simple way to prevent the model predictions from exploding. Radovanovi et al. found that the "hubness phenomenon" is less severe with cosine similarity, although synthetic data was used.