openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #502] Recall (R@k) way lower than the one obtained in papers #426

New issue
Open
opened 2026-10-31 21:14:52 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-31 21:14:52 +00:00
Owner
Copy link

Originally created by @asczdvefb42 on GitHub (Jun 11, 2025).
Original GitHub issue: https://github.com/openai/CLIP/issues/502

Here is what i do :

  • Get test dataset of image - caption (IIITD-20K dataset)
  • calculate embeddings with my fine tuned CLIP
  • calculate cosine distance between each text to all images
  • get the k closest images to the text, if the corresponding image is in it, do +1 to score
  • get the recall by dividing the score by the length of the test dataset.

This is my recall at k. I obtain a R@1 of 17%, while most papers when finetuning CLIP obtain at least 60% recall at 1. Any idea what i could be doing wrong?

Originally created by @asczdvefb42 on GitHub (Jun 11, 2025). Original GitHub issue: https://github.com/openai/CLIP/issues/502 Here is what i do : - Get test dataset of image - caption (IIITD-20K dataset) - calculate embeddings with my fine tuned CLIP - calculate cosine distance between each text to all images - get the k closest images to the text, if the corresponding image is in it, do +1 to score - get the recall by dividing the score by the length of the test dataset. This is my recall at k. I obtain a R@1 of 17%, while most papers when finetuning CLIP obtain at least 60% recall at 1. Any idea what i could be doing wrong?
bsakaguchi commented 2026-10-31 21:14:52 +00:00
Author
Owner
Copy link

@ZJ331456 commented on GitHub (Aug 23, 2025):

Any fix for this? I've been running into the same problem recently.

<!-- gh-comment-id:3217236919 --> @ZJ331456 commented on GitHub (Aug 23, 2025): Any fix for this? I've been running into the same problem recently.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#426
No description provided.