openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #135] Implementation details in few-shot ImageNet evaluation #122

New issue
Open
opened 2026-10-31 21:11:25 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-31 21:11:25 +00:00
Owner
Copy link

Originally created by @TonyLianLong on GitHub (Aug 4, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/135

Thanks for the amazing paper! I have a few questions about the details of the results in Figure 6 in the paper. I tried to use the linear probing code in the README.md on CIFAR to run one-shot evaluation on ImageNet. However, my evaluation implementation, with VIT-B/32, only gets 27% accuracy on ImageNet validation set (not ImageNet V2) instead of about 45% as mentioned in the paper. Therefore, I suspect that I missed some details in the paper experiments and I have a few questions:

  1. What CLIP model is figure 6 using? The zero-shot accuracy makes me think that it's ViT-B/32. However, in my personal evaluation, ViT-B/32 does not provide a 45% one-shot accuracy.
  2. Could you disclose the C parameter in sklearn for this experiment? Although on my end the performance does not vary much in different C values, I will appreciate it if this value could be provided.
  3. How are the instances per class selected? I select the first sample of each class in ImageNet training set, which leads to a dataset size of 1000 and each with feature dimension 768 (which is obtained before the final projection to 512 dimensions). Does my selection approach sound reasonable?

Thanks again for the amazing paper and thanks in advance for helping me.

Originally created by @TonyLianLong on GitHub (Aug 4, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/135 Thanks for the amazing paper! I have a few questions about the details of the results in Figure 6 in the paper. I tried to use the linear probing code in the `README.md` on CIFAR to run one-shot evaluation on ImageNet. However, my evaluation implementation, with `VIT-B/32`, only gets 27% accuracy on ImageNet validation set (not ImageNet V2) instead of about 45% as mentioned in the paper. Therefore, I suspect that I missed some details in the paper experiments and I have a few questions: 1. What CLIP model is figure 6 using? The zero-shot accuracy makes me think that it's `ViT-B/32`. However, in my personal evaluation, `ViT-B/32` does not provide a 45% one-shot accuracy. 2. Could you disclose the `C` parameter in sklearn for this experiment? Although on my end the performance does not vary much in different `C` values, I will appreciate it if this value could be provided. 3. How are the instances per class selected? I select the first sample of each class in ImageNet training set, which leads to a dataset size of 1000 and each with feature dimension 768 (which is obtained before the final projection to 512 dimensions). Does my selection approach sound reasonable? Thanks again for the amazing paper and thanks in advance for helping me.
bsakaguchi commented 2026-10-31 21:11:26 +00:00
Author
Owner
Copy link

@realTaki commented on GitHub (Sep 16, 2021):

I tried run testing on CIFAR100 following by README.md (Zero-Shot Prediction) and also can't achieve the performance in paper. did you solve this problem?

<!-- gh-comment-id:920830186 --> @realTaki commented on GitHub (Sep 16, 2021): I tried run testing on CIFAR100 following by `README.md (Zero-Shot Prediction)` and also can't achieve the performance in paper. did you solve this problem?
bsakaguchi commented 2026-10-31 21:11:26 +00:00
Author
Owner
Copy link

@shyammarjit commented on GitHub (Sep 28, 2023):

Please refer to github.com/openai/CLIP@fcab8b6eb9/notebooks/Prompt_Engineering_for_ImageNet.ipynb for this concern.

<!-- gh-comment-id:1739834414 --> @shyammarjit commented on GitHub (Sep 28, 2023): Please refer to https://github.com/openai/CLIP/blob/fcab8b6eb92af684e7ff0a904464be7b99b49b88/notebooks/Prompt_Engineering_for_ImageNet.ipynb for this concern.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#122
No description provided.