openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #87] Share prompts for other zeroshot eval datasets? #80

New issue
Closed
opened 2026-10-31 21:10:54 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-31 21:10:54 +00:00
Owner
Copy link

Originally created by @hackgoofer on GitHub (Apr 8, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/87

Thank you for providing the prompt engineering file for the ImageNet here

I am curious if you are willing to share the prompts for one other datasets used for the Zeroshot Eval results shown in the paper? I was able to reproduce the ImageNet zero shot eval results but unable to reproduce other results. The zeroshot eval top1 accuracy is 3 to 8 percentages off for the Food101 dataset (show below). This is with the prompt suggested by the paper: "a photo of a {}, a type of food." Curious if you are using an ensemble of prompts to report the results? If so, could you share the ensemble of prompts for one of those datasets?

ours / reported in paper
ViT/B-32: 81.4812  / 84.4 
ResNet50: 73.0099 / 81.1
ResNet101: 75.9644 / 83.9
ResNet50x4: 80.4753 / 86.8
Originally created by @hackgoofer on GitHub (Apr 8, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/87 Thank you for providing the prompt engineering file for the ImageNet [here](https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb) I am curious if you are willing to share the prompts for one other datasets used for the Zeroshot Eval results shown in the paper? I was able to reproduce the ImageNet zero shot eval results but unable to reproduce other results. The zeroshot eval top1 accuracy is 3 to 8 percentages off for the Food101 dataset (show below). This is with the prompt suggested by the paper: "a photo of a {}, a type of food." Curious if you are using an ensemble of prompts to report the results? If so, could you share the ensemble of prompts for one of those datasets? ``` ours / reported in paper ViT/B-32: 81.4812 / 84.4 ResNet50: 73.0099 / 81.1 ResNet101: 75.9644 / 83.9 ResNet50x4: 80.4753 / 86.8 ```
bsakaguchi commented 2026-10-31 21:10:54 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Sep 24, 2021):

The prompts can now be found at https://github.com/openai/CLIP/blob/main/data/prompts.md

<!-- gh-comment-id:926321269 --> @jongwook commented on GitHub (Sep 24, 2021): The prompts can now be found at https://github.com/openai/CLIP/blob/main/data/prompts.md
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#80
No description provided.