openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #204] Zeroshot evaluation CIFAR100 in Colab: Results ~0.5% below values reported in paper #179

New issue
Open
opened 2026-10-31 21:12:04 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-31 21:12:04 +00:00
Owner
Copy link

Originally created by @andsteing on GitHub (Jan 2, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/204

Hi

Thanks so much for providing this repository and the notebooks!

I'm debugging diffs in the zeroshot evaluation results from a JAX port of this repository (scenic.projects.baselines.clip) and as part of this work I'm trying to reproduce the exact numbers published in the paper https://arxiv.org/abs/2103.00020 (Table 11).

I created a short Colab based on the provided notebooks where I'm zeroshot-evaluating CLIP models on the CIFAR100 dataset:
https://colab.research.google.com/github/andsteing/CLIP/blob/zeroshot/notebooks/zeroshot_evaluation.ipynb#scrollTo=Mo-MYo3Flgth

I get the following results:

model dataset 7 prompts 80 prompts table 11
RN50 CIFAR100 40.93 41.04 41.6
B/32 CIFAR100 64.58 64.21 65.1

So the results are still about 0.5% short from what I would expect after reading the paper.

Any idea what this small difference could be due to?

Best, Andreas

Originally created by @andsteing on GitHub (Jan 2, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/204 Hi Thanks so much for providing this repository and the notebooks! I'm debugging diffs in the zeroshot evaluation results from a JAX port of this repository ([`scenic.projects.baselines.clip`](https://github.com/google-research/scenic/tree/main/scenic/projects/baselines/clip)) and as part of this work I'm trying to reproduce the exact numbers published in the paper https://arxiv.org/abs/2103.00020 (Table 11). I created a short Colab based on the provided notebooks where I'm zeroshot-evaluating CLIP models on the CIFAR100 dataset: https://colab.research.google.com/github/andsteing/CLIP/blob/zeroshot/notebooks/zeroshot_evaluation.ipynb#scrollTo=Mo-MYo3Flgth I get the following results: | model | dataset | 7 prompts | 80 prompts | table 11 | | --- | --- | --- | --- | --- | | RN50 | CIFAR100 | 40.93 | 41.04 | 41.6 | | B/32 | CIFAR100 | 64.58 | 64.21 | 65.1 | So the results are still about 0.5% short from what I would expect after reading the paper. Any idea what this small difference could be due to? Best, Andreas
bsakaguchi commented 2026-10-31 21:12:04 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Apr 11, 2022):

Hi,

There can be numerical differences that we cannot fully control, e.g. different CUDA and driver versions, batch sizes, hardware, etc., that may cause the 0.5% difference in evals.

That being said, have you tried with the 18 prompts in this document?

<!-- gh-comment-id:1094469714 --> @jongwook commented on GitHub (Apr 11, 2022): Hi, There can be numerical differences that we cannot fully control, e.g. different CUDA and driver versions, batch sizes, hardware, etc., that may cause the 0.5% difference in evals. That being said, have you tried with the 18 prompts in [this document](https://github.com/openai/CLIP/blob/main/data/prompts.md#cifar100)?
bsakaguchi commented 2026-10-31 21:12:05 +00:00
Author
Owner
Copy link

@Calmepro777 commented on GitHub (Dec 11, 2022):

Hi,

There can be numerical differences that we cannot fully control, e.g. different CUDA and driver versions, batch sizes, hardware, etc., that may cause the 0.5% difference in evals.

That being said, have you tried with the 18 prompts in this document?

Hello,

I would be grateful if the prompt and other settings related to the zero-shot result on Cifar-100, Cifar-10 can be revealed.

In research, we would like to know if our experimental settings are correct and optimal, otherwise reviewers could challenge this.

Regards

<!-- gh-comment-id:1345661708 --> @Calmepro777 commented on GitHub (Dec 11, 2022): > Hi, > > There can be numerical differences that we cannot fully control, e.g. different CUDA and driver versions, batch sizes, hardware, etc., that may cause the 0.5% difference in evals. > > That being said, have you tried with the 18 prompts in [this document](https://github.com/openai/CLIP/blob/main/data/prompts.md#cifar100)? Hello, I would be grateful if the prompt and other settings related to the zero-shot result on Cifar-100, Cifar-10 can be revealed. In research, we would like to know if our experimental settings are correct and optimal, otherwise reviewers could challenge this. Regards
bsakaguchi commented 2026-10-31 21:12:05 +00:00
Author
Owner
Copy link

@Calmepro777 commented on GitHub (Dec 11, 2022):

Here is the best results I obtained:

image encoder: ViT-B/16
prompt: "itap of a {label}."

Dataset Reproduced Acc. Reported Acc. Gap
CIFAR-10 90.51 91.6 1.09
CIFAR-100 68.03 68.7 0.67
<!-- gh-comment-id:1345664851 --> @Calmepro777 commented on GitHub (Dec 11, 2022): Here is the best results I obtained: image encoder: ViT-B/16 prompt: "itap of a {label}." | Dataset | Reproduced Acc. | Reported Acc. | Gap | | ------------- | ------------- | ------------- | ------------- | | CIFAR-10 | 90.51 | 91.6 | 1.09 | | CIFAR-100 | 68.03 | 68.7 | 0.67 |
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#179
No description provided.