[GH-ISSUE #153] the acc result on the CIFAR100 dataset

bsakaguchi commented

2026-10-31 21:11:33 +00:00

Owner

Originally created by @realTaki on GitHub (Sep 13, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/153

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?
Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this?

Originally created by @realTaki on GitHub (Sep 13, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/153 I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this?

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@jongwook commented on GitHub (Sep 24, 2021):

Hi, see https://github.com/openai/CLIP/blob/main/data/prompts.md#cifar100 where you can now find the class names and prompts for ensembling zero-shot predictions for CIFAR100.

@jongwook commented on GitHub (Sep 24, 2021): Hi, see https://github.com/openai/CLIP/blob/main/data/prompts.md#cifar100 where you can now find the class names and prompts for ensembling zero-shot predictions for CIFAR100.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@weiyx16 commented on GitHub (Oct 27, 2021):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?
Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this?

Hello! Have you try to reproduce the results on voc2007? What I got is only 71% mAP with ViT-B/32 using official class names and prompts, which is way below the reported one: 83.1%. Do you have any suggestions? Really thank you for your help

@weiyx16 commented on GitHub (Oct 27, 2021): > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? > Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this? Hello! Have you try to reproduce the results on voc2007? What I got is only 71% mAP with ViT-B/32 using official class names and prompts, which is way below the reported one: 83.1%. Do you have any suggestions? Really thank you for your help

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@xcpeng commented on GitHub (Oct 27, 2021):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?
Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this?

Hello! Have you try to reproduce the results on voc2007? What I got is only 71% mAP with ViT-B/32 using official class names and prompts, which is way below the reported one: 83.1%. Do you have any suggestions? Really thank you for your help

Have you checked the order of categories in voc and prompts? FYI, the default order in prompts:
classes = [
'aeroplane',
'bicycle',
'bird',
'boat',
'bottle',
'bus',
'car',
'cat',
'chair',
'cow',
'dog',
'horse',
'motorbike',
'person',
'sheep',
'sofa',
'diningtable',
'pottedplant',
'train',
'tvmonitor',
]

In VOC, the order is:
pascalvoc2007_classes = [
'aeroplane',
'bicycle',
'bird',
'boat',
'bottle',
'bus',
'car',
'cat',
'chair',
'cow',
'diningtable',
'dog',
'horse',
'motorbike',
'person',
'pottedplant',
'sheep',
'sofa',
'train',
'tvmonitor',
]

Replace the order of prompts with the order in voc, you will get about 82.5 for VIT-B/32

@xcpeng commented on GitHub (Oct 27, 2021): > > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? > > Can you give some details to help me reproduce the acc result in the paper? or do you have any idea how to troubleshoot this? > > Hello! Have you try to reproduce the results on voc2007? What I got is only 71% mAP with ViT-B/32 using official class names and prompts, which is way below the reported one: 83.1%. Do you have any suggestions? Really thank you for your help Have you checked the order of categories in voc and prompts? FYI, the default order in prompts: classes = [ 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'dog', 'horse', 'motorbike', 'person', 'sheep', 'sofa', 'diningtable', 'pottedplant', 'train', 'tvmonitor', ] In VOC, the order is: pascalvoc2007_classes = [ 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor', ] Replace the order of prompts with the order in voc, you will get about 82.5 for VIT-B/32

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@xcpeng commented on GitHub (Oct 27, 2021):

The issue is the index of 'dinningtable' in prompts by the way

@xcpeng commented on GitHub (Oct 27, 2021): The issue is the index of 'dinningtable' in prompts by the way

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@weiyx16 commented on GitHub (Oct 29, 2021):

The issue is the index of 'dinningtable' in prompts by the way

Really thank you for your help!! Actually I have noticed the problem of the prompts and fixed in advance, but I still couldn't reproduce the results. And I need to fix that the point I reproduced is 75%, not 71% mentioned before. The key question is keeping confusing me is how to calculate the 11-point mAP.

Here is how I reproduce the results: https://github.com/weiyx16/vocreproduce/blob/main/reproduce.py Just run reproduce.py and it will download the model and dataset automatically.

@weiyx16 commented on GitHub (Oct 29, 2021): > The issue is the index of 'dinningtable' in prompts by the way Really thank you for your help!! Actually I have noticed the problem of the prompts and fixed in advance, but I still couldn't reproduce the results. And I need to fix that the point I reproduced is 75%, not 71% mentioned before. The key question is keeping confusing me is how to calculate the **11-point mAP**. Here is how I reproduce the results: https://github.com/weiyx16/vocreproduce/blob/main/reproduce.py Just run reproduce.py and it will download the model and dataset automatically.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@weiyx16 commented on GitHub (Nov 2, 2021):

The issue is the index of 'dinningtable' in prompts by the way

Really thank you for your help!! Actually I have noticed the problem of the prompts and fixed in advance, but I still couldn't reproduce the results. And I need to fix that the point I reproduced is 75%, not 71% mentioned before. The key question is keeping confusing me is how to calculate the 11-point mAP.

Here is how I reproduce the results: https://github.com/weiyx16/vocreproduce/blob/main/reproduce.py Just run reproduce.py and it will download the model and dataset automatically.

I think the I fix the bug by adding a softmax after logits. This operation will not effect the accuracy in others dataset, but effect the sorting in mAP calculation.

@weiyx16 commented on GitHub (Nov 2, 2021): > > The issue is the index of 'dinningtable' in prompts by the way > > Really thank you for your help!! Actually I have noticed the problem of the prompts and fixed in advance, but I still couldn't reproduce the results. And I need to fix that the point I reproduced is 75%, not 71% mentioned before. The key question is keeping confusing me is how to calculate the **11-point mAP**. > > Here is how I reproduce the results: https://github.com/weiyx16/vocreproduce/blob/main/reproduce.py Just run reproduce.py and it will download the model and dataset automatically. I think the I fix the bug by adding a softmax after logits. This operation will not effect the accuracy in others dataset, but effect the sorting in mAP calculation.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@machengcheng2016 commented on GitHub (Jun 7, 2022):

@weiyx16 Greetings. Have you ever tried CLIP on the StanfordCars dataset? I can only get a ~47%/~48% acc without/with the "prompt ensembling" trick, far away from ~55% as reported in the original paper. Could you help me with some possible clues, please?

@machengcheng2016 commented on GitHub (Jun 7, 2022): @weiyx16 Greetings. Have you ever tried CLIP on the StanfordCars dataset? I can only get a ~47%/~48% acc without/with the "prompt ensembling" trick, far away from ~55% as reported in the original paper. Could you help me with some possible clues, please?

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@weiyx16 commented on GitHub (Jun 10, 2022):

@machengcheng2016 We tried it before, and it's able to reproduce the zero-shot performance on StanfordCars using R50 backbone (about 55.0% in our experiments). Nothing special needs to check. Have you verified the dataset? It has 8041 test images. And have you reproduced results on other datasets?

@weiyx16 commented on GitHub (Jun 10, 2022): @machengcheng2016 We tried it before, and it's able to reproduce the zero-shot performance on StanfordCars using R50 backbone (about 55.0% in our experiments). Nothing special needs to check. Have you verified the dataset? It has 8041 test images. And have you reproduced results on other datasets?

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@machengcheng2016 commented on GitHub (Jun 10, 2022):

I solved my problem. It is data augmentation that cheats me.

@machengcheng2016 commented on GitHub (Jun 10, 2022): I solved my problem. It is data augmentation that cheats me.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@QMiao-cs commented on GitHub (Nov 26, 2022):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?

I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result?

@QMiao-cs commented on GitHub (Nov 26, 2022): > > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result?

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@machengcheng2016 commented on GitHub (Nov 26, 2022):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?

I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result?

It might come from data augmentation. Please make sure you are using the correct one.

@machengcheng2016 commented on GitHub (Nov 26, 2022): > > > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? > > I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result? It might come from data augmentation. Please make sure you are using the correct one.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@QMiao-cs commented on GitHub (Nov 26, 2022):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?

I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result?

It might come from data augmentation. Please make sure you are using the correct one.

The images are loaded by the following code in the `README.md' :
cifar100 = CIFAR100(root=os.path.expanduser("~/.cache"), download=True, train=False)
image, class_id = cifar100[3637]
Is there any data augmentation?

@QMiao-cs commented on GitHub (Nov 26, 2022): > > > > > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? > > > > > > I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result? > > It might come from data augmentation. Please make sure you are using the correct one. The images are loaded by the following code in the `README.md' : cifar100 = CIFAR100(root=os.path.expanduser("~/.cache"), download=True, train=False) image, class_id = cifar100[3637] Is there any data augmentation?

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@Calmepro777 commented on GitHub (Dec 1, 2022):

I use the code in the README.md (Zero-Shot Prediction) to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper?

I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result?

It might come from data augmentation. Please make sure you are using the correct one.

Hi, I wonder what would be the correct data augmentation setting, I used standard transform settings on validation set of CIFAR100 plus resizing to 224

@Calmepro777 commented on GitHub (Dec 1, 2022): > > > > I use the code in the `README.md (Zero-Shot Prediction)` to test the acc of ViT/B-32 on the CIFAR100 dataset and get the result of about 62% by top1 similarity. It is less than that in the paper (65.1%), I was wondering that how it measures the acc of a zero-shot task in the paper? > > > > > > I also test the acc of ViT/B-32 on CIFAR100 and get the result of 61.67%. I would like to ask if you have solved this problem?Can you give some details to reproduce the result? > > It might come from data augmentation. Please make sure you are using the correct one. Hi, I wonder what would be the correct data augmentation setting, I used standard transform settings on validation set of CIFAR100 plus resizing to 224

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@shyammarjit commented on GitHub (Sep 28, 2023):

Please refer to github.com/openai/CLIP@fcab8b6eb9/notebooks/Prompt_Engineering_for_ImageNet.ipynb for this concern.

@shyammarjit commented on GitHub (Sep 28, 2023): Please refer to https://github.com/openai/CLIP/blob/fcab8b6eb92af684e7ff0a904464be7b99b49b88/notebooks/Prompt_Engineering_for_ImageNet.ipynb for this concern.

bsakaguchi commented

2026-10-31 21:11:34 +00:00

Author

Owner

@jiachengc commented on GitHub (Apr 24, 2024):

Could anyone tell me why "order in prompts" matter? Thanks in advance.

@jiachengc commented on GitHub (Apr 24, 2024): Could anyone tell me why "order in prompts" matter? Thanks in advance.

Rows
Columns

[GH-ISSUE #153] the acc result on the CIFAR100 dataset #135