[GH-ISSUE #272] About Zero-shot CLIP performance on CIFAR100 and ImageNet1K #236

New issue

Closed

opened 2026-10-31 21:12:48 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:12:48 +00:00

Owner

Originally created by @yaohui120 on GitHub (Jul 25, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/272

I can get almost same results provided in Zero-shot Prediction code in More Examples. But when I changed the code and used different backbone(RN50, ViT-B/16, ViT-B/32) to compute all images in test sets of CIFAR100 and ImageNet1K, the accuracies are always lower than result's provided in their paper about 3 or 4 percent. The environment I used to run the code includes torch1.8.1 and torchvision0.9.1. The code is listed below.

# Code
import os
import clip
import torch
from torchvision.datasets import CIFAR100
import numpy as np

# Load the model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load('RN50', device)

# Download the dataset
cifar100 = CIFAR100(root=os.path.expanduser("./Dataset"), download=True, train=False)
text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device)

# Calculate text features
with torch.no_grad():
    text_features = model.encode_text(text_inputs)
text_features /= text_features.norm(dim=-1, keepdim=True)

# Calculate CLIP's Prediction
total, corr = 0, 0
for i in range(len(cifar100)):
    image, class_id = cifar100[i]
    image_input = preprocess(image).unsqueeze(0).to(device)

    with torch.no_grad():
        image_features = model.encode_image(image_input)
    image_features /= image_features.norm(dim=-1, keepdim=True)

    logits = 100.0 * image_features @ text_features.T
    _, preds = torch.max(logits, dim=1)
    if preds.item() == class_id:
        corr += 1
    total += 1
print(np.round(corr*100/total, decimals = 2))

Originally created by @yaohui120 on GitHub (Jul 25, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/272 I can get almost same results provided in Zero-shot Prediction code in More Examples. But when I changed the code and used different backbone(RN50, ViT-B/16, ViT-B/32) to compute all images in test sets of CIFAR100 and ImageNet1K, the accuracies are always lower than result's provided in their paper about 3 or 4 percent. The environment I used to run the code includes torch1.8.1 and torchvision0.9.1. The code is listed below. ```python # Code import os import clip import torch from torchvision.datasets import CIFAR100 import numpy as np # Load the model device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load('RN50', device) # Download the dataset cifar100 = CIFAR100(root=os.path.expanduser("./Dataset"), download=True, train=False) text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device) # Calculate text features with torch.no_grad(): text_features = model.encode_text(text_inputs) text_features /= text_features.norm(dim=-1, keepdim=True) # Calculate CLIP's Prediction total, corr = 0, 0 for i in range(len(cifar100)): image, class_id = cifar100[i] image_input = preprocess(image).unsqueeze(0).to(device) with torch.no_grad(): image_features = model.encode_image(image_input) image_features /= image_features.norm(dim=-1, keepdim=True) logits = 100.0 * image_features @ text_features.T _, preds = torch.max(logits, dim=1) if preds.item() == class_id: corr += 1 total += 1 print(np.round(corr*100/total, decimals = 2)) ```

bsakaguchi closed this issue

2026-10-31 21:12:48 +00:00

bsakaguchi commented

2026-10-31 21:12:49 +00:00

Author

Owner

@yaohui120 commented on GitHub (Jul 25, 2022):

The solution is in https://github.com/openai/CLIP/issues/153. It's the order of classnames in prompt.

@yaohui120 commented on GitHub (Jul 25, 2022): The solution is in https://github.com/openai/CLIP/issues/153. It's the order of classnames in prompt.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#236

No description provided.

Rows
Columns