openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #125] zero-shot results on kinetic-700 dataset #113

New issue

Closed

opened 2026-10-31 21:11:18 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 21:11:18 +00:00

Owner

Originally created by @ray342659093 on GitHub (Jul 6, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/125

Hi, I would like to otain the zero-shot results on the val split of the kinetic-700 dataset. I uniformly sample 20 frames from each video and average their smilarity scores. However, I only get top-1 acc of 38% and top-5 acc of 65%, which is far less than the reported results on the paper (69.6% in terms of the average acc from Table 15). The piece of code I used is as follows. Is there any trick that I need to use in order to boost the performance?


import os
import clip
import torch
from dataset import kinetic
from torch.utils.data import DataLoader
from tqdm import tqdm

device = "cuda" 
model, preprocess = clip.load('/cfs/cfs-phfh0q9j/rayruisu/ViT-B-32.pt', device,jit=False)

model = torch.nn.DataParallel(model,device_ids=[0, 1, 2, 3])

kinetic_data = kinetic(img_dir='val_list.txt', transform=preprocess)

num = 0
corr_1=0
corr_5=0
corr_10 = 0

with torch.no_grad():
    for image, class_label in tqdm(DataLoader(kinetic_data,batch_size=100,num_workers=12,pin_memory=True)):
        b,d,_,_,_ = image.size()
        
        image_input = image.to(device).view(-1,image.size(-3), image.size(-2), image.size(-1))
        text_inputs = torch.cat([clip.tokenize(f"a picture of action {c}") for c in kinetic_data.img_labels]).to(device)
        
        similarity, _ = model(image_input, torch.cat([text_inputs]*4,dim=0))
        similarity = similarity.softmax(dim=-1)
        values_1, indices_1=similarity.view(b,d,-1).mean(dim=1).topk(1,dim=-1)
        values_5, indices_5=similarity.view(b,d,-1).mean(dim=1).topk(5,dim=-1)
        values_10, indices_10=similarity.view(b,d,-1).mean(dim=1).topk(10,dim=-1)
        num += b
        for i in range(b):
            if kinetic_data.img_labels[indices_1[i]] == class_label[i]:
                corr_1 +=1
            top5_list = [kinetic_data.img_labels[cls] for cls in indices_5[i]]
            if class_label[i] in top5_list:
                corr_5+=1
            top10_list = [kinetic_data.img_labels[cls] for cls in indices_10[i]]
            if class_label[i] in top10_list:
                corr_10+=1
                
print(float(corr_1)/num*100)
print(float(corr_5)/num*100)
print(float(corr_10)/num*100)

Originally created by @ray342659093 on GitHub (Jul 6, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/125 Hi, I would like to otain the zero-shot results on the val split of the kinetic-700 dataset. I uniformly sample 20 frames from each video and average their smilarity scores. However, I only get top-1 acc of 38% and top-5 acc of 65%, which is far less than the reported results on the paper (69.6% in terms of the average acc from Table 15). The piece of code I used is as follows. Is there any trick that I need to use in order to boost the performance? ``` import os import clip import torch from dataset import kinetic from torch.utils.data import DataLoader from tqdm import tqdm device = "cuda" model, preprocess = clip.load('/cfs/cfs-phfh0q9j/rayruisu/ViT-B-32.pt', device,jit=False) model = torch.nn.DataParallel(model,device_ids=[0, 1, 2, 3]) kinetic_data = kinetic(img_dir='val_list.txt', transform=preprocess) num = 0 corr_1=0 corr_5=0 corr_10 = 0 with torch.no_grad(): for image, class_label in tqdm(DataLoader(kinetic_data,batch_size=100,num_workers=12,pin_memory=True)): b,d,_,_,_ = image.size() image_input = image.to(device).view(-1,image.size(-3), image.size(-2), image.size(-1)) text_inputs = torch.cat([clip.tokenize(f"a picture of action {c}") for c in kinetic_data.img_labels]).to(device) similarity, _ = model(image_input, torch.cat([text_inputs]*4,dim=0)) similarity = similarity.softmax(dim=-1) values_1, indices_1=similarity.view(b,d,-1).mean(dim=1).topk(1,dim=-1) values_5, indices_5=similarity.view(b,d,-1).mean(dim=1).topk(5,dim=-1) values_10, indices_10=similarity.view(b,d,-1).mean(dim=1).topk(10,dim=-1) num += b for i in range(b): if kinetic_data.img_labels[indices_1[i]] == class_label[i]: corr_1 +=1 top5_list = [kinetic_data.img_labels[cls] for cls in indices_5[i]] if class_label[i] in top5_list: corr_5+=1 top10_list = [kinetic_data.img_labels[cls] for cls in indices_10[i]] if class_label[i] in top10_list: corr_10+=1 print(float(corr_1)/num*100) print(float(corr_5)/num*100) print(float(corr_10)/num*100) ```

bsakaguchi closed this issue

2026-10-31 21:11:19 +00:00

bsakaguchi commented

2026-10-31 21:11:19 +00:00

Author

Owner

@sallymmx commented on GitHub (Jul 7, 2021):

The same question, looking forward for response!

@sallymmx commented on GitHub (Jul 7, 2021): The same question, looking forward for response!

bsakaguchi commented

2026-10-31 21:11:19 +00:00

Author

Owner

@cabaf commented on GitHub (Jul 16, 2021):

I think you should compare with the results in Table 11 (ArXiv paper). Specifically, you should look at the row for the B/32 Clip ViT model. That said, the Clip paper reports a 47.8% performance, i.e., mean(top1, top5), which is already lower than what you have achieved. Perhaps, your 20-frame pooling helps to boost performance.

@cabaf commented on GitHub (Jul 16, 2021): I think you should compare with the results in Table 11 ([ArXiv paper](https://arxiv.org/pdf/2103.00020.pdf)). Specifically, you should look at the row for the B/32 Clip ViT model. That said, the Clip paper reports a 47.8% performance, i.e., mean(top1, top5), which is already lower than what you have achieved. Perhaps, your 20-frame pooling helps to boost performance.

bsakaguchi commented

2026-10-31 21:11:19 +00:00

Author

Owner

@jongwook commented on GitHub (Jul 18, 2021):

@cabaf is correct; apologies about any confusion. Table 11 reported the mean(top1, top5) score evaluated only on the middle frames, whereas table 15 reported the best we can achieve, using the (unreleased) ViT-L/14@336px on all frames. This was following the convention to call the best model simply "CLIP", as in the last sentence of Section 2.

@jongwook commented on GitHub (Jul 18, 2021): @cabaf is correct; apologies about any confusion. Table 11 reported the mean(top1, top5) score evaluated only on the middle frames, whereas table 15 reported the best we can achieve, using the (unreleased) ViT-L/14@336px on all frames. This was following the convention to call the best model simply "CLIP", as in the last sentence of Section 2.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#113

No description provided.

Rows
Columns