mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #207] Inconsistent numbers across tables on UCF101 and Kinetics-700 #182
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#182
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @LuoweiZhou on GitHub (Jan 6, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/207
Hi Team, thanks for open-sourcing this wonderful project. I notice the following zero-shot numbers do not match:
In Tab. 11, row 9, UCF101: 76.9 / Kinetics-700: 61.3
vs.
In Tab. 15, row 9, UCF101: 80.3 / Kinetics-700: 69.6
Link to paper: https://arxiv.org/pdf/2103.00020.pdf (would be Tab. 17 vs Tab. 14 on the OpenAI version).
Assuming the best performing model variant is adopted in Tab. 15 (i.e., CLIP-ViT L/14-336px). Also, assuming the same metrics applied. I did notice the linear probing numbers match. Is it possible due to different model augmentation/ensemble? For example, different numbers of clips/views, numbers of prompts, etc. Or something I am not aware of. Thanks!
@LuoweiZhou commented on GitHub (Jan 6, 2022):
Follow-up: For Tab. 15, from the paper "we report CLIP’s zero-shot performance when averaging predictions across all frames", assuming this is the only difference. The gain is fairly significant, esp. on K700.