openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #156] GTSRB dataset issue #138

New issue
Closed
opened 2026-10-31 21:11:35 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-31 21:11:35 +00:00
Owner
Copy link

Originally created by @pj-ms on GitHub (Sep 23, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/156

According to the official website, there are two versions of GTSRB:

Version num train images num test images
IJCNN 2011 competition 26640 12569
Official 39209 12630

The dataset stats (Table 9, Page 39) seem to suggest it is using train set from IJCNN 2011 version but test set from official version.

Given that Official-Train = IJCNN-Train + IJCNN-Test (Source), is CLIP using IJCNN Train as train set, IJCNN Test set as val set to tune hyper-param, Official Test set as test set? Thanks!

image

Originally created by @pj-ms on GitHub (Sep 23, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/156 According to the [official website](https://sid.erda.dk/public/archives/daaeac0d7ce1152aea9b61d9f1e19370/published-archive.html), there are two versions of GTSRB: | Version| num train images| num test images| | --- | ----------- |----------- | | IJCNN 2011 competition | 26640| 12569 | |Official | 39209 | 12630 | The dataset stats (Table 9, Page 39) seem to suggest it is using train set from IJCNN 2011 version but test set from official version. Given that **_Official-Train = IJCNN-Train + IJCNN-Test_** ([Source](https://benchmark.ini.rub.de/gtsrb_news.html)), is CLIP using IJCNN Train as train set, IJCNN Test set as val set to tune hyper-param, Official Test set as test set? Thanks! ![image](https://user-images.githubusercontent.com/25910409/134549886-2ba77b4d-e22e-4b59-9098-eeb82207646c.png)
bsakaguchi commented 2026-10-31 21:11:36 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Sep 24, 2021):

Thanks for catching this. We were not aware of the distinction and seems like we happened to use the training images from the competition and the testing images from the "official" version.

We'd expect the linear probes (for all models compared) would perform slightly better when it's trained on more images in the official version, so I assume the reported accuracies can still serve as a "baseline" in future studies. We also note that the same set of training images are used across all models, so the comparisons in the paper can still be considered "fair". The zero-shot evaluations are not affected by this, since it only uses the test split.

<!-- gh-comment-id:926323191 --> @jongwook commented on GitHub (Sep 24, 2021): Thanks for catching this. We were not aware of the distinction and seems like we happened to use the training images from the competition and the testing images from the "official" version. We'd expect the linear probes (for all models compared) would perform slightly better when it's trained on more images in the official version, so I assume the reported accuracies can still serve as a "baseline" in future studies. We also note that the same set of training images are used across all models, so the comparisons in the paper can still be considered "fair". The zero-shot evaluations are not affected by this, since it only uses the test split.
bsakaguchi commented 2026-10-31 21:11:36 +00:00
Author
Owner
Copy link

@pj-ms commented on GitHub (Sep 24, 2021):

Thanks for the prompt response! Yeah, it is good enough to know the numbers from the competition train + official test.

Btw, as zero-shot is mentioned here, any chance I could know what Alec eventually used for those traffic signs :-D

image

<!-- gh-comment-id:926361605 --> @pj-ms commented on GitHub (Sep 24, 2021): Thanks for the prompt response! Yeah, it is good enough to know the numbers from the competition train + official test. Btw, as zero-shot is mentioned here, any chance I could know what Alec eventually used for those traffic signs :-D ![image](https://user-images.githubusercontent.com/25910409/134623929-051d40b2-7c8a-4835-b136-6b8fe39b999a.png)
bsakaguchi commented 2026-10-31 21:11:36 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Sep 24, 2021):

We recently uploaded the labels at https://github.com/openai/CLIP/blob/main/data/prompts.md#gtsrb

<!-- gh-comment-id:926362109 --> @jongwook commented on GitHub (Sep 24, 2021): We recently uploaded the labels at https://github.com/openai/CLIP/blob/main/data/prompts.md#gtsrb
bsakaguchi commented 2026-10-31 21:11:36 +00:00
Author
Owner
Copy link

@pj-ms commented on GitHub (Sep 24, 2021):

Thanks! Just found it there as well :-D

<!-- gh-comment-id:926365352 --> @pj-ms commented on GitHub (Sep 24, 2021): Thanks! Just found it there as well :-D
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#138
No description provided.