[GH-ISSUE #23] How is the dataset collected?

bsakaguchi commented

2026-10-31 21:09:59 +00:00

Owner

Originally created by @linzhiqiu on GitHub (Jan 25, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/23

This is a question related to the paper instead of this codebase. In paper section 2.2, it briefly describes how the data are gathered by "...we constructed a new dataset of 400 million (image, text) pairs collected form a variety of publicly available sources on the Internet."

I was wondering what are the publicly available sources (e.g. Google image search, Flickr image search, etc.)?

Originally created by @linzhiqiu on GitHub (Jan 25, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/23 This is a question related to the paper instead of this codebase. In paper section 2.2, it briefly describes how the data are gathered by "_...we constructed a new dataset of 400 million (image, text) pairs collected form a variety of publicly available sources on the Internet._" I was wondering what are the **publicly available sources** (e.g. Google image search, Flickr image search, etc.)?

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@rsomani95 commented on GitHub (Jan 26, 2021):

Tangential to @linzhiqiu's question, I'm also curious how much of the data comes from existing academic datasets / which ones were used for training the model, if any at all?

Is OpenAI open to sharing this information? I think it's crucial to examine the model for bias. For example, if Places365 was used in training the model, it's redundant to test how it performs on that dataset

@rsomani95 commented on GitHub (Jan 26, 2021): Tangential to @linzhiqiu's question, I'm also curious how much of the data comes from existing academic datasets / which ones were used for training the model, if any at all? Is OpenAI open to sharing this information? I think it's crucial to examine the model for bias. For example, if [Places365](https://github.com/CSAILVision/places365) was used in training the model, it's redundant to test how it performs on that dataset

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@meigaoms commented on GitHub (Feb 5, 2021):

This is mentioned in paper: "The base query list is all words occurring at least 100 times in
the English version of Wikipedia. This is augmented with bi-grams
with high pointwise mutual information as well as the names of
all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles.

@meigaoms commented on GitHub (Feb 5, 2021): This is mentioned in paper: "The base query list is all words occurring at least 100 times in the English version of Wikipedia. This is augmented with bi-grams with high pointwise mutual information as well as the names of all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles.

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@Rijgersberg commented on GitHub (Apr 21, 2021):

I too would like to echo the request for more information about the dataset.

As you show in Section 5 of the paper, overlap with the training dataset can be relevant when evaluating the performance and suitability of the pretrained model in novel contexts.

Besides that, the source, reach and quality of the dataset used are important factors in our internal ethics evaluation process, which we are currently unable to assess.

If you are unable to release full details of the dataset, would you be willing to release more partial details? Some suggestions:

a list of links to the images
a list of file hashes
a list of the 500,000 queries used to gather the images
a list of the publicly available sources used (as suggested by the original commenter)

Any additional details would be greatly appreciated and will aid us in our research into the potential impact of models like CLIP in our domain.

@Rijgersberg commented on GitHub (Apr 21, 2021): I too would like to echo the request for more information about the dataset. As you show in Section 5 of the paper, overlap with the training dataset can be relevant when evaluating the performance and suitability of the pretrained model in novel contexts. Besides that, the source, reach and quality of the dataset used are important factors in our internal ethics evaluation process, which we are currently unable to assess. If you are unable to release full details of the dataset, would you be willing to release more partial details? Some suggestions: - a list of links to the images - a list of file hashes - a list of the 500,000 queries used to gather the images - a list of the publicly available sources used (as suggested by the original commenter) Any additional details would be greatly appreciated and will aid us in our research into the potential impact of models like CLIP in our domain.

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@realTaki commented on GitHub (Sep 13, 2021):

This is mentioned in paper: "The base query list is all words occurring at least 100 times in
the English version of Wikipedia. This is augmented with bi-grams
with high pointwise mutual information as well as the names of
all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles.

I want to know how they expand the words to a text for training? I mean if the description of the image was just format from a template(like a photo of {}), the performance would be equal to using a single keyword without using a template?

@realTaki commented on GitHub (Sep 13, 2021): > This is mentioned in paper: "The base query list is all words occurring at least 100 times in > the English version of Wikipedia. This is augmented with bi-grams > with high pointwise mutual information as well as the names of > all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles. I want to know how they expand the words to a text for training? I mean if the description of the image was just format from a template(like `a photo of {}`), the performance would be equal to using a single keyword without using a template?

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@JunweiLiang commented on GitHub (Dec 10, 2021):

If you don't already know, this paper uses the released model to inversely get the closest 400M image-text pairs from web crawl dataset.

@JunweiLiang commented on GitHub (Dec 10, 2021): If you don't already know, [this paper](https://arxiv.org/abs/2111.02114) uses the released model to inversely get the closest 400M image-text pairs from web crawl dataset.

bsakaguchi commented

2026-10-31 21:10:00 +00:00

Author

Owner

@igo312 commented on GitHub (Dec 10, 2021):

This is mentioned in paper: "The base query list is all words occurring at least 100 times in
the English version of Wikipedia. This is augmented with bi-grams
with high pointwise mutual information as well as the names of
all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles.

I want to know how they expand the words to a text for training? I mean if the description of the image was just format from a template(like a photo of {}), the performance would be equal to using a single keyword without using a template?

Maybe OpenAI used some tricks that the template format is more than a photo of {}. e.g. for anmials the template is the animal name is {} . As a result, may template will guided the model?

@igo312 commented on GitHub (Dec 10, 2021): > > This is mentioned in paper: "The base query list is all words occurring at least 100 times in > > the English version of Wikipedia. This is augmented with bi-grams > > with high pointwise mutual information as well as the names of > > all Wikipedia articles above a certain search volume. ". I'm curious about how the high frequency words are augmented with bigrams and wikipedia articles. > > I want to know how they expand the words to a text for training? I mean if the description of the image was just format from a template(like `a photo of {}`), the performance would be equal to using a single keyword without using a template? Maybe OpenAI used some tricks that the template format is more than `a photo of {}`. e.g. for anmials the template is `the animal name is {} `. As a result, may template will guided the model?

bsakaguchi commented

2026-10-31 21:10:01 +00:00

Author

Owner

@haardikdharma10 commented on GitHub (Feb 8, 2025):

I know this is an old thread... but this paper Demystifying CLIP data intends to reveal CLIP’s data curation approach. Also see - https://github.com/facebookresearch/MetaCLIP

@haardikdharma10 commented on GitHub (Feb 8, 2025): I know this is an old thread... but this paper [Demystifying CLIP data](https://arxiv.org/pdf/2309.16671v4) intends to reveal CLIP’s data curation approach. Also see - https://github.com/facebookresearch/MetaCLIP

bsakaguchi referenced this issue

2026-10-31 21:15:06 +00:00

[PR #22] [CLOSED] remove unused/repeated imports #447

Rows
Columns

[GH-ISSUE #23] How is the dataset collected? #22