[GH-ISSUE #436] On a Bag-of-Words baseline and a transformer #373

New issue

Open

opened 2026-10-31 21:14:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:14:28 +00:00

Owner

Originally created by @iburenko on GitHub (Apr 15, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/436

Dear authors,

Thank you very much for the great work!

I am trying to understand, how one could obtain the bag-of-words representations for a caption that are described in Sec 2.3:

we explored training
a system to solve the potentially easier proxy task of predicting only which text as a whole is paired with which
image and not the exact words of that text. Starting with
the same bag-of-words encoding baseline, we swapped the
predictive objective for a contrastive objective in Figure 2
and observed a further 4x efficiency improvement in the rate
of zero-shot transfer to ImageNet.

I wonder how this bag-of-words baseline is trained with the transformer. I guess that we could avoid using positional embeddings at the training phase (obviously, we use them during inference), making the activations of the last layer of the transformer at [EOS] token context-free and, therefore, interpreting them as BoW embeddings. Is this what is happening, or are these BoW representations calculated somehow differently?

Originally created by @iburenko on GitHub (Apr 15, 2024). Original GitHub issue: https://github.com/openai/CLIP/issues/436 Dear authors, Thank you very much for the great work! I am trying to understand, how one could obtain the bag-of-words representations for a caption that are described in Sec 2.3: > we explored training a system to solve the potentially easier proxy task of predicting only which text as a whole is paired with which image and not the exact words of that text. Starting with the same bag-of-words encoding baseline, we swapped the predictive objective for a contrastive objective in Figure 2 and observed a further 4x efficiency improvement in the rate of zero-shot transfer to ImageNet. I wonder how this bag-of-words baseline is trained with the transformer. I guess that we could avoid using positional embeddings at the training phase (obviously, [we use them during inference](https://github.com/openai/CLIP/blob/a1d071733d7111c9c014f024669f959182114e33/clip/model.py#L346)), making the activations of the last layer of the transformer at [EOS] token context-free and, therefore, interpreting them as BoW embeddings. Is this what is happening, or are these BoW representations calculated somehow differently?

bsakaguchi referenced this issue

2026-10-31 21:15:30 +00:00

[PR #373] Update github workflow for PyTorch 2.0.1 #493