[GH-ISSUE #58] Training CLIP-ViT #53

New issue

Closed

opened 2026-10-31 21:10:17 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented 2026-10-31 21:10:17 +00:00

Owner

Copy link

Originally created by @Meteorix on GitHub (Mar 10, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/58

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

Originally created by @Meteorix on GitHub (Mar 10, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/58 @jongwook Thanks for this great work! I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT? In the paper: ``We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.``

bsakaguchi closed this issue 2026-10-31 21:10:17 +00:00

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@yuxulingche commented on GitHub (Mar 22, 2021):

Did you write the training code yourself？

<!-- gh-comment-id:804027891 --> @yuxulingche commented on GitHub (Mar 22, 2021): Did you write the training code yourself？

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@Meteorix commented on GitHub (Mar 24, 2021):

Did you write the training code yourself？

yes, it's not hard.

<!-- gh-comment-id:805581930 --> @Meteorix commented on GitHub (Mar 24, 2021): > Did you write the training code yourself？ yes, it's not hard.

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@jongwook commented on GitHub (Mar 24, 2021):

Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme.

<!-- gh-comment-id:805712587 --> @jongwook commented on GitHub (Mar 24, 2021): Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme.

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@Meteorix commented on GitHub (Mar 24, 2021):

Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme.

Thanks for your reply. We have trained on 100M text-image pairs. It turned out that ViTB32 model outperformed ResNet50 model on some of benchmarks.

<!-- gh-comment-id:805754489 --> @Meteorix commented on GitHub (Mar 24, 2021): > Are you comparing ViT-based and ResNet50-based models trained with the same dataset? Vision transformers tend to underperform ResNet-based models unless they're trained on a huge dataset, so I'd suspect that could have been the reason, rather than the initialization scheme. Thanks for your reply. We have trained on 100M text-image pairs. It turned out that ViTB32 model outperformed ResNet50 model on some of benchmarks.

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@ValerioB88 commented on GitHub (Mar 25, 2021):

@Meteorix what dataset did you use?

<!-- gh-comment-id:806368494 --> @ValerioB88 commented on GitHub (Mar 25, 2021): @Meteorix what dataset did you use?

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@dragen1860 commented on GitHub (Mar 26, 2021):

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

Hi, Meteorix:
I am also working on this exciting project. I found you have implemented your own training code. That's pretty cool. May i ask for sharing your training code? it's would be good if you can kindly send the code to me. liangqu.long@gmail.com . Thank you very much.

<!-- gh-comment-id:808012422 --> @dragen1860 commented on GitHub (Mar 26, 2021): > @jongwook Thanks for this great work! > > I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT? > > In the paper: `We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.` Hi, Meteorix: I am also working on this exciting project. I found you have implemented your own training code. That's pretty cool. May i ask for sharing your training code? it's would be good if you can kindly send the code to me. liangqu.long@gmail.com . Thank you very much.

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@jongwook commented on GitHub (Apr 8, 2021):

@dragen1860 see #83 and also other third-party implementations by @KeremTurgutlu and @lucidrains.

<!-- gh-comment-id:815605513 --> @jongwook commented on GitHub (Apr 8, 2021): @dragen1860 see #83 and also other third-party implementations by [@KeremTurgutlu](https://github.com/KeremTurgutlu/self_supervised/blob/main/self_supervised/multimodal/clip.py) and [@lucidrains](https://github.com/lucidrains/DALLE-pytorch).

bsakaguchi commented 2026-10-31 21:10:18 +00:00

Author

Owner

Copy link

@JACKHAHA363 commented on GitHub (Sep 8, 2021):

Hi, I am also running ViT-B/32 and with the open-clip code on CC3M. I also notice a discrepancy between ResNet and ViT. You can see my curves at https://github.com/mlfoundations/open_clip/issues/14. I am interested to know if you solve this performance gap @jongwook @Meteorix

<!-- gh-comment-id:915417522 --> @JACKHAHA363 commented on GitHub (Sep 8, 2021): Hi, I am also running ViT-B/32 and with the open-clip code on CC3M. I also notice a discrepancy between ResNet and ViT. You can see my curves at https://github.com/mlfoundations/open_clip/issues/14. I am interested to know if you solve this performance gap @jongwook @Meteorix

bsakaguchi commented 2026-10-31 21:10:19 +00:00

Author

Owner

Copy link

@Shelton-Zhou commented on GitHub (Nov 3, 2021):

@jongwook Thanks for this great work!

I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT?

In the paper: We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.

Hi, Meteorix:
I am also working on this exciting project. I found you have implemented your own training code. May i ask for sharing your training code? it's would be good if you can kindly send the code to me at x888632157@163.com . Thanks a lot!

<!-- gh-comment-id:958711380 --> @Shelton-Zhou commented on GitHub (Nov 3, 2021): > @jongwook Thanks for this great work! > > I am trying to train CLIP VIT B/32 from scratch, but cannot get a higher score on imagenet versus CLIP resnet-50. May I ask what initialization you use in training VIT? > > In the paper: `We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.` Hi, Meteorix: I am also working on this exciting project. I found you have implemented your own training code. May i ask for sharing your training code? it's would be good if you can kindly send the code to me at x888632157@163.com . Thanks a lot!

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

codex/pin-github-workflow-refs-20260325-132838

jongwook/issue-396

jongwook-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#53

No description provided.