mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #72] how to distill from CLIP to get a tiny model? #66
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#66
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @dragen1860 on GitHub (Mar 26, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/72
Dear all:
I want to retain a smaller model from CLIP network. However, when i try to run knowledge distillation from Vit32model, I found
model.encode_image(image)get a vector of lenght 512, whilemodel(image_feature, text_feature)network need the text input. I donot know how to get the soft label from CLIP. Anyone get some idea about how to distill knowledge from CLIP? thanks .@jongwook commented on GitHub (Apr 8, 2021):
I guess there are different strategies depending on your goals. If you want to make a smaller vision model that encodes image into the feature space similar to CLIP's, so that you can use CLIP's robust representation with the faster encoder, I'd train the student encoder to minimize L2 or cosine distances to CLIP's features.
If your goal is to make a specific zero-shot image classifier faster with a predetermined set of labels, you can train your image encoder to minimize the KL divergence from the probability distribution resulting from the logits returned by the CLIP model.
@dragen1860 commented on GitHub (Apr 13, 2021):
@jongwook Thank you for the detailed tips. I finally get what you mean. Have you test training a mobile backbone using CLIP setting. Is it possible to get a reasonble zero-shot performance?
@jongwook commented on GitHub (Apr 13, 2021):
We don't have results on smaller model sizes than the released ones, and we haven't experimented CLIP on mobile devices. ResNet50 or ViT-B/32 model requires only a few gigaflops for the forward pass, and you might not need distillation at all if it's fine for your use case.
@dimitry12 commented on GitHub (Aug 17, 2021):
@dragen1860 have you tried distilling CLIP's image-encoder? If yes, which one (ViT or RN based)?
@dimitry12 commented on GitHub (Aug 26, 2021):
I found notes about CLIP's image-encoder distillation in https://tech.pic-collage.com/distillation-of-clip-model-and-other-experiments-f8394b7321ce but authors report loss of model behavior.
@ZJJHYM commented on GitHub (Jan 7, 2023):
Hello, Can you share me the weight after distill or the code about distill the clip model? My email is 13816267895@163.com
Thank you very much!
@justlike-prog commented on GitHub (Aug 2, 2023):
@dragen1860 did you manage to make a succesful distillation?