mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #150] finetune experience or code #132
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#132
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @niatzt on GitHub (Sep 5, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/150
I tried to finetune CLIP on my own small dataset. The results are not satisfying (probably overfitting). After checking the issues here, I notice other people also have similar problems. Can someone share his successful finetune experience or code.
@ABaldrati commented on GitHub (Sep 7, 2021):
Hi @niatzt
In my limited experience of tine-tuning with CLIP I have often noticed how this operation is extremely sensitive to various hyperparameters.
The advice I feel like giving you (which has worked in my case) is:
keep CLIP in evaluation mode even during training (i.e. keeping the normalization layers frozen)
use very low learning rate (even reaching values like 1e-7 or 1e-8)
If you want to fine-tune CLIP together with another network trained from scratch, use different learning rates (lower for CLIP and higher for the other network). You may also consider keeping CLIP frozen a few epochs.
I hope these tips can help you and that your fine-tuning goes successfully.
If you have more questions do not hesitate to ask
@Sierkinhane commented on GitHub (Nov 23, 2021):
I have finetuned CLIP on PASCAL VOC2012 dataset using this implementation. When I changed the weight decay from 0.2 to 0.001, it works well on my task.
@iremonur commented on GitHub (Jan 17, 2022):
Hi, I aim to finetune the clip but I reached %30 of accuracy at max with the learning rate of 5e-6 and weight decay of 0.2, respectively. How is your accuracy in your experiment? What is the size of your data which you use for fine-tuning? I would be glad if you help me with this.
@dongyun-kim-arch commented on GitHub (Feb 20, 2022):
@iremonur Hi! I am wondering how you get the accuracy metrics when training since I could only see the loss... Is there any other function or script I can refer to to plot acc metrics? Thanks!
@jweihe commented on GitHub (Mar 8, 2022):
Could you provide your fine-tune code?thank you~
@Sierkinhane commented on GitHub (Mar 12, 2022):
My code was same with the implementation
@jweihe commented on GitHub (Mar 18, 2022):
ok， thank u
@ErinZhang1998 commented on GitHub (Apr 6, 2022):
@ABaldrati What do you mean that you freeze the normalization layers?
@sanjaygunda13 commented on GitHub (Jul 25, 2022):
I think this helps in training clip from scratch but not finetune. please let me know if i am wrong i am noob :p in all these stuff
@bwanglzu commented on GitHub (Sep 22, 2022):
in case you're searching for some handy solutions, checkout this page https://finetuner.jina.ai/tasks/text-to-image/
it helps help you get your job done.
@zzxslp commented on GitHub (Dec 20, 2022):
Hi, could you let me know the batch size you are using for fine-tuning CLIP? Thanks!
@zzxslp commented on GitHub (Dec 20, 2022):
Hi Alberto, could you share some advice on the batch size for fine-tuning CLIP? For pertaining it is the larger the better in most cases, nor sure if that is the same based on your experience?
@zhentingqi commented on GitHub (Mar 17, 2024):
Hi, any update in 2024? Is there any method to fine-tune the CLIP model using newest huggingface transformers library (4.35.*)? Thanks!
@tengshaofeng commented on GitHub (Feb 18, 2025):
this is my training code, everything gose well. https://github.com/tengshaofeng/finetune-jina-clip-v2