openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #161] Fine-tuning with CLIP and accuracy decreases? #141

New issue
Open
opened 2026-10-31 21:11:37 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-31 21:11:37 +00:00
Owner
Copy link

Originally created by @oh-livia on GitHub (Oct 21, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/161

Has anyone tried to use CLIP fine-tuning as part of a larger model? I am getting some strange issues with accuracy jumping up to around 20% and then... dropping. I cannot tell if this is just an issue with my model setup, or something I'm doing with the CLIP fine-tuning. The model uses the CLIP image embeddings for both images, concatenates them, then passes them through a few FC layers to get an output the same size as the CLIP text embedding. I use a CrossEntropy loss on the cosine measurements between the output and existing caption embeddings I have (and follow the advice from other posts about changing the weight types for CLIP finetuning).

        convert_models_to_fp32(clip_model)
        clip_opt.step()
        clip.model.convert_weights(clip_model)

I don't know how much detail is needed to showcase the issue I'm having, but I'm curious if anyone else has run into issues of accuracy decreasing when trying to fine-tune the CLIP embeddings

Originally created by @oh-livia on GitHub (Oct 21, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/161 Has anyone tried to use CLIP fine-tuning as part of a larger model? I am getting some strange issues with accuracy jumping up to around 20% and then... dropping. I cannot tell if this is just an issue with my model setup, or something I'm doing with the CLIP fine-tuning. The model uses the CLIP image embeddings for both images, concatenates them, then passes them through a few FC layers to get an output the same size as the CLIP text embedding. I use a CrossEntropy loss on the cosine measurements between the output and existing caption embeddings I have (and follow the advice from other posts about changing the weight types for CLIP finetuning). ``` convert_models_to_fp32(clip_model) clip_opt.step() clip.model.convert_weights(clip_model) ``` I don't know how much detail is needed to showcase the issue I'm having, but I'm curious if anyone else has run into issues of accuracy decreasing when trying to fine-tune the CLIP embeddings
bsakaguchi commented 2026-10-31 21:11:38 +00:00
Author
Owner
Copy link

@dongyun-kim-arch commented on GitHub (Feb 20, 2022):

HI! I am also trying to do CLIP fine-tuning. I am wondering how you get the accuracy metrics since when I trained it, I could only see the loss, but it seems many people could also plot accuracy...

<!-- gh-comment-id:1046322156 --> @dongyun-kim-arch commented on GitHub (Feb 20, 2022): HI! I am also trying to do CLIP fine-tuning. I am wondering how you get the accuracy metrics since when I trained it, I could only see the loss, but it seems many people could also plot accuracy...
bsakaguchi commented 2026-10-31 21:11:38 +00:00
Author
Owner
Copy link

@Liaoqing-up commented on GitHub (Apr 17, 2022):

hello, i also want to train the CLIP. How you train it ?

<!-- gh-comment-id:1100829382 --> @Liaoqing-up commented on GitHub (Apr 17, 2022): hello, i also want to train the CLIP. How you train it ?
bsakaguchi commented 2026-10-31 21:11:38 +00:00
Author
Owner
Copy link

@Viditagarwal7479 commented on GitHub (Feb 19, 2024):

HI! I am also trying to do CLIP fine-tuning. I am wondering how you get the accuracy metrics since when I trained it, I could only see the loss, but it seems many people could also plot accuracy...

@dongyun-kim-arch by evaluating it's zero-shot classification accuracy.

<!-- gh-comment-id:1951937512 --> @Viditagarwal7479 commented on GitHub (Feb 19, 2024): > HI! I am also trying to do CLIP fine-tuning. I am wondering how you get the accuracy metrics since when I trained it, I could only see the loss, but it seems many people could also plot accuracy... @dongyun-kim-arch by evaluating it's zero-shot classification accuracy.
bsakaguchi commented 2026-10-31 21:11:38 +00:00
Author
Owner
Copy link

@Viditagarwal7479 commented on GitHub (Feb 19, 2024):

Has anyone tried to use CLIP fine-tuning as part of a larger model? I am getting some strange issues with accuracy jumping up to around 20% and then... dropping. I cannot tell if this is just an issue with my model setup, or something I'm doing with the CLIP fine-tuning. The model uses the CLIP image embeddings for both images, concatenates them, then passes them through a few FC layers to get an output the same size as the CLIP text embedding. I use a CrossEntropy loss on the cosine measurements between the output and existing caption embeddings I have (and follow the advice from other posts about changing the weight types for CLIP finetuning).

        convert_models_to_fp32(clip_model)
        clip_opt.step()
        clip.model.convert_weights(clip_model)

I don't know how much detail is needed to showcase the issue I'm having, but I'm curious if anyone else has run into issues of accuracy decreasing when trying to fine-tune the CLIP embeddings

@oh-livia I am also facing similar issue, the accuracy keeps on dropping even after trying various learning rates and gradient accumulation steps. Did you find any solution for this?

<!-- gh-comment-id:1951940725 --> @Viditagarwal7479 commented on GitHub (Feb 19, 2024): > Has anyone tried to use CLIP fine-tuning as part of a larger model? I am getting some strange issues with accuracy jumping up to around 20% and then... dropping. I cannot tell if this is just an issue with my model setup, or something I'm doing with the CLIP fine-tuning. The model uses the CLIP image embeddings for both images, concatenates them, then passes them through a few FC layers to get an output the same size as the CLIP text embedding. I use a CrossEntropy loss on the cosine measurements between the output and existing caption embeddings I have (and follow the advice from other posts about changing the weight types for CLIP finetuning). > > ``` > convert_models_to_fp32(clip_model) > clip_opt.step() > clip.model.convert_weights(clip_model) > ``` > > I don't know how much detail is needed to showcase the issue I'm having, but I'm curious if anyone else has run into issues of accuracy decreasing when trying to fine-tune the CLIP embeddings @oh-livia I am also facing similar issue, the accuracy keeps on dropping even after trying various learning rates and gradient accumulation steps. Did you find any solution for this?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#141
No description provided.