[GH-ISSUE #252] Fine-tuning CLIP for image object detection. #221

New issue

Open

opened 2026-10-31 21:12:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:12:37 +00:00

Owner

Originally created by @dukaenea on GitHub (Jun 8, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/252

Thank you for publishing such great research and the implementation! This is more of a question rather than an issue per se.

I have used the image encoder of CLIP in the to encode video frames for video action classification. I used it as a frozen backbone and it outperformed all other backbones I was using (ViT being the second best).
Now, I would like to use it again as a backbone for image object detection. This time however, I would like to fine-tune it while training the rest of the model. The dataset I have contains around 200K images. I would like to ask if you think fine-tuning would be feasible in this case since the task I will be fine-tuning on will be quite different than the original task it was trained on.
I am supposing that a lower learning rate (than the one used for the rest of the model), a lower weight decay value and a late start of fine-tuning would be necessary in this case. Any other advice for fine-tuning in this case would be more than welcome :)

Originally created by @dukaenea on GitHub (Jun 8, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/252 Thank you for publishing such great research and the implementation! This is more of a question rather than an issue per se. I have used the image encoder of CLIP in the to encode video frames for video action classification. I used it as a frozen backbone and it outperformed all other backbones I was using (ViT being the second best). Now, I would like to use it again as a backbone for image object detection. This time however, I would like to fine-tune it while training the rest of the model. The dataset I have contains around 200K images. I would like to ask if you think fine-tuning would be feasible in this case since the task I will be fine-tuning on will be quite different than the original task it was trained on. I am supposing that a lower learning rate (than the one used for the rest of the model), a lower weight decay value and a late start of fine-tuning would be necessary in this case. Any other advice for fine-tuning in this case would be more than welcome :)

bsakaguchi commented

2026-10-31 21:12:38 +00:00

Author

Owner

@iasonasxrist commented on GitHub (Jun 14, 2022):

could you share the way you implement the clip image_encoder as backbone?

@iasonasxrist commented on GitHub (Jun 14, 2022): could you share the way you implement the clip image_encoder as backbone?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#221

No description provided.

Rows
Columns