openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #323] Getting overfitted CLIP model while fine-tuning with a large training dataset #278

New issue
Open
opened 2026-10-31 21:13:21 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-31 21:13:21 +00:00
Owner
Copy link

Originally created by @antu-saha on GitHub (Feb 18, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/323

Hi,
I want to fine-tune the CLIP model with my dataset. The size of the dataset is very large. There are more than 2M image-text pairs. I tried various learning rates including too-small learning rates. But during training, the loss is decreasing at first and after 2 or 3 epochs the loss is increasing again. I am using SGD optimizer. If I use a subset of my full dataset (smaller dataset), then the training is going well.
As I have lots of captions for the same image (on average 110 captions for each image), I have created a dataset with a unique batch (all the image-text pairs are unique in the same batch). Due to memory constraints, I am using a batch size of 200. Still, I am getting the same training graph.

Could you please tell me why I am facing this issue? Is it because of my dataset size?
What should I do for getting better fine-tuning?

Thank you very much.

Originally created by @antu-saha on GitHub (Feb 18, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/323 Hi, I want to fine-tune the CLIP model with my dataset. The size of the dataset is very large. There are more than 2M image-text pairs. I tried various learning rates including too-small learning rates. But during training, the loss is decreasing at first and after 2 or 3 epochs the loss is increasing again. I am using SGD optimizer. If I use a subset of my full dataset (smaller dataset), then the training is going well. As I have lots of captions for the same image (on average 110 captions for each image), I have created a dataset with a unique batch (all the image-text pairs are unique in the same batch). Due to memory constraints, I am using a batch size of 200. Still, I am getting the same training graph. Could you please tell me why I am facing this issue? Is it because of my dataset size? What should I do for getting better fine-tuning? Thank you very much.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@Rohinivv96 commented on GitHub (Oct 27, 2023):

Hi @antu-saha Can you please guide me how we can fine tune CLIP model for custom dataset.

<!-- gh-comment-id:1782331953 --> @Rohinivv96 commented on GitHub (Oct 27, 2023): Hi @antu-saha Can you please guide me how we can fine tune CLIP model for custom dataset.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@BaochaoZhu commented on GitHub (Mar 11, 2024):

Hi @antu-saha Can you please guide me how we can fine tune CLIP model for custom dataset.

<!-- gh-comment-id:1987504978 --> @BaochaoZhu commented on GitHub (Mar 11, 2024): Hi @antu-saha Can you please guide me how we can fine tune CLIP model for custom dataset.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@antu-saha commented on GitHub (Mar 11, 2024):

Hi @Rohinivv96 and @BaochaoZhu, my apology for the late reply.
Sure, I can guide you to fine-tune CLIP.
We can fine-tune CLIP using the pretrained model. We just need to prepare our dataset. In my case, I have images with their corresponding captions. I created 2 lists (image paths and captions) and used those lists to create the batches. I can also share the code if needed. Please don't hesitate to reach out if you wanna know more information.
Thank you very much.

<!-- gh-comment-id:1989473142 --> @antu-saha commented on GitHub (Mar 11, 2024): Hi @Rohinivv96 and @BaochaoZhu, my apology for the late reply. Sure, I can guide you to fine-tune CLIP. We can fine-tune CLIP using the pretrained model. We just need to prepare our dataset. In my case, I have images with their corresponding captions. I created 2 lists (image paths and captions) and used those lists to create the batches. I can also share the code if needed. Please don't hesitate to reach out if you wanna know more information. Thank you very much.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@BaochaoZhu commented on GitHub (Mar 12, 2024):

@antu-saha Thank you for your assistance. Would you mind sharing the code with me or emailing it to zhubaochao666888@gmail.com? I would like to learn from it. Thank you very much.

<!-- gh-comment-id:1989761133 --> @BaochaoZhu commented on GitHub (Mar 12, 2024): @antu-saha Thank you for your assistance. Would you mind sharing the code with me or emailing it to zhubaochao666888@gmail.com? I would like to learn from it. Thank you very much.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@anhuigzj commented on GitHub (Jan 3, 2025):

Hi @Rohinivv96 and @BaochaoZhu, my apology for the late reply. Sure, I can guide you to fine-tune CLIP. We can fine-tune CLIP using the pretrained model. We just need to prepare our dataset. In my case, I have images with their corresponding captions. I created 2 lists (image paths and captions) and used those lists to create the batches. I can also share the code if needed. Please don't hesitate to reach out if you wanna know more information. Thank you very much.

Would you mind sharing the code with me or emailing it to 1195688513@qq.com? I would like to learn from it. Thank you very much.

<!-- gh-comment-id:2568597316 --> @anhuigzj commented on GitHub (Jan 3, 2025): > Hi @Rohinivv96 and @BaochaoZhu, my apology for the late reply. Sure, I can guide you to fine-tune CLIP. We can fine-tune CLIP using the pretrained model. We just need to prepare our dataset. In my case, I have images with their corresponding captions. I created 2 lists (image paths and captions) and used those lists to create the batches. I can also share the code if needed. Please don't hesitate to reach out if you wanna know more information. Thank you very much. Would you mind sharing the code with me or emailing it to [1195688513@qq.com](1195688513@qq.com)? I would like to learn from it. Thank you very much.
bsakaguchi commented 2026-10-31 21:13:23 +00:00
Author
Owner
Copy link

@PixelChen24 commented on GitHub (Nov 11, 2025):

笑死我了这哥们本来是提问题的，结果被一堆更加有问题的伸手党要代码了

<!-- gh-comment-id:3515272496 --> @PixelChen24 commented on GitHub (Nov 11, 2025): 笑死我了这哥们本来是提问题的，结果被一堆更加有问题的伸手党要代码了
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#278
No description provided.