[GH-ISSUE #464] What is the best practice to train image with multiple captions/keywords #393

New issue

Open

opened 2026-10-31 21:14:37 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:14:37 +00:00

Owner

Originally created by @pengzhao-life on GitHub (Oct 3, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/464

Hi, I am trying to finetune clip model on my own dataset. My data has multiple texts per image. My understanding is that I need to create each ground truth label to ensure the positive relationship between the image and its every single text. Otherwise, they may be considered as negative pair during the training. Is this correct? Any sample code or post related to this concern?

My second question is that the texts may include image captions, keywords, categories, etc. Do I need to handle them differently? Many thanks!

Originally created by @pengzhao-life on GitHub (Oct 3, 2024). Original GitHub issue: https://github.com/openai/CLIP/issues/464 Hi, I am trying to finetune clip model on my own dataset. My data has multiple texts per image. My understanding is that I need to create each ground truth label to ensure the positive relationship between the image and its every single text. Otherwise, they may be considered as negative pair during the training. Is this correct? Any sample code or post related to this concern? My second question is that the texts may include image captions, keywords, categories, etc. Do I need to handle them differently? Many thanks!

bsakaguchi commented

2026-10-31 21:14:37 +00:00

Author

Owner

@tengshaofeng commented on GitHub (Feb 18, 2025):

@pengzhao-life this is my training code. https://github.com/tengshaofeng/finetune-jina-clip-v2

@tengshaofeng commented on GitHub (Feb 18, 2025): @pengzhao-life this is my training code. https://github.com/tengshaofeng/finetune-jina-clip-v2

bsakaguchi referenced this issue

2026-10-31 21:15:32 +00:00

[PR #393] [CLOSED] a #498