mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #83] CLIP Training Code #73
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#73
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @vinson2233 on GitHub (Apr 8, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/83
Not really an issue, I just want to share my training code since some people still have some difficulties to write the training code. Just modify the code to suit your usage.
Feel free to ask or point out any mistakes in my code.
Code to save the model :
Code to load the saved model :
Alternative training code :
@nikky4D commented on GitHub (Apr 8, 2021):
Very helpful. Thank you
@vkmavani commented on GitHub (Apr 12, 2021):
Hi, Thank you for this training code.
I have a dataset, where I want to check the image similarity, and I want to use the CLIP. But I don't know how to prepare(image_size, embedding_size, transforms, etc) a dataset to feed this training code. Can you please provide me the dataset class if possible?
@vinson2233 commented on GitHub (Apr 12, 2021):
@vkmavani sure. The
preprocessobject from CLIP takes care of all of the preprocessing steps for the image part, so you don't need to worry about image_size or transform(see https://github.com/openai/CLIP/blob/main/clip/clip.py line 58).For example, maybe your data look like this :
where the URL is the path to the image and the caption is the string of the caption.
Here's the dataset class definition for image-text similarity :
With this dataset definition, you can omit the
Image.fromarray()and the preprocess step after loading the batch since the actual data already in tensor formatIf you are interested in doing image-image similarity, just modify the dataset to return pair of images and
for the training code, adjust the code accordingly, a big change will happen in the creating the logits part. Change the forward method
logits_per_image, logits_per_text = model(images, texts)according to https://github.com/openai/CLIP/blob/main/clip/model.py, line 354.@lonngxiang commented on GitHub (Apr 12, 2021):
what is the clip.model.convert_weights meaning? and can you Provide a complete training code if possible
@vinson2233 commented on GitHub (Apr 12, 2021):
@lonngxiang For more information, read https://github.com/openai/CLIP/issues/57, clip.model.convert_weights basically convert the CLIP model weight into float16. This will help accelerate and reduce memory usage during training.
The definition of clip.model.convert_weight can be found at https://github.com/openai/CLIP/blob/main/clip/model.py line 371
I can't give a fully working example code since I'm using a private dataset, but I believe the training code and dataset code that I provided is sufficient.
@lonngxiang commented on GitHub (Apr 12, 2021):
Thank you for your kind reply
@lonngxiang commented on GitHub (Apr 12, 2021):
there is a error when run this train code：
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.JpegImagePlugin.JpegImageFile'>
@vkmavani commented on GitHub (Apr 12, 2021):
Thank you very much. It really helps a lot.
@vinson2233 commented on GitHub (Apr 12, 2021):
@lonngxiang oh you are correct. pardon me, I have edited my code above. The dataset should return something that can be put on PyTorch tensor.
@lonngxiang commented on GitHub (Apr 12, 2021):
one more thing，when you use preprocess in class image_caption_dataset, the torch.stack's preprocess is it still useful?
@lonngxiang commented on GitHub (Apr 12, 2021):
still have a error in images= torch.stack([preprocess(Image.fromarray(img)) for img in list_image],dim=0):
AttributeError: 'Tensor' object has no attribute 'array_interface'
@vinson2233 commented on GitHub (Apr 12, 2021):
Yeah, if already using preprocess inside the class. The result from the batch can be used directly to the CLIP. So that line can be change into this :
images = list_image@lonngxiang commented on GitHub (Apr 12, 2021):
then have anthor error:
RuntimeError: "unfolded2d_copy" not implemented for 'Half'
@vinson2233 commented on GitHub (Apr 12, 2021):
Hmmmm, that error is new for me. Is the error occurred when calculating the loss?
@lonngxiang commented on GitHub (Apr 12, 2021):
yes,the error occurred in this line:
logits_per_image, logits_per_text = model(images, texts)
add model(images.float(), texts.float()) still error:
RuntimeError: "unfolded2d_copy" not implemented for 'Half'
@vinson2233 commented on GitHub (Apr 12, 2021):
Are you using CPU by any chance? The mixed precision training usually don't work on CPU
@lonngxiang commented on GitHub (Apr 12, 2021):
yes, i use it on cpu
@vinson2233 commented on GitHub (Apr 12, 2021):
@lonngxiang I have updated the code again. Basically, remove all code related to mixed-precision training when using CPU instead of GPU
@lonngxiang commented on GitHub (Apr 12, 2021):
ok. so kind of you; Thank you for your patience
@lonngxiang commented on GitHub (Apr 13, 2021):
@lonngxiang commented on GitHub (Apr 13, 2021):
how to set BATCH_SIZE to get ground_truth's label?
@vinson2233 commented on GitHub (Apr 13, 2021):
@lonngxiang Hmmmm, I don't have the faintest idea why the loss is = 0.
BATCH_SIZE is just an integer that you set. Since the image-text are in pairs, the first image will correspond to the first text. So the ground truth for the first image is 0, the second image will correspond to the second image, so the ground truth is 1.
This pattern keeps repeating until the last image-text pair.
So the ground truth is a torch tensor like this :
torch.tensor([0,1,2,3,...,BATCH_SIZE-1]).Since the pre-trained CLIP use a massive batch size, just try to use the largest BATCH_SIZE as your system can take.
You can read more info about cross-entropy loss https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html, especially about the target. Also the CLIP paper, page 5, the upper left part.
@lonngxiang commented on GitHub (Apr 13, 2021):
tks for your reply；so If you have five pairs, so your BATCH_SIZE is five，is right？
@vinson2233 commented on GitHub (Apr 13, 2021):
Your BATCH_SIZE will determince the number of pairs for each batch
For example, If you have 1000 pairs, and set BATCH_SIZE = 20.
Then for each loop of
for batch in train_dataloader, the variablebatchwill give you 20 pairs. The loop will be repeated 50 times to cover all the data for 1 epoch.@lonngxiang commented on GitHub (Apr 13, 2021):
yes，but when I set BATCH_SIZE = 1，the total_loss is always 0，is this right？What's wrong with it
@vinson2233 commented on GitHub (Apr 13, 2021):
Yes, that's the problem. BATCH_SIZE must be greater than 1.
The reason is your prediction will return cosine similarity for that image and that text.
CrossEntropyLoss is combination of softmax with logloss.
Since one row only has 1 prediction(because BATCH_SIZE=1), the softmax will return probability=1 for that entry(It doesn't matter whether the logits is high or low), where it automatically correspond to the correct ground truth.
@lonngxiang commented on GitHub (Apr 13, 2021):
Thank you for helping me a lot and learning a lot
@dmoham1476 commented on GitHub (Apr 13, 2021):
@vinson2233 commented on GitHub (Apr 13, 2021):
@dmoham1476
model.transformer,model.visual.transformer. The text part only using transformers. While the visual part, also using transformers(it's the model.visual.transformer). Loading CLIP will allow you to train all the parts by default. You can freeze some components for example like this :This code will freeze all the visual parts.
I encourage you to see the components of CLIP
@uplusv commented on GitHub (Apr 25, 2021):
Hi, Vinson! Thank you for your code, it helps me a lot! but I met a problem when I fine-tune CLIP on my own data with your code. The task is to classify a 6-class problem so I set batch_size=6. After fine-tuning, the model outputs sample feature for every image, is it the problem of small batch size or fixed order of 6 classes or perhaps something else?
@vinson2233 commented on GitHub (Apr 25, 2021):
@uplusv If you want to modify CLIP as a classifier(Single label, multi class), here's some modification you can do :
ground_truth = torch.arange(BATCH_SIZE).to(device)to integer vector that specify which class your image are on (for exampletorch.tensor([0,1,2,1,2,3,4,5])). With this now you can set your batch size in arbitrary size.total_loss = (loss_img(logits_per_image,ground_truth) + loss_txt(logits_per_text,ground_truth))/2tototal_loss = loss_img(logits_per_image,ground_truth)I'm not sure what you meant by "After fine-tuning, the model outputs sample feature for every image"
@uplusv commented on GitHub (Apr 25, 2021):
Thank you for your reply and advice, I will try it soon! By "After fine-tuning, the model outputs sample feature for every image", I mean that, with "image_features = model.encode_image(image_input)" I print this "image_features" and get image_features: tensor([
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039],
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039],
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039],
...,
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039],
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039],
[ 0.0098, 0.0047, 0.0057, ..., 0.0018, 0.0056, -0.0039]])
while the original model outputs:
image_features: tensor([
[ 0.0304, -0.0169, -0.0383, ..., 0.0927, 0.0261, 0.0203],
[ 0.0013, -0.0067, -0.0524, ..., 0.1029, 0.0028, 0.0169],
[ 0.0115, -0.0006, -0.0392, ..., 0.0616, 0.0317, 0.0171],
...,
[ 0.0173, -0.0152, -0.0431, ..., 0.0836, 0.0405, 0.0268],
[ 0.0287, -0.0236, -0.0401, ..., 0.0856, 0.0119, 0.0287],
[ 0.0150, 0.0013, -0.0537, ..., 0.0792, 0.0104, 0.0062]])
After fine-tuning, the features become same and smaller so I get identical and large logits(like 99.8856) for every image😢.
@vinson2233 commented on GitHub (Apr 25, 2021):
Hmmm, I don't know what caused the model to produce the same value. Maybe something broke inside your data loader.
Whatever the cause is, I hope you can find your solution.
@Zasder3 commented on GitHub (May 23, 2021):
For those looking here in the future, I've made use of @vinson2233's code to create an easy-to-use PyTorch Lightning repo for training your own CLIP model from scratch.
@ChawDoe commented on GitHub (May 23, 2021):
Hey, I try to train it from scratch. But I found that the model is hard to train. The loss remains stable after some iterations. Do you meet the same problem?
@vinson2233 commented on GitHub (May 24, 2021):
@Zasder3 awesome, thanks for the effort 👍. It would be a blast if we can recreate every configuration from the paper since my code still lacks several features
@ChawDoe For me, the training went quite smoothly. I use batch size 512(with 4 GPU), 1million pairs data, and gradient accumulation for 8 steps. First several step give me loss around 2, at 20 epoch my average raining loss is 0.14
@ChawDoe commented on GitHub (May 24, 2021):
@vinson2233 Do you use fp16 training here? I think the problem may be due to my fp16 training and I set lr to 5e-5, which may lead to invalid gradients.
@vinson2233 commented on GitHub (May 24, 2021):
@ChawDoe I'm using fp16 for forward pass and gradient calculation(backward), using fp32 for parameter update(step), just like the code I'm posted. Using full fp32 give slow training time and lower batch size, while using full fp16 give NaN gradient because of underflow. The gradient might differ slightly between fp16 and fp32 but it shouldn't affect the training to the point of wrong step direction.
@lonngxiang commented on GitHub (May 25, 2021):
I use this code training to save PT or PKL files, but how do I load and reuse them later
@vinson2233 commented on GitHub (May 25, 2021):
@lonngxiang i have update the code for save and load, basically to load the model use this code :
Just modify the dict key to match your dict key when saving to .pt file
@lonngxiang commented on GitHub (May 25, 2021):
@vinson2233 commented on GitHub (May 25, 2021):
@lonngxiang actually, you don't need to copy the entire message to reply to a specific message, especially for long message. You can use the copy link function on top right of the message to produce a URL like this that directed to the specific message: this https://github.com/openai/CLIP/issues/83#issue-853114174. Just to make things shorter
For your question, yes k is epoch and loss is total_loss, I just copy-paste from my actual code and forgot to change the variable, will fix that right away.
@lonngxiang commented on GitHub (May 25, 2021):
@vinson2233 tks，https://github.com/openai/CLIP/issues/83#issue-853114174 ，is k equal EPOCH or epoch？i saw you write epoch now
@vinson2233 commented on GitHub (May 25, 2021):
@lonngxiang I save my model in every epoch, so I use the
epochvariable. Also, if the model training completed, then the epoch will equal to EPOCH-1.You also can change it
torch.save({'epoch': epoch+1,...}so the savings start from 1 and the final save will have epoch key equal toEPOCH.Note that this
epochkey will not affect the model behavior after loading, it only stores meta-data for the model.The same also goes for total_loss.
@lonngxiang commented on GitHub (May 25, 2021):
How about the effect of fine-tuning? It seems to affect the previous normal results, and the effect is not good
@vinson2233 commented on GitHub (May 25, 2021):
Some scenario for saving model :
checkpoint['optimizer_state_dict']@lonngxiang commented on GitHub (May 25, 2021):
how to load？like this？
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
@vinson2233 commented on GitHub (May 25, 2021):
@lonngxiang That should be correct. Note that I never try to load the optimizer since I never pause the training.
@lonngxiang commented on GitHub (May 25, 2021):
@vinson2233 commented on GitHub (May 25, 2021):
what is it that doesn't work? does it raise any error? I set jit=False when loading the model for the
clip.load@lonngxiang commented on GitHub (May 26, 2021):
@lonngxiang commented on GitHub (May 26, 2021):
There were no mistakes, but the results were bad
@wilderrodrigues commented on GitHub (May 31, 2021):
Hi @vinson2233 ,
First of all, thanks for the contribution.
I'm implementing a package using part of the code you published here. The add-ons I have are more related to creating the custom dataset, adding unit tests, dockerising the whole thing and also offering a service so other people can have quickly have inference running for their own + some plots.
However, when trying to train on my RTX 2080, I'm getting this:
This is very weird! The code is basically the same. I was using the latest PyTorch, but just now decided to downgrade it and use the same version as the CLIP repo. It still fails. I also tried this:
No effect at all.
And yes, the GPU is available. I can see it when I try this:
Thanks in advance!
@wilderrodrigues commented on GitHub (Jun 1, 2021):
Just another point:
I checked the parameters of both
clip.visual.transformerandclip.transformerblocks and all the tensors are already in the right device.I'm also trying to change the Transformer conv1, having it allocated to the CUDA device. But not helping yet.
Forward pass in the
model.pyof clip is not happy:Will keep digging.
@wilderrodrigues commented on GitHub (Jun 1, 2021):
I found the problem, finally! :) For example, this:
Should be like this:
Or, of course, if one prefers, one can have
.to(device)instead.@wilderrodrigues commented on GitHub (Jun 1, 2021):
Just another quick note: the way the
ground_truthis being taken as a range from theBATCH_SIZEis not right. If one has 10 classes and uses 2 batches, it will always give classes (ground_truth) of 0 and 1. So, it bias the training saying that all the images are 0s and 1s.I changed it and I'm now mapping my prompts (the sentences) to classes in a dictionary: prompt being the key and an integer being the class. It works as expected and with
BATCH_SIZE=1on my MacBook and my GPU. I got 3 novel pictures of airplanes - not in the original dataset - and ran the training for 2 epochs to get this as result:Before my changes were applied, the loss was always the same and the classifier was not working.
@DRSY commented on GitHub (Jun 4, 2021):
Have anyone tried fine-tune CLIP on MS COCO image-text retrieval task? How is the performance compared with other state-of-the-art models?
@lonngxiang commented on GitHub (Jun 4, 2021):
everything goes worse when I fine-tune the CLIP
@DRSY commented on GitHub (Jun 4, 2021):
From my experiment, the zero-shot image retrieval performance is R@1 25.4, R@5 48.7 and R@10 59.9 on the MS COCO
5k test set. After fine-tuning, it slightly improves to R@1 33.6, R@5 62.2 and R@10 73.8. Still lags behind SOTA non-transformer-based models(e.g., VSRN).
@lonngxiang commented on GitHub (Jun 4, 2021):
are you use this issues code to finetune？ https://github.com/openai/CLIP/issues/83；anythin are you changes ？
@vinson2233 commented on GitHub (Jun 4, 2021):
@wilderrodrigues I forgot to include to(device) in my code, thanks for the catch.
Also regarding the ground truth, this ground_truth is designed for image-title embedding to utilize the concept of n-pair-loss, not for image-title classification. I have mentioned this somewhere in this long threads about modifying the CLIP for classification task (https://github.com/openai/CLIP/issues/83#issuecomment-826262221)
@DRSY commented on GitHub (Jun 4, 2021):
I used the PyTorch lightning code, but modify the learning rate for ViT-B/32 from 5e-4 to 5e-5. If I use 5e-4 learning rate, after the first several steps, the model degraded to near-zero accuracy. But if I use 5e-5 learning rate, the model will steadily get improved to the results I posted.
@yangydeng commented on GitHub (Jun 8, 2021):
Hi, thanks for your sharing.
I was wondering what's the difference between your operation (like cast to fp16 and set back to fp32) and automatic-mix-precision(eg. torch.cuda.amp). Because in my code, I adopt "torch.cuda.amp" but it sometimes occurs NAN during training.
@vinson2233 commented on GitHub (Jun 8, 2021):
@yangydeng good question. Your question was discussed in the https://github.com/openai/CLIP/issues/57#issuecomment-794864178.
The Author manually specify what kind of operation done in FP16 and FP32, this can be seen from these part of code
github.com/openai/CLIP@cfcffb90e6/clip/model.py (L371-L392)That's why
torch.cuda.ampgive different result since amp convert everything(I think), while the author filter out which operation done in FP16@yangydeng commented on GitHub (Jun 8, 2021):
Thanks for your reply, I'll try that later.
And I found you adpot quite a large weight-decay (=0.2) as default. Have you try any other number?
@vinson2233 commented on GitHub (Jun 9, 2021):
@yangydeng That's the value that used by the author (see the paper Appendix F, table 18 on page 48). I haven't try weight-decay lower than that. But I have tried AdamW with weight decay=0.1 and it gives a worse result.
@yangydeng commented on GitHub (Jun 9, 2021):
Cool, many details can be found on this table.
@ThakurRajAnand commented on GitHub (Jun 16, 2021):
@vinson2233 Thanks for sharing the code. I am getting following error while using the code. Have you seen this during your training?
RuntimeError: Expected object of scalar type Long but got scalar type Half for argument #2 'target' in call to _thnn_nll_loss_forward@vinson2233 commented on GitHub (Jun 18, 2021):
@ThakurRajAnand Is the error raise on the
loss_img(logits_per_image,ground_truth)part? Are you working on CPU or GPU ? The error you are showing indicates that the target is Half(then your device is not "CPU"), but the logit are Long type, which indicate that your device is indeedCPU. It might be more helpful if you could print the data type at any given step of your training.@uahsan3 commented on GitHub (Jun 21, 2021):
Hi, I am facing the same issue. I verified that my device = 'cuda:0' - I am not sure where I am going wrong. Can you please point out why its giving the error:
@vinson2233 commented on GitHub (Jun 22, 2021):
@ThakurRajAnand @uahsan3 Thanks for the catch. The mistake is from my code. The ground truth should be always long regardless of the device. I've already edited the code. This is the new definition of the ground truth.
@ThakurRajAnand commented on GitHub (Jun 22, 2021):
@vinson2233 Thanks. I got busy and didn't get a chance to reply that I was able to fix it same day by making same change in your code.
I had another question. Can we use the model to predict text from the model I trained on my own data? I tried but it was asking for image and text both.
@vinson2233 commented on GitHub (Jun 22, 2021):
@ThakurRajAnand the forward method that defined on CLIP model is basically composition of
model.encode_text(text),model.encode_image(img)and cosine calculation between 2 embeddings. If you want to use just the text, then theencode_textis the method you are looking for.See this part of code for details
github.com/openai/CLIP@cfcffb90e6/clip/model.py (L354-L356)@ThakurRajAnand commented on GitHub (Jun 22, 2021):
@vinson2233 I think I didn't explain clearly. I am interested in predicting text by providing new image only. Is that possible? Text I trained on is long e.g.
this is a geometry question and we have circles in it@vinson2233 commented on GitHub (Jun 22, 2021):
@ThakurRajAnand
Ohh i see. So regarding text generation, there are 2 types of techniques. The first one is you already have a set of predefined text and try to fetch which text is the most suitable one(text-retrieval style). The second one is to generate new text from the image(generative). I think the CLIP cannot do the latter one, but you can use CLIP for the text-retrieval style.
I think what your are looking for is something like this https://github.com/pzzhang/VinVL or https://github.com/microsoft/Oscar
@lonngxiang commented on GitHub (Jun 30, 2021):
if I only want to save encode_text piece of the model, anyone knows how to do it?
@vinson2233 commented on GitHub (Jun 30, 2021):
@lonngxiang someone have post this question in this threads https://github.com/openai/CLIP/issues/113. I also want to know the answer.
@lonngxiang commented on GitHub (Jun 30, 2021):
tks
@lr19960813 commented on GitHub (Jul 2, 2021):
@vinson2233
Hey vinson, Thank you for sharing, it was a great help. But I encountered a problem during training.
Traceback(most recent call last):File "/CLIP-main/fine_tune.py", line 88, in
logits_per_image, logits_per_text = model(images, texts)
File "/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/CLIP-main/clip/model.py", line 357, in forward
image_features = self.encode_image(image)
File "/CLIP-main/clip/model.py", line 339, in encode_image
return self.visual(image.type(self.dtype))
File "/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/CLIP-main/clip/model.py", line 226, in forward
x = x + self.positional_embedding.to(x.dtype)
RuntimeError: The size of tensor a (1711) must match the size of tensor b (50) at non-singleton dimension 1`
I further tested and finally found that the error occurred in model.py line226：
`
I’m not sure what went wrong. I used pandas to load my csv data set, and some texts are very long. Maybe the reason is the text length is too long to be preprocessed but I am not sure.
@vinson2233 commented on GitHub (Jul 2, 2021):
@lr19960813 first, your error occured during the
image_features = self.encode_image(image)step, so the error caused by the image. Have you make sure to usepreprocessmodule to your images before feeding them to the network? you can put it inside the dataloader when fetching the images.second, regarding the long text, yes, Clip text only accept token length 77 (1 token represent 1-3 characracter). It's better to trim your text first.
@lr19960813 commented on GitHub (Jul 17, 2021):
Thank you for your reply! It is very helpful! I have finiished that experiment. By the way, how I can train clip model with my own dataset from scratch. It seems that this code justs fine tune clip model by using its default parameters.
@vinson2233 commented on GitHub (Jul 18, 2021):
@lr19960813 Then you need to load the model from scratch without loading the trained weight.
Here's the code, you can also set the context_length to a greater number if you have a really long text.
If you confused about how to fill the value (for example :
embed_dim), you can refer to existing model by inspecting thestate_dictof existing model.To see other params, you can refer to this code :
github.com/openai/CLIP@cfcffb90e6/clip/model.py (L395-L424)@gunwooYong commented on GitHub (Jul 22, 2021):
Hello, this is Yong, a researcher from South Korea.
I appreciate your work and it is very helpful for me as I have tried conducting fine-tuning task.
As I studied, CLIP includes a potential risk such as overfiting during fine-tuning with the small number of data.
Thus, I wonder about how much data is necessary at least when applying fine-tuning?
In my case, I have 4 classes and each class includes at most 10 pictures.
In such scenario, can I carry out fine-tuning task? or just use the traditional CLIP?
I am looking forward to your answer and I appreciate your work again.
Thank you.
@lr19960813 commented on GitHub (Jul 22, 2021):
Dear Yong
This is Li Rui a doctor from japan.
I think you are right. I meet the same problem with you.
I try another fine tune task by using 80000+ datas and it can not converge.
I am trying small learning rate and use bigger batch size.
It preforms better than before but still not very good.
I think you can try it.
Li Rui
gunwooYong @.***> 于2021年7月22日周四 下午1:54写道：
@gunwooYong commented on GitHub (Jul 22, 2021):
Dear Rui,
I really appreciate your help.
As described in CLIP-Art(https://github.com/KeremTurgutlu/clip_art), they used 140,000 images for fine-tuning.
So, I think you may solve your problem thorough data augmentation.
However, I cannot acquire more data as it is related with a specific domain.
For this reason, I am trying to conduct fine-tuning with the tiny number of images.
I hope you to overcome this issue.
Best regard,
@vinson2233 commented on GitHub (Jul 22, 2021):
@gunwooYong I might be not the person you are looking for since I'm not the author nor have any affiliation with openai. I think this question is better to discuss in new issue on github page so the real author have more chance to see the question.
But regardless, I did train CLIP as image classification model, where I have 19 categories with around 100k images, with modification where the loss only use the first logits(dim : number_image x num_class) since 1 image only have 1 category but 1 category can be owned by multiple images in the batch. The performance is still way behind a CNN Classification model.
@mitchellnw commented on GitHub (Jul 29, 2021):
In case it is helpful, posting a link to our implementation of CLIP-training code https://github.com/mlfoundations/open_clip
@vinson2233 commented on GitHub (Jul 30, 2021):
@mitchellnw Nice, thanks for sharing, I'll put it on https://github.com/openai/CLIP/issues/83#issue-853114174 so it is easier for people to reach your work.
@sarahESL commented on GitHub (Aug 10, 2021):
Thank for providing this @vinson2233 . I was wondering how come the loss does not include the cosine similarity of the encoded image and text? Following the "Figure 3" from the original clip paper. Or it is included and I am missing something?! 🤔
@vinson2233 commented on GitHub (Aug 10, 2021):
@sarahESL, Actually, the code already include that, but it is not explicit enough on showing that.
This line of code is producing the cosine similarity of the encoded image and text, not the embedding. I'm using the terms 'logits' just to follow the paper and keep it consistent with the example shown in the readme (Actually I didn't like the word logits).
So if we have 10 text and 5 images.
logits_per_imagedimension is 5 x 10, and thelogits_per_textis 10 x 5. Where each entries is cosine similarity times the temperature parameter (you can divide it by 100 or the temperature value itself to get the usual cosine similarity tough).@DRSY commented on GitHub (Aug 12, 2021):
Hi, just wanna share my work on deploying CLIP on iOS for cross-modal text-to-image retrieval/search: https://github.com/DRSY/MTIS. I exported the CLIP image encoder and text encoder as TorchScript format, which can then be loaded via torch's c++ libtorch frontend.
@NingYuanxiang commented on GitHub (Aug 16, 2021):
Hi, I want to revise the structure of clip's model. how to load some special layer and keep it constant in train on new dataset.If you reply me earlier, I will be greatly thankful for you.
@vinson2233 commented on GitHub (Aug 16, 2021):
I'm not sure what you mean by revise. But have you seen this link? https://discuss.pytorch.org/t/how-to-replace-a-layer-or-module-in-a-pretrained-network/60068
So basically you just access specific part of CLIP, and replace it.
To make it constant, I'm assuming what you mean is not update that specific layer during training. Maybe this forum will match your interest https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088. So basically just set require grad = False if the layers name match with your custom layer
@NingYuanxiang commented on GitHub (Aug 16, 2021):
Thank for providing this. I have some confuse, how to verify the revised net have improvement. Which training set is best to train and verify.
@vinson2233 commented on GitHub (Aug 18, 2021):
@NingYuanxiang If I in your position, I will create new image-title dataset, which have not been trained by the CLIP itself (So don't use COCO). This dataset should be totally new and never seen by the CLIP before.
After you have such dataset, try to finetune the original CLIP for the new dataset. Also finetune the revised CLIP for the new dataset, and then compare the performance.
Of course this approach is only based on empirical result, which mean you need to do extensive experiment for validating your modification of CLIP.
@4sunshine commented on GitHub (Aug 18, 2021):
Dear @vinson2233 am I correct that as least train split of MS-COCO (COCO Captions) was used during training of provided checkpoints in this repo?
@vinson2233 commented on GitHub (Aug 18, 2021):
@4sunshine When reading the paper, I'm assume they use every dataset that mentioned in paper in the checkpoint model in this repo. For example MS-COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), YFCC100M (Thomee et al., 2016).
I don't know if the COCO part they are using is only the train part, or maybe they just doing evaluation on COCO. We need to clarify it with the author (And again, I'm not the author).
@4sunshine commented on GitHub (Aug 18, 2021):
Thank you @vinson2233 for quick reply. I want to know the answer because of evaluation setup used in paper https://arxiv.org/abs/2012.04329. This paper is about Scene-Text aware Text-to-Image retrieval and its test samples partly belong to MS-COCO train.
@minmummax commented on GitHub (Aug 25, 2021):
is this demo code used for finetune on small dataset?
@liuhui0401 commented on GitHub (Aug 31, 2021):
I want to know what is the general loss of your code during training? My loss will drop from greater than 2 to greater than 1. Is this normal?
@vinson2233 commented on GitHub (Aug 31, 2021):
@minmummax this is the base code I'm using for private dataset which consist more than 1 million image text pairs. The actual code I'm using involves multi-GPU training
@vinson2233 commented on GitHub (Aug 31, 2021):
@liuhui0401 for my case, it starts with 3, but falls quickly to 1.5, and after 8000 steps (not epoch) the loss oscilating around 0.5
@lr19960813 commented on GitHub (Sep 7, 2021):
Hi,vinson, thank you for your code.
Do you try using a learning rate decay? I think it will get a better result.
@vinson2233 commented on GitHub (Sep 7, 2021):
@lr19960813 no, I haven't try it yet. The learning rate decay they use is this right ? https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html?highlight=cosine#torch.optim.lr_scheduler.CosineAnnealingWarmRestarts
Maybe I'll dive deeper on this.
@lr19960813 commented on GitHub (Sep 7, 2021):
@vinson2233
Yes, that's one type lr scheduler. In my experiment, I use roam scheduler to get a better result in a small test experiment. I will do more experiment to research the performence.
@sarahESL commented on GitHub (Sep 12, 2021):
Isn't it possible to use shorter sequence lengths by adjusting the
context_lengthintokenize? I tried it, but still in encoding the text, it givesRuntimeError: The size of tensor a (_X_) must match the size of tensor b (77) at non-singleton dimension 1!@lr19960813 commented on GitHub (Sep 12, 2021):
Yes，so I think you should trim your text shorter than 77tokens.
Sedigheh (Sarah) Eslami @.***> 于2021年9月12日周日 下午4:23写道：
@vinson2233 commented on GitHub (Sep 14, 2021):
@sarahESL yes you can, but you cannot use pre-trained CLIP if you want to use context_length different than 77. You might need to define CLIP by using this function over here
github.com/openai/CLIP@3b473b0e68/clip/model.py (L239-L240)with fresh weight.
After that, you can use tokenize with context_length longer than 77 and expect CLIP model to able to accept it
@jongwook commented on GitHub (Sep 16, 2021):
@HiGal The two losses are different because the softmax operations are done over different axes.
@nbl97 commented on GitHub (Sep 30, 2021):
@vinson2233 Thanks for your multi-GPU code. However, I don't understand why the similarity matrix is 100x10 in your example if we don't separate the CLIP model. Could you pls explain more? If I want to use DDP, is it necessary to separate the model? Thanks a lot!
@nbl97 commented on GitHub (Sep 30, 2021):
@vinson2233 Some supplement for the above question: I read the official code and found the the forward() function produces the cosine similarity, not the embedding. If I modify the forward() to produce the image and text embedding and calculate the similarity outside the forward(), is it still necessary to separate the CLIP model when multi-GPUs are to be used.
@vinson2233 commented on GitHub (Oct 1, 2021):
@nbl97 Let's understand what DataParallel do. Suppose we have 10 records to inference and we have 2 GPU. Then what data parallel will do is it will create 5 duplicate of model in each GPU, then spread the 10 records to each device. Maybe it will go to something like this :
GPU0 : record 0,1,2,3,4
GPU1 : record 5,6,7,8,9
After that on each device, the model will do inference and then concat all the output.
But for CLIP, this become a bit problematic. On a single model, if we give 10 pairs of text-images, the forward function will produce 10x10 similarity matrix.
But if you work on 2 GPU, then the first GPU will receive first 5 pairs and the second GPU will receive the last 5 pairs.
What happen in first GPU? It will calculate the cosine similarity between 5 pairs, and so does the second GPU.
The result would be 5x5 cosine similarity matrix on each GPU.
The last step is gathering the prediction from each GPU and we will end with 10x5 similarity matrix in this case.
Regarding DDP, i never use it so I cannot give answer on that 🙏
Regarding the modification, yes it should be possible theoretically. I have tried it in the past, but the parallelization only occur with
forwardmethod. Any other function will not get hooked with parallelization. That's why I come with the idea of splitting the model based on their responsibility.@xyxxmb commented on GitHub (Nov 2, 2021):
How did you get the image retrieval result in mscoco？Could you can share your test coding?
@milmin commented on GitHub (Nov 19, 2021):
Hi @vinson2233 thanks a lot for sharing your code! If I understand well you never set the model in training phase
model.train(). Do you do that on purpose to freeze dropout and batchnorm layers during fine-tuning? And second questions, do you have any ideas about good metrics to keep track of during evaluation to understand if fine-tuning is running fine (validation loss apart)?@Wuziyi616 commented on GitHub (Dec 5, 2021):
Hi @vinson2233 thank you so much for sharing the code. I'm just wondering what batch_size are you using to fine-tune CLIP? I'm using DDP training with 4 GPUS, and the batch_size is 128 on each GPU. However the loss isn't decreasing as expected... So I just wonder if that's because my batch_size if too small (so the fine-tuning is unstable) or there are bugs in my DDP code. Thanks in advance!
@akamil-etsy commented on GitHub (Dec 13, 2021):
is it possible to fine-tune pre-trained CLIP model using a loss function different from the original (infoNCE)?
I found an example on Kaggle when CLIP is trained from scratch with triplet loss with pair-wise comparisons (vs. class labels), and I'm wondering if this approach would work for fine-tuning the original pre-trained model from OpenAI.
I.e. freeze lower layers of the model pre-trained with infoNCE and train only a few higher layers with triplet loss on custom data (triplets of images and/or captions) or circle loss instead of the original contrastive loss/infoNCE.
@xyxxmb commented on GitHub (Dec 22, 2021):
I don't think that the classifier idea (1) is correct. Assuming having three categories and five images, and the ground_truth = [0,0,1,2,1]. The logits_per_image would be a matrix A with shape (5,5)，which A(i, j) denotes the similarity of image i and category j. When calculating the F.cross_entropy(logits_per_image, [0,0,1,2,1]), the nll_loss of 3th, 4th and 5th is not correct.
@BrianG13 commented on GitHub (Jan 24, 2022):
@xyxxmb
So, what is the best to do when I have in the same batch images with same category?
For example, batch_size=5, labels/text-features are = [0,0,1,2,1]
ground_truth = torch.arange(BATCH_SIZE).to(device)Is not the best solution also, because for image entries idx 0&1, I want them to be consistent with themselves (diagonal) and with the images of same class
@Unmesh28 commented on GitHub (Feb 19, 2022):
I want to use CLIP for text to art using VQGAN.
Planning to provide custom dataset to it [dataset : different painting styles of different artists ]
If we send a prompt + style = "A girl playing on the beach" + "Thomas Kinkade" , then it should return art of the prompt in style of Thomas Kinkade.
How do I train such a model on the top of existing one ?
@dongyun-kim-arch commented on GitHub (Feb 20, 2022):
Thank you for sharing your training code!
I am wondering how can I do fine-tuning with frozen all layers except for the top layer like linear probe?
@Zasder3 commented on GitHub (Apr 7, 2022):
As a late addendum to this, open_clip recently has some large success in replicating OpenAI's ViT-B/32 model achieving near-identical accuracy on the public LAION-400M dataset.
@Weifeng-Chen commented on GitHub (Apr 7, 2022):
【自动恢复】来信已收到，我将尽快回复你！
@AhmedEwis commented on GitHub (Apr 18, 2022):
Hello my friend
ValueError: Expected input batch_size (111) to match target batch_size (32)
any idea why I am getting this error?
def convert_models_to_fp32(model):
for p in model.parameters():
p.data = p.data.float()
p.grad.data = p.grad.data.float()
device = "cuda:0" if torch.cuda.is_available() else "cpu" # If using GPU then use mixed precision training.
model, preprocess = clip.load("ViT-B/32",device=device,jit=False) #Must set jit=False for training
if device == "cpu":
model.float()
else :
clip.model.convert_weights(model) # Actually this line is unnecessary since clip by default already on float16
loss_img = nn.CrossEntropyLoss()
loss_txt = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=5e-5,betas=(0.9,0.98),eps=1e-6,weight_decay=0.2) #Params used from paper, the lr is smaller, more safe for fine tuning to new dataset
for epoch in range(EPOCH):
for batch in train_dataloader :
optimizer.zero_grad()
Knowing the following:
np_list.shape --> (111, 512, 512, 3)
len(txt_list) --> 111
@BelhalK commented on GitHub (Apr 25, 2022):
Good morning,
Is this chunk of code used for fine-tuning or training?
If it's for fine-tuning, I guess you start from a pretrained ViT.
Hence, where do you introduce the new parameters to be learnt on your private dataset?
I don't understand what parameters are updated using your training script, could you help me figure it out?
Maybe
clip.load("ViT-B/32",device=device,jit=False)loads pretrained model weights and add under the hood some new parameters to fine-tune?`Thank you
@vinson2233 commented on GitHub (Apr 26, 2022):
@BelhalK No, I don't introduce new parameters to make it learn from the new dataset. I use the original architecture, and I don't freeze the weight at all. So all the weight inside the clip (both the image and text ViT) will be adjusted to the image and text from my dataset.
@BelhalK commented on GitHub (Apr 29, 2022):
Thanks for the reply!
So I guess both the text and image models have the same learnt weights since you seem to train a unique model, do you confirm? Thank you
@ElifCerenGokYildirim commented on GitHub (May 2, 2022):
Hi,
I am using exactly the same code to train CIFAR10 dataset. I have defined transform=preprocess while loading the dataset. I defined images=list_image and texts = clip.tokenize(list_txt).to(device). However, when I run the code I get this error: TypeError: len() of a 0-d tensor. My list_txt is equal to this = tensor([6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6, 4, 3, 6, 6,
2, 6, 3, 5, 4, 0, 0, 9]). So I did not understand what is the problem. Thanks in advance :)
@AhmedEwis commented on GitHub (May 8, 2022):
Hello all
I have been using the above code and I have already got my weight and biases model.pt for my custom dataset.
I am trying now to implement my custom model to the following repo:
https://github.com/rmokady/CLIP_prefix_caption
but I am getting the below error:
RuntimeError: Error(s) in loading state_dict for CLIP:
Missing key(s) in state_dict: "positional_embedding", "text_projection", "logit_scale", "visual.class_embedding", "visual.positional_embedding", "visual.proj", "visual.conv1.weight", "visual.ln_pre.weight", "visual.ln_pre.bias", "visual.transformer.resblocks.0.attn.in_proj_weight", "visual.transformer.resblocks.0.attn.in_proj_bias", "visual.transformer.resblocks.0.attn.out_proj.weight", "visual.transformer.resblocks.0.attn.out_proj.bias", "visual.transformer.resblocks.0.ln_1.weight", "visual.transformer.resblocks.0.ln_1.bias", "visual.transformer.resblocks.0.mlp.c_fc.weight", "visual.transformer.resblocks.0.mlp.c_fc.bias", "visual.transformer.resblocks.0.mlp.c_proj.weight", "visual.transformer.resblocks.0.mlp.c_proj.bias", "visual.transformer.resblocks.0.ln_2.weight", "visual.transformer.resblocks.0.ln_2.bias", "visual.transformer.resblocks.1.attn.in_proj_weight", "visual.transformer.resblocks.1.attn.in_proj_bias", "visual.transformer.resblocks.1.attn.out_proj.weight", "visual.transformer.resblocks.1.attn.out_proj.bias", "visual.transformer.resblocks.1.ln_1.weight", "visual.transformer.resblocks.1.ln_1.bias", "visual.transformer.resblocks.1.mlp.c_fc.weight", "visual.transformer.resblocks.1.mlp.c_fc.bias", "visual.transformer.resblocks.1.mlp.c_proj.weight", "visual.transformer.resblocks.1.mlp.c_proj.bias", "visual.transformer.resblocks.1.ln_2.weight", "visual.transformer.resblocks.1.ln_2.bias", "visual.transformer.resblocks.2.attn.in_proj_weight", "visual.transformer.resbl...
Unexpected key(s) in state_dict: "epoch", "model_state_dict", "optimizer_state_dict", "loss".
@AhmedEwis commented on GitHub (May 8, 2022):
Issue fixed by loading only the model state dictionary checkpoints instead of the:
model.load_state_dict(checkpoint['model_state_dict'])
@AhmedEwis commented on GitHub (May 8, 2022):
I am trying to implement a custom clip:
model = clip_model.eval()
device = CUDA(0) if is_gpu else "cpu"
model = clip_model.to(device)
use_beam_search = False
import pandas as pd
prefix_length=10
df = pd.DataFrame([],columns=['image','desc'])
for fname in [i for i in os.listdir('image') if i.endswith('.png')]:
image = io.imread('image/'+fname)
pil_image = PIL.Image.fromarray(image)
image = preprocess(pil_image).unsqueeze(0).to(device)
with torch.no_grad():
prefix = model.encode_image(image).to(device, dtype=torch.float32)
prefix_embed = model.clip_project(prefix).reshape(1, prefix_length, -1)
if use_beam_search:
generated_text_prefix = generate_beam(model, tokenizer, embed=prefix_embed)[0]
else:
generated_text_prefix = generate2(model, tokenizer, embed=prefix_embed)
df.loc[len(df)] = [fname,generated_text_prefix]
print(generated_text_prefix)
Getting error in the below code: #prefix_embed = model.clip_project(prefix).reshape(1, prefix_length, -1)
ModuleAttributeError: 'CLIP' object has no attribute 'clip_project'
@kxkaixin commented on GitHub (May 21, 2022):
Hello, thank you for your work. May I ask you some questions?
My code has several lines like this:
model, preprocess = clip.load("RN50", device=device)
for img, text, lab, id in train_loader:
With the torch. no_grad () :
Image_features = model. Encode_image (img)
Text_features = model. Encode_text (text)
def encode_image (self image) :
return self.visual(image.type(self.dtype)));
the self.visual () network processing the image does not enter the expected network. It jumps straight out. What causes it? I used PyCharm to debug.
I look forward to your reply. I hope you will forgive me for not knowing a lot in my first contact.
@vinson2233 commented on GitHub (May 23, 2022):
@kxkaixin
Do you mean you want the embedding result to become larger right? Since CLIP already trained and the transformer's last layer have dimension of 512, I think one of the simplest way to add the dimensionality is to add another Dense layer at the end. So instead of using forward from CLIP, just use encode_image, then pass to another dense with higher dimension. Do the same with encode_text also (Disclaimer : I never test this approach).
not sure, i never use debugger
I think you wanted to use the embedding from CLIP model for downstream tasks right? I believe the architecture will look like this : Data => CLIP => YourModel => Output.
You can try to optimize yourModel without modifying the CLIP. But if you must adjust the CLIP as well, then I suggest you create a new class, where the class has both CLIP and the model used for the downstream task, and then optimize the new class to optimize both the downstream model as well the CLIP model. Would you use one embedding or both embedding? If you only use one (example image only), then training the visual part of CLIP only will make the embedding between text and images misaligned, and if you wanted to make sure that the CLIP can be fine-tuned to create good image-text representation as well for your downstream task, then I suggest you to use the MultiTaskLearning approach to optimize 2 tasks at the same time.
@kxkaixin commented on GitHub (May 23, 2022):
https://github.com/openai/CLIP/issues/83#issuecomment-1134134811
Thank you very much for your reply. At the same time, thank you for your detailed explanation, which benefited me a lot.
In addition, I am very sorry to ask you a question.
For the first question, I don't mean that the value of [batch_size, emb_dim] obtained by model.encode _ image (img) changes from [100,512] to [100,1024], but whether more multidimensional information can be obtained. For example, can the CLIP model be used to obtain the type of data information such as [batch_size, C, H, W] for the image? (It doesn't have to be that way, I'm not sure about the form of data I can get, so I'm using this clunky example.)Because I couldn't jump to the expected location during debugging, I can only get data in the form of [batch_size, emb_dim] at present. That's why I asked you the second question.
@sarahESL commented on GitHub (May 30, 2022):
Question about the CLIP itself really: does anyone know why they assign random labels in each iteration? I mean why random? Is it mentioned in the original paper?
@vinson2233 commented on GitHub (May 30, 2022):
@kxkaixin do you mean that you want to know how the input changes trough the network? Like how the data changes from [BatchSize,R,G,B] => [BatchSize,XXX,YYY] => ... => [BatchSize,512]. I never try to look at them, but since this repo written in plain Pytorch, I think this stack overflow will be helpful https://stackoverflow.com/questions/42480111/model-summary-in-pytorch
@vinson2233 commented on GitHub (May 30, 2022):
@sarahESL No, it's not a random number. The cross_entropy_loss is accept a label in an integer-based position(not binary one-hot format). For example, let's say I wanted to create a fruit classification. I can mark apple as 0, banana as 1, and melon as 2. If I have [apple,apple,melon,banana], then the labels will become [0,0,2,1].
Now with CLIP, we provide a pair list of images and text. For example, if I have 10 pairs, it means I will have 10 images and 10 texts. The first image should only be matched with the first text, and the second image only to the second text until the 10th image is corresponding to the 10th text. Putting those data will create a logits matrix with the dimension of 10x10.
The way we look at it is, that for the first row, we have the cosine similarity to 10 other values/columns, but the correct value should be the first one (the 0th index).
The second row should have the highest similarity with the second column (the label is 1 for the 2nd column), until the last row which should be matched with the last col (index number: 9).
@kxkaixin commented on GitHub (May 30, 2022):
Ok, thank you for your reply. Much appreciated.
@smith-co commented on GitHub (Jun 29, 2022):
Do you plan to update the snippet to address the above todos?
@Weifeng-Chen commented on GitHub (Jun 29, 2022):
【自动恢复】来信已收到，我将尽快回复你！
@vinson2233 commented on GitHub (Jun 29, 2022):
@smith-co : Nope, I don't plan to at the moment.
@sanjaygunda13 commented on GitHub (Jul 12, 2022):
I want to custome train clip model my data is having captions and images data in b64. i tried training my data using coco but not able to do as i am getting some cuda error can someone help me out please.
@abdullah-jahangir commented on GitHub (Jul 18, 2022):
I am getting the following error when I run the code: AttributeError: 'image_title_dataset' object has no attribute 'list_txt', can you please help with this?
@vinson2233 commented on GitHub (Jul 18, 2022):
@abdullah-jahangir slight typo in my code, i fixed it. just change the len definition inside the class
@vinson2233 commented on GitHub (Jul 18, 2022):
@sanjaygunda13 I never tried the processing in Base64, maybe you need to try to decode the Byte64 into PIL format first.
@abdullah-jahangir commented on GitHub (Jul 18, 2022):
Thank you, however I am now getting a new error: File "train.py", line 32, in_getitem_
image = preprocess(Image.open(self. image_path[idx])) # Image from PIL module. I do not understand this as the number of images and texts are both equal.
@gongshaojie12 commented on GitHub (Jul 28, 2022):
Hi, thank you for your work. I have a question. How to fine-tune with clip in my own chinese dataset? Thanks!
@antu-saha commented on GitHub (Jul 29, 2022):
Hi, thank you very much for the great work.
I am trying to use your code for my data. Could you please give me the Dataset and DataLoader class? or Can you tell how they should look like or what will they do?
@sanjaygunda13 commented on GitHub (Aug 14, 2022):
Hello @vinson2233 can you help me out how to fine tune clip vitb32 model. I am struggling from long time to understand this. thanks
@vgthengane commented on GitHub (Aug 22, 2022):
How can I freeze the clip model weight? So that when I am using CLIP as a teacher model in knowledge distillation, CLIP model weights should not change.
@Asthestarsfalll commented on GitHub (Aug 22, 2022):
@vgthengane Maybe you can use
evalmethod like this:@vgthengane commented on GitHub (Aug 22, 2022):
Do I need to use
torch.no_grad()in that case?Also, I think model.eval() is already there when loading the clip model (
github.com/openai/CLIP@d50d76daa6/clip/clip.py (L129)).Anyway thanks
@pingapplepen commented on GitHub (Oct 10, 2022):
Hi, thanks for the work. I am now facint the issue that training loss doesn't drop. Must we use the loss function provided by you?For those who has difficulty on loss converging when using CLIP, change the learning rate to e-7/-8 may work
@antu-saha commented on GitHub (Oct 10, 2022):
Hi @vinson2233, I am fine-tuning CLIP with my dataset. But after 2 epochs, I am getting the total loss as nan. How can I deal with this issue?
Thank you for the help.
@wntg commented on GitHub (Oct 27, 2022):
Thank you for your work. How does it perform compared to only using image encoder？
@jindongwang commented on GitHub (Nov 1, 2022):
Why is the training loss always 0?
@cgusti commented on GitHub (Nov 30, 2022):
Hi! Thanks alot for this. Do you have a reference to all the import statements you used for this code?
@developer0hye commented on GitHub (Dec 2, 2022):
@vinson2233
Thanks!
I think that we should use AdamW instead of Adam.
@Techsetter commented on GitHub (Feb 15, 2023):
Does anyone have a fine-tuning colab notebook?
@shantanu778 commented on GitHub (Mar 2, 2023):
@vinson2233 It seems pre-training CLIP model instead of fine-tuning the model. Also where did you freeze the model parameters?
According to your script #83, model start training with new dataset which may delete all pretrained weights. Can you please ensure whether it's pre-training or fine-tuning?
@vinson2233 commented on GitHub (Mar 2, 2023):
@shantanu778 :
I don't freeze the parameter, i let all the weight get adjusted for the new dataset.
this is wrong. the model uploaded by the author is already pretrained. just look at the inner code and you can see the first step of the loading the model is importing the pretrained weight.
github.com/openai/CLIP@a9b1bf5920/clip/clip.py (L119-L120)by training the model on new dataset doesn't mean the old weight got wiped out.
@shantanu778 commented on GitHub (Mar 2, 2023):
@vinson2233 Thanks for the reply. I have another question.
Here, we have 1 image with 1 caption. But, if I have 1 image with 3 captions. Then how can I do this?
@chumpblocckami commented on GitHub (Mar 6, 2023):
I'm facing this issue after using the script for finetuning the model on a custom dataset.
Any help would be appreciated :)
@chumpblocckami commented on GitHub (Mar 6, 2023):
You probably want to merge the caption in a single string, such as (" ").join(captions)
@EachOneChew commented on GitHub (Mar 10, 2023):
Thank you! I've two questions about this:
logit_scaleback to a lower value and letting the model relearn it?@DorisBao commented on GitHub (Mar 27, 2023):
Hi, thanks for your code. However, I have a question of the groundtruth part. As your code shows that the groundtruth is
ground_truth = torch.arange(len(images), dtype=torch.long, device=device). What does it means? I think it should be the paired text of each image. So we need to select them to a tensor？ I am confused about this torch.long
@DorisBao commented on GitHub (Mar 28, 2023):
@vinid I mean train a CLIP not finetune.
@vinson2233 commented on GitHub (Mar 29, 2023):
@DorisBao see this https://github.com/openai/CLIP/issues/83#issuecomment-1141139017
@vinson2233 commented on GitHub (Mar 29, 2023):
Got lot of inquiry on how to train image-image, text-text, or even image to multiple text or other way around. The idea is still the same, but we need to make sure that in a batch of training, no contradicting pair can be found. see this diagram for image-image matching. Just make sure we don't put two pair that match in a batch.
The principle is still much the same for image to multiple text.
Letsay image A have text T1, T2 and T3
in batch 1 you wanted to train A with T1,
next batch A with T2
and next batch A with T3. But we don't want to train A with T1 T2 and T3 because it will create contradiction
@DorisBao commented on GitHub (Mar 29, 2023):
I have a confuse about total_loss = (loss_img(logits_per_image,ground_truth) + loss_txt(logits_per_text,ground_truth))/2. I have check the logits_per_image and logits_per_text, they have same value but in different shape. Can we just use one of them?
@vinson2233 commented on GitHub (Mar 30, 2023):
@DorisBao i think i wrote the reason somewhere in this long thread, but can't find it anymore. to answer your question, we need to understand how the PyTorch uses the CrossEntropyLoss loss during backpropagation.
CrossEntropyLoss is popularly used for multi-class classification. We can treat this problem as
given an image, which text is the most suitable oneand alsogiven a text, which image is the most suitable one. We classify it with the index position.for CLIP let's say i lay the image into the rows and the text as the column.
for each row, the image should only have 1 match to a label, that what is the CrossEntropyLoss try to measure.
You do it for all the row and optimize it, then it will solve the
given an image, which text is most suitable one. But it does not try to optimize for the other axis.So we need the transposed version to answer the
given a text, which image is the most suitable one.You can see this as MTL (Multi-Task Learning) because we want to optimize clip for two task, and one simple way to do it is to take average of the looses and backprop through it
@YunfanLing commented on GitHub (Mar 30, 2023):
list_txt = ['description for image1.jpg' , 'description for image2.jpg']
If I have 100 car images, should this be ["car"]*100?
@DorisBao commented on GitHub (Mar 31, 2023):
@vinson2233 Thanks!! So for instance, if I only used CLIP for classification, let's say we only have one text for the whole class( n images) which means we can't match every text to single image. And in this case, the text loss is unnecessary? right?
@DorisBao commented on GitHub (Mar 31, 2023):
The only reason we used both image and text loss is that we want the paired image-text structure to be unique.
@EachOneChew commented on GitHub (Mar 31, 2023):
@vinson2233 I'm wondering if you or anyone else have experimented with freezing certain CLIP layers? As far as I know freezing batch norm during fine tuning is standard because the training distribution is usually different, but I don't see how this applies to layer norm. I've also considered replacing the linear projector in the CLIP encoders with a MLP projector.
At the moment my training loss converges to a lower value during the first epoch, but the decrease in loss overall is tiny (6.8 average loss to 6.5 approximately).
@vinson2233 commented on GitHub (Mar 31, 2023):
@DorisBao correct.
@YunfanLing please refer to https://github.com/openai/CLIP/issues/83#issuecomment-1487820198. If your data only consist of 1 text. then you cannot train the data like this because it will create a contradiction during training.
@EachOneChew no, never do that.
@lamnt2008 commented on GitHub (Apr 5, 2023):
@vinson2233 : Hi, I'm a AI learner. I want to finetune CLIP for image classification. I have a custom dataset but i dont have text dataset for my problem. Can you tell me how to have a good method( create text dataset or finetune model or use CLIP) in order to have best result. Thanks
@EachOneChew commented on GitHub (Apr 5, 2023):
Thank you for the response. It seems like calling
train()oreval()on CLIP makes no difference as LayerNorm is kept on during both training and evaluation, unlike BatchNorm. However, I have experimented with tuning withViT-L/14and keeping the top half transformer layers frozen; the results are better than tuningViT-B/32andViT-B/16with gradients enabled on all layers. I think freezing layers can potentially be a good option for people who do not have enough GPU memory for larger batch sizes and also do not want to deal with the complexity of distributed InfoNCE loss.@EachOneChew commented on GitHub (Apr 5, 2023):
@lamnt2008
Your question is very broad. What is your specific use case? If you are doing image classification, I am assuming that you have in your training dataset images paired with labels? If that is the case, you have two options as I understand:
Use only CLIP's image encoder as a feature extractor and train a MLP classification head on top of it to perform classification. You may want to experiment with using lower learning rate for the CLIP image encoder and higher learning rate for the MLP head. Freezing the CLIP image encoder altogether and only training the MLP head is also a decent option for faster training on simpler classification tasks.
Treat your image-label pairs as positive examples and tune CLIP using the same contrastive approach* as outlined by @vinson2233 in his code. It is probably a good idea to build prompts around your class labels (e.g. banana -> "a photo of a banana").
*For a summary of the contrastive training approach see the CLIP blog post: https://openai.com/research/clip
As for which approach is better, the most recent paper I've read suggests that 2. comes out on top: https://arxiv.org/abs/2212.00638
@lamnt2008 commented on GitHub (Apr 5, 2023):
Thanks @EachOneChew , i see your answer. I understand method 2 from you. I'll try. But method 1, i want to understand deeply, so can you give me example code in order to implement it?
@StonERMax commented on GitHub (Apr 8, 2023):
Hi, l have used your multi-gpu code. But there is bug that the tensors are on different cudas. Some on cuda:0 and some on cuda:1, which causes mistakes.
@cucumberguagua commented on GitHub (Apr 11, 2023):
@vinson2233 If we only want to optimize for the task "given a text, which image is the most suitable one", then shall we only calculate loss as
loss_txt(logits_per_text,ground_truth)as in your training code?@EachOneChew commented on GitHub (Apr 11, 2023):
@cucumberguagua I would argue no; the two tasks are not conflicting. They are related and complimentary.
@EachOneChew commented on GitHub (Apr 11, 2023):
@lamnt2008 sorry, I have been very busy and don't have a quick example on hand. Can you take a look at "linear probing" using a feature extractor? CLIP's image encoder is used as an image extractor in OpenAI's own linear probe evaluations (also found here). They might have some examples, or there might be example code from peripheral research.
Functionally this just means, in the training loop:
I will find some time to give an example later, if possible.
@mananuppadhyay commented on GitHub (Apr 19, 2023):
@vinson2233 @EachOneChew
Hello, I tried using the training code and modified it for my dataset, where there's multiple phrases/captions for an image, but the training loop ends up giving a dimension mismatch error.
Using the same training loop, the error I got is:
permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 4 is not equal to len(dims) = 3
It also points me to encode_text, which probably means there's some text tokenization error.
Any help would be appreciated :)
@one-june commented on GitHub (Apr 23, 2023):
@mananuppadhyay Hi, I had the same issue. In my case it was because there was an extra dimension in my tokenized text, ie my tokenized text had dimensions [batch_size, 1, 77] when it should be [batch_size, 77]. I could fix it by just taking the 0th index of the tokenized text within the dataset. In your case, it would be changing the tokenizing line to
tokenized_caption = self.tokenizer(caption)[0]
Hope this helps
@mananuppadhyay commented on GitHub (Apr 24, 2023):
thanks a lot! this worked
@ghost commented on GitHub (May 7, 2023):
Hi,
I tried to fine-tune the CLIP model on FGVC-Aircraft dataset. After fine-tuning, the accuracy of CLIP on the test set was 3.21. Has anyone gotten similar results when trying to fine-tune CLIP on a small dataset?
@TEnsorTHiru commented on GitHub (May 26, 2023):
@chonky-moose
I tried the same but in final batch, the loss function raises this ValueError: Expected input batch_size (4) to match target batch_size (8).. Did you face a similar error and how did you solve it. My assumption is something is mixed up with the batch dimension since we remove one dimension from tokenized text.
@TEnsorTHiru commented on GitHub (May 27, 2023):
I'm trying in my own dataset, the validation loss doesn't reduce after the first epoch that is there is no learning happening inside the model and it stays wherever it is.
@abhaydoke09 commented on GitHub (Jun 13, 2023):
Thanks for the code snippet. During training, how can we make sure to create random batches of data so that model sees different contrastive pairs in every epoch?
@vinson2233 commented on GitHub (Jun 13, 2023):
@abhaydoke09
we're talking about :
if you're talking between 1 and 2. this is a tradeoff. let's say we train the clip model for clothes data. we can make it super easy for CLIP to train by making sure that in each batch we only fetch 1 jacket, 1 crop top, 1 pants, 1 underwear, 1 hat, and 1 glass. all of those items are very different. but if we keep doing so, the model cannot learn more complex concepts such as "v-neck t-shirt" vs "round neck" t-shirt. One solution is curriculum learning for this problem.
but for 3rd problem, then this is your task to create the data generator so there is no >1 pair with the same label present in a batch. you cannot expect the clip to do so for you.
@abhaydoke09 commented on GitHub (Jun 15, 2023):
I found a elegant solution by adding shuffle=True in the DataLoader solved my question.
train_dataloader = DataLoader(dataset, batch_size = BATCH_SIZE, shuffle=True)This makes sure that before every epoch, train data is shuffled so that pairs for contrastive learning are different in every epoch. This important because it ensures that the model does not learn from the order of training instances.@loseevaya commented on GitHub (Jun 18, 2023):
Hey！thanks for your code, now i want to use CLIP to do some test of prompt generation.
I use other model to generate a prompt (actually, just a decoder in Transformer) to replace the text_encoder in CLIP, and use CLIP image_encoder to get the image_features, i want to freeze the CLIP image_encoder and train the my own model. And the code is as follows, mostly copied from CLIP's github:
I use float32 but the similarity is always NaN, sometime i can train , but i find that the model is not convergencing :(
So, could you please check the code and point out my mistakes (For example, in norm or similarity)? Or get me some advice?
Thank you for your reply!!!!!!!!!!!!!!!!!!!!!
@georgefairs commented on GitHub (Jun 20, 2023):
Hi there and thanks @vinson2233 , I've tried using this implementation to fine-tune the base CLIP model with emotional faces (e.g angry, sad, etc) but the loss never seems to reduce. It may reduce slightly at the beginning but then over each iter. and epoch it just stagnates and has hardly reduced. I've tried using model.eval(), changing the LR and the weight_decay as others have suggested... to no avail! I only have three categories at the moment that I'm trialling fine-tuning with (e.g. angry, sad, neutral with associated prompts ("a photo of an angry face", "a photo of a sad face", etc.). Could the limited number of categories be affecting it?
Any further advice would be greatly appreciated.
@VincentZhou7 commented on GitHub (Jul 10, 2023):
Hi. I want to know if it is the code to finetune the original CLIP on our own dataset, or train the CLIP model from scratch?
@DuongTSon commented on GitHub (Aug 3, 2023):
@georgefairs In my case I have 4 classes and reducing the learning rate to 1e-6 has worked, the loss has been sharply reduced. If it does not work for you, I think you need more training data.
@DuongTSon commented on GitHub (Aug 3, 2023):
@VincentZhou7 this code is like fetching new batches of data into the exported checkpoint of CLIP, so it will update the weights inside the model. It's fair to say it's a finetuning code.
@Benjizhang commented on GitHub (Aug 6, 2023):
Hi, thanks for your helpful codes. In my project, I have a dataset including pairs of text and another modal signal (e.g., tactile information, represented as a one-dimension vector for each tactile signal.). So, if I still use the pipeline of CLIP to align the text embedding and tactile embedding, may I ask for any suggestions or modifications for training this dataset?
@xuntianci commented on GitHub (Aug 17, 2023):
For that 1. ,I use the CLIP's image encoder and then add a MLP head ，make all paramters learnable, but I got a nan loss in the batch 2. when a only train the head, it will be ok.Then I located nan , the crossentropyloss returned a nan value, can you give me some advice?I would appreciate it.
@xuntianci commented on GitHub (Aug 17, 2023):
hey!I have a similar question.Did you solve your problem?I use the CLIP's image encoder and then add a MLP head ，make all paramters learnable, but I got a nan loss in the batch 2. when a only train the head, it will be ok.Then I located nan , the crossentropyloss returned a nan value.
@xiyangyang99 commented on GitHub (Aug 23, 2023):
@vinson2233 >
Firstly, thank you for sharing the fine-tuning code. However, after completing the fine-tuning, in my own dataset, I only fine-tuned the image encoding part, while I did not fine-tune the text encoding part. I used vit base 16 as the pre training weight, but after fine-tuning, the. pt increased by 5 times. Also, how should the. pt model generated after fine-tuning be used for inference? Looking forward to your guidance, thank you.
@TaylorLi123 commented on GitHub (Sep 6, 2023):
Thanks for sharing the fine-tuning code, but when I called the save model to test after fine-tune, the scores for each category were average. how did this happen?Looking forward to your guidance, thank you.
tensor([0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250],
device='cuda:0', dtype=torch.float16)
@xiyangyang99 commented on GitHub (Sep 14, 2023):
@TaylorLi123
What is your dataset like? How many datasets are there? My dataset is very small, and I trained it to be the same as you. The final output value is the same.
@TaylorLi123 commented on GitHub (Sep 14, 2023):
@TaylorLi123 commented on GitHub (Sep 16, 2023):
@vinson2233
How do I output the evaluation indicators accuracy, precision, recall, and F1 during the training process? Does anyone have these indicators equal
@p1k0pan commented on GitHub (Oct 27, 2023):
I have checked the architechture with printing out the
state_dictfrom the model which loaded byclip.load(). I found there are no Dropout layer in the model, although it is a standard layer from a standard ViT and Transformer. Does it mean the model is not complete and not suffient to do fine-tunning?@IliasParas13 commented on GitHub (Nov 5, 2023):
Where can I find a dataset with texts?
@keshavsharma347 commented on GitHub (Dec 15, 2023):
Were you able to finetune for classification task?? If so can you provide some reference. I have my dataset as image, caption and class it belongs to..
@Dinosaurcubs commented on GitHub (Dec 26, 2023):
Really nice reply. I am trying to borrow the image encoder part of the CLIP and finetune the encoder only, because I plan to use it as a feature extraction part of my own model. For detail, I am trying to add some parameters in it and only train the params, but I donnot know how to split the visual encoder part from CLIP and modify it, can u provide some guide? tks
@deadpipe commented on GitHub (Dec 28, 2023):
Hi
Can you please explain the first point you mentioned? Actually i want to fine tune clip for multi labelled image classification where one image may belong to multiple classes
@ItzHaad commented on GitHub (Jan 7, 2024):
Hi, if you could clarify why is that we are using torch arrange, suppose that data is randomly shuffled after every epoch, we will have image pairs at different positions every time, so essentially we are not learning anything apart from position (which is also changing every time and that too randomly), instead this approach makes alot more sense https://github.com/moein-shariatnia/OpenAI-CLIP (with the projection head). Since, we are going to a common representation between image and text embeddings.
@vinson2233 commented on GitHub (Jan 7, 2024):
@ItzHaad I can give you 2 answers:
actual answer
you shuffle the pair of the data instead of shuffling the image and text independently. that's how you retain the labels.
let say the original data is (Image1, Text1), (Image2, Text2), (Image3, Text3)
it will make no difference if you shuffle the data so the order becomes like this (Image2, Text2), (Image3, Text3), (Image1, Text1)
lazy answer : that is what is presented in the paper (read Figure 3 of the paper)
@Heathcliff-saku commented on GitHub (Jan 17, 2024):
Hi！@vinson2233
Thank you very much for your training script. In my code, I have adopted your dataset method. However, my dataset is quite large (about 2 million img-text pairs), which has led to an unusual phenomenon during training: at the start of each epoch, specifically during the loading of the first batch, there is a prolonged delay (approximately 30 to 40 minutes), and the GPU utilization remains at 0%. Have you encountered this issue before? Is this normal? Do you have any recommended solutions to mitigate this?
PS. I have set an appropriate number of workers and enabled pin memory, but this waiting time still seems unavoidable.
@vinson2233 commented on GitHub (Jan 17, 2024):
@Heathcliff-saku its expected because how the dataset object created. if you don't want huge overhead in the front, another way is to do the image preprocess and clip tokenize after the data produced by data loader, but this will create redundancy in every epoch.
If anyone can give recommendations as well then feel free to do so since i don't use CLIP anymore
@manas6266 commented on GitHub (Feb 3, 2024):
How many epochs does clip need to be finetuned and what should be batch size??
@anas2908 commented on GitHub (Mar 3, 2024):
ell In[13], line 64, in image_title_dataset.getitem(self, idx)
63 def getitem(self, idx):
---> 64 image_path = self.image_path[idx] # Get the image path at the specified index
65 image = preprocess(Image.open(image_path)) # Open the image using PIL
66 title = self.title[idx] # Get the title corresponding to the image
TypeError: list indices must be integers or slices, not list
is there any problem in my dataloader i am using from torch.utils.data import DataLoader
i am stucked can you please help me?
@anas2908 commented on GitHub (Mar 3, 2024):
hey can you share me your code
@BaochaoZhu commented on GitHub (Mar 11, 2024):
@vinson2233 ,Hi, I would like to train CLIP with my own custom datasets. Can you please advise me on how many images I need to prepare per class? Thank you.
@lalit-pivotchain commented on GitHub (Mar 15, 2024):
i implemented the above code to load the saved model(.pt) , but i encountered this error
AttributeError: 'CLIP' object has no attribute 'input_resolution'
@Shadedog838 commented on GitHub (Apr 21, 2024):
I have implemented the training code above and have been trying to train the model on the Flickr dataset but my loss keeps increasing until it eventually stays the the same I don't know what the problem is can anyone provide some insight?
@seidasoeun commented on GitHub (Apr 27, 2024):
you just fix your code like below
from torch.utils.data import Dataset@nightrain-vampire commented on GitHub (May 10, 2024):
Hello，I set the Batch Size as 64，and my total loss is always 4.02734375 during the training；I don't know what the problem is can anyone provide some insight?
@pengzhao-life commented on GitHub (Oct 3, 2024):
@vinson2233
Thanks for sharing the code! I am trying to finetune clip model on my own dataset. My data has multiple texts per image. I wonder what the typical solution is to handle multiple texts per image. My understanding is that I need to create each ground truth label to ensure the positive relationship between the image and its every single text. Otherwise, they may be considered as negative pair during the training. Is this correct? Any sample code or post related to this concern?
My second question is that my texts may include image captions, keywords, categories, etc. Should I handle them differently? Many thanks!
@Himanshunitrr commented on GitHub (Dec 30, 2024):
In the paper they multiply the logits with a exponential of temperature too which I dont see here ?
@tengshaofeng commented on GitHub (Feb 10, 2025):
@vinson2233 @jongwook @ThakurRajAnand @nightrain-vampire @pengzhao-life Dears, thanks so much for your useful talking. I have a new question.Because I have only one 4090 card, so the max batch_size for me is 5. It is so small,dose it affect the traing performence?
@Fadela commented on GitHub (Mar 10, 2025):
hi how about training using LoRA? any resources?
@t00350320 commented on GitHub (Mar 24, 2025):
what "ground_truth = torch.arange(len(images),dtype=torch.long,device=device)" make sense?
"ground_truth" seems a meaningless tensor? have no ide ...
@Himanshunitrr commented on GitHub (Mar 24, 2025):
It’s contrastive loss, so every item is GT for itself (we want to pull them together)
@t00350320 commented on GitHub (Mar 24, 2025):
i tested when batch_size=4,then the GT
batch_size=32,then the GT
So "loss_img(logits_per_image,ground_truth) " do what kind of loss with stable ground_truth value? What message should I learn if I want to know the role of "ground_truth" in CrossEntryLoss?
and I find the loss value always print a stable value after many epoch times,"train Loss: 0.1032".
Does this training process make sense for CLIP's parameters training ? Any one have a better practice? Thanks a lot
@tengshaofeng commented on GitHub (Mar 25, 2025):
@jongwook @ThakurRajAnand @cucumberguagua @uahsan3 @Zasder3 dear all, I have written a full training code for finetuning jina-clip-v2. It can surport 8192 tokens for text. Everything is ok.Code is here @ https://github.com/tengshaofeng/finetune-jina-clip-v2.
@t00350320 commented on GitHub (Mar 26, 2025):
for "ground_truth value" ,
we start from logits_per_image. It produced by
then from openai CLIP code we know logits_per_image = image_features(batch_size,512) @text_features.t(512,batch_size)=[batch_size,batch_size]
2. loss_img
as loss_img is nn.CrossEntropyLoss(), it actually work like this
We asume that image.3 and text.3 own a high similarity 27,then it get a proper loss 0 as expectedd. In a word, ground_truth value helps us pay attention to correponded image and text's loss.
@peizhouh commented on GitHub (Sep 16, 2025):
Hi @vinson2233,
Thank you so much for your useful instruction. I have a question about loss calculation part, yet your post exactly follows the CLIP paper did, but what about the loss calculation betow, do you know how much differences will appear in the training? In the loss calculation below, suppose we have image_features and text_features from the model