[GH-ISSUE #57] Best practice for mix precision training #51

New issue

Closed

opened 2026-10-31 21:10:15 +00:00 by bsakaguchi · 17 comments

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Owner

Copy link

Originally created by @qingerVT on GitHub (Mar 10, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/57

I am trying to fine-tune CLIP models on new datasets. What's the best practice for mix precision training?

Using Adam, I got errors either nan or inf since attribute eps is hard to specify for Half and float32. My walk around is to divide parameters into two groups and specify different eps. Any better solutions?

Originally created by @qingerVT on GitHub (Mar 10, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/57 I am trying to fine-tune CLIP models on new datasets. What's the best practice for mix precision training? Using Adam, I got errors either nan or inf since attribute eps is hard to specify for Half and float32. My walk around is to divide parameters into two groups and specify different eps. Any better solutions?

bsakaguchi closed this issue 2026-10-31 21:10:15 +00:00

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@KeremTurgutlu commented on GitHub (Mar 10, 2021):

You can use fastai, which already provides to_fp16() under the hood it uses torch autocast. So it's just as simple as adding that single line to your code. There might be other repos like pytorch lightning, apex, etc.. which allow to do mixed precision training with similar ease.

Also, I've already implemented training code in fastai, you can also take a look at it. You just need to add lines for loading the open source CLIP weights for finetuning, my code is currently for training from scratch but it should be just a couple of lines of change. Code is here, an example sample script.

<!-- gh-comment-id:794730578 --> @KeremTurgutlu commented on GitHub (Mar 10, 2021): You can use fastai, which already provides `to_fp16()` under the hood it uses torch autocast. So it's just as simple as adding that single line to your code. There might be other repos like pytorch lightning, apex, etc.. which allow to do mixed precision training with similar ease. Also, I've already implemented training code in fastai, you can also take a look at it. You just need to add lines for loading the open source CLIP weights for finetuning, my code is currently for training from scratch but it should be just a couple of lines of change. Code is [here](https://github.com/KeremTurgutlu/self_supervised/blob/fastai_update/self_supervised/multimodal/clip.py), an example sample [script](https://github.com/KeremTurgutlu/self_supervised/blob/fastai_update/examples/training_clip.py).

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@qingerVT commented on GitHub (Mar 10, 2021):

Thanks @KeremTurgutlu. The basic idea is to convert parameters into half and then use torch autocast?

<!-- gh-comment-id:794746251 --> @qingerVT commented on GitHub (Mar 10, 2021): Thanks @KeremTurgutlu. The basic idea is to convert parameters into half and then use torch autocast?

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@KeremTurgutlu commented on GitHub (Mar 10, 2021):

Thanks @KeremTurgutlu. The basic idea is to convert parameters into half and then use torch autocast?

You can read materials from https://pytorch.org/docs/stable/amp.html.

<!-- gh-comment-id:794749763 --> @KeremTurgutlu commented on GitHub (Mar 10, 2021): > Thanks @KeremTurgutlu. The basic idea is to convert parameters into half and then use torch autocast? You can read materials from https://pytorch.org/docs/stable/amp.html.

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@jongwook commented on GitHub (Mar 10, 2021):

We did the memory-intensive tensor operations (matmul, convolutions) in fp16, while doing aggregation (batchnorms/layernorms) in fp32. We also kept the model weights and gradients in fp32 (which only takes constant memory w.r.t. the batch size) so that Adam only sees fp32 numbers.

For a general introduction to mixed-precision training, I found this video tutorial and document by NVIDIA and particularly useful.

<!-- gh-comment-id:794864178 --> @jongwook commented on GitHub (Mar 10, 2021): We did the memory-intensive tensor operations (matmul, convolutions) in fp16, while doing aggregation (batchnorms/layernorms) in fp32. We also kept the model weights and gradients in fp32 (which only takes constant memory w.r.t. the batch size) so that Adam only sees fp32 numbers. For a general introduction to mixed-precision training, I found [this video tutorial](https://developer.nvidia.com/blog/video-mixed-precision-techniques-tensor-cores-deep-learning/) and [document](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html) by NVIDIA and particularly useful.

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@qingerVT commented on GitHub (Mar 10, 2021):

@jongwook Thanks. fp32 for optimization and fp16 for forward/backward. Switching back and forth works!

<!-- gh-comment-id:795773404 --> @qingerVT commented on GitHub (Mar 10, 2021): @jongwook Thanks. fp32 for optimization and fp16 for forward/backward. Switching back and forth works!

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@WujiangXu commented on GitHub (Mar 11, 2021):

@jongwook Thanks. fp32 for optimization and fp16 for forward/backward. Switching back and forth works!

hello, I met the same problem with you.
Can you show the specific solution about this problem. I try the code as below but it not work for me.
with amp.autocast(): out = net(photo1,token)

<!-- gh-comment-id:796569816 --> @WujiangXu commented on GitHub (Mar 11, 2021): > @jongwook Thanks. fp32 for optimization and fp16 for forward/backward. Switching back and forth works! hello, I met the same problem with you. Can you show the specific solution about this problem. I try the code as below but it not work for me. `with amp.autocast(): out = net(photo1,token)`

bsakaguchi commented 2026-10-31 21:10:15 +00:00

Author

Owner

Copy link

@qingerVT commented on GitHub (Mar 11, 2021):

@cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

def convert_models_to_mix(model):
    clip.model.convert_weights(model)

---- your train function ------

convert_models_to_fp32(model) 
optimizer.step()
convert_models_to_mix(model)

<!-- gh-comment-id:796915161 --> @qingerVT commented on GitHub (Mar 11, 2021): @cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back ``` def convert_models_to_fp32(model): for p in model.parameters(): p.data = p.data.float() p.grad.data = p.grad.data.float() ``` ``` def convert_models_to_mix(model): clip.model.convert_weights(model) ``` ---- your train function ------ ``` convert_models_to_fp32(model) optimizer.step() convert_models_to_mix(model) ```

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@WujiangXu commented on GitHub (Mar 12, 2021):

@cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

def convert_models_to_mix(model):
    clip.model.convert_weights(model)

---- your train function ------

convert_models_to_fp32(model) 
optimizer.step()
convert_models_to_mix(model)

Thank you for your solution. But there is still a question. Which object is the "clip" represent? I don't understand this function and find no api in the original code.
def convert_models_to_mix(model): clip.model.convert_weights(model)

<!-- gh-comment-id:797209305 --> @WujiangXu commented on GitHub (Mar 12, 2021): > @cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back > > ``` > def convert_models_to_fp32(model): > for p in model.parameters(): > p.data = p.data.float() > p.grad.data = p.grad.data.float() > ``` > > ``` > def convert_models_to_mix(model): > clip.model.convert_weights(model) > ``` > > ---- your train function ------ > > ``` > convert_models_to_fp32(model) > optimizer.step() > convert_models_to_mix(model) > ``` Thank you for your solution. But there is still a question. Which object is the "clip" represent? I don't understand this function and find no api in the original code. `def convert_models_to_mix(model): clip.model.convert_weights(model)`

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@nishanthcgit commented on GitHub (Mar 12, 2021):

@cupcakefromchina here

<!-- gh-comment-id:797270669 --> @nishanthcgit commented on GitHub (Mar 12, 2021): @cupcakefromchina [here](https://github.com/openai/CLIP/blob/beba48f35392a73c6c47ae67ddffced81ad1916d/clip/model.py#L370)

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@nishanthcgit commented on GitHub (Mar 12, 2021):

@qingerVT I was trying out training on my own dataset too and was facing the exact same issue. Thanks for pointing out the stuff about mixed precision training, will definitely try it out, thanks!
I'm curious though, are you able to see the loss decreasing in that case? For me it always ends up blowing up, maybe small batch size is a problem, I have only a single GPU machine.
@KeremTurgutlu Could you weigh in too if training is working out? Also, I've taken a look at your code and it would be really useful to me. Could you maybe point out how I can go about adding the open souce CLIP weights for finetuning? Still quite new to pytorch.

<!-- gh-comment-id:797273203 --> @nishanthcgit commented on GitHub (Mar 12, 2021): @qingerVT I was trying out training on my own dataset too and was facing the exact same issue. Thanks for pointing out the stuff about mixed precision training, will definitely try it out, thanks! I'm curious though, are you able to see the loss decreasing in that case? For me it always ends up blowing up, maybe small batch size is a problem, I have only a single GPU machine. @KeremTurgutlu Could you weigh in too if training is working out? Also, I've taken a look at your code and it would be really useful to me. Could you maybe point out how I can go about adding the open souce CLIP weights for finetuning? Still quite new to pytorch.

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@KeremTurgutlu commented on GitHub (Mar 12, 2021):

@KeremTurgutlu Could you weigh in too if training is working out? Also, I've taken a look at your code and it would be really useful to me. Could you maybe point out how I can go about adding the open souce CLIP weights for finetuning? Still quite new to pytorch.

Yes, I have successfully trained CLIP on my own data and performance is close to open source model with just 5.5 million image-text pairs. I am working on wrapping up the code for both training from scratch and fine-tuning. In the meantime you can watch this branch, should be complete in 1-2 days.

<!-- gh-comment-id:797275581 --> @KeremTurgutlu commented on GitHub (Mar 12, 2021): > @KeremTurgutlu Could you weigh in too if training is working out? Also, I've taken a look at your code and it would be really useful to me. Could you maybe point out how I can go about adding the open souce CLIP weights for finetuning? Still quite new to pytorch. Yes, I have successfully trained CLIP on my own data and performance is close to open source model with just 5.5 million image-text pairs. I am working on wrapping up the code for both training from scratch and fine-tuning. In the meantime you can watch this [branch](https://github.com/KeremTurgutlu/self_supervised/tree/fastai_update), should be complete in 1-2 days.

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@qingerVT commented on GitHub (Mar 12, 2021):

@cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

def convert_models_to_mix(model):
    clip.model.convert_weights(model)

---- your train function ------

convert_models_to_fp32(model) 
optimizer.step()
convert_models_to_mix(model)

Thank you for your solution. But there is still a question. Which object is the "clip" represent? I don't understand this function and find no api in the original code.
def convert_models_to_mix(model): clip.model.convert_weights(model)

yes, loss is decreasing. Function convert_weights() is in clip/model.py. Try
export PYTHONPATH=/your/path/CLIP:$PYTHONPATH

<!-- gh-comment-id:797523067 --> @qingerVT commented on GitHub (Mar 12, 2021): > > @cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back > > ``` > > def convert_models_to_fp32(model): > > for p in model.parameters(): > > p.data = p.data.float() > > p.grad.data = p.grad.data.float() > > ``` > > > > > > ``` > > def convert_models_to_mix(model): > > clip.model.convert_weights(model) > > ``` > > > > > > ---- your train function ------ > > ``` > > convert_models_to_fp32(model) > > optimizer.step() > > convert_models_to_mix(model) > > ``` > > Thank you for your solution. But there is still a question. Which object is the "clip" represent? I don't understand this function and find no api in the original code. > `def convert_models_to_mix(model): clip.model.convert_weights(model)` yes, loss is decreasing. Function convert_weights() is in clip/model.py. Try `export PYTHONPATH=/your/path/CLIP:$PYTHONPATH`

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@vinson2233 commented on GitHub (Mar 29, 2021):

can I know what's the differences between

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

vs
model.float() ?

<!-- gh-comment-id:809311374 --> @vinson2233 commented on GitHub (Mar 29, 2021): can I know what's the differences between ``` def convert_models_to_fp32(model): for p in model.parameters(): p.data = p.data.float() p.grad.data = p.grad.data.float() ``` vs `model.float()` ?

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@qingerVT commented on GitHub (Mar 31, 2021):

can I know what's the differences between

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

vs
model.float() ?

Ah, no difference :)

<!-- gh-comment-id:811127267 --> @qingerVT commented on GitHub (Mar 31, 2021): > can I know what's the differences between > > ``` > def convert_models_to_fp32(model): > for p in model.parameters(): > p.data = p.data.float() > p.grad.data = p.grad.data.float() > ``` > > vs > `model.float()` ? Ah, no difference :)

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@nbl97 commented on GitHub (Oct 1, 2021):

@jongwook. @qingerVT . Hi. I have a simple question. I understand the solution below about the mix precision training. I know the converting to fp16 is to save cuda memory, with negligible loss of Acc. However, what are the advantages of the below referenced method over the usual methods, such as Apex and torch.cuda.amp work well?

@cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back

def convert_models_to_fp32(model): 
    for p in model.parameters(): 
        p.data = p.data.float() 
        p.grad.data = p.grad.data.float() 

def convert_models_to_mix(model):
    clip.model.convert_weights(model)

---- your train function ------

convert_models_to_fp32(model) 
optimizer.step()
convert_models_to_mix(model)

<!-- gh-comment-id:931986682 --> @nbl97 commented on GitHub (Oct 1, 2021): @jongwook. @qingerVT . Hi. I have a simple question. I understand the solution below about the mix precision training. I know the converting to fp16 is to save cuda memory, with negligible loss of Acc. However, what are the advantages of the below referenced method over the usual methods, such as Apex and torch.cuda.amp work well? > @cupcakefromchina Convert parameters and grads to fp 32 before applying Adam, then convert it back > > ``` > def convert_models_to_fp32(model): > for p in model.parameters(): > p.data = p.data.float() > p.grad.data = p.grad.data.float() > ``` > > ``` > def convert_models_to_mix(model): > clip.model.convert_weights(model) > ``` > > ---- your train function ------ > > ``` > convert_models_to_fp32(model) > optimizer.step() > convert_models_to_mix(model) > ```

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@jongwook commented on GitHub (Oct 1, 2021):

@nbl97 Not much, just that we wanted to have a granular control over which operation is in which dtype. Now that torch AMP is mature, I'd use it if I were to start a project from scratch.

<!-- gh-comment-id:932026350 --> @jongwook commented on GitHub (Oct 1, 2021): @nbl97 Not much, just that we wanted to have a granular control over which operation is in which dtype. Now that torch AMP is mature, I'd use it if I were to start a project from scratch.

bsakaguchi commented 2026-10-31 21:10:16 +00:00

Author

Owner

Copy link

@nbl97 commented on GitHub (Oct 3, 2021):

@nbl97 Not much, just that we wanted to have a granular control over which operation is in which dtype. Not that torch AMP is mature, I'd use it if I were to start a project from scratch.

@jongwook As we all know, converting fp32 to fp16 may drop Acc, however, what is the performance if we convert fp16 to fp32 ? When useing CLIP pretrained model on my own task, what is the performance if I always use fp32 and never convert the model? Do you have experience about that? Thanks in advance.

<!-- gh-comment-id:933054054 --> @nbl97 commented on GitHub (Oct 3, 2021): > @nbl97 Not much, just that we wanted to have a granular control over which operation is in which dtype. Not that torch AMP is mature, I'd use it if I were to start a project from scratch. @jongwook As we all know, converting fp32 to fp16 may drop Acc, however, what is the performance if we convert fp16 to fp32 ? When useing CLIP pretrained model on my own task, what is the performance if I always use fp32 and never convert the model? Do you have experience about that? Thanks in advance.

bsakaguchi referenced this issue 2026-10-31 21:10:52 +00:00

[GH-ISSUE #85] Linear projection & normalization after encoder #76

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

codex/pin-github-workflow-refs-20260325-132838

jongwook/issue-396

jongwook-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#51

No description provided.