[GH-ISSUE #63] NAN in downstream detection finetuneing task

bsakaguchi commented

2026-10-31 21:10:20 +00:00

Owner

Originally created by @launchauto on GitHub (Mar 22, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/63

I try to use you RN50.pt as a backbone in Faster-RCNN-coco-detection task in mmdetection arichtecture.
Firstly, I try to modified the layer name of RN50.pt and resaved them so that the parameters can be laoded correctly.
Then, I modified ResNetV1d to match your ModifiedResNet in clip model.
During the fineturning procedure, I do not use your attention layer and just use the output of reslayer4,reslayer3,reslayer2 reslayer1 to FPN.

Originally created by @launchauto on GitHub (Mar 22, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/63 I try to use you RN50.pt as a backbone in Faster-RCNN-coco-detection task in mmdetection arichtecture. Firstly, I try to modified the layer name of RN50.pt and resaved them so that the parameters can be laoded correctly. Then, I modified ResNetV1d to match your ModifiedResNet in clip model. During the fineturning procedure, I do not use your attention layer and just use the output of reslayer4,reslayer3,reslayer2 reslayer1 to FPN.

bsakaguchi closed this issue

2026-10-31 21:10:20 +00:00

bsakaguchi commented

2026-10-31 21:10:20 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 22, 2021):

During the training procedure the loss will be NAN right from the start

@launchauto commented on GitHub (Mar 22, 2021): During the training procedure the loss will be NAN right from the start

bsakaguchi commented

2026-10-31 21:10:20 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 22, 2021):

Did you try #40 ?

@jongwook commented on GitHub (Mar 22, 2021): Did you try #40 ?

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 22, 2021):

Did you try #40 ?

When I load your pretrained model and resave it I use
clip_r50_model=torch.jit.load('/data1/pretrained/model/clip/RN50.pt',map_location='cpu').eval()
Then I modified some layer names and resaved it.
After build model, I add model=model.float()
It still NAN right from the start.
Does it work?

@launchauto commented on GitHub (Mar 22, 2021): > Did you try #40 ? When I load your pretrained model and resave it I use clip_r50_model=torch.jit.load('/data1/pretrained/model/clip/RN50.pt',map_location='cpu').eval() Then I modified some layer names and resaved it. After build model, I add model=model.float() It still NAN right from the start. Does it work?

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 22, 2021):

The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling .float(); this was why I had to patch those variables in some hacky way to make it runnable on CPU.

Use clip.load() with jit=False to load the model in a non-JIT mode, and try again.

@jongwook commented on GitHub (Mar 22, 2021): The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling `.float()`; this was why I had to patch those variables in [some hacky way](https://github.com/openai/CLIP/blob/main/clip/clip.py#L137-L157) to make it runnable on CPU. Use `clip.load()` with `jit=False` to load the model in a non-JIT mode, and try again.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 23, 2021):

The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling .float(); this was why I had to patch those variables in some hacky way to make it runnable on CPU.

Use clip.load() with jit=False to load the model in a non-JIT mode, and try again.

Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and warmup step =1000 batchsize=16(ImageNet supervision is 0.02 for batchsize=16). Hopefully this time I can get reasonable result.

@launchauto commented on GitHub (Mar 23, 2021): > The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling `.float()`; this was why I had to patch those variables in [some hacky way](https://github.com/openai/CLIP/blob/main/clip/clip.py#L137-L157) to make it runnable on CPU. > > Use `clip.load()` with `jit=False` to load the model in a non-JIT mode, and try again. Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and warmup step =1000 batchsize=16(ImageNet supervision is 0.02 for batchsize=16). Hopefully this time I can get reasonable result.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 23, 2021):

The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling .float(); this was why I had to patch those variables in some hacky way to make it runnable on CPU.
Use clip.load() with jit=False to load the model in a non-JIT mode, and try again.

Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and batchsize=16(ImageNet supervision is 0.02 for batchsize=16). Hopefully this time I can get reasonable result.

This time it will not be NAN.

@launchauto commented on GitHub (Mar 23, 2021): > > The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling `.float()`; this was why I had to patch those variables in [some hacky way](https://github.com/openai/CLIP/blob/main/clip/clip.py#L137-L157) to make it runnable on CPU. > > Use `clip.load()` with `jit=False` to load the model in a non-JIT mode, and try again. > > Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and batchsize=16(ImageNet supervision is 0.02 for batchsize=16). Hopefully this time I can get reasonable result. This time it will not be NAN.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 23, 2021):

The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling .float(); this was why I had to patch those variables in some hacky way to make it runnable on CPU.
Use clip.load() with jit=False to load the model in a non-JIT mode, and try again.

Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and batchsize=16(ImageNet supervision is 0.2 for batchsize=16). Hopefully this time I can get reasonable result.

This time it will not be NAN.

BTW, do you have some suggestion to learning rate and optimizer when transfer to detection model. Thx~

@launchauto commented on GitHub (Mar 23, 2021): > > > The JIT code contains hard-coded dtype variables and will not work if you simply cast by calling `.float()`; this was why I had to patch those variables in [some hacky way](https://github.com/openai/CLIP/blob/main/clip/clip.py#L137-L157) to make it runnable on CPU. > > > Use `clip.load()` with `jit=False` to load the model in a non-JIT mode, and try again. > > > > > > Thx! I try jit=False and let the learning_rate to be 0.002 for 8GPU and batchsize=16(ImageNet supervision is 0.2 for batchsize=16). Hopefully this time I can get reasonable result. > > This time it will not be NAN. BTW, do you have some suggestion to learning rate and optimizer when transfer to detection model. Thx~

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 23, 2021):

There's no universal rule on this, but I'd start with 10% the learning rate used for pre-training.

@jongwook commented on GitHub (Mar 23, 2021): There's no universal rule on this, but I'd start with 10% the learning rate used for pre-training.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@launchauto commented on GitHub (Mar 23, 2021):

There's no universal rule on this, but I'd start with 10% the learning rate used for pre-training.

Thx. I will try.

@launchauto commented on GitHub (Mar 23, 2021): > There's no universal rule on this, but I'd start with 10% the learning rate used for pre-training. Thx. I will try.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@dragen1860 commented on GitHub (Apr 1, 2021):

@launchauto Hi, What's the mAP when you adopt the clip backbone? is it gain improment? I would like to here some suggestion from you . Thank you.

@dragen1860 commented on GitHub (Apr 1, 2021): @launchauto Hi, What's the mAP when you adopt the clip backbone? is it gain improment? I would like to here some suggestion from you . Thank you.

bsakaguchi commented

2026-10-31 21:10:21 +00:00

Author

Owner

@Edwardmark commented on GitHub (Apr 12, 2021):

@launchauto Have you made any progress in detection using clip models?

@Edwardmark commented on GitHub (Apr 12, 2021): @launchauto Have you made any progress in detection using clip models?

bsakaguchi referenced this issue

2026-10-31 21:10:33 +00:00

[GH-ISSUE #83] CLIP Training Code #73

bsakaguchi referenced this issue

2026-10-31 21:10:55 +00:00

[GH-ISSUE #92] Half precision error on forward pass of RN50 and RN101 models #84

Rows
Columns

[GH-ISSUE #63] NAN in downstream detection finetuneing task #57