openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #144] NaN on encode_image #125

New issue
Closed
opened 2026-10-31 21:11:27 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-31 21:11:27 +00:00
Owner
Copy link

Originally created by @sarahESL on GitHub (Aug 28, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/144

I fine-tuned CLIP on my own dataset and plan to use it as a pretrained network in order to encode images. After loading the model, I get NaN values when using encode_image on my images. More interestingly, this happens only since the second batch, so the first batch is alright. I checked the loaded images from the second batch, and they are fine. Any ideas what might be the issue?

Originally created by @sarahESL on GitHub (Aug 28, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/144 I fine-tuned CLIP on my own dataset and plan to use it as a pretrained network in order to encode images. After loading the model, I get `NaN` values when using `encode_image` on my images. More interestingly, this happens only since the second batch, so the first batch is alright. I checked the loaded images from the second batch, and they are fine. Any ideas what might be the issue?
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Aug 30, 2021):

Not sure. Are you using the encoder in .eval() mode, and did you use jit=False (now the default) when loading the model? On the ResNet-based models the batchnorm layers might be outputting unintended results in the train mode.

Casting the entire model to fp32 by model.float() or checking if any of the weights have a NaN element before and after the encode_image call might be helpful.

You can also try checking the output of each layer to see where the NaN started to happen during encoding.

<!-- gh-comment-id:908013334 --> @jongwook commented on GitHub (Aug 30, 2021): Not sure. Are you using the encoder in `.eval()` mode, and did you use `jit=False` (now the default) when loading the model? On the ResNet-based models the batchnorm layers might be outputting unintended results in the train mode. Casting the entire model to fp32 by `model.float()` or checking if any of the weights have a `NaN` element before and after the `encode_image` call might be helpful. You can also try checking the output of each layer to see where the `NaN` started to happen during encoding.
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@sarahESL commented on GitHub (Aug 30, 2021):

Thanks. Casting the model to fp32 seems to have solved the problem for now.

<!-- gh-comment-id:908072708 --> @sarahESL commented on GitHub (Aug 30, 2021): Thanks. Casting the model to fp32 seems to have solved the problem for now.
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@NotNANtoN commented on GitHub (Oct 1, 2021):

@sarahESL Hey, I'm curious about fine-tuning CLIP.

Could you share some details on how many image-text pairs you used and what batch size? Did it work to fine-tune it? I'd appreciate any input! :)

<!-- gh-comment-id:932635210 --> @NotNANtoN commented on GitHub (Oct 1, 2021): @sarahESL Hey, I'm curious about fine-tuning CLIP. Could you share some details on how many image-text pairs you used and what batch size? Did it work to fine-tune it? I'd appreciate any input! :)
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@sarahESL commented on GitHub (Oct 4, 2021):

@NotNANtoN Hey, sure. I used more than 60k image-text pair with batchsizes of 16, 32, 64. Yes, the finetuning worked. I was getting NaNs on encoding images in test phase, but then casting the model to fp32 by model.float() worked, as suggested by @jongwook.

<!-- gh-comment-id:933752159 --> @sarahESL commented on GitHub (Oct 4, 2021): @NotNANtoN Hey, sure. I used more than 60k image-text pair with batchsizes of 16, 32, 64. Yes, the finetuning worked. I was getting `NaN`s on encoding images in test phase, but then casting the model to fp32 by `model.float()` worked, as suggested by @jongwook.
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@NotNANtoN commented on GitHub (Oct 11, 2021):

Hi Sarah, thanks a lot! That's great to hear. The dataset and batch size both seem to be of a manageable size.

<!-- gh-comment-id:939998338 --> @NotNANtoN commented on GitHub (Oct 11, 2021): Hi Sarah, thanks a lot! That's great to hear. The dataset and batch size both seem to be of a manageable size.
bsakaguchi commented 2026-10-31 21:11:28 +00:00
Author
Owner
Copy link

@sarahESL commented on GitHub (Jun 6, 2022):

Just wanted to document that this issue happens on encode_text as well and casting to fp32 is still the fix.

<!-- gh-comment-id:1147693702 --> @sarahESL commented on GitHub (Jun 6, 2022): Just wanted to document that this issue happens on `encode_text` as well and casting to fp32 is still the fix.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#125
No description provided.