mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #171] Change dtype on inference #149
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#149
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @MaxKochanov on GitHub (Nov 2, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/171
Hello!
In the issue #30 you answered: "Empirically, we have been doing CLIP inference in fp16 without much problem, and that's how the model was trained for anyway".
In this case I have two questions:
Is there any chance to change dtype of CLIP weights from float32 to float16 and use it on GPU? You said "without much problem", and I`ve already dived into the code, but I have not succeed to change the dtype correctly.
Do I understand right that you trained CLIP on float32?
@jongwook commented on GitHub (Nov 16, 2021):
For most of weights it's safe to convert the parameter dtypes to float16, but some operations like LayerNorm needs to be done in fp32 for stable training. We convert some of the weights depending on the layer's type; see convert_weights().
We used mixed precision training for training CLIP, as mentioned in Section 2.5 of the paper.