mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #452] Issue with Text Encoder Output Dimensions in Fine-Tuned CLIP Model When Using with Stable Diffusion #387
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#387
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @QXGeraldMo on GitHub (Jul 5, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/452
I'm encountering an issue with the dimensions of the text encoder output in a fine-tuned CLIP model. The fine-tuning output of my CLIP model based on RN50 is (1, 1024), whereas the output from CLIPTextModel in transformers is (1, 77, 768). This causes difficulties when integrating with Stable Diffusion models, as the encoder_hidden_states parameter in U-Net requires an embedding of shape (x, x, 768).
Can someone help me deal with this problem? Would truncating the output of encode_text to (1, 1, 768) potentially resolve this issue?
@supersonicMaclaurin commented on GitHub (Jul 23, 2024):
same question
@99991 commented on GitHub (Jan 23, 2025):
CLIP has many models with different sizes. See also here for HuggingFace CLIP models.
That aside, if you finetune a vision encoder like ResNet50 without also finetuning the text encoder at the same time, their embeddings might become desynchronized and you get bad results. But truncating would definitely not work at all unless the model is specifically trained for this task (see e.g. Matryoshka embeddings).