mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #460] How to convert CLIP vectors to an LLM text token embeddings? #390
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#390
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xoraizw on GitHub (Aug 29, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/460
I'm looking to embed multiple modalities into your conventional text based LLMs. For that I need to convert any modality into a CLIP vector which I have done, now I need to convert this vector into an LLM text token embedding. Can anyone help me out with this conversion?
@Asynchronousx commented on GitHub (Jan 16, 2025):
Don't know if this could still help, but you do this by using a mapper F that projects your vision embeddings to something that could be given in input to an LLM (such as GPT2, naming some lightweight LLMs). Basically you're translating the vision embeddings to a text one suitable for your LLM of choice. You do this by training a mapper (that could be a linear layer to a transformer) and use the loss of the prediction of the LLM to upgrade the weight of such mapper. This is the general schema, hope it helps!