[GH-ISSUE #381] how to convert a picture to text embedding, without training these image model #326

New issue

Open

opened 2026-10-31 21:13:54 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:13:54 +00:00

Owner

Originally created by @yanchaoguo on GitHub (Aug 4, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/381

Now, to generate specific concepts, like personal objects or artistic styles, by describing them using new "words" in the embedding space of pre-trained text-to-image models. These can be used in new sentences, just like any other word.

how to repace text embedding using a image without training

Originally created by @yanchaoguo on GitHub (Aug 4, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/381 Now, to generate specific concepts, like personal objects or artistic styles, by describing them using new "words" in the embedding space of pre-trained text-to-image models. These can be used in new sentences, just like any other word. **how to repace text embedding using a image without training**

bsakaguchi commented

2026-10-31 21:13:55 +00:00

Author

Owner

@wunting-chan commented on GitHub (Jan 20, 2024):

Not sure if you're looking for a solution without any training done on your end (meaning using models pretrained by others), or you're looking for a true no additional training solution.

If it's the former, you could look into ClipCap. They've trained a network to learn a prefix that gets added to an image's clip embedding for caption generation.

@wunting-chan commented on GitHub (Jan 20, 2024): Not sure if you're looking for a solution without any training done on your end (meaning using models pretrained by others), or you're looking for a true no additional training solution. If it's the former, you could look into [ClipCap](https://github.com/rmokady/CLIP_prefix_caption). They've trained a network to learn a prefix that gets added to an image's clip embedding for caption generation.

bsakaguchi commented

2026-10-31 21:13:55 +00:00

Author

Owner

@LinMa910 commented on GitHub (Feb 21, 2025):

I also have the same question, how do you solvove this? Is there any presures that can share you solutions. very thank you for your help!

@LinMa910 commented on GitHub (Feb 21, 2025): I also have the same question, how do you solvove this? Is there any presures that can share you solutions. very thank you for your help!

bsakaguchi referenced this issue

2026-10-31 21:15:28 +00:00

[PR #326] [MERGED] Update README.md #488