[GH-ISSUE #113] Is there a way to only use the text encoder ?

bsakaguchi commented

2026-10-31 21:11:08 +00:00

Owner

Originally created by @ranran9991 on GitHub (Jun 9, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/113

Hey!
I'd like to use only one part of the model, specifically the text encoder in my work. I don't want to store the whole model in GPU memory just to use the text encoding part, is there a simple way to do that? or will I have to dive into the code myself

Thanks for the help ! :)

Originally created by @ranran9991 on GitHub (Jun 9, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/113 Hey! I'd like to use only one part of the model, specifically the text encoder in my work. I don't want to store the whole model in GPU memory just to use the text encoding part, is there a simple way to do that? or will I have to dive into the code myself Thanks for the help ! :)

bsakaguchi closed this issue

2026-10-31 21:11:08 +00:00

bsakaguchi commented

2026-10-31 21:11:08 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jun 9, 2021):

What I have done is to set the model.visual = None to remove all the visual part. But this will raise error since dtype property is dependent on the .visual part.

github.com/openai/CLIP@cfcffb90e6/clip/model.py (L332-L334)

I think if we can set all the visual part into None except for model.visual.conv1.weight then the encode_text will work perfectly without the need to store the visual part.

@vinson2233 commented on GitHub (Jun 9, 2021): What I have done is to set the `model.visual = None` to remove all the visual part. But this will raise error since dtype property is dependent on the `.visual` part. https://github.com/openai/CLIP/blob/cfcffb90e69f37bf2ff1e988237a0fbe41f33c04/clip/model.py#L332-L334 I think if we can set all the visual part into None except for `model.visual.conv1.weight` then the encode_text will work perfectly without the need to store the visual part.

bsakaguchi commented

2026-10-31 21:11:08 +00:00

Author

Owner

@ranran9991 commented on GitHub (Jun 14, 2021):

What I have done is to set the model.visual = None to remove all the visual part. But this will raise error since dtype property is dependent on the .visual part.

github.com/openai/CLIP@cfcffb90e6/clip/model.py (L332-L334)

I think if we can set all the visual part into None except for model.visual.conv1.weight then the encode_text will work perfectly without the need to store the visual part.

Thank you for the help, hopefully this feature will be added soon

@ranran9991 commented on GitHub (Jun 14, 2021): > > > What I have done is to set the `model.visual = None` to remove all the visual part. But this will raise error since dtype property is dependent on the `.visual` part. > > https://github.com/openai/CLIP/blob/cfcffb90e69f37bf2ff1e988237a0fbe41f33c04/clip/model.py#L332-L334 > > I think if we can set all the visual part into None except for `model.visual.conv1.weight` then the encode_text will work perfectly without the need to store the visual part. Thank you for the help, hopefully this feature will be added soon

bsakaguchi commented

2026-10-31 21:11:08 +00:00

Author

Owner

@lonngxiang commented on GitHub (Jun 30, 2021):

same needs

@lonngxiang commented on GitHub (Jun 30, 2021): same needs

bsakaguchi commented

2026-10-31 21:11:08 +00:00

Author

Owner

@jongwook commented on GitHub (Jun 30, 2021):

You can replace self.visual.conv1.weight.dtype with next(self.parameters()).dtype and such, which will avoid the error. I'll plan to replace it in the next round of updates.

@jongwook commented on GitHub (Jun 30, 2021): You can replace `self.visual.conv1.weight.dtype` with `next(self.parameters()).dtype` and such, which will avoid the error. I'll plan to replace it in the next round of updates.

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@lonngxiang commented on GitHub (Jun 30, 2021):

You can replace self.visual.conv1.weight.dtype with next(self.parameters()).dtype and such, which will avoid the error. I'll plan to replace it in the next round of updates.

thumb up

@lonngxiang commented on GitHub (Jun 30, 2021): > You can replace `self.visual.conv1.weight.dtype` with `next(self.parameters()).dtype` and such, which will avoid the error. I'll plan to replace it in the next round of updates. thumb up

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@lonngxiang commented on GitHub (Jul 5, 2021):

finally, I succeed @jongwook @vinson2233 @ranran9991

model.py

class CLIP(nn.Module):
    def __init__(self,
                embed_dim:int,
                 context_length: int,
                 vocab_size: int,
                 transformer_width: int,
                 transformer_heads: int,
                 transformer_layers: int
                 ):
        super().__init__()

   self.visual=None

   @property
    def dtype(self):
        return next(self.parameters()).dtype

   def forward(self, text):
        # image_features = self.encode_image(image)
        text_features = self.encode_text(text)

        # normalized features
        # image_features = image_features / image_features.norm(dim=-1, keepdim=True)
        text_features = text_features / text_features.norm(dim=-1, keepdim=True)

        # cosine similarity as logits
        # logit_scale = self.logit_scale.exp()
        # logits_per_image = logit_scale * image_features @ text_features.t()
        # logits_per_text = logit_scale * text_features @ image_features.t()

        # shape = [global_batch_size, global_batch_size]
        # return logits_per_image, logits_per_text
        return text_features

save.py

device = "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

dict_trained = model.state_dict()    # trained model
trained_lst = list(dict_trained.keys())

model_txt = CLIP(embed_dim=512,context_length=77,vocab_size=49408,transformer_width=512,transformer_heads=16,transformer_layers=6)
dict_txt = model_txt.state_dict()
print(dict_txt)
for key in dict_txt:
    dict_txt[key] = dict_trained[key]
model_txt.load_state_dict(dict_txt) 
torch.save(model_txt, "./single_model_text1.pkl")

@lonngxiang commented on GitHub (Jul 5, 2021): finally, I succeed @jongwook @vinson2233 @ranran9991 model.py ``` class CLIP(nn.Module): def __init__(self, embed_dim:int, context_length: int, vocab_size: int, transformer_width: int, transformer_heads: int, transformer_layers: int ): super().__init__() self.visual=None @property def dtype(self): return next(self.parameters()).dtype def forward(self, text): # image_features = self.encode_image(image) text_features = self.encode_text(text) # normalized features # image_features = image_features / image_features.norm(dim=-1, keepdim=True) text_features = text_features / text_features.norm(dim=-1, keepdim=True) # cosine similarity as logits # logit_scale = self.logit_scale.exp() # logits_per_image = logit_scale * image_features @ text_features.t() # logits_per_text = logit_scale * text_features @ image_features.t() # shape = [global_batch_size, global_batch_size] # return logits_per_image, logits_per_text return text_features ``` save.py ``` device = "cpu" model, preprocess = clip.load("ViT-B/32", device=device) dict_trained = model.state_dict() # trained model trained_lst = list(dict_trained.keys()) model_txt = CLIP(embed_dim=512,context_length=77,vocab_size=49408,transformer_width=512,transformer_heads=16,transformer_layers=6) dict_txt = model_txt.state_dict() print(dict_txt) for key in dict_txt: dict_txt[key] = dict_trained[key] model_txt.load_state_dict(dict_txt) torch.save(model_txt, "./single_model_text1.pkl") ```

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jul 5, 2021):

@lonngxiang To highlight the code part, you can use ``` instead of `, put it on top and bottom part of code, all will be highlighted as code.
But I get the idea. Thanks.

@vinson2233 commented on GitHub (Jul 5, 2021): @lonngxiang To highlight the code part, you can use ``` instead of `, put it on top and bottom part of code, all will be highlighted as code. But I get the idea. Thanks.

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@lonngxiang commented on GitHub (Jul 5, 2021):

@lonngxiang To highlight the code part, you can use ``` instead of `, put it on top and bottom part of code, all will be highlighted as code.
But I get the idea. Thanks.

ok，tks

@lonngxiang commented on GitHub (Jul 5, 2021): > @lonngxiang To highlight the code part, you can use ``` instead of `, put it on top and bottom part of code, all will be highlighted as code. > But I get the idea. Thanks. ok，tks

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@laurenspriem commented on GitHub (Feb 28, 2023):

finally, I succeed @jongwook @vinson2233 @ranran9991

model.py

class CLIP(nn.Module):
    def __init__(self,
                embed_dim:int,
                 context_length: int,
                 vocab_size: int,
                 transformer_width: int,
                 transformer_heads: int,
                 transformer_layers: int
                 ):
        super().__init__()

   self.visual=None

   @property
    def dtype(self):
        return next(self.parameters()).dtype

   def forward(self, text):
        # image_features = self.encode_image(image)
        text_features = self.encode_text(text)

        # normalized features
        # image_features = image_features / image_features.norm(dim=-1, keepdim=True)
        text_features = text_features / text_features.norm(dim=-1, keepdim=True)

        # cosine similarity as logits
        # logit_scale = self.logit_scale.exp()
        # logits_per_image = logit_scale * image_features @ text_features.t()
        # logits_per_text = logit_scale * text_features @ image_features.t()

        # shape = [global_batch_size, global_batch_size]
        # return logits_per_image, logits_per_text
        return text_features

save.py

device = "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

dict_trained = model.state_dict()    # trained model
trained_lst = list(dict_trained.keys())

model_txt = CLIP(embed_dim=512,context_length=77,vocab_size=49408,transformer_width=512,transformer_heads=16,transformer_layers=6)
dict_txt = model_txt.state_dict()
print(dict_txt)
for key in dict_txt:
    dict_txt[key] = dict_trained[key]
model_txt.load_state_dict(dict_txt) 
torch.save(model_txt, "./single_model_text1.pkl")

I followed this approach and got a text encoder. However, the embeddings that the model gives are completely wrong. @lonngxiang have you verified that the resulting embeddings match those of the original model?

@jongwook is the above method still the recommended approach, or is there a better way by now?

@laurenspriem commented on GitHub (Feb 28, 2023): > finally, I succeed @jongwook @vinson2233 @ranran9991 > > model.py > > ``` > class CLIP(nn.Module): > def __init__(self, > embed_dim:int, > context_length: int, > vocab_size: int, > transformer_width: int, > transformer_heads: int, > transformer_layers: int > ): > super().__init__() > > self.visual=None > > @property > def dtype(self): > return next(self.parameters()).dtype > > def forward(self, text): > # image_features = self.encode_image(image) > text_features = self.encode_text(text) > > # normalized features > # image_features = image_features / image_features.norm(dim=-1, keepdim=True) > text_features = text_features / text_features.norm(dim=-1, keepdim=True) > > # cosine similarity as logits > # logit_scale = self.logit_scale.exp() > # logits_per_image = logit_scale * image_features @ text_features.t() > # logits_per_text = logit_scale * text_features @ image_features.t() > > # shape = [global_batch_size, global_batch_size] > # return logits_per_image, logits_per_text > return text_features > ``` > > save.py > > ``` > device = "cpu" > model, preprocess = clip.load("ViT-B/32", device=device) > > dict_trained = model.state_dict() # trained model > trained_lst = list(dict_trained.keys()) > > model_txt = CLIP(embed_dim=512,context_length=77,vocab_size=49408,transformer_width=512,transformer_heads=16,transformer_layers=6) > dict_txt = model_txt.state_dict() > print(dict_txt) > for key in dict_txt: > dict_txt[key] = dict_trained[key] > model_txt.load_state_dict(dict_txt) > torch.save(model_txt, "./single_model_text1.pkl") > ``` I followed this approach and got a text encoder. However, the embeddings that the model gives are completely wrong. @lonngxiang have you verified that the resulting embeddings match those of the original model? @jongwook is the above method still the recommended approach, or is there a better way by now?

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@laurenspriem commented on GitHub (Feb 28, 2023):

Since the above solution didn't work for me, I've used another workaround that was kind of suggested earlier in this thread.

Basically, you can strip away most of the image encoder by setting model.visual.transformer = None. Contrary to model.visual = None, this doesn't give an error.

Downsides are that you're still left with some useless weights from the image encoder, and that you have to use model.encode_text(text_input) instead of just model(text_input).

@laurenspriem commented on GitHub (Feb 28, 2023): Since the above solution didn't work for me, I've used another workaround that was kind of suggested earlier in this thread. Basically, you can strip away most of the image encoder by setting `model.visual.transformer = None`. Contrary to `model.visual = None`, this doesn't give an error. Downsides are that you're still left with some useless weights from the image encoder, and that you have to use `model.encode_text(text_input)` instead of just `model(text_input)`.

bsakaguchi commented

2026-10-31 21:11:09 +00:00

Author

Owner

@ranran9991 commented on GitHub (Feb 28, 2023):

Since the above solution didn't work for me, I've used another workaround that was kind of suggested earlier in this thread.

Basically, you can strip away most of the image encoder by setting model.visual.transformer = None. Contrary to model.visual = None, this doesn't give an error.

Downsides are that you're still left with some useless weights from the image encoder, and that you have to use model.encode_text(text_input) instead of just model(text_input).

This is the workaround that I use as well.

@ranran9991 commented on GitHub (Feb 28, 2023): > Since the above solution didn't work for me, I've used another workaround that was kind of suggested earlier in this thread. > > Basically, you can strip away most of the image encoder by setting `model.visual.transformer = None`. Contrary to `model.visual = None`, this doesn't give an error. > > Downsides are that you're still left with some useless weights from the image encoder, and that you have to use `model.encode_text(text_input)` instead of just `model(text_input)`. This is the workaround that I use as well.

Rows
Columns

[GH-ISSUE #113] Is there a way to only use the text encoder ? #101