[GH-ISSUE #250] [Question] Compatibility with Huggingface CLIPModel #219

New issue

Open

opened 2026-10-31 21:12:35 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:12:35 +00:00

Owner

Originally created by @saga1214 on GitHub (Jun 6, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/250

Hi,

I am trying to use a training script from another repo, where it uses Huggingface models.
There exists a model card at Huggingface about CLIP, and it seems that the pretrained weight at HF CLIP and this original repo are almost same; the resulting features of input data have a very small difference while the total number of parameters are equal.
However, there is a discripancy between the model name via model.named_parameters(). The modules from this repo contains 302 modules while the ones from HF contains 398.

So, my question is: Is it able to load the saved models from HF to this repo or vice versa? If available, I would like to train the HF CLIP models and load to the model of this repo.

Thanks.

Originally created by @saga1214 on GitHub (Jun 6, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/250 Hi, I am trying to use a training script from another repo, where it uses Huggingface models. There exists a model card at Huggingface about CLIP, and it seems that the pretrained weight at HF CLIP and this original repo are almost same; the resulting features of input data have a very small difference while the total number of parameters are equal. However, there is a **discripancy** between the model name via model.named_parameters(). The modules from this repo contains **302 modules** while the ones from HF contains **398**. So, my question is: Is it able to **load the saved models from HF to this repo** or vice versa? If available, I would like to train the HF CLIP models and load to the model of this repo. Thanks.

bsakaguchi commented

2026-10-31 21:12:36 +00:00

Author

Owner

@wren93 commented on GitHub (Nov 4, 2022):

Some of the weight implementations are different. E.g. for the vision transformer part, in the HF model each transformer layer has the query, key and value weights. In this repo these three weight matrices are concatenated as a single matrix called "attn.in_proj_weight". It's possible to load weights from the HF model but requires some modification to the state dict.

@wren93 commented on GitHub (Nov 4, 2022): Some of the weight implementations are different. E.g. for the vision transformer part, in the HF model each transformer layer has the query, key and value weights. In this repo these three weight matrices are concatenated as a single matrix called "attn.in_proj_weight". It's possible to load weights from the HF model but requires some modification to the state dict.

bsakaguchi commented

2026-10-31 21:12:36 +00:00

Author

Owner

@junwang-wish commented on GitHub (Apr 1, 2023):

The resulting image features seem to be completely different between Huggingface and OpenAI CLIP, is it expected? cross-posted to https://github.com/huggingface/transformers/issues/22505

url = "https://canary.contestimg.wish.com/api/webimage/61b241a3a4ee2ecaf2f63c77-large.jpg?cache_buster=bbeee1fdb460a1d12bc266824914e030"

# get HF image fearures
from PIL import Image
import requests
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")

outputs = model.get_image_features(**inputs)
pooled_output_hf = outputs.detach().cpu().numpy()

# get OpenAI image features
import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
image = preprocess(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0).to(device)


with torch.no_grad():
   image_features = model.encode_image(image)
pooled_output_clip = image_features.detach().cpu().numpy()

# check difference
assert np.allclose(pooled_output_hf, pooled_output_clip, atol=0.1), "hf and clip too different"

@junwang-wish commented on GitHub (Apr 1, 2023): The resulting image features seem to be completely different between Huggingface and OpenAI CLIP, is it expected? cross-posted to https://github.com/huggingface/transformers/issues/22505 ``` url = "https://canary.contestimg.wish.com/api/webimage/61b241a3a4ee2ecaf2f63c77-large.jpg?cache_buster=bbeee1fdb460a1d12bc266824914e030" # get HF image fearures from PIL import Image import requests from transformers import CLIPProcessor, CLIPModel model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") image = Image.open(requests.get(url, stream=True).raw) inputs = processor(images=image, return_tensors="pt") outputs = model.get_image_features(**inputs) pooled_output_hf = outputs.detach().cpu().numpy() # get OpenAI image features import torch import clip from PIL import Image device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device) image = preprocess(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0).to(device) with torch.no_grad(): image_features = model.encode_image(image) pooled_output_clip = image_features.detach().cpu().numpy() # check difference assert np.allclose(pooled_output_hf, pooled_output_clip, atol=0.1), "hf and clip too different" ```

bsakaguchi referenced this issue

2026-10-31 21:15:21 +00:00

[PR #219] [MERGED] Patch clip model for ONNX compatibility #473