openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[PR #497] [New Feature] Loading HuggingFace .safetensors and .bin variants for CLIP models #515

New issue

Open

opened 2026-10-31 21:15:39 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:15:39 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/openai/CLIP/pull/497
Author: @bogoconic1
Created: 6/4/2025
Status: 🔄 Open

Base: main ← Head: main

📝 Commits (3)

d57daf9 Update requirements.txt
c2bdbb6 load safetensors version in clip
1486984 add support for huggingface bin/safetensors variant

📊 Changes

2 files changed (+226 additions, -10 deletions)

View changed files

📝 clip/clip.py (+225 -10)
📝 requirements.txt (+1 -0)

📄 Description

Issue

I tried to load HuggingFace safetensors/bin checkpoints into CLIP, but failed due to this error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File /opt/conda/lib/python3.11/site-packages/clip/clip.py:129, in load(name, device, jit, download_root)
    127 try:
    128     # loading JIT archive
--> 129     model = torch.jit.load(opened_file, map_location=device if jit else "cpu").eval()
    130     state_dict = None

File /opt/conda/lib/python3.11/site-packages/torch/jit/_serialization.py:165, in load(f, map_location, _extra_files, _restore_shapes)
    164 else:
--> 165     cpp_module = torch._C.import_ir_module_from_buffer(
    166         cu, f.read(), map_location, _extra_files, _restore_shapes
    167     )  # type: ignore[call-arg]
    169 # TODO: Pretty sure this approach loses ConstSequential status and such

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

Fix

Load the .bin and .safetensors files in a different way

if model_path.endswith('.bin'):
    state_dict = torch.load(model_path, map_location="cpu")
elif model_path.endswith('.safetensors'):
    with safe_open(model_path, framework="pt", device="cpu") as f:
        state_dict = {key: f.get_tensor(key) for key in f.keys()}

But this is insufficient, because the HuggingFace and OpenAI have different naming conventions and storage format for the state_dict. An additional logic is added to convert the HuggingFace state_dict to the OpenAI state_dict format.

Tests

I tested the enhancements on

using the following snippet, checking that the state_dicts for the original model and the huggingface model are exactly the same. It passed for all 4 models.

import torch
import clip
from tqdm.auto import tqdm

official_model, _ = clip.load("ViT-B/16", device="cpu")
official_state_dict = official_model.state_dict()

dir = "<your local directory>"
try:
    model, _ = clip.load(f"{dir}/pytorch_model.bin", device="cpu")
    state_dict = model.state_dict()

    assert len(state_dict) == len(official_state_dict)
    
    for key in tqdm(official_state_dict, total=len(official_state_dict)):
        assert key in state_dict
        assert torch.equal(official_state_dict[key], state_dict[key])

    model, _ = clip.load(f"{dir}/model.safetensors", device="cpu")
    state_dict = model.state_dict()
    
    for key in tqdm(official_state_dict, total=len(official_state_dict)):
        assert key in state_dict
        assert torch.equal(official_state_dict[key], state_dict[key])

except Exception as e:
    print(e)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/openai/CLIP/pull/497 **Author:** [@bogoconic1](https://github.com/bogoconic1) **Created:** 6/4/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (3) - [`d57daf9`](https://github.com/openai/CLIP/commit/d57daf938a5635e4629bb07c2ea84bc2ae0cedda) Update requirements.txt - [`c2bdbb6`](https://github.com/openai/CLIP/commit/c2bdbb6ded7d92819f938655543d643082e8bea7) load safetensors version in clip - [`1486984`](https://github.com/openai/CLIP/commit/1486984f3ac7969f15b0d5ad26c3f9660bf66393) add support for huggingface bin/safetensors variant ### 📊 Changes **2 files changed** (+226 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `clip/clip.py` (+225 -10) 📝 `requirements.txt` (+1 -0) </details> ### 📄 Description ## Issue I tried to load HuggingFace safetensors/bin checkpoints into CLIP, but failed due to this error ``` --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) File /opt/conda/lib/python3.11/site-packages/clip/clip.py:129, in load(name, device, jit, download_root) 127 try: 128 # loading JIT archive --> 129 model = torch.jit.load(opened_file, map_location=device if jit else "cpu").eval() 130 state_dict = None File /opt/conda/lib/python3.11/site-packages/torch/jit/_serialization.py:165, in load(f, map_location, _extra_files, _restore_shapes) 164 else: --> 165 cpp_module = torch._C.import_ir_module_from_buffer( 166 cu, f.read(), map_location, _extra_files, _restore_shapes 167 ) # type: ignore[call-arg] 169 # TODO: Pretty sure this approach loses ConstSequential status and such RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory ``` ## Fix 1. Load the .bin and .safetensors files in a different way ``` if model_path.endswith('.bin'): state_dict = torch.load(model_path, map_location="cpu") elif model_path.endswith('.safetensors'): with safe_open(model_path, framework="pt", device="cpu") as f: state_dict = {key: f.get_tensor(key) for key in f.keys()} ``` 2. But this is insufficient, because the HuggingFace and OpenAI have different naming conventions and storage format for the state_dict. An additional logic is added to convert the HuggingFace state_dict to the OpenAI state_dict format. ## Tests I tested the enhancements on - https://huggingface.co/openai/clip-vit-base-patch16 - https://huggingface.co/openai/clip-vit-base-patch32 - https://huggingface.co/openai/clip-vit-large-patch14-336 - https://huggingface.co/openai/clip-vit-large-patch14 using the following snippet, checking that the state_dicts for the original model and the huggingface model are exactly the same. It passed for all 4 models. ``` import torch import clip from tqdm.auto import tqdm official_model, _ = clip.load("ViT-B/16", device="cpu") official_state_dict = official_model.state_dict() dir = "<your local directory>" try: model, _ = clip.load(f"{dir}/pytorch_model.bin", device="cpu") state_dict = model.state_dict() assert len(state_dict) == len(official_state_dict) for key in tqdm(official_state_dict, total=len(official_state_dict)): assert key in state_dict assert torch.equal(official_state_dict[key], state_dict[key]) model, _ = clip.load(f"{dir}/model.safetensors", device="cpu") state_dict = model.state_dict() for key in tqdm(official_state_dict, total=len(official_state_dict)): assert key in state_dict assert torch.equal(official_state_dict[key], state_dict[key]) except Exception as e: print(e) ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 21:15:39 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#515

No description provided.

Rows
Columns