[PR #219] [MERGED] Patch clip model for ONNX compatibility #473

New issue

Closed

opened 2026-10-31 21:15:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:15:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/openai/CLIP/pull/219
Author: @chajath
Created: 2/18/2022
Status: ✅ Merged
Merged: 4/10/2022
Merged by: @jongwook

Base: main ← Head: chajath/onnx-compatibility-patches

📝 Commits (2)

1937532 Patch clip model for ONNX compatibility
772e202 Add compatibility fix for torch 1.7

📊 Changes

2 files changed (+9 additions, -5 deletions)

View changed files

📝 clip/clip.py (+7 -3)
📝 clip/model.py (+2 -2)

📄 Description

Team,

When productionizing AI models like CLIP, it is often useful to be able to export to ONNX so that we can utilize training and serving ecosystem built around ONNX runtime. This PR includes changes that are necessary to make ONNX compilation and runtime working without needing to further patch the code:

Changes to use INT32 for tokenization, since ONNX doesn't yet support ArgMax(INT64). See https://github.com/microsoft/onnxruntime/issues/10068
Use explicit dimension for norm

With these changes, we can now compile and run ONNX models. For example:

import clip
import torch

m, pre = clip.load("RN50")
npx = m.visual.input_resolution
dummy_image = torch.randn(10, 3, npx, npx)
dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"])
m.forward(dummy_image ,dummy_texts) # Original CLIP result (1)

torch.onnx.export(m, (dummy_image, dummy_texts), "clip_resnet.onnx", export_params=True,
  input_names=["IMAGE", "TEXT"],
  output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"],
  opset_version=14,
  dynamic_axes={
      "IMAGE": {
          0: "image_batch_size",
      },
      "TEXT": {
          0: "text_batch_size",
      },
      "LOGITS_PER_IMAGE": {
          0: "image_batch_size",
          1: "text_batch_size",
      },
      "LOGITS_PER_TEXT": {
          0: "text_batch_size",
          1: "image_batch_size",
      },
  }
)

# Now run onnxruntime to verify
import onnxruntime as ort

ort_sess = ort.InferenceSession("clip_resnet.onnx")
result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], 
  {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()})
result # verify that result is comparable to (1)

I've locally verified the result. I see that CLIP has a github action to run pytest. I will be happy to further contribute by adding onnx-specific tests to see if the model can be compiled and the resulting model is correct.

I understand that ONNX compatibility might not be the primary goal of CLIP repo, but maintaining compatibility with ONNX will be immensely useful to the ML practitioners out in the wild, so please take a look at the change.

Thanks!

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/openai/CLIP/pull/219 **Author:** [@chajath](https://github.com/chajath) **Created:** 2/18/2022 **Status:** ✅ Merged **Merged:** 4/10/2022 **Merged by:** [@jongwook](https://github.com/jongwook) **Base:** `main` ← **Head:** `chajath/onnx-compatibility-patches` --- ### 📝 Commits (2) - [`1937532`](https://github.com/openai/CLIP/commit/1937532d60b7f61f1e102ed535561ab252a02755) Patch clip model for ONNX compatibility - [`772e202`](https://github.com/openai/CLIP/commit/772e2023b7d58193e5590591f848410c45b04520) Add compatibility fix for torch 1.7 ### 📊 Changes **2 files changed** (+9 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `clip/clip.py` (+7 -3) 📝 `clip/model.py` (+2 -2) </details> ### 📄 Description Team, When productionizing AI models like CLIP, it is often useful to be able to export to ONNX so that we can utilize training and serving ecosystem built around ONNX runtime. This PR includes changes that are necessary to make ONNX compilation and runtime working without needing to further patch the code: * Changes to use INT32 for tokenization, since ONNX doesn't yet support ArgMax(INT64). See https://github.com/microsoft/onnxruntime/issues/10068 * Use explicit dimension for norm With these changes, we can now compile and run ONNX models. For example: ```python import clip import torch m, pre = clip.load("RN50") npx = m.visual.input_resolution dummy_image = torch.randn(10, 3, npx, npx) dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"]) m.forward(dummy_image ,dummy_texts) # Original CLIP result (1) torch.onnx.export(m, (dummy_image, dummy_texts), "clip_resnet.onnx", export_params=True, input_names=["IMAGE", "TEXT"], output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], opset_version=14, dynamic_axes={ "IMAGE": { 0: "image_batch_size", }, "TEXT": { 0: "text_batch_size", }, "LOGITS_PER_IMAGE": { 0: "image_batch_size", 1: "text_batch_size", }, "LOGITS_PER_TEXT": { 0: "text_batch_size", 1: "image_batch_size", }, } ) # Now run onnxruntime to verify import onnxruntime as ort ort_sess = ort.InferenceSession("clip_resnet.onnx") result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()}) result # verify that result is comparable to (1) ``` I've locally verified the result. I see that CLIP has a github action to run pytest. I will be happy to further contribute by adding onnx-specific tests to see if the model can be compiled and the resulting model is correct. I understand that ONNX compatibility might not be the primary goal of `CLIP` repo, but maintaining compatibility with ONNX will be immensely useful to the ML practitioners out in the wild, so please take a look at the change. Thanks! --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 21:15:21 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#473

No description provided.

Rows
Columns