[GH-ISSUE #122] How to transform clip model into onnx format? #109

Closed
opened 2026-10-31 21:11:16 +00:00 by bsakaguchi · 8 comments
Owner

Originally created by @lonngxiang on GitHub (Jun 30, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/122

Originally created by @lonngxiang on GitHub (Jun 30, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/122
Author
Owner

@jongwook commented on GitHub (Jun 30, 2021):

We don't have experience with the ONNX format and can only refer you to the manual

<!-- gh-comment-id:871268079 --> @jongwook commented on GitHub (Jun 30, 2021): We don't have experience with the ONNX format and can only refer you to [the manual](https://pytorch.org/docs/stable/onnx.html)
Author
Owner

@xyxxmb commented on GitHub (Sep 8, 2021):

No description provided.

The same problem with you. Have you got any good solutions to the problem?

<!-- gh-comment-id:915130640 --> @xyxxmb commented on GitHub (Sep 8, 2021): > _No description provided._ The same problem with you. Have you got any good solutions to the problem?
Author
Owner

@cendelian commented on GitHub (Dec 15, 2021):

have you solve the problem? how to convert clip model to onnx format? thanks

<!-- gh-comment-id:994325691 --> @cendelian commented on GitHub (Dec 15, 2021): have you solve the problem? how to convert clip model to onnx format? thanks
Author
Owner

@lonngxiang commented on GitHub (Dec 15, 2021):

#113,then you can convert clip model to onnx format

<!-- gh-comment-id:994332341 --> @lonngxiang commented on GitHub (Dec 15, 2021): #113,then you can convert clip model to onnx format
Author
Owner

@Lednik7 commented on GitHub (Jan 5, 2022):

Hi, over the weekend I solved the CLIP export issue, tried to pack everything into the library. Implementation here

<!-- gh-comment-id:1005843071 --> @Lednik7 commented on GitHub (Jan 5, 2022): Hi, over the weekend I solved the CLIP export issue, tried to pack everything into the library. Implementation [here](https://github.com/Lednik7/CLIP-ONNX)
Author
Owner

@hanxiao commented on GitHub (Apr 10, 2022):

For ONNX support, please try https://github.com/jina-ai/clip-as-service/

<!-- gh-comment-id:1094349549 --> @hanxiao commented on GitHub (Apr 10, 2022): For ONNX support, please try https://github.com/jina-ai/clip-as-service/
Author
Owner

@deshwalmahesh commented on GitHub (Jul 2, 2022):

No description provided.

You can do this as:

! pip install onnx onnxruntime

import clip
import torch

m, pre = clip.load("RN50")
npx = m.visual.input_resolution
dummy_image = torch.randn(10, 3, npx, npx)
dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"])
m.forward(dummy_image,dummy_texts) # Original CLIP result (1)

torch.onnx.export(m, (dummy_image, dummy_texts), "clip_resnet.onnx", export_params=True,
  input_names=["IMAGE", "TEXT"],
  output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"],
  opset_version=14,
  dynamic_axes={
      "IMAGE": {
          0: "image_batch_size",
      },
      "TEXT": {
          0: "text_batch_size",
      },
      "LOGITS_PER_IMAGE": {
          0: "image_batch_size",
          1: "text_batch_size",
      },
      "LOGITS_PER_TEXT": {
          0: "text_batch_size",
          1: "image_batch_size",
      },
  }
)

# Now run onnxruntime to verify
import onnxruntime as ort

ort_sess = ort.InferenceSession("clip_resnet.onnx")
result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], 
  {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()})
result # verify that result is comparable to (1)

NOT MY CODE.

<!-- gh-comment-id:1172928859 --> @deshwalmahesh commented on GitHub (Jul 2, 2022): > _No description provided._ You can do this as: ``` ! pip install onnx onnxruntime import clip import torch m, pre = clip.load("RN50") npx = m.visual.input_resolution dummy_image = torch.randn(10, 3, npx, npx) dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"]) m.forward(dummy_image,dummy_texts) # Original CLIP result (1) torch.onnx.export(m, (dummy_image, dummy_texts), "clip_resnet.onnx", export_params=True, input_names=["IMAGE", "TEXT"], output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], opset_version=14, dynamic_axes={ "IMAGE": { 0: "image_batch_size", }, "TEXT": { 0: "text_batch_size", }, "LOGITS_PER_IMAGE": { 0: "image_batch_size", 1: "text_batch_size", }, "LOGITS_PER_TEXT": { 0: "text_batch_size", 1: "image_batch_size", }, } ) # Now run onnxruntime to verify import onnxruntime as ort ort_sess = ort.InferenceSession("clip_resnet.onnx") result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()}) result # verify that result is comparable to (1) ``` NOT MY CODE.
Author
Owner

@JanWerder commented on GitHub (Jan 29, 2024):

https://github.com/Lednik7/CLIP-ONNX

<!-- gh-comment-id:1914180325 --> @JanWerder commented on GitHub (Jan 29, 2024): https://github.com/Lednik7/CLIP-ONNX
Sign in to join this conversation.
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#109
No description provided.