[GH-ISSUE #263] How do I use model.encode_image() function in Onnx to get output instead of using default model.forward() function #229

New issue

Open

opened 2026-10-31 21:12:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 21:12:44 +00:00

Owner

Originally created by @deshwalmahesh on GitHub (Jul 2, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/263

I have quantized the model using your official snippet from the merge request and it working great. Now I want CLIP for encoding images and texts than classification. How can I do that?

Below is the code to quantize and predict:


import clip
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'

model, preprocess = clip.load("RN50", device=device) # Load any model
model = model.eval() # Inference Only

img_size = model.visual.input_resolution
dummy_image = torch.randn(10, 3, img_size, img_size).to(device)
image_embedding = model.encode_image(dummy_image).to(device))

dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"]).to(device)
model.encode_text(dummy_texts)

and it works fine giving me [Batch, 1024] tensors for both for the loaded model.

Now I have quantized my model in Onnx as:

model.forward(dummy_image,dummy_texts) # Original CLIP result (1)

torch.onnx.export(model, (dummy_image, dummy_texts), "model.onnx", export_params=True,
  input_names=["IMAGE", "TEXT"],
  output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"],
  opset_version=14,
  dynamic_axes={
      "IMAGE": {
          0: "image_batch_size",
      },
      "TEXT": {
          0: "text_batch_size",
      },
      "LOGITS_PER_IMAGE": {
          0: "image_batch_size",
          1: "text_batch_size",
      },
      "LOGITS_PER_TEXT": {
          0: "text_batch_size",
          1: "image_batch_size",
      },
  }
)

and the model is saved.

When I test the model as :

# Now run onnxruntime to verify
import onnxruntime as ort

ort_sess = ort.InferenceSession("model.onnx")
result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], 
  {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()})

It gives me a list of length 2, one for each image and text and the result[0] has shape of [Batch,2].

Originally created by @deshwalmahesh on GitHub (Jul 2, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/263 I have quantized the model using your official snippet from the merge request and it working great. Now I want `CLIP` for encoding images and texts than classification. How can I do that? Below is the code to quantize and predict: ``` import clip import torch device = 'cuda' if torch.cuda.is_available() else 'cpu' model, preprocess = clip.load("RN50", device=device) # Load any model model = model.eval() # Inference Only img_size = model.visual.input_resolution dummy_image = torch.randn(10, 3, img_size, img_size).to(device) image_embedding = model.encode_image(dummy_image).to(device)) dummy_texts = clip.tokenize(["quick brown fox", "lorem ipsum"]).to(device) model.encode_text(dummy_texts) ``` and it works fine giving me `[Batch, 1024]` tensors for both for the loaded model. Now I have quantized my model in `Onnx` as: ``` model.forward(dummy_image,dummy_texts) # Original CLIP result (1) torch.onnx.export(model, (dummy_image, dummy_texts), "model.onnx", export_params=True, input_names=["IMAGE", "TEXT"], output_names=["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], opset_version=14, dynamic_axes={ "IMAGE": { 0: "image_batch_size", }, "TEXT": { 0: "text_batch_size", }, "LOGITS_PER_IMAGE": { 0: "image_batch_size", 1: "text_batch_size", }, "LOGITS_PER_TEXT": { 0: "text_batch_size", 1: "image_batch_size", }, } ) ``` and the model is saved. When I test the model as : ``` # Now run onnxruntime to verify import onnxruntime as ort ort_sess = ort.InferenceSession("model.onnx") result=ort_sess.run(["LOGITS_PER_IMAGE", "LOGITS_PER_TEXT"], {"IMAGE": dummy_image.numpy(), "TEXT": dummy_texts.numpy()}) ``` It gives me a list of length 2, one for each image and text and the `result[0]` has shape of `[Batch,2]`. ```

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#229

No description provided.

Rows
Columns