[GH-ISSUE #175] having trouble when doing batch inference #152

New issue

Open

opened 2026-10-31 21:11:45 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-31 21:11:45 +00:00

Owner

Originally created by @WilliamHoo on GitHub (Nov 7, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/175

Hi CLIP authors,

so i was trying to run the clip using the code below, but when i doing this in batch, i am having trouble with model.encode_text method, as it will show error message below
Screen Shot 2021-11-07 at 12 49 07 PM

my only difference with your sample code below is that, my text has a shape of (batch_size, num_class, n_ctx). My code will run perfectly is my input is of shape (num_class, n_ctx), but i was trying to speed up the process by doing it in batch.
Any advice or help will be really appreciated!

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

print("Label probs:", probs)  # prints: [[0.9927937  0.00421068 0.00299572]]

Originally created by @WilliamHoo on GitHub (Nov 7, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/175 Hi CLIP authors, so i was trying to run the clip using the code below, but when i doing this in batch, i am having trouble with model.encode_text method, as it will show error message below <img width="775" alt="Screen Shot 2021-11-07 at 12 49 07 PM" src="https://user-images.githubusercontent.com/21349011/140661449-44fac7c7-6809-47c8-a21a-ea0ad57aa699.png"> my only difference with your sample code below is that, my text has a shape of (batch_size, num_class, n_ctx). My code will run perfectly is my input is of shape (num_class, n_ctx), but i was trying to speed up the process by doing it in batch. Any advice or help will be really appreciated! ``` import torch import clip from PIL import Image device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device) image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device) text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device) with torch.no_grad(): image_features = model.encode_image(image) text_features = model.encode_text(text) logits_per_image, logits_per_text = model(image, text) probs = logits_per_image.softmax(dim=-1).cpu().numpy() print("Label probs:", probs) # prints: [[0.9927937 0.00421068 0.00299572]] ```

bsakaguchi commented

2026-10-31 21:11:46 +00:00

Author

Owner

@jongwook commented on GitHub (Nov 16, 2021):

Can you try reshaping the array to (batch_size * num_class, n_ctx) and feed it to the model?

@jongwook commented on GitHub (Nov 16, 2021): Can you try reshaping the array to (batch_size * num_class, n_ctx) and feed it to the model?

bsakaguchi commented

2026-10-31 21:11:46 +00:00

Author

Owner

@hanxiao commented on GitHub (Apr 10, 2022):

for batch inference, you can use https://github.com/jina-ai/clip-as-service/

@hanxiao commented on GitHub (Apr 10, 2022): for batch inference, you can use https://github.com/jina-ai/clip-as-service/

bsakaguchi commented

2026-10-31 21:11:46 +00:00

Author

Owner

@Angtrim commented on GitHub (Feb 28, 2023):

Did you find a solution?

@Angtrim commented on GitHub (Feb 28, 2023): Did you find a solution?

bsakaguchi commented

2026-10-31 21:11:46 +00:00

Author

Owner

@ahmadmustafaanis commented on GitHub (Apr 19, 2023):

The best working solution is to torch.stack to do batch inference.

imgs = [preprocess(img) for img in imgs]

logits_per_image, logits_per_text = model(
        torch.stack(imgs).to(device), tokenized_text
    )

@ahmadmustafaanis commented on GitHub (Apr 19, 2023): The best working solution is to torch.stack to do batch inference. ```python3 imgs = [preprocess(img) for img in imgs] logits_per_image, logits_per_text = model( torch.stack(imgs).to(device), tokenized_text ) ```

bsakaguchi commented

2026-10-31 21:11:46 +00:00

Author

Owner

@lucasjinreal commented on GitHub (Aug 14, 2024):

imgs = [preprocess(img) for img in imgs]

this func is very slow.

@lucasjinreal commented on GitHub (Aug 14, 2024): > imgs = [preprocess(img) for img in imgs] this func is very slow.

bsakaguchi referenced this issue

2026-10-31 21:12:52 +00:00

[GH-ISSUE #280] Discussion on the transposing trick used in transformer #243