openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #55] problem using RN50x4 - tensor sizes #50

New issue
Closed
opened 2026-10-31 21:10:15 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-31 21:10:15 +00:00
Owner
Copy link

Originally created by @nikjetchev on GitHub (Mar 7, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/55

Dear all,

the newly released RN50x4 model gives me an error -- but my exact same code works ok with the RN 101 or ViT-B/32 models.
What could be wrong?

perceptor, preprocess = clip.load('RN50x4', jit=True)#gives error in encode_image()
#perceptor, preprocess = clip.load('RN101', jit=True)#works OK with encode_image()
#perceptor, preprocess = clip.load('ViT-B/32', jit=True)# works OK with encode_image()

perceptor.encode_image(torch.zeros(1,3,224,224))

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/multimodal/model/multimodal_transformer.py", line 19, in encode_image
    _0 = self.visual
    x = torch.to(image, torch.device("cuda:0"), 5, False, False, None)
    return (_0).forward(x, )
            ~~~~~~~~~~~ <--- HERE
  def encode_text(self: __torch__.multimodal.model.multimodal_transformer.Multimodal,
    input: Tensor) -> Tensor:
  File "code/__torch__/multimodal/model/modified_resnet.py", line 39, in forward
    _16 = (_10).forward2((_6).forward((_7).forward(_15, ), ), )
    _17 = (_3).forward((_4).forward((_5).forward(_16, ), ), )
    _18 = (_0).forward((_1).forward((_2).forward(_17, ), ), )
           ~~~~~~~~~~~ <--- HERE
    return _18
  def forward1(self: __torch__.multimodal.model.modified_resnet.ModifiedResNet,
  File "code/__torch__/multimodal/model/modified_resnet.py", line 143, in forward
    _81 = torch.slice(torch.unsqueeze(_80, 1), 2, 0, 9223372036854775807, 1)
    _82 = torch.to(_81, 5, False, False, None)
    x1 = torch.add(x0, _82, alpha=1)
         ~~~~~~~~~ <--- HERE
    in_proj_bias = torch.cat([_70, _69, _68], 0)
    tgt_len = ops.prim.NumToTensor(torch.size(x1, 0))

Traceback of TorchScript, original code (most recent call last):
/root/multimodal-pytorch/multimodal/model/modified_resnet.py(76): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/multimodal-pytorch/multimodal/model/checkpointing.py(61): checkpoint
/root/multimodal-pytorch/multimodal/model/modified_resnet.py(154): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/multimodal-pytorch/multimodal/model/multimodal_transformer.py(221): visual_forward
/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py(940): trace_module
export_torchscript_models.py(37): export_torchscript_models
/opt/conda/lib/python3.7/site-packages/fire/core.py(672): _CallAndUpdateTrace
/opt/conda/lib/python3.7/site-packages/fire/core.py(468): _Fire
/opt/conda/lib/python3.7/site-packages/fire/core.py(138): Fire
export_torchscript_models.py(43): <module>
RuntimeError: The size of tensor a (50) must match the size of tensor b (82) at non-singleton dimension 0
Originally created by @nikjetchev on GitHub (Mar 7, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/55 Dear all, the newly released RN50x4 model gives me an error -- but my exact same code works ok with the RN 101 or ViT-B/32 models. What could be wrong? perceptor, preprocess = clip.load('RN50x4', jit=True)#gives error in encode_image() #perceptor, preprocess = clip.load('RN101', jit=True)#works OK with encode_image() #perceptor, preprocess = clip.load('ViT-B/32', jit=True)# works OK with encode_image() perceptor.encode_image(torch.zeros(1,3,224,224)) ``` RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript, serialized code (most recent call last): File "code/__torch__/multimodal/model/multimodal_transformer.py", line 19, in encode_image _0 = self.visual x = torch.to(image, torch.device("cuda:0"), 5, False, False, None) return (_0).forward(x, ) ~~~~~~~~~~~ <--- HERE def encode_text(self: __torch__.multimodal.model.multimodal_transformer.Multimodal, input: Tensor) -> Tensor: File "code/__torch__/multimodal/model/modified_resnet.py", line 39, in forward _16 = (_10).forward2((_6).forward((_7).forward(_15, ), ), ) _17 = (_3).forward((_4).forward((_5).forward(_16, ), ), ) _18 = (_0).forward((_1).forward((_2).forward(_17, ), ), ) ~~~~~~~~~~~ <--- HERE return _18 def forward1(self: __torch__.multimodal.model.modified_resnet.ModifiedResNet, File "code/__torch__/multimodal/model/modified_resnet.py", line 143, in forward _81 = torch.slice(torch.unsqueeze(_80, 1), 2, 0, 9223372036854775807, 1) _82 = torch.to(_81, 5, False, False, None) x1 = torch.add(x0, _82, alpha=1) ~~~~~~~~~ <--- HERE in_proj_bias = torch.cat([_70, _69, _68], 0) tgt_len = ops.prim.NumToTensor(torch.size(x1, 0)) Traceback of TorchScript, original code (most recent call last): /root/multimodal-pytorch/multimodal/model/modified_resnet.py(76): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/multimodal-pytorch/multimodal/model/checkpointing.py(61): checkpoint /root/multimodal-pytorch/multimodal/model/modified_resnet.py(154): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/multimodal-pytorch/multimodal/model/multimodal_transformer.py(221): visual_forward /opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py(940): trace_module export_torchscript_models.py(37): export_torchscript_models /opt/conda/lib/python3.7/site-packages/fire/core.py(672): _CallAndUpdateTrace /opt/conda/lib/python3.7/site-packages/fire/core.py(468): _Fire /opt/conda/lib/python3.7/site-packages/fire/core.py(138): Fire export_torchscript_models.py(43): <module> RuntimeError: The size of tensor a (50) must match the size of tensor b (82) at non-singleton dimension 0 ```
bsakaguchi commented 2026-10-31 21:10:15 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Mar 8, 2021):

The RN50x4 model expects an 288x288 image input. You can use the preprocess transform returned by clip.load to center-crop and resize images as the model requires.

Please refer to the appendix section F of the paper for the input resolution and other hyperparameters corresponding to each model.

<!-- gh-comment-id:792392425 --> @jongwook commented on GitHub (Mar 8, 2021): The RN50x4 model expects an 288x288 image input. You can use the `preprocess` transform returned by `clip.load` to center-crop and resize images as the model requires. Please refer to the appendix section F of the [paper](https://arxiv.org/abs/2103.00020) for the input resolution and other hyperparameters corresponding to each model.
bsakaguchi commented 2026-10-31 21:10:15 +00:00
Author
Owner
Copy link

@nikjetchev commented on GitHub (Mar 9, 2021):

Great, thanks for the quick help, it all works now.

<!-- gh-comment-id:793692954 --> @nikjetchev commented on GitHub (Mar 9, 2021): Great, thanks for the quick help, it all works now.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#50
No description provided.