[GH-ISSUE #197] Torch TensortRT #169

New issue

Open

opened 2026-10-31 21:11:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-31 21:11:57 +00:00

Owner

Copy link

Originally created by @hfawaz on GitHub (Dec 16, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/197

Trying to use CLIP model with the new library Torch-TensorRT

We have encountered the following error:

Traceback (most recent call last):
  File "benchmark.py", line 155, in <module>
    trt_ts_module = torch_tensorrt.compile(scripted_model,
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 97, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 119, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/multimodal/model/multimodal_transformer.py", line 34, in forward
    x2 = torch.add(x1, torch.to(_4, 5, False, False, None), alpha=1)
    x3 = torch.permute((_3).forward(x2, ), [1, 0, 2])
    x4 = torch.permute((_2).forward(x3, ), [1, 0, 2])
                        ~~~~~~~~~~~ <--- HERE
    _15 = torch.slice(x4, 0, 0, 9223372036854775807, 1)
    x5 = torch.slice(torch.select(_15, 1, 0), 1, 0, 9223372036854775807, 1)
  File "code/__torch__/multimodal/model/multimodal_transformer/___torch_mangle_9477.py", line 8, in forward
  def forward(self: __torch__.multimodal.model.multimodal_transformer.___torch_mangle_9477.Transformer,
    x: Tensor) -> Tensor:
    return (self.resblocks).forward(x, )
            ~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
  def forward1(self: __torch__.multimodal.model.multimodal_transformer.___torch_mangle_9477.Transformer,
    x: Tensor) -> Tensor:
  File "code/__torch__/torch/nn/modules/container/___torch_mangle_9476.py", line 29, in forward
    _8 = getattr(self, "3")
    _9 = getattr(self, "2")
    _10 = (getattr(self, "1")).forward((getattr(self, "0")).forward(x, ), )
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    _11 = (_7).forward((_8).forward((_9).forward(_10, ), ), )
    _12 = (_4).forward((_5).forward((_6).forward(_11, ), ), )
  File "code/__torch__/multimodal/model/multimodal_transformer/___torch_mangle_9376.py", line 13, in forward
    _0 = self.mlp
    _1 = self.ln_2
    _2 = (self.attn).forward((self.ln_1).forward(x, ), )
          ~~~~~~~~~~~~~~~~~~ <--- HERE
    x0 = torch.add(x, _2, alpha=1)
    x1 = torch.add(x0, (_0).forward((_1).forward(x0, ), ), alpha=1)
  File "code/__torch__/torch/nn/modules/activation/___torch_mangle_9369.py", line 34, in forward
    _13 = torch.contiguous(k, memory_format=0)
    _14 = [-1, int(torch.mul(bsz, CONSTANTS.c0)), _9]
    k0 = torch.transpose(torch.view(_13, _14), 0, 1)
                         ~~~~~~~~~~ <--- HERE
    _15 = torch.contiguous(v, memory_format=0)
    _16 = [-1, int(torch.mul(bsz, CONSTANTS.c0)), _8]

Traceback of TorchScript, original code (most recent call last):
/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py(4265): multi_head_attention_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/activation.py(985): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(45): attention
/root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(48): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py(117): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(63): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(93): forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl
/root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(221): visual_forward
/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py(940): trace_module
<ipython-input-1-40b054242c5d>(36): export_torchscript_models
<ipython-input-2-808c11c4d1cf>(3): <module>
/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3418): run_code
/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3338): run_ast_nodes
/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3147): run_cell_async
/opt/conda/lib/python3.7/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(2923): _run_cell
/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(2878): run_cell
/opt/conda/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py(555): interact
/opt/conda/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py(564): mainloop
/opt/conda/lib/python3.7/site-packages/IPython/terminal/ipapp.py(356): start
/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py(845): launch_instance
/opt/conda/lib/python3.7/site-packages/IPython/__init__.py(126): start_ipython
/opt/conda/bin/ipython(8): <module>
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

I am running using the scripted version already

import clip 
from torchvision import transforms
import torch_tensorrt
import torch


device = "cuda:0"

batch_size = 4

clip_model_name = "ViT-B/32"

scripted_model , preprocess = clip.load(clip_model_name, device, jit=True)

scripted_model = scripted_model.visual.to(device)

preprocess = transforms.Compose([
    preprocess,
    lambda x: x.half()
    ])

trt_ts_module = torch_tensorrt.compile(scripted_model,
                inputs = [
                    torch_tensorrt.Input( # Specify input object with shape and dtype
                        shape=[batch_size, 3, 224, 224],
                        dtype=torch.half) # Datatype of input tensor. Allowed options torch.(float|half|int8|int32|bool)
                ])

Environment:

nvidia-dlprof-pytorch-nvtx @ file:///nvidia/opt/dlprof/bin/nvidia_dlprof_pytorch_nvtx-1.6.0-py3-none-any.whl
onnx @ file:///opt/pytorch/pytorch/third_party/onnx
pytorch-quantization==2.1.0
torch==1.10.0a0+0aef44c
torch-tensorrt @ file:///workspace/torch_tensorrt-1.1.0a0%2B733a4b1c-cp38-cp38-linux_x86_64.whl
torchtext @ file:///opt/pytorch/text
torchvision @ file:///opt/pytorch/vision
clip @ git+https://github.com/openai/CLIP.git@573315e83f07b53a61ff5098757e8fc885f1703e

I have build my docker image using base image 21.10 with nvidia driver 470.86.

docker build --build-arg BASE=21.10 -f docker/Dockerfile -t torch_tensorrt:latest .

Originally created by @hfawaz on GitHub (Dec 16, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/197 Trying to use CLIP model with the new library [Torch-TensorRT](https://github.com/NVIDIA/Torch-TensorRT) We have encountered the following error: ``` Traceback (most recent call last): File "benchmark.py", line 155, in <module> trt_ts_module = torch_tensorrt.compile(scripted_model, File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 97, in compile return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 119, in compile compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec)) RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript, serialized code (most recent call last): File "code/__torch__/multimodal/model/multimodal_transformer.py", line 34, in forward x2 = torch.add(x1, torch.to(_4, 5, False, False, None), alpha=1) x3 = torch.permute((_3).forward(x2, ), [1, 0, 2]) x4 = torch.permute((_2).forward(x3, ), [1, 0, 2]) ~~~~~~~~~~~ <--- HERE _15 = torch.slice(x4, 0, 0, 9223372036854775807, 1) x5 = torch.slice(torch.select(_15, 1, 0), 1, 0, 9223372036854775807, 1) File "code/__torch__/multimodal/model/multimodal_transformer/___torch_mangle_9477.py", line 8, in forward def forward(self: __torch__.multimodal.model.multimodal_transformer.___torch_mangle_9477.Transformer, x: Tensor) -> Tensor: return (self.resblocks).forward(x, ) ~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE def forward1(self: __torch__.multimodal.model.multimodal_transformer.___torch_mangle_9477.Transformer, x: Tensor) -> Tensor: File "code/__torch__/torch/nn/modules/container/___torch_mangle_9476.py", line 29, in forward _8 = getattr(self, "3") _9 = getattr(self, "2") _10 = (getattr(self, "1")).forward((getattr(self, "0")).forward(x, ), ) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE _11 = (_7).forward((_8).forward((_9).forward(_10, ), ), ) _12 = (_4).forward((_5).forward((_6).forward(_11, ), ), ) File "code/__torch__/multimodal/model/multimodal_transformer/___torch_mangle_9376.py", line 13, in forward _0 = self.mlp _1 = self.ln_2 _2 = (self.attn).forward((self.ln_1).forward(x, ), ) ~~~~~~~~~~~~~~~~~~ <--- HERE x0 = torch.add(x, _2, alpha=1) x1 = torch.add(x0, (_0).forward((_1).forward(x0, ), ), alpha=1) File "code/__torch__/torch/nn/modules/activation/___torch_mangle_9369.py", line 34, in forward _13 = torch.contiguous(k, memory_format=0) _14 = [-1, int(torch.mul(bsz, CONSTANTS.c0)), _9] k0 = torch.transpose(torch.view(_13, _14), 0, 1) ~~~~~~~~~~ <--- HERE _15 = torch.contiguous(v, memory_format=0) _16 = [-1, int(torch.mul(bsz, CONSTANTS.c0)), _8] Traceback of TorchScript, original code (most recent call last): /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py(4265): multi_head_attention_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/activation.py(985): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(45): attention /root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(48): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py(117): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(63): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(93): forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(709): _slow_forward /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py(725): _call_impl /root/workspace/multimodal-pytorch/multimodal/model/multimodal_transformer.py(221): visual_forward /opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py(940): trace_module <ipython-input-1-40b054242c5d>(36): export_torchscript_models <ipython-input-2-808c11c4d1cf>(3): <module> /opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3418): run_code /opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3338): run_ast_nodes /opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(3147): run_cell_async /opt/conda/lib/python3.7/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner /opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(2923): _run_cell /opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py(2878): run_cell /opt/conda/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py(555): interact /opt/conda/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py(564): mainloop /opt/conda/lib/python3.7/site-packages/IPython/terminal/ipapp.py(356): start /opt/conda/lib/python3.7/site-packages/traitlets/config/application.py(845): launch_instance /opt/conda/lib/python3.7/site-packages/IPython/__init__.py(126): start_ipython /opt/conda/bin/ipython(8): <module> RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead. ``` I am running using the scripted version already ``` import clip from torchvision import transforms import torch_tensorrt import torch device = "cuda:0" batch_size = 4 clip_model_name = "ViT-B/32" scripted_model , preprocess = clip.load(clip_model_name, device, jit=True) scripted_model = scripted_model.visual.to(device) preprocess = transforms.Compose([ preprocess, lambda x: x.half() ]) trt_ts_module = torch_tensorrt.compile(scripted_model, inputs = [ torch_tensorrt.Input( # Specify input object with shape and dtype shape=[batch_size, 3, 224, 224], dtype=torch.half) # Datatype of input tensor. Allowed options torch.(float|half|int8|int32|bool) ]) ``` Environment: ``` nvidia-dlprof-pytorch-nvtx @ file:///nvidia/opt/dlprof/bin/nvidia_dlprof_pytorch_nvtx-1.6.0-py3-none-any.whl onnx @ file:///opt/pytorch/pytorch/third_party/onnx pytorch-quantization==2.1.0 torch==1.10.0a0+0aef44c torch-tensorrt @ file:///workspace/torch_tensorrt-1.1.0a0%2B733a4b1c-cp38-cp38-linux_x86_64.whl torchtext @ file:///opt/pytorch/text torchvision @ file:///opt/pytorch/vision clip @ git+https://github.com/openai/CLIP.git@573315e83f07b53a61ff5098757e8fc885f1703e ``` I have build my docker image using base image `21.10` with nvidia driver `470.86`. ``` docker build --build-arg BASE=21.10 -f docker/Dockerfile -t torch_tensorrt:latest . ```

bsakaguchi referenced this issue 2026-10-31 21:15:18 +00:00

[PR #170] [CLOSED] Fixes #169: Compare version using integers #469

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

codex/pin-github-workflow-refs-20260325-132838

jongwook/issue-396

jongwook-patch-1

No results found.

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#169

No description provided.