openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #247] forward pass on torch.device("mps") #214

New issue
Open
opened 2026-10-31 21:12:32 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-31 21:12:32 +00:00
Owner
Copy link

Originally created by @dribnet on GitHub (May 24, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/247

Would be great to get this running on pytorch M1. Has anyone managed to get a forward pass on torch.device("mps")? Just posting this issue here in case others want to compare notes on this.

I gave it a shot but so far I haven't been able to get a forward pass through CLIP to work for images or text when tensors and model are on the mps device. Here's my stacktrace.

File ~/projects/CLIP/notebooks/../clip/model.py:366, in CLIP.encode_text(self, text, stops)
    364 x = x + self.positional_embedding.type(self.dtype)
    365 x = x.permute(1, 0, 2)  # NLD -> LND
--> 366 x = self.transformer(x)
    367 x = x.permute(1, 0, 2)  # LND -> NLD
    368 x = self.ln_final(x).type(self.dtype)

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/projects/CLIP/notebooks/../clip/model.py:206, in Transformer.forward(self, x)
    205 def forward(self, x: torch.Tensor):
--> 206     return self.resblocks(x)

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/container.py:139, in Sequential.forward(self, input)
    137 def forward(self, input):
    138     for module in self:
--> 139         input = module(input)
    140     return input

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/projects/CLIP/notebooks/../clip/model.py:193, in ResidualAttentionBlock.forward(self, x)
    192 def forward(self, x: torch.Tensor):
--> 193     x = x + self.attention(self.ln_1(x))
    194     x = x + self.mlp(self.ln_2(x))
    195     return x

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/projects/CLIP/notebooks/../clip/model.py:165, in LayerNorm.forward(self, x)
    163 print(x)
    164 print(x.shape)
--> 165 ret = super().forward(x.type(torch.float32))
    166 return ret.type(orig_type)

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/normalization.py:189, in LayerNorm.forward(self, input)
    188 def forward(self, input: Tensor) -> Tensor:
--> 189     return F.layer_norm(
    190         input, self.normalized_shape, self.weight, self.bias, self.eps)

File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/functional.py:2452, in layer_norm(input, normalized_shape, weight, bias, eps)
   2448 if has_torch_function_variadic(input, weight, bias):
   2449     return handle_torch_function(
   2450         layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
   2451     )
-> 2452 return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

As suggested for this error online, I tried calling .contiguous().reshape on the tensor before passing it to model.encode_image / model.encode_text but this didn't seem to help. My guess is that the memory layout on the tensor is somehow different that expected deep within that call but not sure if there is a workaround.

Originally created by @dribnet on GitHub (May 24, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/247 Would be great to get this running on [pytorch M1](https://pytorch.org/docs/master/notes/mps.html). Has anyone managed to get a forward pass on `torch.device("mps")`? Just posting this issue here in case others want to compare notes on this. I gave it a shot but so far I haven't been able to get a forward pass through CLIP to work for images or text when tensors and model are on the mps device. Here's my stacktrace. ```python File ~/projects/CLIP/notebooks/../clip/model.py:366, in CLIP.encode_text(self, text, stops) 364 x = x + self.positional_embedding.type(self.dtype) 365 x = x.permute(1, 0, 2) # NLD -> LND --> 366 x = self.transformer(x) 367 x = x.permute(1, 0, 2) # LND -> NLD 368 x = self.ln_final(x).type(self.dtype) File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs) 1126 # If we don't have any hooks, we want to skip the rest of the logic in 1127 # this function, and just call forward. 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(*input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], [] File ~/projects/CLIP/notebooks/../clip/model.py:206, in Transformer.forward(self, x) 205 def forward(self, x: torch.Tensor): --> 206 return self.resblocks(x) File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs) 1126 # If we don't have any hooks, we want to skip the rest of the logic in 1127 # this function, and just call forward. 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(*input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], [] File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/container.py:139, in Sequential.forward(self, input) 137 def forward(self, input): 138 for module in self: --> 139 input = module(input) 140 return input File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs) 1126 # If we don't have any hooks, we want to skip the rest of the logic in 1127 # this function, and just call forward. 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(*input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], [] File ~/projects/CLIP/notebooks/../clip/model.py:193, in ResidualAttentionBlock.forward(self, x) 192 def forward(self, x: torch.Tensor): --> 193 x = x + self.attention(self.ln_1(x)) 194 x = x + self.mlp(self.ln_2(x)) 195 return x File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs) 1126 # If we don't have any hooks, we want to skip the rest of the logic in 1127 # this function, and just call forward. 1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1129 or _global_forward_hooks or _global_forward_pre_hooks): -> 1130 return forward_call(*input, **kwargs) 1131 # Do not call functions when jit is used 1132 full_backward_hooks, non_full_backward_hooks = [], [] File ~/projects/CLIP/notebooks/../clip/model.py:165, in LayerNorm.forward(self, x) 163 print(x) 164 print(x.shape) --> 165 ret = super().forward(x.type(torch.float32)) 166 return ret.type(orig_type) File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/modules/normalization.py:189, in LayerNorm.forward(self, input) 188 def forward(self, input: Tensor) -> Tensor: --> 189 return F.layer_norm( 190 input, self.normalized_shape, self.weight, self.bias, self.eps) File /opt/anaconda3/envs/pixray2/lib/python3.9/site-packages/torch/nn/functional.py:2452, in layer_norm(input, normalized_shape, weight, bias, eps) 2448 if has_torch_function_variadic(input, weight, bias): 2449 return handle_torch_function( 2450 layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps 2451 ) -> 2452 return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead. ``` As suggested for this error online, I tried calling `.contiguous().reshape` on the tensor before passing it to `model.encode_image` / `model.encode_text` but this didn't seem to help. My guess is that the memory layout on the tensor is somehow different that expected deep within that call but not sure if there is a workaround.
bsakaguchi commented 2026-10-31 21:12:33 +00:00
Author
Owner
Copy link

@lowfuel commented on GitHub (Jun 13, 2022):

I haven't seen that. I get this error on even the most basic implementation:

RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

<!-- gh-comment-id:1154070610 --> @lowfuel commented on GitHub (Jun 13, 2022): I haven't seen that. I get this error on even the most basic implementation: `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#214
No description provided.