openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #9] Plans to release the model code? #7

New issue
Closed
opened 2026-10-31 21:09:50 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-31 21:09:50 +00:00
Owner
Copy link

Originally created by @thoppe on GitHub (Jan 11, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/9

Thanks for sharing OpenAI! I'd like to adapt the code to use in other downstream models, but I noticed you haven't defined it anywhere. I can peak at the forward pass a it a bit with model.code

def forward(self,
    image: Tensor,
    input: Tensor) -> Tuple[Tensor, Tensor]:
  _0 = self.logit_scale
  _1 = self.text_projection
  _2 = self.ln_final
  _3 = self.transformer
  _4 = self.positional_embedding
  _5 = self.token_embedding
  _6 = self.visual
  input0 = torch.to(image, torch.device("cuda"), 5, False, False, None)
  _7 = (_6).forward1(input0, )
  x = torch.to((_5).forward1(input, ), torch.device("cuda"), 5, False, False, None)
  _8 = torch.to(_4, torch.device("cuda"), 5, False, False, None)
  x4 = torch.add(x, _8, alpha=1)
  x5 = torch.permute(x4, [1, 0, 2])
  x6 = torch.permute((_3).forward1(x5, ), [1, 0, 2])
  x7 = torch.to((_2).forward1(x6, ), torch.device("cuda"), 5, False, False, None)
  _9 = ops.prim.NumToTensor(torch.size(x7, 0))
  _10 = torch.arange(annotate(number, _9), dtype=None, layout=0, device=torch.device("cpu"), pin_memory=False)
  _11 = torch.argmax(input, -1, False)
  _12 = torch.to(_10, dtype=4, layout=0, device=torch.device("cuda"), pin_memory=None, non_blocking=False, copy=False, memory_format=None)
  _13 = torch.to(_11, dtype=4, layout=0, device=torch.device("cuda"), pin_memory=None, non_blocking=False, copy=False, memory_format=None)
  _14 = annotate(List[Optional[Tensor]], [_12, _13])
  input1 = torch.matmul(torch.index(x7, _14), _1)
  image_features = torch.div(_7, torch.frobenius_norm(_7, [-1], True))
  _15 = torch.frobenius_norm(input1, [-1], True)
  text_features = torch.div(input1, _15)
  logit_scale = torch.exp(_0)
  _16 = torch.mul(logit_scale, image_features)
  _17 = torch.matmul(_16, torch.t(text_features))
  _18 = torch.matmul(torch.mul(logit_scale, text_features), torch.t(image_features))
  return (_17, _18)

but it's hard to work with and adapt! Any chance that the model code itself will be released?

Originally created by @thoppe on GitHub (Jan 11, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/9 Thanks for sharing OpenAI! I'd like to adapt the code to use in other downstream models, but I noticed you haven't defined it anywhere. I can peak at the forward pass a it a bit with `model.code` ``` python def forward(self, image: Tensor, input: Tensor) -> Tuple[Tensor, Tensor]: _0 = self.logit_scale _1 = self.text_projection _2 = self.ln_final _3 = self.transformer _4 = self.positional_embedding _5 = self.token_embedding _6 = self.visual input0 = torch.to(image, torch.device("cuda"), 5, False, False, None) _7 = (_6).forward1(input0, ) x = torch.to((_5).forward1(input, ), torch.device("cuda"), 5, False, False, None) _8 = torch.to(_4, torch.device("cuda"), 5, False, False, None) x4 = torch.add(x, _8, alpha=1) x5 = torch.permute(x4, [1, 0, 2]) x6 = torch.permute((_3).forward1(x5, ), [1, 0, 2]) x7 = torch.to((_2).forward1(x6, ), torch.device("cuda"), 5, False, False, None) _9 = ops.prim.NumToTensor(torch.size(x7, 0)) _10 = torch.arange(annotate(number, _9), dtype=None, layout=0, device=torch.device("cpu"), pin_memory=False) _11 = torch.argmax(input, -1, False) _12 = torch.to(_10, dtype=4, layout=0, device=torch.device("cuda"), pin_memory=None, non_blocking=False, copy=False, memory_format=None) _13 = torch.to(_11, dtype=4, layout=0, device=torch.device("cuda"), pin_memory=None, non_blocking=False, copy=False, memory_format=None) _14 = annotate(List[Optional[Tensor]], [_12, _13]) input1 = torch.matmul(torch.index(x7, _14), _1) image_features = torch.div(_7, torch.frobenius_norm(_7, [-1], True)) _15 = torch.frobenius_norm(input1, [-1], True) text_features = torch.div(input1, _15) logit_scale = torch.exp(_0) _16 = torch.mul(logit_scale, image_features) _17 = torch.matmul(_16, torch.t(text_features)) _18 = torch.matmul(torch.mul(logit_scale, text_features), torch.t(image_features)) return (_17, _18) ``` but it's hard to work with and adapt! Any chance that the model code itself will be released?
bsakaguchi commented 2026-10-31 21:09:50 +00:00
Author
Owner
Copy link

@weiyx16 commented on GitHub (Jan 11, 2021):

Hi, according to this issue #5 , I think they won't release a Non-JiT model. So I paid some effort to reproduce one, you can refer to this repo.

<!-- gh-comment-id:757716945 --> @weiyx16 commented on GitHub (Jan 11, 2021): Hi, according to this issue #5 , I think they won't release a Non-JiT model. So I paid some effort to reproduce one, you can refer to this [repo](https://github.com/weiyx16/CLIP-pytorch).
bsakaguchi commented 2026-10-31 21:09:50 +00:00
Author
Owner
Copy link

@SeanPedersen commented on GitHub (Jan 11, 2021):

What's the advantage of non-JIT model vs JIT model? Is there a difference in performance?

<!-- gh-comment-id:757849492 --> @SeanPedersen commented on GitHub (Jan 11, 2021): What's the advantage of non-JIT model vs JIT model? Is there a difference in performance?
bsakaguchi commented 2026-10-31 21:09:50 +00:00
Author
Owner
Copy link

@weiyx16 commented on GitHub (Jan 11, 2021):

As far as I know, a JIT model is a little faster during inference, and easy to deploy without dependency on certain python environment. a non-JIT model can enable you do any modification of the model..

<!-- gh-comment-id:757924782 --> @weiyx16 commented on GitHub (Jan 11, 2021): As far as I know, a JIT model is a little faster during inference, and easy to deploy without dependency on certain python environment. a non-JIT model can enable you do any modification of the model..
bsakaguchi commented 2026-10-31 21:09:51 +00:00
Author
Owner
Copy link

@thoppe commented on GitHub (Jan 11, 2021):

@weiyx16 this is amazing and thank you! My main reason for wanting this is to cache some of the intermediate results for faster inference -- from the current model signature it looks like "text" is inferred, but the "image" is processed every time before it's feed into the joint model.

Any tips on how you translated JIT into plain torch? I couldn't figure out the primitives like ops.prim.NumToTensor

Closing as this solves my problem.

<!-- gh-comment-id:758006745 --> @thoppe commented on GitHub (Jan 11, 2021): @weiyx16 this is amazing and thank you! My main reason for wanting this is to cache some of the intermediate results for faster inference -- from the current model signature it looks like "text" is inferred, but the "image" is processed every time before it's feed into the joint model. Any tips on how you translated JIT into plain torch? I couldn't figure out the primitives like `ops.prim.NumToTensor` Closing as this solves my problem.
bsakaguchi commented 2026-10-31 21:09:51 +00:00
Author
Owner
Copy link

@weiyx16 commented on GitHub (Jan 12, 2021):

I think if you dig further in pytorch src code, you will find ops.prim.NumToTensor but in format of ops::prim::NumToTensor, like here. I don't think it will make much difference, for these kind of ops can be passed, since we know the full architecture of the CLIP model.

<!-- gh-comment-id:758349026 --> @weiyx16 commented on GitHub (Jan 12, 2021): I think if you dig further in pytorch src code, you will find ```ops.prim.NumToTensor``` but in format of ```ops::prim::NumToTensor```, like [here](https://github.com/pytorch/pytorch/blob/8530c65e2560c5a5d464a5e21a70b42fb8469b6d/torch/csrc/jit/passes/erase_number_types.cpp#L46). I don't think it will make much difference, for these kind of ops can be passed, since we know the full architecture of the CLIP model.
bsakaguchi commented 2026-10-31 21:09:51 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Jan 12, 2021):

Hello all, thanks for your interest in CLIP model code. We are preparing to release the model code this week, which will allow users to inspect and modify the architecture more conveniently.

<!-- gh-comment-id:758385818 --> @jongwook commented on GitHub (Jan 12, 2021): Hello all, thanks for your interest in CLIP model code. We are preparing to release the model code this week, which will allow users to inspect and modify the architecture more conveniently.
bsakaguchi commented 2026-10-31 21:09:51 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Jan 12, 2021):

The model code is now available in model.py

<!-- gh-comment-id:759105818 --> @jongwook commented on GitHub (Jan 12, 2021): The model code is now available in [model.py](https://github.com/openai/CLIP/blob/main/model.py)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#7
No description provided.