openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #280] Discussion on the transposing trick used in transformer #243

New issue
Closed
opened 2026-10-31 21:12:51 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-31 21:12:51 +00:00
Owner
Copy link

Originally created by @Joy-Lunkad on GitHub (Aug 12, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/280

I was hoping to understand the benefits of the "double transpose trick" employed in model.py and its potential effects on layernorm inside and outside the transformer.

I would greatly appreciate if you could point me towards a good resource that explains this.

x = self.ln_pre(x)
x = x.permute(1, 0, 2)  # NLD -> LND
x = self.transformer(x)
x = x.permute(1, 0, 2)  # LND -> NLD
x = self.ln_post(x[:, 0, :])

Thank you!

Originally created by @Joy-Lunkad on GitHub (Aug 12, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/280 I was hoping to understand the benefits of the "double transpose trick" employed in [model.py](https://github.com/openai/CLIP/blob/main/clip/model.py) and its potential effects on layernorm inside and outside the transformer. I would greatly appreciate if you could point me towards a good resource that explains this. ```python3 x = self.ln_pre(x) x = x.permute(1, 0, 2) # NLD -> LND x = self.transformer(x) x = x.permute(1, 0, 2) # LND -> NLD x = self.ln_post(x[:, 0, :]) ``` Thank you!
bsakaguchi commented 2026-10-31 21:12:52 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Aug 12, 2022):

Hi, it was just to conform with the input shape expected by the nn.MultiheadAttention module, which is (L=sequence length, N=batch size, D=model width) by default. I just noticed that PyTorch has added the batch_first=True option in 1.9.0 which would eliminate the need to transpose.

<!-- gh-comment-id:1213572291 --> @jongwook commented on GitHub (Aug 12, 2022): Hi, it was just to conform with the input shape expected by the `nn.MultiheadAttention` module, which is `(L=sequence length, N=batch size, D=model width)` by default. I just noticed that PyTorch has added the `batch_first=True` option in 1.9.0 which would eliminate the need to transpose.
bsakaguchi commented 2026-10-31 21:12:52 +00:00
Author
Owner
Copy link

@Joy-Lunkad commented on GitHub (Aug 13, 2022):

Thank you for your reply! I have just a couple more questions :)
LN inside transformer:-

self.ln_1 = LayerNorm(d_model)

LN outside transformer:-

self.ln_pre = LayerNorm(width)

Am I missing something obvious here since these do not inform the layer norm module which axis is the batch axis that it needs to ignore?

There is also ln_1 directly after ln_pre which makes me wonder what benefits might taking the norm of a norm have.

<!-- gh-comment-id:1213638456 --> @Joy-Lunkad commented on GitHub (Aug 13, 2022): Thank you for your reply! I have just a couple more questions :) LN inside transformer:- ```python3 self.ln_1 = LayerNorm(d_model) ``` LN outside transformer:- ```python3 self.ln_pre = LayerNorm(width) ``` Am I missing something obvious here since these do not inform the [layer norm module](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html ) which axis is the batch axis that it needs to ignore? There is also ln_1 directly after ln_pre which makes me wonder what benefits might taking the norm of a norm have.
bsakaguchi commented 2026-10-31 21:12:52 +00:00
Author
Owner
Copy link

@jongwook commented on GitHub (Aug 13, 2022):

The LayerNorm module will use the last dimension(s), and in our case the model width is always the last dimension before or after the transposes.

It was pointed out in #152 (with a similar discussion about the redundancy of ln_pre) that common implementations of vision transformers don't use ln_pre. Model-wise, there's a slight difference with and without ln_pre, but I think it wouldn't make much difference when trained without ln_pre.

<!-- gh-comment-id:1213643558 --> @jongwook commented on GitHub (Aug 13, 2022): The LayerNorm module will use the last dimension(s), and in our case the model width is always the last dimension before or after the transposes. It was pointed out in #152 (with a similar discussion about the redundancy of `ln_pre`) that common implementations of vision transformers don't use `ln_pre`. Model-wise, there's a slight difference with and without `ln_pre`, but I think it wouldn't make much difference when trained without `ln_pre`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#243
No description provided.