[GH-ISSUE #91] Layer Norm before Residual #82

New issue

Closed

opened 2026-10-31 21:10:55 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:10:55 +00:00

Owner

Originally created by @skull8888888 on GitHub (Apr 10, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/91

Hello, I noticed that on line 186 in model.py you put layer norm before MultiHeadAttention and Residual, as opposed to putting Layer Norm after Residual as described in the original Transformer paper.

def forward(self, x: torch.Tensor):
    x = x + self.attention(self.ln_1(x))
    x = x + self.mlp(self.ln_2(x))
    return x

In my research I found this paper which argues that such configuration improves convergence and removes the need for careful warmup.

In my experiment when I used your implementation of Transformer Block for seq2seq model 1 layer works pretty nice, however when I stuck 2 layers performance significantly degrades.

So my questions are: What is the reasoning behind it? And how to make it work with stacked layers?

Originally created by @skull8888888 on GitHub (Apr 10, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/91 Hello, I noticed that on line 186 in `model.py` you put `layer norm` before `MultiHeadAttention` and Residual, as opposed to putting Layer Norm after Residual as described in the original Transformer paper. ``` def forward(self, x: torch.Tensor): x = x + self.attention(self.ln_1(x)) x = x + self.mlp(self.ln_2(x)) return x ``` In my research I found this [paper](https://openreview.net/pdf?id=B1x8anVFPr) which argues that such configuration improves convergence and removes the need for careful warmup. In my experiment when I used your implementation of Transformer Block for seq2seq model 1 layer works pretty nice, however when I stuck 2 layers performance significantly degrades. So my questions are: What is the reasoning behind it? And how to make it work with stacked layers?

bsakaguchi closed this issue

2026-10-31 21:10:55 +00:00

bsakaguchi commented

2026-10-31 21:10:56 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 21, 2021):

The linked paper shows some reasoning behind using normalizations in the residual branch as opposed to the main trunk. One intuition is that keeping an identity mapping flow helps stabilize the training, as suggested by He et al. (2016), as opposed to going through multiple layer normalization ops proportional to the depth. This approach is adopted in most of OpenAI models since GPT-2, as outlined in page 4, column 2 of the GPT-2 paper and Figure 8d of the Jukebox paper.

As for your 1-layer vs 2-layer results, I can't say anything definite because it's not a precise science in the first place, and your mileage may vary vastly depending on various factors such as model architecture, training hyperparameters, datasets, etc.

@jongwook commented on GitHub (Apr 21, 2021): The linked paper shows some reasoning behind using normalizations in the residual branch as opposed to the main trunk. One intuition is that keeping an identity mapping flow helps stabilize the training, as suggested by [He et al. (2016)](https://arxiv.org/abs/1603.05027), as opposed to going through multiple layer normalization ops proportional to the depth. This approach is adopted in most of OpenAI models since GPT-2, as outlined in [page 4, column 2 of the GPT-2 paper](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf) and [Figure 8d of the Jukebox paper](https://arxiv.org/pdf/2005.00341.pdf). As for your 1-layer vs 2-layer results, I can't say anything definite because it's not a precise science in the first place, and your mileage may vary vastly depending on various factors such as model architecture, training hyperparameters, datasets, etc.

bsakaguchi referenced this issue

2026-10-31 21:10:58 +00:00

[GH-ISSUE #93] Is CLIP capable for object detection task? #86

bsakaguchi referenced this issue

2026-10-31 21:11:34 +00:00

[GH-ISSUE #155] Is there away where CLIP gives use multiple outputs like Multiple Object Detection #133

bsakaguchi referenced this issue

2026-10-31 21:11:38 +00:00

[GH-ISSUE #162] Questions about the spatial features of vision encoder in CLIP #140

bsakaguchi referenced this issue

2026-10-31 21:13:04 +00:00

[GH-ISSUE #299] Clip draw bounding box #256