Masked self-attention was used in the text encoder to preserve the ability to initialize with a pre-trained language model or add language modeling as an auxiliary objective, though exploration of this is left as future work.

@jongwook commented on GitHub (Nov 16, 2021): This was explained in section 2.4 of the paper: > Masked self-attention was used in the text encoder to preserve the ability to initialize with a pre-trained language model or add language modeling as an auxiliary objective, though exploration of this is left as future work.

bsakaguchi commented

2026-10-31 21:11:47 +00:00

Author

Owner

@LifeIsStrange commented on GitHub (May 2, 2022):

@jongwook Masked language modeling significantly reduce performance, see the alternative solution: https://arxiv.org/abs/1906.08237
Therefore, the successor of CLIP should be XLnet-like.

@LifeIsStrange commented on GitHub (May 2, 2022): @jongwook Masked language modeling significantly reduce performance, see the alternative solution: https://arxiv.org/abs/1906.08237 Therefore, the successor of CLIP should be XLnet-like.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#154

No description provided.

Rows
Columns