mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #488] attention mask #412
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#412
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ZOUHAN1 on GitHub (Mar 11, 2025).
Original GitHub issue: https://github.com/openai/CLIP/issues/488
I would like to understand why masking is used in the text encoder. This doesn't seem necessary for CLIP since it does not perform an autoregressive task. Maybe my understanding is incomplete. The relevant code is located at line 286 in model.py.
@wadkisson commented on GitHub (Apr 3, 2025):
What up @ZOUHAN1. The paper released with the repo mentions on page five that "Masked self-attention was used in the text encoder to preserve the ability to initialize with a pre-trained language model or add language modeling as an auxiliary objective...". From what I'm hearing, the authors implemented masked attention just in case they wanted to experiment with using pre-trained models down the line.
@roysubhankar commented on GitHub (Sep 7, 2025):
Hi @ZOUHAN1, even though its not an AR task, masking is still needed to mask the PAD tokens or the tokens that correspond to id = 0. If masking was not used, the text encoder will attend to the PAD tokens (or tokens with id=0) and it has no meaning whatsoever.
Now you may ask, why there are PAD tokens in the first place? Its because the text encoder expects a fixed sequence length of 77 tokens in CLIP. If there aren't enough real tokens to reach the length 77, it will simply add these PAD tokens.
This is true for any kind of text transformers, not just CLIP. I hope it answers your question.