mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #306] why tokenized_prompts.argmax =49407，'<|endoftext|>' #262
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#262
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Harzva on GitHub (Dec 16, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/306
Can <|endoftext|> represent global information of tokenized_prompts？why tokenized_prompts.argmax(dim=-1)
'<|endoftext|>': 49407
like cls_token of transformer?
Thanks
@jongwook commented on GitHub (Dec 17, 2022):
Yes.
argmaxselects the largest value in the input which is the EOT token. Because of the autoregressive mask, SOT (or CLS for the same purpose) at the beginning position will not be able to aggregate the global information, so the network is instead trained to produce the text features at the position of EOT.