mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #51] Multimodal projection matrix with ModifiedResNet #46
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#46
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @PaulLerner on GitHub (Mar 3, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/51
Hi,
There's something unclear to me about the code:
It's seems quite clear that the image is projected in the multimodal space here with the VisualTransformer and that the text is projected there but what about ModifiedResNet? It doesn't seem to project the visual features in the multimodal space, did I miss something?
Bests,
@jongwook commented on GitHub (Mar 4, 2021):
It's not explicitly written as the other operations, but the projection matrix corresponding to the ResNet models is here, within the self-attention pooling operation. This reduces the dimensionality from the resnet width to the embedding dimension.
@PaulLerner commented on GitHub (Mar 4, 2021):
Oh ok, I thought there would be an extra layer after this, as in VisualTransformer, thanks !