mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[PR #271] [MERGED] Remove inefficient computation from AttentionPool2d Module #480
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#480
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
📋 Pull Request Information
Original PR: https://github.com/openai/CLIP/pull/271
Author: @jenkspt
Created: 7/21/2022
Status: ✅ Merged
Merged: 7/21/2022
Merged by: @jongwook
Base:
main← Head:fix-attention-pool2d📝 Commits (3)
8c0b98afix inefficient attention computation5eb1828remove erroneous formatting2fc776csimplified flatten📊 Changes
1 file changed (+3 additions, -4 deletions)
View changed files
📝
clip/model.py(+3 -4)📄 Description
This is a simple fix that removes unnecessary attention computation from the
AttentionPool2dModule.In the existing version, self attention is calculated on the full spatial + average embedding sequence with shape
[(HW+1), N, C]. In the proposed fix, attention is calculated with the average embedding[1, N, C]as the query and the spatial + average embedding sequence[(HW+1), N, C]as the key/value.I created this gist: https://gist.github.com/jenkspt/3a09cc150ab531781c6084c166047639 to demonstrate the equivalence of the existing implementation and the proposed one. There is parity in both the computation and the parameter state -- so there shouldn't be any breaking changes introduced.
I realize that
AttentionPool2dis only used once in the CLIP model, so this fix will not have a huge impact -- however I arrived here from https://github.com/openai/guided-diffusion/blob/main/guided_diffusion/unet.py#L22-L51, which is based on the clip version (and has the same problem) -- so I think there is the added benefit for posterity🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.