openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #353] Linear probe pre or post-projection? #305

New issue
Open
opened 2026-10-31 21:13:40 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-31 21:13:40 +00:00
Owner
Copy link

Originally created by @ghost on GitHub (Apr 20, 2023).
Original GitHub issue: https://github.com/openai/CLIP/issues/353

Hi,

You mention in the paper that for linear probe evaluation For CLIP-ViT models, we used the features before the linear projection to the embedding space, which corresponds to I_f in Figure 3.. Yet in the README it uses features from model.encode_image

def get_features(dataset):
    all_features = []
    all_labels = []
    
    with torch.no_grad():
        for images, labels in tqdm(DataLoader(dataset, batch_size=100)):
            features = model.encode_image(images.to(device))

            all_features.append(features)
            all_labels.append(labels)

    return torch.cat(all_features).cpu().numpy(), torch.cat(all_labels).cpu().numpy()

which uses the visual module
github.com/openai/CLIP@d50d76daa6/clip/model.py (L340)
Which is a VisionTransformer with the self.proj projection layer to the common representation space used to match text and images, see
github.com/openai/CLIP@d50d76daa6/clip/model.py (L221)
and github.com/openai/CLIP@d50d76daa6/clip/model.py (L368)

Hence, it seems that you train LinearRegression post-projector (image features in the common embedding space). Could you clarify how you would recommend doing linear probe?

Originally created by @ghost on GitHub (Apr 20, 2023). Original GitHub issue: https://github.com/openai/CLIP/issues/353 Hi, You mention in the paper that for linear probe evaluation `For CLIP-ViT models, we used the features before the linear projection to the embedding space, which corresponds to I_f in Figure 3.`. Yet in the README it uses features from `model.encode_image` ``` def get_features(dataset): all_features = [] all_labels = [] with torch.no_grad(): for images, labels in tqdm(DataLoader(dataset, batch_size=100)): features = model.encode_image(images.to(device)) all_features.append(features) all_labels.append(labels) return torch.cat(all_features).cpu().numpy(), torch.cat(all_labels).cpu().numpy() ``` which uses the visual module https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/clip/model.py#L340 Which is a VisionTransformer with the `self.proj` projection layer to the common representation space used to match text and images, see https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/clip/model.py#L221 and https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/clip/model.py#L368 Hence, it seems that you train LinearRegression post-projector (image features in the common embedding space). Could you clarify how you would recommend doing linear probe?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#305
No description provided.