[GH-ISSUE #211] Are linear probes for ResNet50 also before projection ? #184

New issue

Open

opened 2026-10-31 21:12:08 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:12:08 +00:00

Owner

Originally created by @YannDubs on GitHub (Feb 5, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/211

If I understand correctly when performing linear probing you take the representations before the linear projection heads. In the code, this can be done very nicely thanks to this line: github.com/openai/CLIP@3b473b0e68/clip/model.py (L233)

It is less clean for ResNet architecture, which makes me wonder whether you do the same for ResNets?

Here's the code I'm currently using assuming that I have to remove the projection head for both ViT and ResNet:

model, preprocess = clip.load(model, device, jit=False)  
encoder = model.visual  # only keep the image model

if hasattr(encoder, "proj"):
    encoder.proj = None
else:
    # set manually the projection head to identity while ensuring that still linear layer:
    N = encoder.attnpool.c_proj.in_features
    identity = torch.nn.Linear(N, N)
    nn.init.zeros_(identity.bias)
    identity.weight.data.copy_(torch.eye(N))
    encoder.attnpool.c_proj = identity

Thanks for the easily usable code + pretrained weights ♡

Originally created by @YannDubs on GitHub (Feb 5, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/211 If I understand correctly when performing linear probing you take the representations before the linear projection heads. In the code, this can be done very nicely thanks to this line: https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py#L233 It is less clean for ResNet architecture, which makes me wonder whether you do the same for ResNets? Here's the code I'm currently using assuming that I have to remove the projection head for both ViT and ResNet: ```python model, preprocess = clip.load(model, device, jit=False) encoder = model.visual # only keep the image model if hasattr(encoder, "proj"): encoder.proj = None else: # set manually the projection head to identity while ensuring that still linear layer: N = encoder.attnpool.c_proj.in_features identity = torch.nn.Linear(N, N) nn.init.zeros_(identity.bias) identity.weight.data.copy_(torch.eye(N)) encoder.attnpool.c_proj = identity ``` Thanks for the easily usable code + pretrained weights ♡

bsakaguchi commented

2026-10-31 21:12:09 +00:00

Author

Owner

@AverageName commented on GitHub (Feb 14, 2022):

I'm working on linear probe fine-tuning too, but I can't see where encoder.proj is being turned off in the official linear probe code, can you show me please?

@AverageName commented on GitHub (Feb 14, 2022): I'm working on linear probe fine-tuning too, but I can't see where encoder.proj is being turned off in the official linear probe code, can you show me please?

bsakaguchi commented

2026-10-31 21:12:09 +00:00

Author

Owner

@kinggongzilla commented on GitHub (Jun 6, 2022):

I have the same question, after reading the CLIP paper.
What exactly does "linear probing on ResNet" mean?

@kinggongzilla commented on GitHub (Jun 6, 2022): I have the same question, after reading the CLIP paper. What exactly does "linear probing on ResNet" mean?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#184

No description provided.

Rows
Columns