[GH-ISSUE #39] Hyperparameter sweep in Evaluation (linear probe)

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Owner

Originally created by @meigaoms on GitHub (Feb 10, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/39

Hi there,
I'm trying to reproduce evaluation scores in this paper, particularly table 10. A.3. Evaluation in Page 38 mentioned L2 regularization strength lambda is determined with a hyperparameter sweep.
(1) Only maximum 1,000 iterations is mentioned in L-BFGS. Do other parameters matter, like the learning rate?
(2) For parametric binary search, is the cost function monotonic to lambda?

Thank you!

Originally created by @meigaoms on GitHub (Feb 10, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/39 Hi there, I'm trying to reproduce evaluation scores in this paper, particularly table 10. A.3. Evaluation in Page 38 mentioned L2 regularization strength lambda is determined with a hyperparameter sweep. (1) Only maximum 1,000 iterations is mentioned in L-BFGS. Do other parameters matter, like the learning rate? (2) For parametric binary search, is the cost function monotonic to lambda? Thank you!

bsakaguchi closed this issue

2026-10-31 21:10:07 +00:00

bsakaguchi commented

2026-10-31 21:10:08 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 12, 2021):

Hi! Thanks for your interest in CLIP!

(1) We used scikit-learn's LogisticRegression implementation for linear probes, and we used the default arguments for anything not mentioned. Under the hood, it calls scipy.minimize and in turn scipy.optimize.fmin_l_bfgs_b, which performs line search along the quasi-Newton direction to find the minimum in the convex objective. So a learning rate / step length is not required.

(2) The cost function is convex in weights and linear in lambda (or C), so if you take min_w over it, the function is still convex in lambda. But cost functions using different lambda values are not comparable anyway. Also, this does not always guarantee that there will be a single peak accuracy/ROCAUC/etc. with respect to C.

However, we empirically found that the validation curves almost always have a single peak w.r.t. C, barring some noise, so we went ahead with parametric binary search to save time.

@jongwook commented on GitHub (Feb 12, 2021): Hi! Thanks for your interest in CLIP! (1) We used scikit-learn's [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) implementation for linear probes, and we used the default arguments for anything not mentioned. Under the hood, it calls `scipy.minimize` and in turn [`scipy.optimize.fmin_l_bfgs_b`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html), which performs line search along the quasi-Newton direction to find the minimum in the convex objective. So a learning rate / step length is not required. (2) The cost function is convex in weights and linear in lambda (or C), so if you take min_w over it, the function is still convex in lambda. But cost functions using different lambda values are not comparable anyway. Also, this does not always guarantee that there will be a single peak accuracy/ROCAUC/etc. with respect to C. However, we empirically found that the validation curves almost always have a single peak w.r.t. C, barring some noise, so we went ahead with parametric binary search to save time.

bsakaguchi commented

2026-10-31 21:10:08 +00:00

Author

Owner

@meigaoms commented on GitHub (Feb 16, 2021):

With Tensorflow flower 102 dataset and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way?

Thank you!

@meigaoms commented on GitHub (Feb 16, 2021): With [Tensorflow flower 102 dataset](https://www.tensorflow.org/datasets/catalog/oxford_flowers102) and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way? Thank you!

bsakaguchi commented

2026-10-31 21:10:08 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 16, 2021):

We experienced that linear probe accuracy can vary ±0.5% depending on various factors that we can't perfectly control, such as the exact versions of CUDA, cudnn, etc. Considering that, I'd say the 0.2%P difference is quite normal.

@jongwook commented on GitHub (Feb 16, 2021): We experienced that linear probe accuracy can vary ±0.5% depending on various factors that we can't perfectly control, such as the exact versions of CUDA, cudnn, etc. Considering that, I'd say the 0.2%P difference is quite normal.

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Author

Owner

@nikky4D commented on GitHub (Feb 16, 2021):

With Tensorflow flower 102 dataset and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way?

Thank you!

Would you have code for this? I am working on recreating the same thing but stuck at hyperparameter sweep

@nikky4D commented on GitHub (Feb 16, 2021): > With [Tensorflow flower 102 dataset](https://www.tensorflow.org/datasets/catalog/oxford_flowers102) and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way? > > Thank you! Would you have code for this? I am working on recreating the same thing but stuck at hyperparameter sweep

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Author

Owner

@meigaoms commented on GitHub (Feb 17, 2021):

We experienced that linear probe accuracy can vary ±0.5% depending on various factors that we can't perfectly control, such as the exact versions of CUDA, cudnn, etc. Considering that, I'd say the 0.2%P difference is quite normal.

Thanks for the explanation.
I ran the linear probe with ResNet50 and multiple datasets, accuracy/mean-per-class score from some datasets are the same as what shown in Table 10, such as Food-101, CIFAR10, CIFAR100, and MNIST. Some are not: FGVC Aircraft has the largest difference (CLIP paper reported 48.5, while I got 56.4). CLIP paper mentioned that "For the datasets that contains a validation split in addition to from the test split, we use the provided validation set to perform the hyperparameter search".
(1) After hyper parameter search, do you combine the validation set with training set together to train the final LinearRegression? Or use the training set only?
(2) I currently use transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop for train, test, and val (if any) datasets. (Thanks of answering my question about normalization in another thread) Since there are couple of different strategies, like moco, bit. Could you share details about data augmentation in linear probe?

@meigaoms commented on GitHub (Feb 17, 2021): > We experienced that linear probe accuracy can vary ±0.5% depending on various factors that we can't perfectly control, such as the exact versions of CUDA, cudnn, etc. Considering that, I'd say the 0.2%P difference is quite normal. Thanks for the explanation. I ran the linear probe with ResNet50 and multiple datasets, accuracy/mean-per-class score from some datasets are the same as what shown in Table 10, such as Food-101, CIFAR10, CIFAR100, and MNIST. Some are not: FGVC Aircraft has the largest difference (CLIP paper reported 48.5, while I got 56.4). CLIP paper mentioned that "For the datasets that contains a validation split in addition to from the test split, we use the provided validation set to perform the hyperparameter search". (1) After hyper parameter search, do you combine the validation set with training set together to train the final LinearRegression? Or use the training set only? (2) I currently use transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop for train, test, and val (if any) datasets. (Thanks of answering my question about normalization in another thread) Since there are couple of different strategies, like [moco](https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_lincls.py#L266), [bit](https://github.com/google-research/big_transfer/blob/49afe42338b62af9fbe18f0258197a33ee578a6b/bit_pytorch/train.py#L52). Could you share details about data augmentation in linear probe?

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Author

Owner

@meigaoms commented on GitHub (Feb 17, 2021):

With Tensorflow flower 102 dataset and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way?
Thank you!

Would you have code for this? I am working on recreating the same thing but stuck at hyperparameter sweep

Here is a sketch of my implementation:

    def hyperparameter_sweep():
        l2_lambda_list = np.logspace(-6, 6, num=97).tolist()
        
        def find_peak(l2_lambda_idx_list):
        """Calculate accuracy on all indexes and return the peak index"""
        return peak_idx

        l2_lambda_init_idx = [i for i,val in enumerate(l2_lambda_list) if val in set(np.logspace(-6, 6, num=7))]
        peak_idx = find_peak(l2_lambda_init_idx)
        step_span = 8
        while step_span > 0:
            left, right = max(peak_idx - step_span, 0), min(peak_idx + step_span, len(l2_lambda_list)-1)
            peak_idx = find_peak([left, peak_idx, right])
            step_span //= 2
     return l2_lambda_list[peak_idx]`

@meigaoms commented on GitHub (Feb 17, 2021): > > With [Tensorflow flower 102 dataset](https://www.tensorflow.org/datasets/catalog/oxford_flowers102) and Pytorch ResNet 50 pre-trained model model, transforms of ImageNet normalization and Resize(224,interpolation=PIL.Image.BICUBIC) + CenterCrop, fixed rand_state=0 in LogisticRegression, I expected to reproduce the number 90.8 in Table 10 exactly. However, I got 91.0. Did I miss some details? or Is this difference explainable in some way? > > Thank you! > > Would you have code for this? I am working on recreating the same thing but stuck at hyperparameter sweep Here is a sketch of my implementation: def hyperparameter_sweep(): l2_lambda_list = np.logspace(-6, 6, num=97).tolist() def find_peak(l2_lambda_idx_list): """Calculate accuracy on all indexes and return the peak index""" return peak_idx l2_lambda_init_idx = [i for i,val in enumerate(l2_lambda_list) if val in set(np.logspace(-6, 6, num=7))] peak_idx = find_peak(l2_lambda_init_idx) step_span = 8 while step_span > 0: left, right = max(peak_idx - step_span, 0), min(peak_idx + step_span, len(l2_lambda_list)-1) peak_idx = find_peak([left, peak_idx, right]) step_span //= 2 return l2_lambda_list[peak_idx]`

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 18, 2021):

(1) Yes. For the final test scores reported, we combined the validation split back to the training split, after the hyperparameter sweep.

(2) We didn't use any data augmentation, other than center-cropping and resizing the images as in the code. (During pre-training, we used random square crops) One exception is the KITTI dataset which contains very wide images and we stretch-resized the images into squares.

@jongwook commented on GitHub (Feb 18, 2021): (1) Yes. For the final test scores reported, we combined the validation split back to the training split, after the hyperparameter sweep. (2) We didn't use any data augmentation, other than center-cropping and resizing the images as in the code. (During pre-training, we used random square crops) One exception is the KITTI dataset which contains very wide images and we stretch-resized the images into squares.

bsakaguchi commented

2026-10-31 21:10:09 +00:00

Author

Owner

@meigaoms commented on GitHub (Feb 19, 2021):

Thank you very much!

@meigaoms commented on GitHub (Feb 19, 2021): Thank you very much!

Rows
Columns

[GH-ISSUE #39] Hyperparameter sweep in Evaluation (linear probe) #37