mirror of
https://github.com/openai/CLIP.git
synced 2026-11-05 01:37:26 +00:00
[GH-ISSUE #64] Evaluation with ImageNet #60
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
openai/CLIP#60
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @meigaoms on GitHub (Mar 22, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/64
Hi there,
"A logistic regression classifier using scikit-learn’s L-BFGS implementation" is used in A.3 Evaluation. It's a brilliant choice for its simplicity for a fair comparison. This classifier works well for most datasets, but takes me a very long time for ImageNet dataset, about 8 days with a single CPU. Could you shed some lights on how to speed it up?
Thank you in advance.
@jongwook commented on GitHub (Mar 22, 2021):
We used a quite beefy machine with many CPU cores to run each logistic regression taking about 2 days, but it took more than a week for us to finish the hyperparameter sweep. This was okay for this project as a one-off task, but we're looking into ways to perform logistic regression on distributed GPUs so that it can produce equivalent results as sklearn's while running faster.
Using Adam or SGD with momentum instead of L-BFGS allows to use faster mini-batch training instead of full-batch training, but it's tricky to get the optimal result from it, and the gap seems to vary between models/datasets. This was precisely the reason why we sticked to scikit-learn.
During earlier iterations on the project, we used a subset of 50,000 random samples from the dataset for validation, which can be run quite faster.
Btw, we haven't got a chance to look into #45 yet. Thanks for your patience!
@meigaoms commented on GitHub (Mar 25, 2021):
Exactly! The hyperparameter sweep takes most of the time. Thanks a lot.
I found cuML has a GPU version logistic regression with L-bfgs for multinomial classification. Unfortunately, its L-bfgs implementation with L2 regularization is not exactly the same as sklearn's, so it fails to produce an equivalent result still.