openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #167] Reproduce zeroshot results on KITTI-Distance dataset. #147

New issue
Open
opened 2026-10-31 21:11:41 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-31 21:11:41 +00:00
Owner
Copy link

Originally created by @xcpeng on GitHub (Oct 27, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/167

Thank you for your work on CLIP!

I was trying to reproduce the zeroshot prediction results listed in Table 11 in the paper. Though I can reproduce most of the results in the Table 11, I found there are huge gaps on KITTI-Distance dataset.

We have tried:

  • Use JIT when loading CLIP model or not
  • Using different image preprocessing, i.e. center-crop or not center-crop
  • We have confirmed the order of categories in the promts.py is consistent with the dataset

But we can still can not reproduce the reported numbers in table 11. Any hints will be greatly appreciated, thank you!

Model name ResNet50 ResNet101 RN50x4 RN50x16 ViT-B/16 ViT-B/32
Dataset CLIP Ours Delta CLIP Ours Delta CLIP Ours Delta CLIP Ours Delta CLIP Ours Delta CLIP Ours Delta
KITTI-Distance 42.2 9.4 -24.4 33.3 30.7 -2.6 26.0 22.1 -3.9 33.9 24.1 -9.8 44.0 26.7 -17.3 39.4 28.9 -10.5

JIT applied or Not when loading CLIP model

Model name ResNet50 ResNet101 RN50x4 RN50x16 ViT-B/16 ViT-B/32
Dataset w/ JIT w/o JIT Delta w/ JIT w/o JIT Delta w/ JIT w/o JIT Delta w/ JIT w/o JIT Delta w/ JIT w/o JIT Delta w/ JIT w/o JIT Delta
KITTI-Distance 9.6 9.4 0.2 30.9 30.7 0.2 22.1 22.1 0 24.2 24.1 0.1 26.6 26.7 -0.1 29.0 28.9 0.1

Image Preprocessing: Center Crop v.s. No Center Crop

Model name ResNet50 ResNet101 RN50x4 RN50x16 ViT-B/16 ViT-B/32
Dataset w/ Crop w/o Crop Delta w/ Crop w/o Crop Delta w/ Crop w/o Crop Delta w/ Crop w/o Crop Delta w/ Crop w/o Crop Delta w/ Crop w/o Crop Delta
KITTI-Distance 17.6 9.4 -8.2 26.0 30.7 4.7 23.2 22.1 -1.1 23.1 24.1 1 26.0 26.7 0.7 27.6 28.9 1.3

Order of categories

Dataset Order Inconsistent Order Fixed
KITTI-Distance N NA
Originally created by @xcpeng on GitHub (Oct 27, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/167 Thank you for your work on CLIP! I was trying to reproduce the zeroshot prediction results listed in Table 11 in the paper. Though I can reproduce most of the results in the Table 11, I found there are huge gaps on **KITTI-Distance** dataset. We have tried: - Use JIT when loading CLIP model or not - Using different image preprocessing, i.e. center-crop or not center-crop - We have confirmed the order of categories in the promts.py is consistent with the dataset But we can still can not reproduce the reported numbers in table 11. Any hints will be greatly appreciated, thank you! | Model name | ResNet50 | | | ResNet101 | | |RN50x4 | | | RN50x16 | | | ViT-B/16 | | | ViT-B/32 | | | | ------------------------------- | -------- | ---- | ------ | -------- | ---- | ------ |-------- | ---- | ------ | --------| ---- | ------ | -------- | ---- | ------ | -------- | ---- | ------ | | Dataset | CLIP | Ours | Delta | CLIP | Ours | Delta | CLIP | Ours | Delta | CLIP | Ours | Delta | CLIP | Ours | Delta | CLIP | Ours | Delta | | KITTI-Distance | 42.2 | 9.4 | \-24.4 | 33.3 | 30.7| \-2.6 | 26.0 | 22.1 | \-3.9 | 33.9 | 24.1 | \-9.8 | 44.0 | 26.7 | \-17.3 | 39.4 | 28.9 | \-10.5 | ### JIT applied or Not when loading CLIP model | Model name | ResNet50 | | | ResNet101| | |RN50x4 | | | RN50x16 | | | ViT-B/16 | | | ViT-B/32 | | | | ------------------------------- | -------- | ---- | ------ | -------- | ---- | ------ |-------- | ---- | ------ | --------| ---- | ------ | -------- | ---- | ------ | -------- | ---- | ------ | | Dataset | w/ JIT |w/o JIT| Delta | w/ JIT |w/o JIT| Delta | w/ JIT |w/o JIT| Delta | w/ JIT |w/o JIT| Delta | w/ JIT |w/o JIT| Delta | w/ JIT |w/o JIT| Delta | | KITTI-Distance | 9.6 | 9.4 | 0.2 | 30.9 | 30.7 | 0.2 | 22.1 | 22.1 | 0 | 24.2 | 24.1 | 0.1 | 26.6 | 26.7 | -0.1 | 29.0 | 28.9 | 0.1 | ### Image Preprocessing: Center Crop v.s. No Center Crop | Model name | ResNet50 | | | ResNet101 | | |RN50x4 | | | RN50x16 | | | ViT-B/16 | | | ViT-B/32 | | | | ------------------------------- | -------- | ---- | ------ | -------- | ---- | ------ |-------- | ---- | ------ | -------- | ---- | ------ | -------- | ---- | ------ | -------- | ---- | ------ | | Dataset | w/ Crop |w/o Crop| Delta | w/ Crop |w/o Crop| Delta | w/ Crop |w/o Crop| Delta | w/ Crop |w/o Crop| Delta | w/ Crop |w/o Crop| Delta | w/ Crop |w/o Crop| Delta | | KITTI-Distance | 17.6 | 9.4 | -8.2 | 26.0 | 30.7 | 4.7 | 23.2 | 22.1 | -1.1 | 23.1 | 24.1 | 1 | 26.0 | 26.7 | 0.7 | 27.6 | 28.9 | 1.3 | ### Order of categories |Dataset | Order Inconsistent | Order Fixed | | --------------------------------- | ------------- | -------- | | KITTI-Distance | N | NA |
bsakaguchi commented 2026-10-31 21:11:42 +00:00
Author
Owner
Copy link

@gabrielilharco commented on GitHub (Apr 20, 2022):

Hey @xcpeng, I'm wondering if you ever got to the bottom of this? I'm currently trying to reproduce numbers for this datasets, and I'm getting similar numbers as what you present in your first table. Any advice is appreciated!

<!-- gh-comment-id:1103276117 --> @gabrielilharco commented on GitHub (Apr 20, 2022): Hey @xcpeng, I'm wondering if you ever got to the bottom of this? I'm currently trying to reproduce numbers for this datasets, and I'm getting similar numbers as what you present in your first table. Any advice is appreciated!
bsakaguchi commented 2026-10-31 21:11:42 +00:00
Author
Owner
Copy link

@msxingpeng commented on GitHub (Apr 20, 2022):

Hey @xcpeng, I'm wondering if you ever got to the bottom of this? I'm currently trying to reproduce numbers for this datasets, and I'm getting similar numbers as what you present in your first table. Any advice is appreciated!

No, I haven't fix this

<!-- gh-comment-id:1103378624 --> @msxingpeng commented on GitHub (Apr 20, 2022): > Hey @xcpeng, I'm wondering if you ever got to the bottom of this? I'm currently trying to reproduce numbers for this datasets, and I'm getting similar numbers as what you present in your first table. Any advice is appreciated! No, I haven't fix this
bsakaguchi commented 2026-10-31 21:11:42 +00:00
Author
Owner
Copy link

@haooooooqi commented on GitHub (Nov 11, 2022):

Hi please refer to my latest issue

<!-- gh-comment-id:1311323461 --> @haooooooqi commented on GitHub (Nov 11, 2022): Hi please refer to my latest issue
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#147
No description provided.