openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #459] Preprocessor - How does it work? #388

New issue
Open
opened 2026-10-31 21:14:35 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-31 21:14:35 +00:00
Owner
Copy link

Originally created by @whishei on GitHub (Aug 13, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/459

I have been exploring the source code of different preprocessors for ViT models and I am struggling to understand the exact pre-processing steps an image goes through in the ImageProcessor / CLIPProcessor . How is the image resized to 224x224? Is it similar to skimage.transform.resize() where it performs either up-sampling or down-sampling based on the image size? Thanks in advance.

Looking into the image_processing_clip.py through huggingface I found the following explanation:
"Resize an image. The shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio."

Originally created by @whishei on GitHub (Aug 13, 2024). Original GitHub issue: https://github.com/openai/CLIP/issues/459 I have been exploring the source code of different preprocessors for ViT models and I am struggling to understand the exact pre-processing steps an image goes through in the ImageProcessor / CLIPProcessor . How is the image resized to 224x224? Is it similar to skimage.transform.resize() where it performs either up-sampling or down-sampling based on the image size? Thanks in advance. Looking into the image_processing_clip.py through huggingface I found the following explanation: "Resize an image. The shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio."
bsakaguchi commented 2026-10-31 21:14:36 +00:00
Author
Owner
Copy link

@99991 commented on GitHub (Jan 23, 2025):

CLIP has been trained on images of size 224x224 and therefore works best at this size. If you want best results, your images have to be converted to this size also.

https://github.com/openai/CLIP/blob/main/clip/clip.py#L79-L86

First, the image is resized such that the size of the smaller side becomes 224. For example, if the image is of size (448, 2240), the smaller side is 448, so the image has to be resized by a factor of 2 down to (224, 1120). The next step is to crop the center from this image, so the top and bottom are discarded, leaving only a (224, 224) image.

Next, the range of the pixels is converted from [0, 255] (data type uint8) down to [0.0, 1.0] (float32).

Lastly, the image is normalized, which is cargo cult from back in the days when people hadn't figured out weight initialization yet.

Here is a preprocessing implementation without TorchVision using only the Pillow library and NumPy, which might be a bit easier to understand.

<!-- gh-comment-id:2609915965 --> @99991 commented on GitHub (Jan 23, 2025): CLIP has been trained on images of size 224x224 and therefore works best at this size. If you want best results, your images have to be converted to this size also. https://github.com/openai/CLIP/blob/main/clip/clip.py#L79-L86 First, the image is resized such that the size of the smaller side becomes 224. For example, if the image is of size (448, 2240), the smaller side is 448, so the image has to be resized by a factor of 2 down to (224, 1120). The next step is to crop the center from this image, so the top and bottom are discarded, leaving only a (224, 224) image. Next, the range of the pixels is converted from [0, 255] (data type uint8) down to [0.0, 1.0] (float32). Lastly, the image is [normalized](https://en.wikipedia.org/wiki/Normalization_(statistics)), which is [cargo cult](https://en.wikipedia.org/wiki/Cargo_cult) from back in the days when people hadn't figured out weight initialization yet. [Here](https://github.com/99991/NumPyCLIP/blob/main/numpyclip.py#L154-L184) is a preprocessing implementation without TorchVision using only the Pillow library and NumPy, which might be a bit easier to understand.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#388
No description provided.