[GH-ISSUE #262] What is the limit for text sequences? #227

New issue

Open

opened 2026-10-31 21:12:42 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 21:12:42 +00:00

Owner

Originally created by @smith-co on GitHub (Jun 29, 2022).
Original GitHub issue: https://github.com/openai/CLIP/issues/262

I am trying to apply CLIP on a very specific dataset and need to fine tune. I am doing fine tuning following the steps here https://github.com/openai/CLIP/issues/83.

But cannot figure out what is maximum size of the text sequence length. Can it handle large texts? Or I have to restict the text to a pre-defined limit.

Any help is appreciated.

Originally created by @smith-co on GitHub (Jun 29, 2022). Original GitHub issue: https://github.com/openai/CLIP/issues/262 I am trying to apply CLIP on a **very** specific dataset and need to fine tune. I am doing fine tuning following the steps here https://github.com/openai/CLIP/issues/83. But cannot figure out what is maximum size of the `text` sequence length. Can it handle large texts? Or I have to restict the text to a pre-defined limit. Any help is appreciated.

bsakaguchi commented

2026-10-31 21:12:43 +00:00

Author

Owner

@akskuchi commented on GitHub (Jul 19, 2022):

According to this function, it can tokenize sequences up to the length of 77 (i.e., 77 words/tokens) and raises a runtime exception for anything beyond.

@akskuchi commented on GitHub (Jul 19, 2022): According to [this function](https://github.com/openai/CLIP/blob/4d120f3ec35b30bd0f992f5d8af2d793aad98d2a/clip/clip.py#L206), it can tokenize sequences up to the length of `77` (i.e., 77 words/tokens) and raises a runtime exception for anything beyond.

bsakaguchi commented

2026-10-31 21:12:43 +00:00

Author

Owner

@hrithickcodes commented on GitHub (Jul 8, 2023):

The default length is 77, you might need to perform some more fine-tuning to accept larger text sequences. Below is how you can do it.

from transformers import (
    CLIPTextConfig,
    CLIPVisionConfig,
    CLIPTextModelWithProjection,
    CLIPVisionModelWithProjection
)

CLIP_CHECKPOINTS = "openai/clip-vit-base-patch32"  
PROJECTION_DIM = 512  # Replace with your desired projection dimension
padding_max_length = 100  # Replace with your desired maximum position embeddings

textConfig = CLIPTextConfig.from_pretrained(CLIP_CHECKPOINTS)
textConfig.projection_dim = PROJECTION_DIM
textConfig.max_position_embeddings = padding_max_length

visionConfig = CLIPVisionConfig.from_pretrained(CLIP_CHECKPOINTS)
visionConfig.projection_dim = PROJECTION_DIM

# Using CLIP text model with a projection head on top
clipTextModel = CLIPTextModelWithProjection.from_pretrained(
    pretrained_model_name_or_path=CLIP_CHECKPOINTS,
    config=textConfig,
    ignore_mismatched_sizes=True
)

# Using CLIP vision ViT model with a projection head on top 
clipVisionModel = CLIPVisionModelWithProjection.from_pretrained(
    pretrained_model_name_or_path=CLIP_CHECKPOINTS,
    config=visionConfig,
    ignore_mismatched_sizes=True
)

This code snippet can help.

@hrithickcodes commented on GitHub (Jul 8, 2023): The default length is 77, you might need to perform some more fine-tuning to accept larger text sequences. Below is how you can do it. ```python from transformers import ( CLIPTextConfig, CLIPVisionConfig, CLIPTextModelWithProjection, CLIPVisionModelWithProjection ) CLIP_CHECKPOINTS = "openai/clip-vit-base-patch32" PROJECTION_DIM = 512 # Replace with your desired projection dimension padding_max_length = 100 # Replace with your desired maximum position embeddings textConfig = CLIPTextConfig.from_pretrained(CLIP_CHECKPOINTS) textConfig.projection_dim = PROJECTION_DIM textConfig.max_position_embeddings = padding_max_length visionConfig = CLIPVisionConfig.from_pretrained(CLIP_CHECKPOINTS) visionConfig.projection_dim = PROJECTION_DIM # Using CLIP text model with a projection head on top clipTextModel = CLIPTextModelWithProjection.from_pretrained( pretrained_model_name_or_path=CLIP_CHECKPOINTS, config=textConfig, ignore_mismatched_sizes=True ) # Using CLIP vision ViT model with a projection head on top clipVisionModel = CLIPVisionModelWithProjection.from_pretrained( pretrained_model_name_or_path=CLIP_CHECKPOINTS, config=visionConfig, ignore_mismatched_sizes=True ) ``` This code snippet can help.

bsakaguchi commented

2026-10-31 21:12:43 +00:00

Author

Owner

@chuyihuan commented on GitHub (Jul 19, 2023):

Hello, I would like to ask where this code should be added? Is transformers a library? Looking forward to your response

@chuyihuan commented on GitHub (Jul 19, 2023): > Hello, I would like to ask where this code should be added? Is transformers a library? Looking forward to your response

bsakaguchi commented

2026-10-31 21:12:43 +00:00

Author

Owner

@hrithickcodes commented on GitHub (Jul 19, 2023):

@chuyihuan If you are looking forward to fine-tuning CLIP then the snippet can be useful. yes, transformers is a library.

@hrithickcodes commented on GitHub (Jul 19, 2023): @chuyihuan If you are looking forward to fine-tuning CLIP then the snippet can be useful. yes, transformers is a library.

bsakaguchi referenced this issue

2026-10-31 21:15:21 +00:00

[PR #227] [MERGED] fix utf8 username on windows #472