openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #468] RuntimeError: Input is too long for context length 77. No truncation passed #399

New issue

Open

opened 2026-10-31 21:14:40 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 21:14:40 +00:00

Owner

Originally created by @hessaAlawwad on GitHub (Oct 21, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/468

Hello,
So I am trying to embed text using CLIP, I got the error that my text is too long but from the huggingface I see that I can fix the variable:

max_position_embeddings (int, optional, defaults to 77) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).

any ideas on how can I do this?

My code is as follow:

import clip
client = OpenAI(api_key = "",)
#load model on device. The device you are running inference/training on is either a CPU or GPU if you have.
device = "cpu"
model, preprocess = clip.load("ViT-B/32",device=device)
def clip_get_features_from_single_image(image_path):
    image = preprocess(Image.open(image_path).convert("RGB"))
    image_input = torch.tensor(image).unsqueeze(0)
    with torch.no_grad():
        image_features = model.encode_image(image_input).float()
    return image_features

def clip_get_single_text_embedding(text):
    # inputs = clip.tokenize(text, context_length=77, truncate=True)
    inputs = clip.tokenize(text).to(device)
    with torch.no_grad():
        text_features = model.encode_text(inputs)
    return text_features

and do I need to adjust the image embeddings size too in order to calculate the similarty with the text?

Thanks in advance

Originally created by @hessaAlawwad on GitHub (Oct 21, 2024). Original GitHub issue: https://github.com/openai/CLIP/issues/468 Hello, So I am trying to embed text using CLIP, I got the error that my text is too long but from the huggingface I see that I can fix the variable: `max_position_embeddings (int, optional, defaults to 77) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).` any ideas on how can I do this? My code is as follow: ``` import clip client = OpenAI(api_key = "",) #load model on device. The device you are running inference/training on is either a CPU or GPU if you have. device = "cpu" model, preprocess = clip.load("ViT-B/32",device=device) def clip_get_features_from_single_image(image_path): image = preprocess(Image.open(image_path).convert("RGB")) image_input = torch.tensor(image).unsqueeze(0) with torch.no_grad(): image_features = model.encode_image(image_input).float() return image_features def clip_get_single_text_embedding(text): # inputs = clip.tokenize(text, context_length=77, truncate=True) inputs = clip.tokenize(text).to(device) with torch.no_grad(): text_features = model.encode_text(inputs) return text_features ``` and do I need to adjust the image embeddings size too in order to calculate the similarty with the text? Thanks in advance

bsakaguchi commented

2026-10-31 21:14:40 +00:00

Author

Owner

@DeadLineChaser commented on GitHub (Jan 7, 2025):

hello，i met the same problem as you did,have，do you know how to change 'max_position_embeddings ' now? waiting for your replay，thanks in advance

@DeadLineChaser commented on GitHub (Jan 7, 2025): hello，i met the same problem as you did,have，do you know how to change 'max_position_embeddings ' now? waiting for your replay，thanks in advance

bsakaguchi commented

2026-10-31 21:14:40 +00:00

Author

Owner

@99991 commented on GitHub (Jan 23, 2025):

Your code example is for openai/CLIP, but max_position_embeddings is for HuggingFace CLIP, which are different implementations.

Either way, there simply are no positional embeddings for tokens in longer texts. If you change context_length, you need to retrain the model (or at least the new parameters of positional_embedding).

github.com/openai/CLIP@dcba3cb2e2/clip/model.py (L291)

You have a few options:

Make your text shorter. Small LLMs are probably good enough for this task.
Train a new model.
Split your text into chunks smaller than 77 tokens, compute embeddings for each chunk and then average the embeddings, as described here for example. Note that embeddings get worse the more you average them, but it might be good enough for small texts.
Use a different model.

@99991 commented on GitHub (Jan 23, 2025): Your code example is for openai/CLIP, but `max_position_embeddings` is for HuggingFace CLIP, which are different implementations. Either way, there simply are no positional embeddings for tokens in longer texts. If you change `context_length`, you need to retrain the model (or at least the new parameters of `positional_embedding`). https://github.com/openai/CLIP/blob/dcba3cb2e2827b402d2701e7e1c7d9fed8a20ef1/clip/model.py#L291 You have a few options: * Make your text shorter. Small LLMs are probably good enough for this task. * Train a new model. * Split your text into chunks smaller than 77 tokens, compute embeddings for each chunk and then average the embeddings, as described [here](https://cookbook.openai.com/examples/embedding_long_inputs) for example. Note that embeddings get worse the more you average them, but it might be good enough for small texts. * Use a different model.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

openai/CLIP#399

No description provided.

Rows
Columns