openai/CLIP
1
0
Fork 0
mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00
Code Issues 274 Projects Packages Wiki Activity

[GH-ISSUE #429] encode_text gives different clip features for the same text, single batch vs mutliple batch #367

New issue
Open
opened 2026-10-31 21:14:25 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-31 21:14:25 +00:00
Owner
Copy link

Originally created by @KevinNWalker on GitHub (Feb 27, 2024).
Original GitHub issue: https://github.com/openai/CLIP/issues/429

I seem to get different results from encode_text when providing text as a single batch, or as part of several batches.

See the code sample below:

` import clip
import numpy

device = 'cuda'
clip_model, _ = clip.load('ViT-B/32', device)

clip_model.eval()

# Process 'I am happy' as a single batch
text_1 = clip.tokenize( 'I am happy', truncate=True).to(device)
feature_1 = clip_model.encode_text(text_1)
feature_1_np = feature_1.detach().cpu().numpy()
text_1_np = text_1.detach().cpu().numpy()

# Process 'I am happy' with a second batch
text_2 = clip.tokenize(['I am happy', 'I am happy'], truncate=True).to(device)
feature_2 = clip_model.encode_text(text_2)
feature_2_np = feature_2.detach().cpu().numpy()[0]
text_2_np = text_2.detach().cpu().numpy()[0]

print(f'Max diff in tokens {numpy.abs(text_2_np-text_1_np).max()}')
print(f'Max diff in features {numpy.abs(feature_2_np-feature_1_np).max()}')`

When I run this I get the following results:

Max diff in tokens 0 Max diff in features 0.000732421875

Is this to be expected or am I using the code incorrectly?

Many thanks

Originally created by @KevinNWalker on GitHub (Feb 27, 2024). Original GitHub issue: https://github.com/openai/CLIP/issues/429 I seem to get different results from encode_text when providing text as a single batch, or as part of several batches. See the code sample below: ` import clip import numpy device = 'cuda' clip_model, _ = clip.load('ViT-B/32', device) clip_model.eval() # Process 'I am happy' as a single batch text_1 = clip.tokenize( 'I am happy', truncate=True).to(device) feature_1 = clip_model.encode_text(text_1) feature_1_np = feature_1.detach().cpu().numpy() text_1_np = text_1.detach().cpu().numpy() # Process 'I am happy' with a second batch text_2 = clip.tokenize(['I am happy', 'I am happy'], truncate=True).to(device) feature_2 = clip_model.encode_text(text_2) feature_2_np = feature_2.detach().cpu().numpy()[0] text_2_np = text_2.detach().cpu().numpy()[0] print(f'Max diff in tokens {numpy.abs(text_2_np-text_1_np).max()}') print(f'Max diff in features {numpy.abs(feature_2_np-feature_1_np).max()}')` When I run this I get the following results: `Max diff in tokens 0 Max diff in features 0.000732421875` Is this to be expected or am I using the code incorrectly? Many thanks
bsakaguchi commented 2026-10-31 21:14:26 +00:00
Author
Owner
Copy link

@bonjour-npy commented on GitHub (Mar 2, 2024):

Hi there👋

I think your code is correct and the result of your code is also correct.

To the best of my knowledge (I can't be certain it's 100% right), the tokenization of the text in CLIP model is taking corresponding values from the fixed dict vocab. In other words, if the input text is the same, the tokenizer will always return the same result. That's why text_1, text_2[0] and text_2[1] are exactly the same.

But when it comes to function encode_text, it's generated by nn.Embedding, and the output of the Embedding Layer may be affected by the batch_size and context of the input or something else like that.

Here's a simple test:

import torch
import numpy
import clip

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
clip_model, _ = clip.load('ViT-B/32', device)

clip_model.eval()

text_1 = clip.tokenize('I am happy', truncate=True).to(device)
feature_1 = clip_model.encode_text(text_1)

text_2 = clip.tokenize(['I am happy', 'I am happy'], truncate=True).to(device)
feature_2 = clip_model.encode_text(text_2)

text_3 = clip.tokenize(['I am happy', 'I am happy', 'I am happy'], truncate=True).to(device)
feature_3 = clip_model.encode_text(text_3)

text_4 = clip.tokenize(['I am happy', 'I am sad', 'I am angry'], truncate=True).to(device)
feature_4 = clip_model.encode_text(text_4)

print((text_1 - text_2[0]).sum(), '\n', (text_1[0] - text_3[0]).sum(), '\n', (text_1[0] - text_4[0]).sum())
print((feature_1 - feature_2[0]).sum(), '\n', (feature_1[0] - feature_3[0]).sum(), '\n', (feature_1[0] - feature_4[0]).sum())

And my output is shown below:

tensor(0, device='cuda:0') 
 tensor(0, device='cuda:0') 
 tensor(0, device='cuda:0')
tensor(0.0089, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>) 
 tensor(0.0180, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>) 
 tensor(0.0180, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>)

We can jump to a conclusion (not meticulous enough apparently) from the result of text3 and text4, the context didn't affect the output of encode_text, it was the batch_size.

If you'd like to further communicate, feel free to reach out to me at nipeiyang@163.com.

<!-- gh-comment-id:1974837213 --> @bonjour-npy commented on GitHub (Mar 2, 2024): Hi there👋 I think your code is correct and the result of your code is also correct. To the best of my knowledge (I can't be certain it's 100% right), the tokenization of the text in CLIP model is taking corresponding values from the `fixed` dict `vocab`. In other words, if the input text is the same, the tokenizer will always return the same result. That's why `text_1`, `text_2[0]` and `text_2[1]` are exactly the same. But when it comes to function `encode_text`, it's generated by `nn.Embedding`, and the output of the Embedding Layer may be affected by the batch_size and context of the input or something else like that. Here's a simple test: ```python import torch import numpy import clip device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') clip_model, _ = clip.load('ViT-B/32', device) clip_model.eval() text_1 = clip.tokenize('I am happy', truncate=True).to(device) feature_1 = clip_model.encode_text(text_1) text_2 = clip.tokenize(['I am happy', 'I am happy'], truncate=True).to(device) feature_2 = clip_model.encode_text(text_2) text_3 = clip.tokenize(['I am happy', 'I am happy', 'I am happy'], truncate=True).to(device) feature_3 = clip_model.encode_text(text_3) text_4 = clip.tokenize(['I am happy', 'I am sad', 'I am angry'], truncate=True).to(device) feature_4 = clip_model.encode_text(text_4) print((text_1 - text_2[0]).sum(), '\n', (text_1[0] - text_3[0]).sum(), '\n', (text_1[0] - text_4[0]).sum()) print((feature_1 - feature_2[0]).sum(), '\n', (feature_1[0] - feature_3[0]).sum(), '\n', (feature_1[0] - feature_4[0]).sum()) ``` And my output is shown below: ```powershell tensor(0, device='cuda:0') tensor(0, device='cuda:0') tensor(0, device='cuda:0') tensor(0.0089, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>) tensor(0.0180, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>) tensor(0.0180, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward0>) ``` We can jump to a conclusion (not meticulous enough apparently) from the result of text3 and text4, the context didn't affect the output of encode_text, it was the batch_size. If you'd like to further communicate, feel free to reach out to me at `nipeiyang@163.com`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/pin-github-workflow-refs-20260325-132838
jongwook/issue-396
jongwook-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
openai/CLIP#367
No description provided.