[GH-ISSUE #198] Can I know why clip model outputs are changed by batch size?? #173

New issue

Open

opened 2026-10-31 21:11:59 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 21:11:59 +00:00

Owner

Originally created by @larcane97 on GitHub (Dec 16, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/198

Hi, first of all, Thank you for sharing clip model coed and brilliant idea!! :)
during the model experiments, I found that clip model's outputs are changed by batch size,
But I couldn't come up with why this values are changed.. could you tell me why this is happening?

below is the code to reproduce !

# The length of token_labels : 11
with torch.no_grad():
    dummy = torch.rand((64,3,224,224)).cuda()
    image_logit,text_logit = clip_model.forward(dummy[0].unsqueeze(0),token_labels)
    batch_image_logit,batch_text_logit = clip_model.forward(dummy,token_labels)

print(image_logit.squeeze().tolist())
print(batch_image_logit[0].tolist())
print()
print(text_logit.squeeze().tolist())
print(batch_text_logit.T[0].tolist())

output:
[23.125, 22.1875, 21.375, 21.84375, 22.578125, 22.4375, 20.625, 23.203125, 20.5, 24.140625, 19.453125]
[23.125, 22.171875, 21.375, 21.828125, 22.5625, 22.4375, 20.625, 23.1875, 20.484375, 24.140625, 19.453125]

[23.125, 22.1875, 21.375, 21.84375, 22.578125, 22.4375, 20.625, 23.203125, 20.5, 24.140625, 19.453125]
[23.125, 22.171875, 21.375, 21.828125, 22.5625, 22.4375, 20.625, 23.1875, 20.484375, 24.140625, 19.453125]

Originally created by @larcane97 on GitHub (Dec 16, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/198 Hi, first of all, Thank you for sharing clip model coed and brilliant idea!! :) during the model experiments, I found that clip model's outputs are changed by batch size, But I couldn't come up with why this values are changed.. could you tell me why this is happening? below is the code to reproduce ! ``` # The length of token_labels : 11 with torch.no_grad(): dummy = torch.rand((64,3,224,224)).cuda() image_logit,text_logit = clip_model.forward(dummy[0].unsqueeze(0),token_labels) batch_image_logit,batch_text_logit = clip_model.forward(dummy,token_labels) print(image_logit.squeeze().tolist()) print(batch_image_logit[0].tolist()) print() print(text_logit.squeeze().tolist()) print(batch_text_logit.T[0].tolist()) ``` ``` output: [23.125, 22.1875, 21.375, 21.84375, 22.578125, 22.4375, 20.625, 23.203125, 20.5, 24.140625, 19.453125] [23.125, 22.171875, 21.375, 21.828125, 22.5625, 22.4375, 20.625, 23.1875, 20.484375, 24.140625, 19.453125] [23.125, 22.1875, 21.375, 21.84375, 22.578125, 22.4375, 20.625, 23.203125, 20.5, 24.140625, 19.453125] [23.125, 22.171875, 21.375, 21.828125, 22.5625, 22.4375, 20.625, 23.1875, 20.484375, 24.140625, 19.453125] ```

bsakaguchi commented

2026-10-31 21:11:59 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 11, 2022):

This is unfortunately true for most of CUDA programs. The runtime is free to select different implementations (for matrix multiplication, etc.) depending on the batch size, even between subsequent calls, and they might introduces slightly numerical differences similar to what you're seeing. Running models in fp32 will likely reduce the discrepancy, and check out some documentations regarding the determinism issue from NVIDIA and PyTorch which might also help.

@jongwook commented on GitHub (Apr 11, 2022): This is unfortunately true for most of CUDA programs. The runtime is free to select different implementations (for matrix multiplication, etc.) depending on the batch size, even between subsequent calls, and they might introduces slightly numerical differences similar to what you're seeing. Running models in fp32 will likely reduce the discrepancy, and check out some documentations regarding the determinism issue from [NVIDIA](https://github.com/NVIDIA/framework-determinism/blob/master/doc/pytorch.md) and [PyTorch](https://pytorch.org/docs/stable/notes/randomness.html) which might also help.

bsakaguchi referenced this issue

2026-10-31 21:15:18 +00:00

[PR #173] [MERGED] Fix PyTorch version check for nightly builds #160 #468

bsakaguchi referenced this issue

2026-10-31 21:15:18 +00:00

[PR #191] [MERGED] use pkg_resources for PyTorch version checks in notebooks #470