[GH-ISSUE #114] Embedding is not deterministic

bsakaguchi commented

2026-10-31 21:11:11 +00:00

Owner

Originally created by @ArrowLuo on GitHub (Jun 9, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/114

Thanks for this amazing model.

I encountered a reproduce issue when extracting features. The results are different randomly with a small probability. Is there something wrong with my running?

A toy example is shown as the last block in colab.

test_tensor = None
model.eval()
with torch.no_grad():
  for i in range(10):
    image = preprocess(Image.open(raw_path)).unsqueeze(0).cuda()
    batch_size = np.random.randint(2, 10)
    image = image.expand(batch_size, -1, -1, -1)

    image_features = model.encode_image(image)
    print(image_features.size())
    
    sequence_output_cpu = image_features.detach().cpu().numpy()
    if test_tensor is None:
      test_tensor = sequence_output_cpu[0]
    for itm in sequence_output_cpu:
      if np.array_equal(test_tensor, itm):
        print("==========")
      else:
        print("**********")

The results are like,

torch.Size([3, 512])
==========
==========
==========
torch.Size([4, 512])
**********
**********
**********
**********
...

Thanks.

Originally created by @ArrowLuo on GitHub (Jun 9, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/114 Thanks for this amazing model. I encountered a reproduce issue when extracting features. The results are different randomly with a small probability. Is there something wrong with my running? A toy example is shown as the last block in [colab](https://colab.research.google.com/drive/1XFMxkjlcLAUkeaPUQoyxZynCNulHsifP?usp=sharing). ``` test_tensor = None model.eval() with torch.no_grad(): for i in range(10): image = preprocess(Image.open(raw_path)).unsqueeze(0).cuda() batch_size = np.random.randint(2, 10) image = image.expand(batch_size, -1, -1, -1) image_features = model.encode_image(image) print(image_features.size()) sequence_output_cpu = image_features.detach().cpu().numpy() if test_tensor is None: test_tensor = sequence_output_cpu[0] for itm in sequence_output_cpu: if np.array_equal(test_tensor, itm): print("==========") else: print("**********") ``` The results are like, ``` torch.Size([3, 512]) ========== ========== ========== torch.Size([4, 512]) ********** ********** ********** ********** ... ``` Thanks.

bsakaguchi closed this issue

2026-10-31 21:11:11 +00:00

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jun 9, 2021):

Have you read about this ? https://pytorch.org/docs/stable/notes/randomness.html

Btw I seen your code and a bit confused. So you breaking a single image to multiple different patches, and then compare the embedding between the different patches to the first patches ?
in this partnp.array_equal(test_tensor, itm):,
itm is every part of sequence_output_cpu while test_tensor is only the first part of sequence_output_cpu

@vinson2233 commented on GitHub (Jun 9, 2021): Have you read about this ? https://pytorch.org/docs/stable/notes/randomness.html Btw I seen your code and a bit confused. So you breaking a single image to multiple different patches, and then compare the embedding between the different patches to the first patches ? in this part`np.array_equal(test_tensor, itm):`, `itm` is every part of `sequence_output_cpu` while `test_tensor` is only the first part of `sequence_output_cpu`

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@ArrowLuo commented on GitHub (Jun 9, 2021):

Thanks for your quick reply.

Yes, I have read the link you provide.

The code runs 10 turns, each turn has the same images with different batch sizes. test_tensor is the feature from the first image of the first turn, which is used to compare with the images of the rest turns.

Because the images in each turn are the same, we naturally hope the CLIP returns the same features. But the results are not deterministic, which occurs without a pattern.

Are there some methods to make the generated feature deterministic from pretrained CLIP?

@ArrowLuo commented on GitHub (Jun 9, 2021): Thanks for your quick reply. Yes, I have read the link you provide. The code runs 10 turns, each turn has the same images with different batch sizes. `test_tensor` is the feature from the first image of the first turn, which is used to compare with the images of the rest turns. Because the images in each turn are the same, we naturally hope the CLIP returns the same features. But the results are not deterministic, which occurs without a pattern. Are there some methods to make the generated feature deterministic from pretrained CLIP?

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jun 9, 2021):

Unless I'm wrongly interpreted it, I think the code you use for testing is at fault here.

batch_size = np.random.randint(2, 10)
image = image.expand(batch_size, -1, -1, -1)

let's say for this turn batch_size is 2,
then the image is split into 2 (I don't know how maybe the first part is the top half of the image and the second part is the bottom half).

So the image itself is not same in each turn. I think what you want is to set the batch_size to 1 in this case, since the batch_size is relevant on multiple images, not single image

@vinson2233 commented on GitHub (Jun 9, 2021): Unless I'm wrongly interpreted it, I think the code you use for testing is at fault here. ```python batch_size = np.random.randint(2, 10) image = image.expand(batch_size, -1, -1, -1) ``` let's say for this turn batch_size is 2, then the image is split into 2 (I don't know how maybe the first part is the top half of the image and the second part is the bottom half). So the image itself is not same in each turn. I think what you want is to set the batch_size to 1 in this case, since the batch_size is relevant on multiple images, not single image

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@ArrowLuo commented on GitHub (Jun 9, 2021):

I print the image size at each turn, the size reflects the images are not split operation. The colab has been updated.

torch.Size([9, 3, 224, 224])
torch.Size([9, 512])
==========
==========
==========
==========
==========
==========
==========
==========
==========
torch.Size([5, 3, 224, 224])
torch.Size([5, 512])
**********
**********
**********
**********
**********

Also, when replace sequence_output_cpu = image_features.detach().cpu().numpy() with sequence_output_cpu = image .detach().cpu().numpy(), the print shows the images are the same. So, the difference is the output of the model.encode_image(image).

@ArrowLuo commented on GitHub (Jun 9, 2021): I print the image size at each turn, the size reflects the images are not split operation. The [colab](https://colab.research.google.com/drive/1XFMxkjlcLAUkeaPUQoyxZynCNulHsifP?usp=sharing) has been updated. ``` torch.Size([9, 3, 224, 224]) torch.Size([9, 512]) ========== ========== ========== ========== ========== ========== ========== ========== ========== torch.Size([5, 3, 224, 224]) torch.Size([5, 512]) ********** ********** ********** ********** ********** ``` Also, when replace `sequence_output_cpu = image_features.detach().cpu().numpy()` with `sequence_output_cpu = image .detach().cpu().numpy()`, the print shows the images are the same. So, the difference is the output of the `model.encode_image(image)`.

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@vinson2233 commented on GitHub (Jun 10, 2021):

I see, I mistook the expand method with the reshape method/

Hmmm, that's weird, the issue doesn't happen in my environment. I'm using this code :

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess  = clip.load("ViT-B/32", device=device,jit=True) 

import requests

with torch.no_grad():
    for i in range(10):
        img = preprocess(Image.open(requests.get("http://images.cocodataset.org/val2017/000000439715.jpg",stream=True).raw))
        batch_size = np.random.randint(2, 10)
        img = img.expand(batch_size,-1,-1,-1).to(device)
        result = model.encode_image(img)
        
        # check for discrepancy in GPU
        for i in range(batch_size):
            for j in range(i,batch_size):
                if not torch.eq(result[i],result[j]).all() :
                    print("Different")
       # check for discrepancy in CPU
        result_numpy = result.detach().cpu().numpy()
        for i in range(batch_size):
            for j in range(i,batch_size):
                if not np.array_equal(result_numpy[i],result_numpy[j]) :
                    print("Different")

This print nothing

@vinson2233 commented on GitHub (Jun 10, 2021): I see, I mistook the expand method with the reshape method/ Hmmm, that's weird, the issue doesn't happen in my environment. I'm using this code : ```python device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device,jit=True) import requests with torch.no_grad(): for i in range(10): img = preprocess(Image.open(requests.get("http://images.cocodataset.org/val2017/000000439715.jpg",stream=True).raw)) batch_size = np.random.randint(2, 10) img = img.expand(batch_size,-1,-1,-1).to(device) result = model.encode_image(img) # check for discrepancy in GPU for i in range(batch_size): for j in range(i,batch_size): if not torch.eq(result[i],result[j]).all() : print("Different") # check for discrepancy in CPU result_numpy = result.detach().cpu().numpy() for i in range(batch_size): for j in range(i,batch_size): if not np.array_equal(result_numpy[i],result_numpy[j]) : print("Different") ``` This print nothing

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@ArrowLuo commented on GitHub (Jun 10, 2021):

The outputs are the same in the same batch, but not at different turns even for the same image.
So, the testing should save the first tensor from the image, and use it to compare with the rest of the outputs.

The difference occurs randomly, so rerun several times if nothing prints.

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess  = clip.load("ViT-B/32", device=device,jit=True) 

import requests
model.eval()

# save the first turn tensor
first_gpu_tensor, first_cpu_tensor = None, None

with torch.no_grad():
    for i in range(10):
        img = preprocess(Image.open(requests.get("http://images.cocodataset.org/val2017/000000439715.jpg",stream=True).raw))
        batch_size = np.random.randint(2, 10)
        img = img.expand(batch_size,-1,-1,-1).to(device)
        result = model.encode_image(img)
        
        # save the first turn tensor
        if first_gpu_tensor is None: first_gpu_tensor = result[0]
        if first_cpu_tensor is None: first_cpu_tensor = result[0].detach().cpu().numpy()

        # check for discrepancy in GPU
        for i in range(batch_size):
            if not torch.eq(result[i],first_gpu_tensor).all() :
                print("Different")
       # check for discrepancy in CPU
        result_numpy = result.detach().cpu().numpy()
        for i in range(batch_size):
            if not np.array_equal(result_numpy[i],first_cpu_tensor) :
                print("Different")

@ArrowLuo commented on GitHub (Jun 10, 2021): The outputs are the same in the same batch, but not at different turns even for the same image. So, the testing should save the first tensor from the image, and use it to compare with the rest of the outputs. The difference occurs randomly, so rerun several times if nothing prints. ``` device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device,jit=True) import requests model.eval() # save the first turn tensor first_gpu_tensor, first_cpu_tensor = None, None with torch.no_grad(): for i in range(10): img = preprocess(Image.open(requests.get("http://images.cocodataset.org/val2017/000000439715.jpg",stream=True).raw)) batch_size = np.random.randint(2, 10) img = img.expand(batch_size,-1,-1,-1).to(device) result = model.encode_image(img) # save the first turn tensor if first_gpu_tensor is None: first_gpu_tensor = result[0] if first_cpu_tensor is None: first_cpu_tensor = result[0].detach().cpu().numpy() # check for discrepancy in GPU for i in range(batch_size): if not torch.eq(result[i],first_gpu_tensor).all() : print("Different") # check for discrepancy in CPU result_numpy = result.detach().cpu().numpy() for i in range(batch_size): if not np.array_equal(result_numpy[i],first_cpu_tensor) : print("Different") ```

bsakaguchi commented

2026-10-31 21:11:12 +00:00

Author

Owner

@jongwook commented on GitHub (Jun 30, 2021):

Related: #13. It can be frustrating, but there are unfortunately many factors in GPU we can't fully control. We empirically find that those differences do not make a significant impact on downstream tasks.

@jongwook commented on GitHub (Jun 30, 2021): Related: #13. It can be frustrating, but there are unfortunately many factors in GPU we can't fully control. We empirically find that those differences do not make a significant impact on downstream tasks.

Rows
Columns

[GH-ISSUE #114] Embedding is not deterministic #103