[GH-ISSUE #37] Can't get GPU speedup

bsakaguchi commented

2026-10-31 21:10:06 +00:00

Owner

Originally created by @steremma on GitHub (Feb 8, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/37

It appears that the code fails to utilise GPUs on my setup: 8 V100 GPUs | nvidia driver 450 | cuda 11.0.

While torch correctly identifies the GPU (the 0th one but I can tweak this with CUDA_VISIBLE_DEVICES externally), when running nvidia-smi during calls to model.forward or encode_images I observe the GPU memory filling up but utilization stays at 0, while htop shows my cpu cores running. As a result I get the same timing results when device is "cpu" or "cuda", the only difference being that device = "cuda" will fail with GPU memory error if called for more than 500-1000 images.

Is this expected, or do you have any idea how to troubleshoot this?

Originally created by @steremma on GitHub (Feb 8, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/37 It appears that the code fails to utilise GPUs on my setup: **8 V100 GPUs | nvidia driver 450 | cuda 11.0**. While torch correctly identifies the GPU (the 0th one but I can tweak this with `CUDA_VISIBLE_DEVICES` externally), when running nvidia-smi during calls to `model.forward` or `encode_images` I observe the GPU memory filling up but utilization stays at 0, while `htop` shows my cpu cores running. As a result I get the same timing results when device is `"cpu"` or `"cuda"`, the only difference being that `device = "cuda"` will fail with GPU memory error if called for more than 500-1000 images. Is this expected, or do you have any idea how to troubleshoot this?

bsakaguchi closed this issue

2026-10-31 21:10:06 +00:00

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Author

Owner

@jongwook commented on GitHub (Feb 12, 2021):

GPU usage can be low depending on various factors, especially when the batch size is small, and it's normal to see a CPU core hitting 100% in htop during active communications between GPU.

Also, nvidia-smi calls may not give an accurate measurement and might not be measured at the exact moment when the GPU kernels are being executed.

Can you see if the issue persists when you run multiple batches in a loop, such as the CIFAR-100 example in README.md, using the code before the logistic regression? I see that on our V100 GPU it takes ~80 seconds to calculate train_features in GPU and ~15 minutes in CPU.

You can also try NVIDIA Visual Profiler to get a better sense of what's happening in GPU.

@jongwook commented on GitHub (Feb 12, 2021): GPU usage can be low depending on various factors, especially when the batch size is small, and it's normal to see a CPU core hitting 100% in `htop` during active communications between GPU. Also, `nvidia-smi` calls may not give an accurate measurement and might not be measured at the exact moment when the GPU kernels are being executed. Can you see if the issue persists when you run multiple batches in a loop, such as the [CIFAR-100 example in README.md](https://github.com/openai/CLIP#linear-probe-evaluation), using the code before the logistic regression? I see that on our V100 GPU it takes ~80 seconds to calculate `train_features` in GPU and ~15 minutes in CPU. You can also try [NVIDIA Visual Profiler](https://developer.nvidia.com/nvidia-visual-profiler) to get a better sense of what's happening in GPU.

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Author

Owner

@steremma commented on GitHub (Mar 3, 2021):

My code actually resembles the linked one as I am indeed trying to embed many (100k - 10m) images in a loop. The only difference is that I do .cpu() at the end of each batch in order to write to the file and release the memory. I tuned the batch size to 256 just below where it results in GPU-OOM error. Here is the simplified code:

    with open(output, "a", encoding="utf-8") as f:
        for i, batch in enumerate(batch_reader(data, batch_size)):
            batch = [path for path in batch if os.path.basename(path) not in existing]
            images = torch.stack([preprocess(Image.open(path)) for path in batch]).to(device)
            with torch.no_grad():
                # Is this .cpu() suspicious?
                embeddings = model.encode_image(images).cpu()
                embeddings = embeddings / embeddings.norm(dim=-1, keepdim=True)

            for path, embedding in zip(batch, embeddings):
                name = os.path.basename(path)
                f.write(name + "\t" + "\t".join([str(x) for x in embedding.tolist()]) + "\n")

@steremma commented on GitHub (Mar 3, 2021): My code actually resembles the linked one as I am indeed trying to embed many (100k - 10m) images in a loop. The only difference is that I do `.cpu()` at the end of each batch in order to write to the file and release the memory. I tuned the batch size to 256 just below where it results in GPU-OOM error. Here is the simplified code: ``` with open(output, "a", encoding="utf-8") as f: for i, batch in enumerate(batch_reader(data, batch_size)): batch = [path for path in batch if os.path.basename(path) not in existing] images = torch.stack([preprocess(Image.open(path)) for path in batch]).to(device) with torch.no_grad(): # Is this .cpu() suspicious? embeddings = model.encode_image(images).cpu() embeddings = embeddings / embeddings.norm(dim=-1, keepdim=True) for path, embedding in zip(batch, embeddings): name = os.path.basename(path) f.write(name + "\t" + "\t".join([str(x) for x in embedding.tolist()]) + "\n") ```

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Author

Owner

@nreimers commented on GitHub (Mar 19, 2021):

For me the bottleneck is the preprocess(Image.open()), this is sadly an extremely slow operation which is executed on the CPU.

Sadly did not find a solution yet how to speed up preprocess

@nreimers commented on GitHub (Mar 19, 2021): For me the bottleneck is the preprocess(Image.open()), this is sadly an extremely slow operation which is executed on the CPU. Sadly did not find a solution yet how to speed up preprocess

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Author

Owner

@jongwook commented on GitHub (Mar 23, 2021):

@steremma I suspect the string operations and file writing operations crammed in the last line are taking long and making the GPU underutilized. I'd recommend saving them in a .pt or .npy format, and use a separate conversion script if you must have the text format. It will probably work better if you collect the embeddings in a list and save all of them outside the for loop.

@nreimers That can be a problem when images are large. Pillow-SIMD may give you 2-3x boost in throughput, but in case you're working on a fixed dataset, I'd recommend resizing the images offline to match the model's input resolution (224px for smaller models), so that you can drop the expensive bicubic resize operations while loading the images. It'll load faster if you use PNG or npy format to save the resized images.

@jongwook commented on GitHub (Mar 23, 2021): @steremma I suspect the string operations and file writing operations crammed in the last line are taking long and making the GPU underutilized. I'd recommend saving them in a `.pt` or `.npy` format, and use a separate conversion script if you must have the text format. It will probably work better if you collect the embeddings in a list and save all of them outside the `for` loop. @nreimers That can be a problem when images are large. [Pillow-SIMD](https://pypi.org/project/Pillow-SIMD/) may give you 2-3x boost in throughput, but in case you're working on a fixed dataset, I'd recommend resizing the images offline to match the model's input resolution (224px for smaller models), so that you can drop the expensive bicubic resize operations while loading the images. It'll load faster if you use PNG or npy format to save the resized images.

bsakaguchi commented

2026-10-31 21:10:07 +00:00

Author

Owner

@steremma commented on GitHub (Mar 28, 2021):

The issue was indeed related to Image IO and preprocessing, thanks for all
the tips!

On Sun 28. Mar 2021 at 08:32, Jong Wook Kim @.***>
wrote:

Closed #37 https://github.com/openai/CLIP/issues/37.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openai/CLIP/issues/37#event-4517854685, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACG2NY5KGVVZ5GQGDBQAT23TF3EORANCNFSM4XJDLAZA
.

@steremma commented on GitHub (Mar 28, 2021): The issue was indeed related to Image IO and preprocessing, thanks for all the tips! On Sun 28. Mar 2021 at 08:32, Jong Wook Kim ***@***.***> wrote: > Closed #37 <https://github.com/openai/CLIP/issues/37>. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/openai/CLIP/issues/37#event-4517854685>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACG2NY5KGVVZ5GQGDBQAT23TF3EORANCNFSM4XJDLAZA> > . >

Rows
Columns

[GH-ISSUE #37] Can't get GPU speedup #35