openai/CLIP

Fork 0

mirror of https://github.com/openai/CLIP.git synced 2026-11-05 01:37:26 +00:00

[GH-ISSUE #88] CUDA out of memory when running the training code on ubuntu 18.04 LTS #79

New issue

Closed

opened 2026-10-31 21:10:54 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-31 21:10:54 +00:00

Owner

Originally created by @lycutter on GitHub (Apr 9, 2021).
Original GitHub issue: https://github.com/openai/CLIP/issues/88

When I re-train the clip network on Windows, it works well. But I re-train it on unbuntu, it always reminds me of CUDA out of memory no matter how I set the batch size. The configuration information of Windows and ubuntu are showed as follows:
Windows:

the type of graphic card：GTX950M
pytorch version: 1.7.1
cuda version: 10.2

Ubuntu:

the type of graphic card: RTX2080
pytorch version: 1.7.1
cuda version: 10.1

Besides, other pytorch projects work well on the Ubuntu. So I think the graphic card also works well and the memory is also sufficient. I try to update the pytorch version to 1.8, but it seems that it only support the version 1.7.1.

Originally created by @lycutter on GitHub (Apr 9, 2021). Original GitHub issue: https://github.com/openai/CLIP/issues/88 When I re-train the clip network on Windows, it works well. But I re-train it on unbuntu, it always reminds me of CUDA out of memory no matter how I set the batch size. The configuration information of Windows and ubuntu are showed as follows: Windows: 1. the type of graphic card：GTX950M 2. pytorch version: 1.7.1 3. cuda version: 10.2 Ubuntu: 1. the type of graphic card: RTX2080 2. pytorch version: 1.7.1 3. cuda version: 10.1 Besides, other pytorch projects work well on the Ubuntu. So I think the graphic card also works well and the memory is also sufficient. I try to update the pytorch version to 1.8, but it seems that it only support the version 1.7.1.

bsakaguchi closed this issue

2026-10-31 21:10:54 +00:00

bsakaguchi commented

2026-10-31 21:10:54 +00:00

Author

Owner

@vinson2233 commented on GitHub (Apr 9, 2021):

Can you share your training code?
Have you look the CUDA memory when loading the data, loading the model and doing a single forward?

@vinson2233 commented on GitHub (Apr 9, 2021): Can you share your training code? Have you look the CUDA memory when loading the data, loading the model and doing a single forward?

bsakaguchi commented

2026-10-31 21:10:54 +00:00

Author

Owner

@lycutter commented on GitHub (Apr 9, 2021):

Can you share your training code?
Have you look the CUDA memory when loading the data, loading the model and doing a single forward?

Hello, my training code can be available at https://github.com/lycutter/CLIP/tree/master.
When I ran forward step on ubuntu, the allocated memory of graphic card didn't change.

@lycutter commented on GitHub (Apr 9, 2021): > Can you share your training code? > Have you look the CUDA memory when loading the data, loading the model and doing a single forward? Hello, my training code can be available at https://github.com/lycutter/CLIP/tree/master. When I ran forward step on ubuntu, the allocated memory of graphic card didn't change.

bsakaguchi commented

2026-10-31 21:10:54 +00:00

Author

Owner

@dragen1860 commented on GitHub (Apr 13, 2021):

@lycutter Hi, do you have multiple GPUs on ubuntu. YOu may need to block other GPUs in case of full GPU used.

@dragen1860 commented on GitHub (Apr 13, 2021): @lycutter Hi, do you have multiple GPUs on ubuntu. YOu may need to block other GPUs in case of full GPU used.

bsakaguchi commented

2026-10-31 21:10:55 +00:00

Author

Owner

@jongwook commented on GitHub (Apr 13, 2021):

Try memory profiling using pytorch_memlab to identify which operation causes the highest memory jump, and since you have the Windows version working, comparing the profiling results on Windows vs Linux would help. Setting environment variable CUDA_LAUNCH_BLOCKING=1 will also help getting more accurate profiling.

@jongwook commented on GitHub (Apr 13, 2021): Try memory profiling using [pytorch_memlab](https://github.com/Stonesjtu/pytorch_memlab) to identify which operation causes the highest memory jump, and since you have the Windows version working, comparing the profiling results on Windows vs Linux would help. [Setting environment variable `CUDA_LAUNCH_BLOCKING=1`](https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution) will also help getting more accurate profiling.

bsakaguchi commented

2026-10-31 21:10:55 +00:00

Author

Owner

@lycutter commented on GitHub (Apr 14, 2021):

Try memory profiling using pytorch_memlab to identify which operation causes the highest memory jump, and since you have the Windows version working, comparing the profiling results on Windows vs Linux would help. Setting environment variable CUDA_LAUNCH_BLOCKING=1 will also help getting more accurate profiling.

It works, thanks you!

@lycutter commented on GitHub (Apr 14, 2021): > Try memory profiling using [pytorch_memlab](https://github.com/Stonesjtu/pytorch_memlab) to identify which operation causes the highest memory jump, and since you have the Windows version working, comparing the profiling results on Windows vs Linux would help. [Setting environment variable `CUDA_LAUNCH_BLOCKING=1`](https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution) will also help getting more accurate profiling. It works, thanks you!

bsakaguchi referenced this issue

2026-10-31 21:10:56 +00:00

[GH-ISSUE #92] Half precision error on forward pass of RN50 and RN101 models #84