RVC-Project/Retrieval-based-Voice-Conversion-WebUI
1
0
Fork 0
mirror of https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git synced 2026-11-05 00:27:30 +00:00
Code Issues 588 Projects Packages Wiki Activity

[GH-ISSUE #1016] CUDA memory error, not using most of available memory #662

New issue
Open
opened 2026-10-20 10:01:41 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-20 10:01:41 +00:00
Owner
Copy link

Originally created by @Myst1ck on GitHub (Aug 15, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1016

Hello there!

I get the CUDA out of memory error, but when I look at my GPU's memory usage, I see that it only uses up to 4GB of memory before crashing, while my GPU's dedicated memory is 8GB (my shared memory is also 8GB if that's somehow relevant).
My GPU is 1070 GTX which is supported by CUDA and I have installed the latest drivers and all that.

Moreover, when I use cache it actually does work and uses up to 7GB without changing the batch size (I'm using 4).
I am using WSL 2 for running RVC on my machine, and as far as I know, the WSL isn't limiting GPU memory usage, so theoretically it should use up to 100% of my 8GB of memories.

If someone might have a clue about what's going on and how can I fix it, I would be happy to know!

Originally created by @Myst1ck on GitHub (Aug 15, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1016 Hello there! I get the CUDA out of memory error, but when I look at my GPU's memory usage, I see that it only **uses up to 4GB** of memory before crashing, while my **GPU's dedicated memory is 8GB** (my shared memory is also 8GB if that's somehow relevant). My GPU is 1070 GTX which is supported by CUDA and I have installed the latest drivers and all that. Moreover, when I use cache it actually does work and **uses up to 7GB** without changing the batch size (I'm using 4). I am using WSL 2 for running RVC on my machine, and as far as I know, the WSL isn't limiting GPU memory usage, so theoretically it should use up to 100% of my 8GB of memories. If someone might have a clue about what's going on and how can I fix it, I would be happy to know!
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@truszko1 commented on GitHub (Aug 15, 2023):

Can you post the entire error stack trace?

I am using WSL with 8GB VRAM (albeit using a 3080 laptop card).

<!-- gh-comment-id:1679749962 --> @truszko1 commented on GitHub (Aug 15, 2023): Can you post the entire error stack trace? I am using WSL with 8GB VRAM (albeit using a 3080 laptop card).
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@truszko1 commented on GitHub (Aug 15, 2023):

What does nvidia-smi display from WSL?

<!-- gh-comment-id:1679750587 --> @truszko1 commented on GitHub (Aug 15, 2023): What does `nvidia-smi` display from WSL?
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@truszko1 commented on GitHub (Aug 15, 2023):

What is your "resting" VRAM usage when your WSL is shut down? You can shut it down from powershell with wsl --shutdown. I do this often to clear the memory. Bear in mind that "Model Inference", for example, will load things into memory even if you're done using it.

<!-- gh-comment-id:1679751358 --> @truszko1 commented on GitHub (Aug 15, 2023): What is your "resting" VRAM usage when your WSL is shut down? You can shut it down from powershell with `wsl --shutdown`. I do this often to clear the memory. Bear in mind that "Model Inference", for example, will load things into memory even if you're done using it.
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@Myst1ck commented on GitHub (Aug 16, 2023):

Yes, sorry, here are all the stuff:

Entire error stack trace 

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/myst1ck/RVC/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 228, in run
    train_and_evaluate(
  File "/home/myst1ck/RVC/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 456, in train_and_evaluate
    scaler.scale(loss_gen_all).backward()
  File "/home/myst1ck/.local/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/home/myst1ck/.local/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

nvidia-smi in WSL (RVC not running) 

Wed Aug 16 10:38:18 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98.01              Driver Version: 536.99       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070        On  | 00000000:02:00.0  On |                  N/A |
|  0%   48C    P0              37W / 180W |    802MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

nvidia-smi in Windows (WSL not running) 

Wed Aug 16 10:32:50 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 536.99                 Driver Version: 536.99       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070      WDDM  | 00000000:02:00.0  On |                  N/A |
| 20%   48C    P8              11W / 180W |    775MiB /  8192MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1280    C+G   C:\Windows\System32\dwm.exe               N/A      |
|    0   N/A  N/A      4972    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A      8904    C+G   ....Search_cw5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A      9800    C+G   ...5n1h2txyewy\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     10568    C+G   ...t.LockApp_cw5n1h2txyewy\LockApp.exe    N/A      |
|    0   N/A  N/A     11656    C+G   ...b3d8bbwe\Microsoft.Media.Player.exe    N/A      |
|    0   N/A  N/A     11684    C+G   ...ekyb3d8bbwe\PhoneExperienceHost.exe    N/A      |
|    0   N/A  N/A     12372    C+G   ...CBS_cw5n1h2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     12444    C+G   ...GeForce Experience\NVIDIA Share.exe    N/A      |
|    0   N/A  N/A     12724    C+G   ...oogle\Chrome\Application\chrome.exe    N/A      |
|    0   N/A  N/A     13320    C+G   ...cal\Microsoft\OneDrive\OneDrive.exe    N/A      |
|    0   N/A  N/A     15568    C+G   ...AppData\Roaming\Spotify\Spotify.exe    N/A      |
|    0   N/A  N/A     16292    C+G   ...al\Discord\app-1.0.9016\Discord.exe    N/A      |
|    0   N/A  N/A     16600    C+G   ...siveControlPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A     17340    C+G   ...\Local\Programs\todoist\Todoist.exe    N/A      |
|    0   N/A  N/A     17364    C+G   ...inaries\Win64\EpicGamesLauncher.exe    N/A      |
|    0   N/A  N/A     17680    C+G   ...crosoft\Edge\Application\msedge.exe    N/A      |
|    0   N/A  N/A     18000    C+G   ...__8wekyb3d8bbwe\WindowsTerminal.exe    N/A      |
|    0   N/A  N/A     18632    C+G   ...ne\Binaries\Win64\EpicWebHelper.exe    N/A      |
|    0   N/A  N/A     18792    C+G   ...e Stream\79.0.2.0\GoogleDriveFS.exe    N/A      |
|    0   N/A  N/A     21080    C+G   ... Synapse 3 Host\Razer Synapse 3.exe    N/A      |
+---------------------------------------------------------------------------------------+

nvidia-smi running RVC without cache (latest point before the crush, batch size 1) 

Wed Aug 16 10:40:51 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98.01              Driver Version: 536.99       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070        On  | 00000000:02:00.0  On |                  N/A |
| 21%   53C    P2              47W / 180W |   4853MiB /  8192MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    0   N/A  N/A      4309      C   /python3.10                               N/A      |
+---------------------------------------------------------------------------------------+

After that, the app crush and the next nvidia-smi I will do will be the same as in the nvidia-smi WSL when no RVC is running.

nvidia-smi running RVC with cache (no crush, batch size 1) 

Wed Aug 16 10:44:46 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98.01              Driver Version: 536.99       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070        On  | 00000000:02:00.0  On |                  N/A |
| 31%   58C    P2              74W / 180W |   5682MiB /  8192MiB |     84%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    0   N/A  N/A      4427      C   /python3.10                               N/A      |
+---------------------------------------------------------------------------------------+

Usually I can do batch size 4 when I do caching and then it will go up to using about 7GB of GPU memory.

I am not using model inference or any of the RVC when I train the model, so I don't think the problem is because of that

<!-- gh-comment-id:1680124419 --> @Myst1ck commented on GitHub (Aug 16, 2023): Yes, sorry, here are all the stuff: <details><summary>Entire error stack trace</summary> <p> ```shell Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/myst1ck/RVC/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 228, in run train_and_evaluate( File "/home/myst1ck/RVC/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 456, in train_and_evaluate scaler.scale(loss_gen_all).backward() File "/home/myst1ck/.local/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/home/myst1ck/.local/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` </p> </details> <details><summary>nvidia-smi in WSL (RVC not running)</summary> <p> ```shell Wed Aug 16 10:38:18 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.98.01 Driver Version: 536.99 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1070 On | 00000000:02:00.0 On | N/A | | 0% 48C P0 37W / 180W | 802MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 23 G /Xwayland N/A | +---------------------------------------------------------------------------------------+ ``` </p> </details> <details><summary>nvidia-smi in Windows (WSL not running)</summary> <p> ```shell Wed Aug 16 10:32:50 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 536.99 Driver Version: 536.99 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1070 WDDM | 00000000:02:00.0 On | N/A | | 20% 48C P8 11W / 180W | 775MiB / 8192MiB | 1% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 1280 C+G C:\Windows\System32\dwm.exe N/A | | 0 N/A N/A 4972 C+G C:\Windows\explorer.exe N/A | | 0 N/A N/A 8904 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A | | 0 N/A N/A 9800 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A | | 0 N/A N/A 10568 C+G ...t.LockApp_cw5n1h2txyewy\LockApp.exe N/A | | 0 N/A N/A 11656 C+G ...b3d8bbwe\Microsoft.Media.Player.exe N/A | | 0 N/A N/A 11684 C+G ...ekyb3d8bbwe\PhoneExperienceHost.exe N/A | | 0 N/A N/A 12372 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A | | 0 N/A N/A 12444 C+G ...GeForce Experience\NVIDIA Share.exe N/A | | 0 N/A N/A 12724 C+G ...oogle\Chrome\Application\chrome.exe N/A | | 0 N/A N/A 13320 C+G ...cal\Microsoft\OneDrive\OneDrive.exe N/A | | 0 N/A N/A 15568 C+G ...AppData\Roaming\Spotify\Spotify.exe N/A | | 0 N/A N/A 16292 C+G ...al\Discord\app-1.0.9016\Discord.exe N/A | | 0 N/A N/A 16600 C+G ...siveControlPanel\SystemSettings.exe N/A | | 0 N/A N/A 17340 C+G ...\Local\Programs\todoist\Todoist.exe N/A | | 0 N/A N/A 17364 C+G ...inaries\Win64\EpicGamesLauncher.exe N/A | | 0 N/A N/A 17680 C+G ...crosoft\Edge\Application\msedge.exe N/A | | 0 N/A N/A 18000 C+G ...__8wekyb3d8bbwe\WindowsTerminal.exe N/A | | 0 N/A N/A 18632 C+G ...ne\Binaries\Win64\EpicWebHelper.exe N/A | | 0 N/A N/A 18792 C+G ...e Stream\79.0.2.0\GoogleDriveFS.exe N/A | | 0 N/A N/A 21080 C+G ... Synapse 3 Host\Razer Synapse 3.exe N/A | +---------------------------------------------------------------------------------------+ ``` </p> </details> <details><summary>nvidia-smi running RVC without cache (latest point before the crush, batch size 1)</summary> <p> ```shell Wed Aug 16 10:40:51 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.98.01 Driver Version: 536.99 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1070 On | 00000000:02:00.0 On | N/A | | 21% 53C P2 47W / 180W | 4853MiB / 8192MiB | 3% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 23 G /Xwayland N/A | | 0 N/A N/A 4309 C /python3.10 N/A | +---------------------------------------------------------------------------------------+ ``` After that, the app crush and the next nvidia-smi I will do will be the same as in the nvidia-smi WSL when no RVC is running. </p> </details> <details><summary>nvidia-smi running RVC with cache (no crush, batch size 1)</summary> <p> ```shell Wed Aug 16 10:44:46 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.98.01 Driver Version: 536.99 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1070 On | 00000000:02:00.0 On | N/A | | 31% 58C P2 74W / 180W | 5682MiB / 8192MiB | 84% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 23 G /Xwayland N/A | | 0 N/A N/A 4427 C /python3.10 N/A | +---------------------------------------------------------------------------------------+ ``` Usually I can do batch size 4 when I do caching and then it will go up to using about 7GB of GPU memory. </p> </details> I am not using model inference or any of the RVC when I train the model, so I don't think the problem is because of that
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@rubenvillarreal commented on GitHub (Jun 6, 2024):

Just to help anyone with a similar issue. I had similar Cuda memory issues (RTX 3060) when trying to use WSL Ubuntu system on windows 10. After trying many things, I gave up trying to use WSL and just installed everything on windows powershell. Finally got it to work. So if you have trouble with WSL system making your GPU run out of memory, then it is not your GPU, and you need to either resolve the driver issue (do not install linux nvidia driver, just use the windows nvidia driver on Windows OS), or you need to just start from scratch on powershell

<!-- gh-comment-id:2153268141 --> @rubenvillarreal commented on GitHub (Jun 6, 2024): Just to help anyone with a similar issue. I had similar Cuda memory issues (RTX 3060) when trying to use WSL Ubuntu system on windows 10. After trying many things, I gave up trying to use WSL and just installed everything on windows powershell. Finally got it to work. So if you have trouble with WSL system making your GPU run out of memory, then it is not your GPU, and you need to either resolve the driver issue (do not install linux nvidia driver, just use the windows nvidia driver on Windows OS), or you need to just start from scratch on powershell
bsakaguchi commented 2026-10-20 10:01:42 +00:00
Author
Owner
Copy link

@DrBrule commented on GitHub (Dec 4, 2025):

I'm having the same issue on an A100 card, only for the Feature Extraction Step. Has anyone found a solution for this?

using A100, pytorch2.4.0

<!-- gh-comment-id:3609935752 --> @DrBrule commented on GitHub (Dec 4, 2025): I'm having the same issue on an A100 card, only for the Feature Extraction Step. Has anyone found a solution for this? using A100, pytorch2.4.0
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
20240604
v1
2.3.260718
2.2.231006
2.1.230814
2.0.230618
2.0.230528
1.2.230428
1.1.230416
1.0.230410
Labels
Clear labels   
AMD/Intel

   
bug

   
bug

   
bug

   
colab

   
dependencies

   
documentation

   
duplicate

   
enhancement

   
following up

   
good first issue

   
help wanted

   
invalid

   
question

   
stale

   
stale

   
stale

   
todo list

   
wontfix
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#662
No description provided.