mirror of
https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git
synced 2026-11-05 00:27:30 +00:00
[GH-ISSUE #215] [Bug] RuntimeError: CUDA error: an illegal instruction was encountered #100
Labels
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#100
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @mr-segfault on GitHub (May 2, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/215
Using an Ubuntu system, 2x3060 (12g ea) and the latest version of RVC, commit
c4a1810During training, after a few epochs complete, a CUDA error is thrown:
@mr-segfault commented on GitHub (May 2, 2023):
After doing a little digging, other people online have encountered this bug with dual GPU setups; I have disabled dual GPU training (only training on one) and that seemed to work up until it hits the error associated with #167
INFO:user-test-3:[14000, 9.976276699833672e-05]
INFO:user-test-3:loss_disc=2.972, loss_gen=3.147, loss_fm=8.040,loss_mel=17.520, loss_kl=0.971
INFO:user-test-3:Saving model and optimizer state at epoch 20 to ./logs/user-test-3/G_14060.pth
INFO:user-test-3:Saving model and optimizer state at epoch 20 to ./logs/user-test-3/D_14060.pth
INFO:user-test-3:====> Epoch: 20
INFO:user-test-3:Training is done. The program is closed.
INFO:user-test-3:saving final ckpt:Success.
Traceback (most recent call last):
File "/home/user/rvc-test/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 534, in
main()
File "/home/user/rvc-test/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 50, in main
mp.spawn(
File "/home/user/rvc-test/Retrieval-based-Voice-Conversion-WebUI/venv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/user/rvc-test/Retrieval-based-Voice-Conversion-WebUI/venv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/home/user/rvc-test/Retrieval-based-Voice-Conversion-WebUI/venv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 149, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with exit code 149
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d
`
@Tps-F commented on GitHub (May 3, 2023):
What is the version of Pytorch?
This bug was found in 1.5.0
@mr-segfault commented on GitHub (May 3, 2023):
I am using a venv - so whatever was installed via the first preamble git instructions + pip requirements -r is what it should be using.
I used this to check: python -c "import torch; print(torch.version)"
Received: 2.0.0+cu117
@Tps-F commented on GitHub (May 3, 2023):
https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/167#issuecomment-1528941884
Which of these GPUs did this occur on?
@mr-segfault commented on GitHub (May 3, 2023):
#167 occurs in both environments I have - one environment is a dual 3060 (12gb ea) and another is a single 1660ti -- I initially try out things on the lower powered card and if they don't work move to the more memory available system and try there.
It looks like the 1660ti doesn't do several steps due to lack of resources, which is fine, but when it goes to the file handling process it crashes. It also happens on the 3060x2 system that can actually do all of the processing/training steps (except for the dual train issue identified in this 215 issue) but then crashes when file writing.
@mr-segfault commented on GitHub (May 11, 2023):
Dual GPUs still broken on current build
@paulerbear commented on GitHub (Oct 27, 2023):
Encountering same issue running a single RTX4090