RVC-Project/Retrieval-based-Voice-Conversion-WebUI

Fork 0

mirror of https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git synced 2026-11-05 00:27:30 +00:00

[GH-ISSUE #1199] Training does not continue, but begins anew. #773

New issue

Open

opened 2026-10-20 10:02:54 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-20 10:02:54 +00:00

Owner

Originally created by @danekbiglike on GitHub (Sep 6, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1199

My training stopped due to a full disk, I cleared the space, entered all the same settings, it says that point 660 is loaded, but it starts anyway from the 1st epoch.

runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e "my-voice3.0" -sr 48k -f0 1 -bs 28 -g 0 -te 1000 -se 20 -pg pretrained_v2/f0G48k.pth -pd pretrained_v2/f0D48k.pth -l 0 -c 1 -sw 1 -v v2
INFO:my-voice3.0:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 28, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 17280, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 48000, 'filter_length': 2048, 'hop_length': 480, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs\\my-voice3.0/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [12, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs\\my-voice3.0', 'experiment_dir': './logs\\my-voice3.0', 'save_every_epoch': 20, 'name': 'my-voice3.0', 'total_epoch': 1000, 'pretrainG': 'pretrained_v2/f0G48k.pth', 'pretrainD': 'pretrained_v2/f0D48k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '48k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '1', 'if_cache_data_in_gpu': 1}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
gin_channels: 256 self.spk_embed_dim: 109
./logs\my-voice3.0\D_31680.pth
INFO:my-voice3.0:Loaded model weights
INFO:my-voice3.0:Loaded checkpoint './logs\my-voice3.0\D_31680.pth' (epoch 660)
INFO:my-voice3.0:loaded D
./logs\my-voice3.0\G_32160.pth
INFO:my-voice3.0:loaded pretrained pretrained_v2/f0G48k.pth
<All keys matched successfully>
INFO:my-voice3.0:loaded pretrained pretrained_v2/f0D48k.pth
<All keys matched successfully>
E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\autograd\__init__.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:337.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
INFO:my-voice3.0:Train Epoch: 1 [0%]
INFO:my-voice3.0:[0, 0.0001]
INFO:my-voice3.0:loss_disc=3.944, loss_gen=3.080, loss_fm=12.032,loss_mel=27.263, loss_kl=7.555
DEBUG:matplotlib:matplotlib data path: E:\ai\RVC0813Nvidia\runtime\lib\site-packages\matplotlib\mpl-data
DEBUG:matplotlib:CONFIGDIR=C:\Users\remot\.matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is win32
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
Keyboard interruption in main thread... closing server.
Process Process-1:

Originally created by @danekbiglike on GitHub (Sep 6, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1199 My training stopped due to a full disk, I cleared the space, entered all the same settings, it says that point 660 is loaded, but it starts anyway from the 1st epoch. ``` runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e "my-voice3.0" -sr 48k -f0 1 -bs 28 -g 0 -te 1000 -se 20 -pg pretrained_v2/f0G48k.pth -pd pretrained_v2/f0D48k.pth -l 0 -c 1 -sw 1 -v v2 INFO:my-voice3.0:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 28, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 17280, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 48000, 'filter_length': 2048, 'hop_length': 480, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs\\my-voice3.0/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [12, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs\\my-voice3.0', 'experiment_dir': './logs\\my-voice3.0', 'save_every_epoch': 20, 'name': 'my-voice3.0', 'total_epoch': 1000, 'pretrainG': 'pretrained_v2/f0G48k.pth', 'pretrainD': 'pretrained_v2/f0D48k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '48k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '1', 'if_cache_data_in_gpu': 1} INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. gin_channels: 256 self.spk_embed_dim: 109 ./logs\my-voice3.0\D_31680.pth INFO:my-voice3.0:Loaded model weights INFO:my-voice3.0:Loaded checkpoint './logs\my-voice3.0\D_31680.pth' (epoch 660) INFO:my-voice3.0:loaded D ./logs\my-voice3.0\G_32160.pth INFO:my-voice3.0:loaded pretrained pretrained_v2/f0G48k.pth <All keys matched successfully> INFO:my-voice3.0:loaded pretrained pretrained_v2/f0D48k.pth <All keys matched successfully> E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\autograd\__init__.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance. grad.sizes() = [64, 1, 4], strides() = [4, 1, 1] bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:337.) Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass INFO:my-voice3.0:Train Epoch: 1 [0%] INFO:my-voice3.0:[0, 0.0001] INFO:my-voice3.0:loss_disc=3.944, loss_gen=3.080, loss_fm=12.032,loss_mel=27.263, loss_kl=7.555 DEBUG:matplotlib:matplotlib data path: E:\ai\RVC0813Nvidia\runtime\lib\site-packages\matplotlib\mpl-data DEBUG:matplotlib:CONFIGDIR=C:\Users\remot\.matplotlib DEBUG:matplotlib:interactive is False DEBUG:matplotlib:platform is win32 INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. Keyboard interruption in main thread... closing server. Process Process-1: ```

bsakaguchi added the

following up

label

2026-10-20 10:02:54 +00:00

bsakaguchi commented

2026-10-20 10:02:54 +00:00

Author

Owner

@danekbiglike commented on GitHub (Sep 6, 2023):

I installed Mangio-RVC-Fork and it continued training, instead of starting from epoch 1, what was the problem in the original, I did not understand.

@danekbiglike commented on GitHub (Sep 6, 2023): I installed Mangio-RVC-Fork and it continued training, instead of starting from epoch 1, what was the problem in the original, I did not understand.

bsakaguchi commented

2026-10-20 10:02:55 +00:00

Author

Owner

@Fratermalou commented on GitHub (Sep 7, 2023):

i am a little noob in this but does the pth trained in the first place was an extracted one ? this error occured for me when i used an extracted model. you can correct it by enter the number of the epoch manually by modifying pth info in chkpnt proccessing, by replacinf the status "exctarcted" by epochs660

@Fratermalou commented on GitHub (Sep 7, 2023): i am a little noob in this but does the pth trained in the first place was an extracted one ? this error occured for me when i used an extracted model. you can correct it by enter the number of the epoch manually by modifying pth info in chkpnt proccessing, by replacinf the status "exctarcted" by epochs660

bsakaguchi commented

2026-10-20 10:02:55 +00:00

Author

Owner

@RVC-Boss commented on GitHub (Sep 13, 2023):

It may be caused by the last saved model is not fully saved. So it can't be loaded successfully when continue training.

@RVC-Boss commented on GitHub (Sep 13, 2023): It may be caused by the last saved model is not fully saved. So it can't be loaded successfully when continue training.

bsakaguchi commented

2026-10-20 10:02:55 +00:00

Author

Owner

@evorios commented on GitHub (Dec 10, 2023):

May be it might check files there for do not lost many-hours-progress?

@evorios commented on GitHub (Dec 10, 2023): May be it might check files there for do not lost many-hours-progress?

bsakaguchi commented

2026-10-20 10:02:55 +00:00

Author

Owner

@Disorbs commented on GitHub (Dec 29, 2023):

i also did the same thing my stopped cus disk was full and i already hit pre train index and hit train model (idk if i fucked up)
but it starts back at one instead of 380 epoch do i need to go in there and remove a file or edit something form 1 > 380?

name + settings is the same and before the disk issue it stopped at 250 successfully and i continued to 350 but got the disk error and now it starts at 1

EDIT: Fixed! i guess i went back and deleted two files in the /logs/(Name)/ with the last D_###.pth and G_###.pth whatever last one you had and it worked fined starting from 370 now which is better than nothing!

@Disorbs commented on GitHub (Dec 29, 2023): i also did the same thing my stopped cus disk was full and i already hit pre train index and hit train model (idk if i fucked up) but it starts back at one instead of 380 epoch do i need to go in there and remove a file or edit something form 1 > 380? name + settings is the same and before the disk issue it stopped at 250 successfully and i continued to 350 but got the disk error and now it starts at 1 EDIT: Fixed! i guess i went back and deleted two files in the /logs/(Name)/ with the last D_###.pth and G_###.pth whatever last one you had and it worked fined starting from 370 now which is better than nothing!

bsakaguchi commented

2026-10-20 10:02:55 +00:00

Author

Owner

@danekbiglike commented on GitHub (Dec 30, 2023):

My problem was that the last save G and D files did not match. (There was a conditional file G_8400, and D_8000) I deleted files that did not have a pair, perhaps this is what helped. I still haven't figured out why these files were missing.

@danekbiglike commented on GitHub (Dec 30, 2023): My problem was that the last save G and D files did not match. (There was a conditional file G_8400, and D_8000) I deleted files that did not have a pair, perhaps this is what helped. I still haven't figured out why these files were missing.