mirror of
https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git
synced 2026-11-05 00:27:30 +00:00
[GH-ISSUE #1085] does not load beyond 1 epoch #718
Labels
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#718
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Froks3 on GitHub (Aug 25, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1085
my video card is not detected, but it was not a problem. last time I calmly trained a working model in a couple of hours.
now when I try to run, doing everything the same as yesterday, nothing works. do not load further than the first epoch
C:\RVC>runtime\python.exe infer-web.py --pycmd runtime\python.exe --port 7897 --dml
No supported Nvidia GPU found
use DirectML instead
Use Language: en_US
Running on local URL: http://0.0.0.0:7897
runtime\python.exe trainset_preprocess_pipeline_print.py "C:\RVC\sampl\mayni-model\sample.mp3" 40000 3 "C:\RVC/logs/mayni123" False
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model\sample.mp3', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model\sample.mp3', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
Fail. Traceback (most recent call last):
File "C:\RVC\trainset_preprocess_pipeline_print.py", line 111, in pipeline_mp_inp_dir
for idx, name in enumerate(sorted(list(os.listdir(inp_root))))
NotADirectoryError: [WinError 267] Неверно задано имя папки: 'C:\RVC\sampl\mayni-model\sample.mp3'
end preprocess
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model\sample.mp3', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
Fail. Traceback (most recent call last):
File "C:\RVC\trainset_preprocess_pipeline_print.py", line 111, in pipeline_mp_inp_dir
for idx, name in enumerate(sorted(list(os.listdir(inp_root))))
NotADirectoryError: [WinError 267] Неверно задано имя папки: 'C:\RVC\sampl\mayni-model\sample.mp3'
end preprocess
runtime\python.exe trainset_preprocess_pipeline_print.py "C:\RVC\sampl\mayni-model" 40000 3 "C:\RVC/logs/mayni123" False
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
C:\RVC\sampl\mayni-model/sample.mp3->Suc.
end preprocess
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
C:\RVC\sampl\mayni-model/sample.mp3->Suc.
end preprocess
runtime\python.exe extract_f0_rmvpe_dml.py "C:\RVC/logs/mayni123"
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
loading rmvpe model
2023-08-25 17:02:07.0276152 [W:onnxruntime:, session_state.cc:1169 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-08-25 17:02:07.0382099 [W:onnxruntime:, session_state.cc:1171 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
C:\RVC\lib\rmvpe.py:120: UserWarning: The operator 'aten::reflection_pad3d.out' is not currently supported on the DML backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at D:\a_work\1\s\pytorch-directml-plugin\torch_directml\csrc\dml\dml_cpu_fallback.cpp:17.)
input_data = F.pad(input_data.unsqueeze(1),(self.pad_amount, self.pad_amount, 0, 0,0,0),mode='reflect').squeeze(1)
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
runtime\python.exe extract_feature_print.py privateuseone:0 1 0 "C:\RVC/logs/mayni123" v2
['extract_feature_print.py', 'privateuseone:0', '1', '0', 'C:\RVC/logs/mayni123', 'v2']
C:\RVC/logs/mayni123
load model(s) from hubert_base.pt
move model to privateuseone:0
all-feature-78
now-78,all-0,0_0.wav,(149, 768)
now-78,all-7,0_15.wav,(149, 768)
now-78,all-14,0_22.wav,(149, 768)
now-78,all-21,0_3.wav,(149, 768)
now-78,all-28,0_36.wav,(149, 768)
now-78,all-35,0_42.wav,(149, 768)
now-78,all-42,0_49.wav,(149, 768)
now-78,all-49,0_55.wav,(149, 768)
now-78,all-56,0_61.wav,(149, 768)
now-78,all-63,0_69.wav,(149, 768)
now-78,all-70,0_76.wav,(149, 768)
now-78,all-77,0_9.wav,(149, 768)
all-feature-done
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
['extract_feature_print.py', 'privateuseone:0', '1', '0', 'C:\RVC/logs/mayni123', 'v2']
C:\RVC/logs/mayni123
load model(s) from hubert_base.pt
move model to privateuseone:0
all-feature-78
now-78,all-0,0_0.wav,(149, 768)
now-78,all-7,0_15.wav,(149, 768)
now-78,all-14,0_22.wav,(149, 768)
now-78,all-21,0_3.wav,(149, 768)
now-78,all-28,0_36.wav,(149, 768)
now-78,all-35,0_42.wav,(149, 768)
now-78,all-42,0_49.wav,(149, 768)
now-78,all-49,0_55.wav,(149, 768)
now-78,all-56,0_61.wav,(149, 768)
now-78,all-63,0_69.wav,(149, 768)
now-78,all-70,0_76.wav,(149, 768)
now-78,all-77,0_9.wav,(149, 768)
all-feature-done
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
['C:\RVC\trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
C:\RVC\sampl\mayni-model/sample.mp3->Suc.
end preprocess
start preprocess
['trainset_preprocess_pipeline_print.py', 'C:\RVC\sampl\mayni-model', '40000', '3', 'C:\RVC/logs/mayni123', 'False']
C:\RVC\sampl\mayni-model/sample.mp3->Suc.
end preprocess
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
['extract_feature_print.py', 'privateuseone:0', '1', '0', 'C:\RVC/logs/mayni123', 'v2']
C:\RVC/logs/mayni123
load model(s) from hubert_base.pt
move model to privateuseone:0
all-feature-78
all-feature-done
['extract_f0_rmvpe_dml.py', 'C:\RVC/logs/mayni123']
todo-f0-78
f0ing,now-0,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_0.wav
f0ing,now-15,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_24.wav
f0ing,now-30,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_38.wav
f0ing,now-45,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_51.wav
f0ing,now-60,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_66.wav
f0ing,now-75,all-78,-C:\RVC/logs/mayni123/1_16k_wavs/0_80.wav
['extract_feature_print.py', 'privateuseone:0', '1', '0', 'C:\RVC/logs/mayni123', 'v2']
C:\RVC/logs/mayni123
load model(s) from hubert_base.pt
move model to privateuseone:0
all-feature-78
all-feature-done
NO GPU DETECTED: falling back to CPU - this may take a while
INFO:mayni123:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 3, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 12800, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 40000, 'filter_length': 2048, 'hop_length': 400, 'win_length': 2048, 'n_mel_channels': 125, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs\mayni123/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': 1, 3, 5], [1, 3, 5], [1, 3, 5, 'upsample_rates': [10, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs\mayni123', 'experiment_dir': './logs\mayni123', 'save_every_epoch': 5, 'name': 'mayni123', 'total_epoch': 100, 'pretrainG': 'pretrained_v2/f0G40k.pth', 'pretrainD': 'pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 1, 'save_every_weights': '1', 'if_cache_data_in_gpu': 0}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
gin_channels: 256 self.spk_embed_dim: 109
INFO:mayni123:loaded pretrained pretrained_v2/f0G40k.pth
INFO:mayni123:loaded pretrained pretrained_v2/f0D40k.pth
C:\RVC\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\RVC\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\RVC\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\RVC\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\RVC\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
C:\RVC\runtime\lib\site-packages\torch\autograd_init.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:337.)
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
INFO:mayni123:Train Epoch: 1 [0%]
INFO:mayni123:[0, 0.0001]
INFO:mayni123:loss_disc=3.761, loss_gen=5.026, loss_fm=13.513,loss_mel=22.202, loss_kl=6.184
DEBUG:matplotlib:matplotlib data path: C:\RVC\runtime\lib\site-packages\matplotlib\mpl-data
DEBUG:matplotlib:CONFIGDIR=C:\Users\admin.matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is win32
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
@RVC-Boss commented on GitHub (Aug 30, 2023):
You should use dataset dir instead of dataset audio name.