RVC-Project/Retrieval-based-Voice-Conversion-WebUI
1
0
Fork 0
mirror of https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git synced 2026-11-05 00:27:30 +00:00
Code Issues 588 Projects Packages Wiki Activity

[GH-ISSUE #1557] Dual GPU - RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer #1028

New issue
Closed
opened 2026-10-20 10:05:15 +00:00 by bsakaguchi · 11 comments
bsakaguchi commented 2026-10-20 10:05:15 +00:00
Owner
Copy link

Originally created by @hpnyaggerman on GitHub (Nov 17, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1557

Issue: Training Process Halts with Errors

Environment Specifications:

  • Operating System: Debian GNU/Linux 12 (bookworm) x86_64 6.1.0-9-amd64
  • Python Version: 3.10
  • GPU: Dual NVIDIA GeForce RTX 4090s
  • Python Environment: Conda virtual environment specifically for RVC-WebUI

Steps to Reproduce:

  1. Created a conda environment with Python version 3.10.
  2. Cloned RVC's git repository.
  3. Run run.sh and install dependencies, and then run RVC-WebUI.
  4. Navigated to the training tab, performed preprocessing, and then performed feature extraction.
  5. Started the training process.

Issue Encountered:
During training, I encountered a series of errors, causing the training process to halt. The detailed error log is as follows:

2023-11-18 02:32:55 | INFO | __main__ | Use gpus: 0-1
2023-11-18 02:32:55 | INFO | __main__ | "python3" infer/modules/train/train.py -e "model-test" -sr 48k -f0 1 -bs 12 -g 0-1 -te 1000 -se 1 -pg /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth -pd /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth -l 0 -c 1 -sw 0 -v v2
INFO:model-test:{'data': {'filter_length': 2048, 'hop_length': 480, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 128, 'sampling_rate': 48000, 'win_length': 2048, 'training_files': './logs/model-test/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'upsample_rates': [12, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': True, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 17280, 'warmup_epochs': 0}, 'model_dir': './logs/model-test', 'experiment_dir': './logs/model-test', 'save_every_epoch': 1, 'name': 'model-test', 'total_epoch': 1000, 'pretrainG': '/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth', 'pretrainD': '/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth', 'version': 'v2', 'gpus': '0-1', 'sample_rate': '48k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 1}
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
INFO:model-test:loaded pretrained /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth
Process Process-2:
Traceback (most recent call last):
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 213, in run
    utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 213, in latest_checkpoint_path
    x = f_list[-1]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 232, in run
    logger.info(
UnboundLocalError: local variable 'logger' referenced before assignment
INFO:model-test:<All keys matched successfully>
INFO:model-test:loaded pretrained /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth
INFO:model-test:<All keys matched successfully>
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
Process Process-1:
Traceback (most recent call last):
  File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 271, in run
    train_and_evaluate(
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 484, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [127.0.0.1]:27868

Any assistance or guidance on resolving these errors would be greatly appreciated.

Originally created by @hpnyaggerman on GitHub (Nov 17, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1557 ### Issue: Training Process Halts with Errors **Environment Specifications:** - **Operating System:** Debian GNU/Linux 12 (bookworm) x86_64 6.1.0-9-amd64 - **Python Version:** 3.10 - GPU: Dual NVIDIA GeForce RTX 4090s - Python Environment: Conda virtual environment specifically for RVC-WebUI **Steps to Reproduce:** 1. Created a conda environment with Python version 3.10. 2. Cloned RVC's git repository. 3. Run run.sh and install dependencies, and then run RVC-WebUI. 4. Navigated to the training tab, performed preprocessing, and then performed feature extraction. 5. Started the training process. **Issue Encountered:** During training, I encountered a series of errors, causing the training process to halt. The detailed error log is as follows: ``` 2023-11-18 02:32:55 | INFO | __main__ | Use gpus: 0-1 2023-11-18 02:32:55 | INFO | __main__ | "python3" infer/modules/train/train.py -e "model-test" -sr 48k -f0 1 -bs 12 -g 0-1 -te 1000 -se 1 -pg /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth -pd /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth -l 0 -c 1 -sw 0 -v v2 INFO:model-test:{'data': {'filter_length': 2048, 'hop_length': 480, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 128, 'sampling_rate': 48000, 'win_length': 2048, 'training_files': './logs/model-test/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'upsample_rates': [12, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': True, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 17280, 'warmup_epochs': 0}, 'model_dir': './logs/model-test', 'experiment_dir': './logs/model-test', 'save_every_epoch': 1, 'name': 'model-test', 'total_epoch': 1000, 'pretrainG': '/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth', 'pretrainD': '/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth', 'version': 'v2', 'gpus': '0-1', 'sample_rate': '48k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 1} /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm. warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.") /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm. warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.") DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109 DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109 INFO:model-test:loaded pretrained /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0G48k.pth Process Process-2: Traceback (most recent call last): File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 213, in run utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 213, in latest_checkpoint_path x = f_list[-1] IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 232, in run logger.info( UnboundLocalError: local variable 'logger' referenced before assignment INFO:model-test:<All keys matched successfully> INFO:model-test:loaded pretrained /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/assets/pretrained_v2/f0D48k.pth INFO:model-test:<All keys matched successfully> /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] /home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error. Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined] Process Process-1: Traceback (most recent call last): File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/hkyouma/miniconda3/envs/voicegen-rvc/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 271, in run train_and_evaluate( File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 484, in train_and_evaluate scaler.scale(loss_disc).backward() File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/home/hkyouma/ai/voicegen/Retrieval-based-Voice-Conversion-WebUI/.venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [127.0.0.1]:27868 ``` Any assistance or guidance on resolving these errors would be greatly appreciated.
bsakaguchi 2026-10-20 10:05:15 +00:00
  • closed this issue
  • added the
    stale
         label
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@hpnyaggerman commented on GitHub (Nov 18, 2023):

The error seems to stem from using two GPUs for training. Using just one makes the issue go away.

<!-- gh-comment-id:1817279370 --> @hpnyaggerman commented on GitHub (Nov 18, 2023): The error seems to stem from using two GPUs for training. Using just one makes the issue go away.
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@Irennnne commented on GitHub (Nov 19, 2023):

I changed to gpu:0 but still get the same error:

(cvr) kk1@kk1:/media/disk1_ssd/Rvc$ python infer-web.py
2023-11-19 23:53:30 | INFO | configs.config | Found GPU NVIDIA GeForce RTX 3090
is_half:True, device:cuda:0
2023-11-19 23:53:31 | INFO | __main__ | Use Language: en_US
Running on local URL:  http://0.0.0.0:7865
2023-11-19 23:54:12 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-19 23:54:12 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/preprocess.py "/media/disk1_ssd/voc" 40000 22 "/media/disk1_ssd/Rvc/logs/trial" False 3.0
['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0']
start preprocess
['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0']
/media/disk1_ssd/voc/.DS_Store->Traceback (most recent call last):
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio
    audio2(f, out, "f32le", sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2
    inp = av.open(i, "rb")
  File "av/container/core.pyx", line 401, in av.container.core.open
  File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 285, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "infer/modules/train/preprocess.py", line 87, in pipeline
    audio = load_audio(path, self.sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 73, in load_audio
    raise RuntimeError(traceback.format_exc())
RuntimeError: Traceback (most recent call last):
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio
    audio2(f, out, "f32le", sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2
    inp = av.open(i, "rb")
  File "av/container/core.pyx", line 401, in av.container.core.open
  File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 285, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store'


/media/disk1_ssd/voc/vocal_6.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_5.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_1.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_4.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_3.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_7.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_2.mp3_10.wav->Suc.
end preprocess
2023-11-19 23:54:18 | INFO | __main__ | start preprocess
['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0']
/media/disk1_ssd/voc/.DS_Store->Traceback (most recent call last):
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio
    audio2(f, out, "f32le", sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2
    inp = av.open(i, "rb")
  File "av/container/core.pyx", line 401, in av.container.core.open
  File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 285, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "infer/modules/train/preprocess.py", line 87, in pipeline
    audio = load_audio(path, self.sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 73, in load_audio
    raise RuntimeError(traceback.format_exc())
RuntimeError: Traceback (most recent call last):
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio
    audio2(f, out, "f32le", sr)
  File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2
    inp = av.open(i, "rb")
  File "av/container/core.pyx", line 401, in av.container.core.open
  File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__
  File "av/container/core.pyx", line 285, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store'


/media/disk1_ssd/voc/vocal_6.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_5.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_1.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_4.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_3.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_7.mp3_10.wav->Suc.
/media/disk1_ssd/voc/vocal_2.mp3_10.wav->Suc.
end preprocess

2023-11-19 23:54:18 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 0 0 "/media/disk1_ssd/Rvc/logs/trial" True 
2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 1 1 "/media/disk1_ssd/Rvc/logs/trial" True 
2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 2 0 "/media/disk1_ssd/Rvc/logs/trial" True 
2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 3 1 "/media/disk1_ssd/Rvc/logs/trial" True 
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav
Loading rmvpe model
Loading rmvpe model
Loading rmvpe model
Loading rmvpe model
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav
2023-11-19 23:54:26 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav

2023-11-19 23:54:26 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract_feature_print.py cuda:0 1 0 0 "/media/disk1_ssd/Rvc/logs/trial" v2
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'v2']
/media/disk1_ssd/Rvc/logs/trial
load model(s) from assets/hubert/hubert_base.pt
2023-11-19 23:54:26 | INFO | fairseq.tasks.hubert_pretraining | current directory is /media/disk1_ssd/Rvc
2023-11-19 23:54:26 | INFO | fairseq.tasks.hubert_pretraining | HubertPretrainingTask Config {'_name': 'hubert_pretraining', 'data': 'metadata', 'fine_tuning': False, 'labels': ['km'], 'label_dir': 'label', 'label_rate': 50.0, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_keep_size': None, 'max_sample_size': 250000, 'min_sample_size': 32000, 'single_target': False, 'random_crop': True, 'pad_audio': False}
2023-11-19 23:54:26 | INFO | fairseq.models.hubert.hubert | HubertModel Config: {'_name': 'hubert', 'label_rate': 50.0, 'extractor_mode': default, 'encoder_layers': 12, 'encoder_embed_dim': 768, 'encoder_ffn_embed_dim': 3072, 'encoder_attention_heads': 12, 'activation_fn': gelu, 'layer_type': transformer, 'dropout': 0.1, 'attention_dropout': 0.1, 'activation_dropout': 0.0, 'encoder_layerdrop': 0.05, 'dropout_input': 0.1, 'dropout_features': 0.1, 'final_dim': 256, 'untie_final_proj': True, 'layer_norm_first': False, 'conv_feature_layers': '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2', 'conv_bias': False, 'logit_temp': 0.1, 'target_glu': False, 'feature_grad_mult': 0.1, 'mask_length': 10, 'mask_prob': 0.8, 'mask_selection': static, 'mask_other': 0.0, 'no_mask_overlap': False, 'mask_min_space': 1, 'mask_channel_length': 10, 'mask_channel_prob': 0.0, 'mask_channel_selection': static, 'mask_channel_other': 0.0, 'no_mask_channel_overlap': False, 'mask_channel_min_space': 1, 'conv_pos': 128, 'conv_pos_groups': 16, 'latent_temp': [2.0, 0.5, 0.999995], 'skip_masked': False, 'skip_nomask': False, 'checkpoint_activations': False, 'required_seq_len_multiple': 2, 'depthwise_conv_kernel_size': 31, 'attn_type': '', 'pos_enc_type': 'abs', 'fp16': False}
move model to cuda
all-feature-326
now-326,all-0,1_0.wav,(149, 768)
now-326,all-32,2_11.wav,(155, 768)
now-326,all-64,2_148.wav,(134, 768)
now-326,all-96,2_56.wav,(105, 768)
now-326,all-128,2_94.wav,(149, 768)
now-326,all-160,3_42.wav,(149, 768)
now-326,all-192,4_18.wav,(149, 768)
now-326,all-224,5_16.wav,(149, 768)
now-326,all-256,7_1.wav,(149, 768)
now-326,all-288,7_50.wav,(113, 768)
now-326,all-320,7_9.wav,(140, 768)
all-feature-done
2023-11-19 23:54:33 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-82
f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True']
todo-f0-81
f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav
f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav
f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav
f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav
f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav
f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav
f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav
f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav
f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav
f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav
f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'v2']
/media/disk1_ssd/Rvc/logs/trial
load model(s) from assets/hubert/hubert_base.pt
move model to cuda
all-feature-326
now-326,all-0,1_0.wav,(149, 768)
now-326,all-32,2_11.wav,(155, 768)
now-326,all-64,2_148.wav,(134, 768)
now-326,all-96,2_56.wav,(105, 768)
now-326,all-128,2_94.wav,(149, 768)
now-326,all-160,3_42.wav,(149, 768)
now-326,all-192,4_18.wav,(149, 768)
now-326,all-224,5_16.wav,(149, 768)
now-326,all-256,7_1.wav,(149, 768)
now-326,all-288,7_50.wav,(113, 768)
now-326,all-320,7_9.wav,(140, 768)
all-feature-done

2023-11-19 23:54:33 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-19 23:54:33 | INFO | __main__ | Use gpus: 0
2023-11-19 23:54:33 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/train.py -e "trial" -sr 40k -f0 1 -bs 12 -g 0 -te 20 -se 5 -pg assets/pretrained_v2/f0G40k.pth -pd assets/pretrained_v2/f0D40k.pth -l 0 -c 0 -sw 0 -v v2
INFO:trial:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/trial/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/trial', 'experiment_dir': './logs/trial', 'save_every_epoch': 5, 'name': 'trial', 'total_epoch': 20, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Process Process-2:
Traceback (most recent call last):
  File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/media/disk1_ssd/Rvc/infer/modules/train/train.py", line 137, in run
    torch.cuda.set_device(rank)
  File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
Process Process-1:
Traceback (most recent call last):
  File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/media/disk1_ssd/Rvc/infer/modules/train/train.py", line 205, in run
    net_g = DDP(net_g, device_ids=[rank])
  File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [fca1:de0c:6dd9:56e9:448a::1]:21721
<!-- gh-comment-id:1817897556 --> @Irennnne commented on GitHub (Nov 19, 2023): I changed to gpu:0 but still get the same error: ``` (cvr) kk1@kk1:/media/disk1_ssd/Rvc$ python infer-web.py 2023-11-19 23:53:30 | INFO | configs.config | Found GPU NVIDIA GeForce RTX 3090 is_half:True, device:cuda:0 2023-11-19 23:53:31 | INFO | __main__ | Use Language: en_US Running on local URL: http://0.0.0.0:7865 2023-11-19 23:54:12 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK" 2023-11-19 23:54:12 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/preprocess.py "/media/disk1_ssd/voc" 40000 22 "/media/disk1_ssd/Rvc/logs/trial" False 3.0 ['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0'] start preprocess ['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0'] /media/disk1_ssd/voc/.DS_Store->Traceback (most recent call last): File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio audio2(f, out, "f32le", sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2 inp = av.open(i, "rb") File "av/container/core.pyx", line 401, in av.container.core.open File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__ File "av/container/core.pyx", line 285, in av.container.core.Container.err_check File "av/error.pyx", line 336, in av.error.err_check av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "infer/modules/train/preprocess.py", line 87, in pipeline audio = load_audio(path, self.sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 73, in load_audio raise RuntimeError(traceback.format_exc()) RuntimeError: Traceback (most recent call last): File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio audio2(f, out, "f32le", sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2 inp = av.open(i, "rb") File "av/container/core.pyx", line 401, in av.container.core.open File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__ File "av/container/core.pyx", line 285, in av.container.core.Container.err_check File "av/error.pyx", line 336, in av.error.err_check av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store' /media/disk1_ssd/voc/vocal_6.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_5.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_1.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_4.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_3.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_7.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_2.mp3_10.wav->Suc. end preprocess 2023-11-19 23:54:18 | INFO | __main__ | start preprocess ['infer/modules/train/preprocess.py', '/media/disk1_ssd/voc', '40000', '22', '/media/disk1_ssd/Rvc/logs/trial', 'False', '3.0'] /media/disk1_ssd/voc/.DS_Store->Traceback (most recent call last): File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio audio2(f, out, "f32le", sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2 inp = av.open(i, "rb") File "av/container/core.pyx", line 401, in av.container.core.open File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__ File "av/container/core.pyx", line 285, in av.container.core.Container.err_check File "av/error.pyx", line 336, in av.error.err_check av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "infer/modules/train/preprocess.py", line 87, in pipeline audio = load_audio(path, self.sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 73, in load_audio raise RuntimeError(traceback.format_exc()) RuntimeError: Traceback (most recent call last): File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 63, in load_audio audio2(f, out, "f32le", sr) File "/media/disk1_ssd/Rvc/infer/lib/audio.py", line 34, in audio2 inp = av.open(i, "rb") File "av/container/core.pyx", line 401, in av.container.core.open File "av/container/core.pyx", line 265, in av.container.core.Container.__cinit__ File "av/container/core.pyx", line 285, in av.container.core.Container.err_check File "av/error.pyx", line 336, in av.error.err_check av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: '/media/disk1_ssd/voc/.DS_Store' /media/disk1_ssd/voc/vocal_6.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_5.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_1.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_4.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_3.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_7.mp3_10.wav->Suc. /media/disk1_ssd/voc/vocal_2.mp3_10.wav->Suc. end preprocess 2023-11-19 23:54:18 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK" 2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 0 0 "/media/disk1_ssd/Rvc/logs/trial" True 2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 1 1 "/media/disk1_ssd/Rvc/logs/trial" True 2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 2 0 "/media/disk1_ssd/Rvc/logs/trial" True 2023-11-19 23:54:18 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract/extract_f0_rmvpe.py 4 3 1 "/media/disk1_ssd/Rvc/logs/trial" True ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav Loading rmvpe model Loading rmvpe model Loading rmvpe model Loading rmvpe model f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav 2023-11-19 23:54:26 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav 2023-11-19 23:54:26 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/extract_feature_print.py cuda:0 1 0 0 "/media/disk1_ssd/Rvc/logs/trial" v2 ['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'v2'] /media/disk1_ssd/Rvc/logs/trial load model(s) from assets/hubert/hubert_base.pt 2023-11-19 23:54:26 | INFO | fairseq.tasks.hubert_pretraining | current directory is /media/disk1_ssd/Rvc 2023-11-19 23:54:26 | INFO | fairseq.tasks.hubert_pretraining | HubertPretrainingTask Config {'_name': 'hubert_pretraining', 'data': 'metadata', 'fine_tuning': False, 'labels': ['km'], 'label_dir': 'label', 'label_rate': 50.0, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_keep_size': None, 'max_sample_size': 250000, 'min_sample_size': 32000, 'single_target': False, 'random_crop': True, 'pad_audio': False} 2023-11-19 23:54:26 | INFO | fairseq.models.hubert.hubert | HubertModel Config: {'_name': 'hubert', 'label_rate': 50.0, 'extractor_mode': default, 'encoder_layers': 12, 'encoder_embed_dim': 768, 'encoder_ffn_embed_dim': 3072, 'encoder_attention_heads': 12, 'activation_fn': gelu, 'layer_type': transformer, 'dropout': 0.1, 'attention_dropout': 0.1, 'activation_dropout': 0.0, 'encoder_layerdrop': 0.05, 'dropout_input': 0.1, 'dropout_features': 0.1, 'final_dim': 256, 'untie_final_proj': True, 'layer_norm_first': False, 'conv_feature_layers': '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2', 'conv_bias': False, 'logit_temp': 0.1, 'target_glu': False, 'feature_grad_mult': 0.1, 'mask_length': 10, 'mask_prob': 0.8, 'mask_selection': static, 'mask_other': 0.0, 'no_mask_overlap': False, 'mask_min_space': 1, 'mask_channel_length': 10, 'mask_channel_prob': 0.0, 'mask_channel_selection': static, 'mask_channel_other': 0.0, 'no_mask_channel_overlap': False, 'mask_channel_min_space': 1, 'conv_pos': 128, 'conv_pos_groups': 16, 'latent_temp': [2.0, 0.5, 0.999995], 'skip_masked': False, 'skip_nomask': False, 'checkpoint_activations': False, 'required_seq_len_multiple': 2, 'depthwise_conv_kernel_size': 31, 'attn_type': '', 'pos_enc_type': 'abs', 'fp16': False} move model to cuda all-feature-326 now-326,all-0,1_0.wav,(149, 768) now-326,all-32,2_11.wav,(155, 768) now-326,all-64,2_148.wav,(134, 768) now-326,all-96,2_56.wav,(105, 768) now-326,all-128,2_94.wav,(149, 768) now-326,all-160,3_42.wav,(149, 768) now-326,all-192,4_18.wav,(149, 768) now-326,all-224,5_16.wav,(149, 768) now-326,all-256,7_1.wav,(149, 768) now-326,all-288,7_50.wav,(113, 768) now-326,all-320,7_9.wav,(140, 768) all-feature-done 2023-11-19 23:54:33 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_0.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '3', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_12.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '1', '1', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-82 f0ing,now-0,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_10.wav ['infer/modules/train/extract/extract_f0_rmvpe.py', '4', '2', '0', '/media/disk1_ssd/Rvc/logs/trial', 'True'] todo-f0-81 f0ing,now-0,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/1_11.wav f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_15.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_17.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_95.wav f0ing,now-16,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_16.wav f0ing,now-16,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_148.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_98.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_19.wav f0ing,now-32,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_94.wav f0ing,now-32,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/2_96.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_20.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_10.wav f0ing,now-48,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_18.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_12.wav f0ing,now-48,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/4_2.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_90.wav f0ing,now-64,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_1.wav f0ing,now-64,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_11.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_92.wav f0ing,now-80,all-82,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_9.wav f0ing,now-80,all-81,-/media/disk1_ssd/Rvc/logs/trial/1_16k_wavs/7_91.wav ['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/media/disk1_ssd/Rvc/logs/trial', 'v2'] /media/disk1_ssd/Rvc/logs/trial load model(s) from assets/hubert/hubert_base.pt move model to cuda all-feature-326 now-326,all-0,1_0.wav,(149, 768) now-326,all-32,2_11.wav,(155, 768) now-326,all-64,2_148.wav,(134, 768) now-326,all-96,2_56.wav,(105, 768) now-326,all-128,2_94.wav,(149, 768) now-326,all-160,3_42.wav,(149, 768) now-326,all-192,4_18.wav,(149, 768) now-326,all-224,5_16.wav,(149, 768) now-326,all-256,7_1.wav,(149, 768) now-326,all-288,7_50.wav,(113, 768) now-326,all-320,7_9.wav,(140, 768) all-feature-done 2023-11-19 23:54:33 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK" 2023-11-19 23:54:33 | INFO | __main__ | Use gpus: 0 2023-11-19 23:54:33 | INFO | __main__ | "/home/miniconda3/envs/cvr/bin/python" infer/modules/train/train.py -e "trial" -sr 40k -f0 1 -bs 12 -g 0 -te 20 -se 5 -pg assets/pretrained_v2/f0G40k.pth -pd assets/pretrained_v2/f0D40k.pth -l 0 -c 0 -sw 0 -v v2 INFO:trial:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/trial/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/trial', 'experiment_dir': './logs/trial', 'save_every_epoch': 5, 'name': 'trial', 'total_epoch': 20, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0} INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. Process Process-2: Traceback (most recent call last): File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/media/disk1_ssd/Rvc/infer/modules/train/train.py", line 137, in run torch.cuda.set_device(rank) File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device torch._C._cuda_setDevice(device) RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109 Process Process-1: Traceback (most recent call last): File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap self.run() File "/home/miniconda3/envs/cvr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/media/disk1_ssd/Rvc/infer/modules/train/train.py", line 205, in run net_g = DDP(net_g, device_ids=[rank]) File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in __init__ _verify_param_shape_across_processes(self.process_group, parameters) File "/home/miniconda3/envs/cvr/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes return dist._verify_params_across_processes(process_group, tensors, logger) RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [fca1:de0c:6dd9:56e9:448a::1]:21721 ```
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@jnjnnjzch commented on GitHub (Nov 26, 2023):

My solution: use one GPU to train for one or more epoches to create a checkpoint. Then continue training on multi-GPUs.
It seems that RVC can train on multi-GPUs by loading an existed checkpoint.

<!-- gh-comment-id:1826483690 --> @jnjnnjzch commented on GitHub (Nov 26, 2023): My solution: use one GPU to train for one or more epoches to create a checkpoint. Then continue training on multi-GPUs. It seems that RVC can train on multi-GPUs by loading an existed checkpoint.
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@ospreyclaw commented on GitHub (Dec 21, 2023):

I have the same problem. Have you fixed this problem?

<!-- gh-comment-id:1865812807 --> @ospreyclaw commented on GitHub (Dec 21, 2023): I have the same problem. Have you fixed this problem?
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@1044690543 commented on GitHub (Jan 18, 2024):

I also have the same problem, trying to solve

<!-- gh-comment-id:1898012389 --> @1044690543 commented on GitHub (Jan 18, 2024): I also have the same problem, trying to solve
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@Irennnne commented on GitHub (Jan 18, 2024):

I have solved this problem and I think adding “ CUDA_LAUNCH_BLOCKING=1” works.

<!-- gh-comment-id:1898232311 --> @Irennnne commented on GitHub (Jan 18, 2024): I have solved this problem and I think adding “ CUDA_LAUNCH_BLOCKING=1” works.
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@1044690543 commented on GitHub (Jan 19, 2024):

I also have the same problem, trying to solve

Using the latest TAG can solve the problem

<!-- gh-comment-id:1899809292 --> @1044690543 commented on GitHub (Jan 19, 2024): > I also have the same problem, trying to solve Using the latest TAG can solve the problem
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@aiejebrazorf commented on GitHub (Jan 21, 2024):

same error for me, did anyone find a solution ?

<!-- gh-comment-id:1902590239 --> @aiejebrazorf commented on GitHub (Jan 21, 2024): same error for me, did anyone find a solution ?
bsakaguchi commented 2026-10-20 10:05:16 +00:00
Author
Owner
Copy link

@ymmbb8882ymmbb commented on GitHub (Mar 11, 2024):

我也有同样的问题，正在尝试解决

使用最新的TAG即可解决问题

我用了10月份的tar包，还是遇到了这个问题，并且他并不支持多卡训练IndexError: list index out of range

<!-- gh-comment-id:1988258429 --> @ymmbb8882ymmbb commented on GitHub (Mar 11, 2024): > > 我也有同样的问题，正在尝试解决 > > 使用最新的TAG即可解决问题 我用了10月份的tar包，还是遇到了这个问题，并且他并不支持多卡训练IndexError: list index out of range
bsakaguchi commented 2026-10-20 10:05:17 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Apr 28, 2024):

This issue was closed because it has been inactive for 15 days since being marked as stale.

<!-- gh-comment-id:2081318767 --> @github-actions[bot] commented on GitHub (Apr 28, 2024): This issue was closed because it has been inactive for 15 days since being marked as stale.
bsakaguchi commented 2026-10-20 10:05:17 +00:00
Author
Owner
Copy link

@saiisa commented on GitHub (Jun 19, 2025):

Use only 1 GPU for training by changing 3 GPU index configs in web UI (0, 0-0), the error gone.

<!-- gh-comment-id:2987211692 --> @saiisa commented on GitHub (Jun 19, 2025): Use only 1 GPU for training by changing 3 GPU index configs in web UI (0, 0-0), the error gone.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
20240604
v1
2.3.260718
2.2.231006
2.1.230814
2.0.230618
2.0.230528
1.2.230428
1.1.230416
1.0.230410
Labels
Clear labels   
AMD/Intel

   
bug

   
bug

   
bug

   
colab

   
dependencies

   
documentation

   
duplicate

   
enhancement

   
following up

   
good first issue

   
help wanted

   
invalid

   
question

   
stale

   
stale

   
stale

   
todo list

   
wontfix
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#1028
No description provided.