RVC-Project/Retrieval-based-Voice-Conversion-WebUI
1
0
Fork 0
mirror of https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git synced 2026-11-05 00:27:30 +00:00
Code Issues 588 Projects Packages Wiki Activity

[GH-ISSUE #1333] Runtime Error when trying to train RVC #860

New issue
Closed
opened 2026-10-20 10:03:44 +00:00 by bsakaguchi · 13 comments
bsakaguchi commented 2026-10-20 10:03:44 +00:00
Owner
Copy link

Originally created by @treya-lin on GitHub (Sep 26, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1333

Hi I ran into the following Runtime error when training on linux system (ubuntu16.04). I am working with py3.10, torch1.12.1+cu102, cuda 10.2. I am trying to train on 2 GPU (1080Ti*2).

The final RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] seems to be the cause of the problem, but I am not sure what is going wrong. Could you please take a look and give me some advice. Thank you very much!!!!

Below is the whold training log.

2023-09-26 11:37:58 | INFO | __main__ | Use gpus: 2-3
2023-09-26 11:37:58 | INFO | __main__ | "/home/user/anaconda3/envs/env_vits_torch112/bin/python" infer/modules/train/train.py -e "A01010_20230926_pitchguide-false_v2" -sr 48k -f0 0 -bs 5 -g 2-3 -te 20 -se 5 -pg assets/pretrained_v2/G48k.pth -pd assets/pretrained_v2/D48k.pth -l 0 -c 0 -sw 0 -v v2
INFO:A01010_20230926_pitchguide-false_v2:{'data': {'filter_length': 2048, 'hop_length': 480, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 128, 'sampling_rate': 48000, 'win_length': 2048, 'training_files': './logs/A01010_20230926_pitchguide-false_v2/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'upsample_rates': [12, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 5, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 17280, 'warmup_epochs': 0}, 'model_dir': './logs/A01010_20230926_pitchguide-false_v2', 'experiment_dir': './logs/A01010_20230926_pitchguide-false_v2', 'save_every_epoch': 5, 'name': 'A01010_20230926_pitchguide-false_v2', 'total_epoch': 20, 'pretrainG': 'assets/pretrained_v2/G48k.pth', 'pretrainD': 'assets/pretrained_v2/D48k.pth', 'version': 'v2', 'gpus': '2-3', 'sample_rate': '48k', 'if_f0': 0, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
INFO:A01010_20230926_pitchguide-false_v2:loaded pretrained assets/pretrained_v2/G48k.pth
Process Process-2:
Traceback (most recent call last):
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 213, in run
    utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 213, in latest_checkpoint_path
    x = f_list[-1]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 232, in run
    logger.info(
UnboundLocalError: local variable 'logger' referenced before assignment
INFO:A01010_20230926_pitchguide-false_v2:<All keys matched successfully>
INFO:A01010_20230926_pitchguide-false_v2:loaded pretrained assets/pretrained_v2/D48k.pth
INFO:A01010_20230926_pitchguide-false_v2:<All keys matched successfully>
Process Process-1:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 271, in run
    train_and_evaluate(
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 484, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:3230

ps: My setup:
target sampling rate: 48k, pitch_guide:false, version:v2
pretrained models: assets/pretrained_v2/G48k.pth and assets/pretrained_v2/D48k.pth

Originally created by @treya-lin on GitHub (Sep 26, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1333 Hi I ran into the following Runtime error when training on linux system (ubuntu16.04). I am working with py3.10, torch1.12.1+cu102, cuda 10.2. I am trying to train on 2 GPU (1080Ti*2). The final `RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] ` seems to be the cause of the problem, but I am not sure what is going wrong. Could you please take a look and give me some advice. Thank you very much!!!! Below is the whold training log. ``` 2023-09-26 11:37:58 | INFO | __main__ | Use gpus: 2-3 2023-09-26 11:37:58 | INFO | __main__ | "/home/user/anaconda3/envs/env_vits_torch112/bin/python" infer/modules/train/train.py -e "A01010_20230926_pitchguide-false_v2" -sr 48k -f0 0 -bs 5 -g 2-3 -te 20 -se 5 -pg assets/pretrained_v2/G48k.pth -pd assets/pretrained_v2/D48k.pth -l 0 -c 0 -sw 0 -v v2 INFO:A01010_20230926_pitchguide-false_v2:{'data': {'filter_length': 2048, 'hop_length': 480, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 128, 'sampling_rate': 48000, 'win_length': 2048, 'training_files': './logs/A01010_20230926_pitchguide-false_v2/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'upsample_rates': [12, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 5, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 17280, 'warmup_epochs': 0}, 'model_dir': './logs/A01010_20230926_pitchguide-false_v2', 'experiment_dir': './logs/A01010_20230926_pitchguide-false_v2', 'save_every_epoch': 5, 'name': 'A01010_20230926_pitchguide-false_v2', 'total_epoch': 20, 'pretrainG': 'assets/pretrained_v2/G48k.pth', 'pretrainD': 'assets/pretrained_v2/D48k.pth', 'version': 'v2', 'gpus': '2-3', 'sample_rate': '48k', 'if_f0': 0, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0} INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109 DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109 INFO:A01010_20230926_pitchguide-false_v2:loaded pretrained assets/pretrained_v2/G48k.pth Process Process-2: Traceback (most recent call last): File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 213, in run utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 213, in latest_checkpoint_path x = f_list[-1] IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 232, in run logger.info( UnboundLocalError: local variable 'logger' referenced before assignment INFO:A01010_20230926_pitchguide-false_v2:<All keys matched successfully> INFO:A01010_20230926_pitchguide-false_v2:loaded pretrained assets/pretrained_v2/D48k.pth INFO:A01010_20230926_pitchguide-false_v2:<All keys matched successfully> Process Process-1: Traceback (most recent call last): File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 271, in run train_and_evaluate( File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 484, in train_and_evaluate scaler.scale(loss_disc).backward() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:3230 ``` ps: My setup: target sampling rate: 48k, pitch_guide:false, version:v2 pretrained models: assets/pretrained_v2/G48k.pth and assets/pretrained_v2/D48k.pth
bsakaguchi commented 2026-10-20 10:03:44 +00:00
Author
Owner
Copy link

@treya-lin commented on GitHub (Sep 26, 2023):

Update: It works after I switched to v1 model and used one GPU

<!-- gh-comment-id:1735251767 --> @treya-lin commented on GitHub (Sep 26, 2023): Update: It works after I switched to v1 model and used one GPU
bsakaguchi commented 2026-10-20 10:03:44 +00:00
Author
Owner
Copy link

@treya-lin commented on GitHub (Sep 27, 2023):

It seems the error is related to distributed training, so now I can only use one GPU to train. Do you have this issue in your local environment when using multi-GPU too?

<!-- gh-comment-id:1736553343 --> @treya-lin commented on GitHub (Sep 27, 2023): It seems the error is related to distributed training, so now I can only use one GPU to train. Do you have this issue in your local environment when using multi-GPU too?
bsakaguchi commented 2026-10-20 10:03:44 +00:00
Author
Owner
Copy link

@ChasonJiang commented on GitHub (Sep 27, 2023):

Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch.

<!-- gh-comment-id:1736789293 --> @ChasonJiang commented on GitHub (Sep 27, 2023): Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch.
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@treya-lin commented on GitHub (Oct 7, 2023):

Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch.

Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference?

<!-- gh-comment-id:1751575707 --> @treya-lin commented on GitHub (Oct 7, 2023): > Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch. Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference?
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@ChasonJiang commented on GitHub (Oct 7, 2023):

Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch.

Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference?

@treya-lin Because yesterday, the dev branch was merged into the main branch

<!-- gh-comment-id:1751591218 --> @ChasonJiang commented on GitHub (Oct 7, 2023): > > Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch. > > Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference? @treya-lin Because yesterday, the dev branch was merged into the main branch
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@treya-lin commented on GitHub (Oct 7, 2023):

Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch.

Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference?

@treya-lin Because yesterday, the dev branch was merged into the main branch

Hi thank you very much for the quick reply. I see. I will update my local repo accordingly.

By the way, could you kindly point out in which PR actually fixed this issue? I have looked it up in the PR logs and commit history for a while but couldn't locate it. I wanna understand what went wrong and how it was fixed, so if you could kindly point out the links of relevant PRs I will really appreciate that! Thanks!

<!-- gh-comment-id:1751595581 --> @treya-lin commented on GitHub (Oct 7, 2023): > > > Yes, this problem can occur when using the main branch and training with multi-GPU. This issue should have been fixed in the dev branch. > > > > > > Hi thanks for the reply. Do you mean that this issue was fixed in the dev branch even before I posted this issue? But as I can see here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/compare/dev , the dev and main branch do not seem to have any difference? > > @treya-lin Because yesterday, the dev branch was merged into the main branch Hi thank you very much for the quick reply. I see. I will update my local repo accordingly. By the way, could you kindly point out in which PR actually fixed this issue? I have looked it up in the PR logs and commit history for a while but couldn't locate it. I wanna understand what went wrong and how it was fixed, so if you could kindly point out the links of relevant PRs I will really appreciate that! Thanks!
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@ChasonJiang commented on GitHub (Oct 7, 2023):

@treya-lin here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/pull/1291/commits/9b3f210c9c66157153c4c332c526f3945f05538c

<!-- gh-comment-id:1751601764 --> @ChasonJiang commented on GitHub (Oct 7, 2023): @treya-lin here: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/pull/1291/commits/9b3f210c9c66157153c4c332c526f3945f05538c
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@treya-lin commented on GitHub (Oct 7, 2023):

@treya-lin here: 9b3f210

Hi thanks!

Now I have pulled the latest version of the repo, and trying it with 4 GPU (epoch=20,batch_size=4). But the training log is very strange and there was some other errors and warnings. Could you take a look?

  • First, nothing was written and saved in the train.log file in the log directory. (When I was using single gpu there was no this issue).

  • Second, the training procedure did started and proceeded (very fast) and saved some checkpoint, but as it reached epoch 20, it only saved "G_180.pth", is it normal? And it only took like a few minutes. When I was using the same training dataset and trained with one GPU, the final checkpoint was G_1520.pth" and it took 20 minutes or so.

  • Third, after it saved the final checkpoints, it threw some other runtime error again.

Below is the log. I would really appreciate your advice!

INFO:root:Saving model and optimizer state at epoch 20 to ./logs/test_20231007-gpu4_pitchguide-false_v1/G_180.pth
INFO:root:Saving model and optimizer state at epoch 20 to ./logs/test_20231007-gpu4_pitchguide-false_v1/D_180.pth
INFO:test_20231007-gpu4_pitchguide-false_v1:====> Epoch: 20 [2023-10-07 16:48:17] | (0:00:29.155022)
INFO:test_20231007-gpu4_pitchguide-false_v1:Training is done. The program is closed.
INFO:test_20231007-gpu4_pitchguide-false_v1:saving final ckpt:Success.
Process Process-4:
Process Process-2:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run
    train_and_evaluate(
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:31903
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run
    train_and_evaluate(
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:36560
Process Process-3:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run
    train_and_evaluate(
  File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate
    scaler.scale(loss_disc).backward()
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:19892
2023-10-07 16:48:20 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-10-07 16:48:20 | INFO | httpx | HTTP Request: POST http://localhost:7865/reset "HTTP/1.1 200 OK"
/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
<!-- gh-comment-id:1751658239 --> @treya-lin commented on GitHub (Oct 7, 2023): > @treya-lin here: [9b3f210](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/commit/9b3f210c9c66157153c4c332c526f3945f05538c) Hi thanks! Now I have pulled the latest version of the repo, and trying it with 4 GPU (epoch=20,batch_size=4). But the training log is very strange and there was some other errors and warnings. Could you take a look? * First, nothing was written and saved in the `train.log` file in the log directory. (When I was using single gpu there was no this issue). * Second, the training procedure did started and proceeded (very fast) and saved some checkpoint, but as it reached epoch 20, it only saved "G_180.pth", is it normal? And it only took like a few minutes. When I was using the same training dataset and trained with one GPU, the final checkpoint was G_1520.pth" and it took 20 minutes or so. * Third, after it saved the final checkpoints, it threw some other runtime error again. Below is the log. I would really appreciate your advice! ``` INFO:root:Saving model and optimizer state at epoch 20 to ./logs/test_20231007-gpu4_pitchguide-false_v1/G_180.pth INFO:root:Saving model and optimizer state at epoch 20 to ./logs/test_20231007-gpu4_pitchguide-false_v1/D_180.pth INFO:test_20231007-gpu4_pitchguide-false_v1:====> Epoch: 20 [2023-10-07 16:48:17] | (0:00:29.155022) INFO:test_20231007-gpu4_pitchguide-false_v1:Training is done. The program is closed. INFO:test_20231007-gpu4_pitchguide-false_v1:saving final ckpt:Success. Process Process-4: Process Process-2: Traceback (most recent call last): File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run train_and_evaluate( File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate scaler.scale(loss_disc).backward() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:31903 Traceback (most recent call last): File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run train_and_evaluate( File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate scaler.scale(loss_disc).backward() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:36560 Process Process-3: Traceback (most recent call last): File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 284, in run train_and_evaluate( File "/newdisk1_8T/yanfazu/tts_space/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 483, in train_and_evaluate scaler.scale(loss_disc).backward() File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/site-packages/torch/autograd/__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.4.169]:19892 2023-10-07 16:48:20 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK" 2023-10-07 16:48:20 | INFO | httpx | HTTP Request: POST http://localhost:7865/reset "HTTP/1.1 200 OK" /home/user/anaconda3/envs/env_vits_torch112/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' ```
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@ChasonJiang commented on GitHub (Oct 7, 2023):

Hello, I tried to reproduce and found the following results:

  • First, there is indeed nothing in train.log, which should be caused by a bug.
  • Second, it should be normal for the file names of these two files to be inconsistent, but the calculation method is different.
  • Third, after it saved the final checkpoints, I did not find any error messages.

Finally, thank you very much for the issues.

<!-- gh-comment-id:1751728829 --> @ChasonJiang commented on GitHub (Oct 7, 2023): Hello, I tried to reproduce and found the following results: - First, there is indeed nothing in train.log, which should be caused by a bug. - Second, it should be normal for the file names of these two files to be inconsistent, but the calculation method is different. - Third, after it saved the final checkpoints, I did not find any error messages. Finally, thank you very much for the issues.
bsakaguchi commented 2026-10-20 10:03:45 +00:00
Author
Owner
Copy link

@RVC-Boss commented on GitHub (Oct 9, 2023):

logger issue has been fixed now.

<!-- gh-comment-id:1752458755 --> @RVC-Boss commented on GitHub (Oct 9, 2023): logger issue has been fixed now.
bsakaguchi commented 2026-10-20 10:03:46 +00:00
Author
Owner
Copy link

@InconsolableCellist commented on GitHub (Oct 24, 2023):

@ChasonJiang it looks like your changes didn't make it to main: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/infer/modules/train/train.py#L107

Also Github says: "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository."

I just ran into this issue and found this thread about it.

<!-- gh-comment-id:1778189574 --> @InconsolableCellist commented on GitHub (Oct 24, 2023): @ChasonJiang it looks like your changes didn't make it to `main`: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/infer/modules/train/train.py#L107 Also Github says: "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository." I just ran into this issue and found this thread about it.
bsakaguchi commented 2026-10-20 10:03:46 +00:00
Author
Owner
Copy link

@ChasonJiang commented on GitHub (Oct 25, 2023):

@ChasonJiang it looks like your changes didn't make it to main: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/infer/modules/train/train.py#L107

Also Github says: "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository."

I just ran into this issue and found this thread about it.

@InconsolableCellist
Sorry, I haven't been involved in the development of RVC for a long time. I remember it was merged into the main branch before. You can contact @RVC-Boss to inquire about the situation.

<!-- gh-comment-id:1778592475 --> @ChasonJiang commented on GitHub (Oct 25, 2023): > @ChasonJiang it looks like your changes didn't make it to `main`: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/infer/modules/train/train.py#L107 > > Also Github says: "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository." > > I just ran into this issue and found this thread about it. @InconsolableCellist Sorry, I haven't been involved in the development of RVC for a long time. I remember it was merged into the main branch before. You can contact @RVC-Boss to inquire about the situation.
bsakaguchi commented 2026-10-20 10:03:46 +00:00
Author
Owner
Copy link

@barryhunt commented on GitHub (Dec 4, 2023):

@treya-lin这里：9b3f210

it can work for me, thanks

<!-- gh-comment-id:1838196304 --> @barryhunt commented on GitHub (Dec 4, 2023): > @treya-lin这里：[9b3f210](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/commit/9b3f210c9c66157153c4c332c526f3945f05538c) it can work for me, thanks
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
20240604
v1
2.3.260718
2.2.231006
2.1.230814
2.0.230618
2.0.230528
1.2.230428
1.1.230416
1.0.230410
Labels
Clear labels   
AMD/Intel

   
bug

   
bug

   
bug

   
colab

   
dependencies

   
documentation

   
duplicate

   
enhancement

   
following up

   
good first issue

   
help wanted

   
invalid

   
question

   
stale

   
stale

   
stale

   
todo list

   
wontfix
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#860
No description provided.