[GH-ISSUE #840] Very long epoch generation #536

New issue

Open

opened 2026-10-20 10:00:23 +00:00 by bsakaguchi · 18 comments

bsakaguchi commented 2026-10-20 10:00:23 +00:00

Owner

Copy link

Originally created by @Trolus55 on GitHub (Jul 23, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/840

I have 1277 around 5s wav files for training. On my RTX 3080 it took 13 hours to generate 10 epochs, using 26 batch size. Is it normal for it to take so long with this dataset? I want to train on total 200 epochs, how do I go about it then?

Originally created by @Trolus55 on GitHub (Jul 23, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/840 I have 1277 around 5s wav files for training. On my RTX 3080 it took 13 hours to generate 10 epochs, using 26 batch size. Is it normal for it to take so long with this dataset? I want to train on total 200 epochs, how do I go about it then?

bsakaguchi added the

following up

todo list

help wanted

bug

enhancement

question

labels 2026-10-20 10:00:23 +00:00

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@Hivemind11 commented on GitHub (Jul 23, 2023):

Same problem. mine is 6 ins per epoch but the total training time would be 40 hours for training 15 min sample for 400 epochs. 16 threads and 40 batch size on my 3080.

<!-- gh-comment-id:1646920405 --> @Hivemind11 commented on GitHub (Jul 23, 2023): Same problem. mine is 6 ins per epoch but the total training time would be 40 hours for training 15 min sample for 400 epochs. 16 threads and 40 batch size on my 3080.

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@Trolus55 commented on GitHub (Jul 24, 2023):

Same problem. mine is 6 ins per epoch but the total training time would be 40 hours for training 15 min sample for 400 epochs. 16 threads and 40 batch size on my 3080.

I uesd 20 threads and on 40 batch I'm getting out of CUDA memory on 10GB 3080, so im clueless why it is that way.

<!-- gh-comment-id:1646999670 --> @Trolus55 commented on GitHub (Jul 24, 2023): > Same problem. mine is 6 ins per epoch but the total training time would be 40 hours for training 15 min sample for 400 epochs. 16 threads and 40 batch size on my 3080. I uesd 20 threads and on 40 batch I'm getting out of CUDA memory on 10GB 3080, so im clueless why it is that way.

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@fumiama commented on GitHub (Jul 24, 2023):

You can open your task manager and monitor the CPU/GPU using status when training. It's better to provide some screenshots. A low GPU load may relate to the RVC codes' bad GPU utilization.

<!-- gh-comment-id:1647606379 --> @fumiama commented on GitHub (Jul 24, 2023): You can open your task manager and monitor the CPU/GPU using status when training. It's better to provide some screenshots. A low GPU load may relate to the RVC codes' bad GPU utilization.

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@Trolus55 commented on GitHub (Jul 24, 2023):

So this are my settings. I didn't attach data process step because there are no settings there

I attached GPU usage to (sorry it's in polish) you can see that 3D usage its at max and the memory is used to. My only concern is that it only heats up 10 C above idle, so this could be a sign that GPU is not used as it should be. (Should it heat more while training? idk)

Driver if that matters:

Here is CPU

And here is the terminal, you can see that it is at 1 epoch and it will stay like that for an hour, and it is like that with every epoch. Basically every epoch takes more than an hour. And I guess it should not be this slow with 3080, on collab I use V100 GPU and its waaaay faster, so really im lost 😂

<!-- gh-comment-id:1648007195 --> @Trolus55 commented on GitHub (Jul 24, 2023): So this are my settings. I didn't attach data process step because there are no settings there   I attached GPU usage to (sorry it's in polish) you can see that 3D usage its at max and the memory is used to. My only concern is that it only heats up 10 C above idle, so this could be a sign that GPU is not used as it should be. (Should it heat more while training? idk)  Driver if that matters:  Here is CPU  And here is the terminal, you can see that it is at 1 epoch and it will stay like that for an hour, and it is like that with every epoch. Basically every epoch takes more than an hour. And I guess it should not be this slow with 3080, on collab I use V100 GPU and its waaaay faster, so really im lost 😂 

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@flashbackpoductions commented on GitHub (Jul 25, 2023):

same here
INFO:karyan:Train Epoch: 1 [0%]
INFO:karyan:[0, 0.0001]
INFO:karyan:loss_disc=4.345, loss_gen=4.110, loss_fm=12.065,loss_mel=24.973, loss_kl=6.884
DEBUG:matplotlib:matplotlib data path: E:\RVC-beta0717\runtime\lib\site-packages\matplotlib\mpl-data
DEBUG:matplotlib:CONFIGDIR=C:\Users\User.matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is win32
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
INFO:karyan:====> Epoch: 1 [2023-07-25 23:39:10] | (0:01:42.878054)
INFO:karyan:====> Epoch: 2 [2023-07-25 23:41:25] | (0:02:14.549031)
INFO:karyan:====> Epoch: 3 [2023-07-25 23:45:02] | (0:03:36.795793)
INFO:karyan:====> Epoch: 4 [2023-07-25 23:48:04] | (0:03:02.016243)

is it supposed to take so long? for 200 eptoch its gonna take forever...
is it a bug?

3070TI

100% gpu

<!-- gh-comment-id:1650546565 --> @flashbackpoductions commented on GitHub (Jul 25, 2023): same here INFO:karyan:Train Epoch: 1 [0%] INFO:karyan:[0, 0.0001] INFO:karyan:loss_disc=4.345, loss_gen=4.110, loss_fm=12.065,loss_mel=24.973, loss_kl=6.884 DEBUG:matplotlib:matplotlib data path: E:\RVC-beta0717\runtime\lib\site-packages\matplotlib\mpl-data DEBUG:matplotlib:CONFIGDIR=C:\Users\User\.matplotlib DEBUG:matplotlib:interactive is False DEBUG:matplotlib:platform is win32 INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. INFO:karyan:====> Epoch: 1 [2023-07-25 23:39:10] | (0:01:42.878054) INFO:karyan:====> Epoch: 2 [2023-07-25 23:41:25] | (0:02:14.549031) INFO:karyan:====> Epoch: 3 [2023-07-25 23:45:02] | (0:03:36.795793) INFO:karyan:====> Epoch: 4 [2023-07-25 23:48:04] | (0:03:02.016243) is it supposed to take so long? for 200 eptoch its gonna take forever... is it a bug? 3070TI 100% gpu

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@RVC-Boss commented on GitHub (Jul 26, 2023):

Check if you use fp16 config.
in configs/xxk.json (xx is the sample rate of the model you used)
fp16_run
changing it to true can greatly improve the training speed in N cards such as V100, 3080

<!-- gh-comment-id:1650953404 --> @RVC-Boss commented on GitHub (Jul 26, 2023): Check if you use fp16 config. in configs/xxk.json (xx is the sample rate of the model you used) fp16_run changing it to true can greatly improve the training speed in N cards such as V100, 3080

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@utilbird commented on GitHub (Jul 26, 2023):

It looks like you're using shared GPU memory, which means your batch size is too big.

Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka shared GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance.
Try halving your batch size to start with and work your way up from there.

IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is.

<!-- gh-comment-id:1651746558 --> @utilbird commented on GitHub (Jul 26, 2023): It looks like you're using shared GPU memory, which means your batch size is too big. Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka _shared_ GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance. Try halving your batch size to start with and work your way up from there. IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is.

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@flashbackpoductions commented on GitHub (Jul 26, 2023):

It looks like you're using shared GPU memory, which means your batch size is too big.

Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka shared GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance. Try halving your batch size to start with and work your way up from there.

IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is.

yes, this is the prob.
I managed to decipher the issue last night by trial and error, and I still haven't had time to report it here. Now I also understand the logic behind it.
I think in one of the updates, some kind of warning should be written about it. Because everyone is just trying to load as much memory as possible in training.

<!-- gh-comment-id:1651931645 --> @flashbackpoductions commented on GitHub (Jul 26, 2023): > It looks like you're using shared GPU memory, which means your batch size is too big. > > Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka _shared_ GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance. Try halving your batch size to start with and work your way up from there. > > IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is. yes, this is the prob. I managed to decipher the issue last night by trial and error, and I still haven't had time to report it here. Now I also understand the logic behind it. I think in one of the updates, some kind of warning should be written about it. Because everyone is just trying to load as much memory as possible in training.

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@Trolus55 commented on GitHub (Jul 26, 2023):

Ok, so I set up virtual enviroment instead and used a batch size of 10 and yeah it's faster, one epoch takes around 3 minutes. So if my 3080 has 10GB VRAM I should use no more than 10 batch size, correct?

Side question: There is now rmvpe added, when I use it I get this output:

Extraction ends normally, should I ignore it?

<!-- gh-comment-id:1652566735 --> @Trolus55 commented on GitHub (Jul 26, 2023): Ok, so I set up virtual enviroment instead and used a batch size of 10 and yeah it's faster, one epoch takes around 3 minutes. So if my 3080 has 10GB VRAM I should use no more than 10 batch size, correct? Side question: There is now rmvpe added, when I use it I get this output:  Extraction ends normally, should I ignore it?

bsakaguchi commented 2026-10-20 10:00:24 +00:00

Author

Owner

Copy link

@RVC-Boss commented on GitHub (Jul 27, 2023):

Ok, so I set up virtual enviroment instead and used a batch size of 10 and yeah it's faster, one epoch takes around 3 minutes. So if my 3080 has 10GB VRAM I should use no more than 10 batch size, correct?

Side question: There is now rmvpe added, when I use it I get this output:

Extraction ends normally, should I ignore it?

You can ignore it, it happens because the audio is too short that is not suitable as training set
It will not affect the processing of the remaining normal data

<!-- gh-comment-id:1652860645 --> @RVC-Boss commented on GitHub (Jul 27, 2023): > Ok, so I set up virtual enviroment instead and used a batch size of 10 and yeah it's faster, one epoch takes around 3 minutes. So if my 3080 has 10GB VRAM I should use no more than 10 batch size, correct? > > Side question: There is now rmvpe added, when I use it I get this output:  > > Extraction ends normally, should I ignore it? You can ignore it, it happens because the audio is too short that is not suitable as training set It will not affect the processing of the remaining normal data

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@RVC-Boss commented on GitHub (Jul 27, 2023):

It looks like you're using shared GPU memory, which means your batch size is too big.

Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka shared GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance. Try halving your batch size to start with and work your way up from there.

IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is.

wow, good job!
Do you know how to disable shared GPU memory for pytorch? I think RVC should notice GPU memory error instead of training very very slowly.

<!-- gh-comment-id:1652861099 --> @RVC-Boss commented on GitHub (Jul 27, 2023): > It looks like you're using shared GPU memory, which means your batch size is too big. > > Because you're trying to allocate more than 10GB of gpu memory, it'll be allocating the 'overflow' memory into the normal system RAM (aka _shared_ GPU memory). This massively increases memory latency (among other things), which greatly reduces your performance. Try halving your batch size to start with and work your way up from there. > > IMO the library shouldn't let you use shared ram in the first place, it's almost never desired behavior due to how abysmal the performance is. wow, good job! Do you know how to disable shared GPU memory for pytorch? I think RVC should notice GPU memory error instead of training very very slowly.

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@balu100 commented on GitHub (Aug 16, 2023):

Same issue here, shared memory slows down the training extremely.
I've tried to search for a way to disable this but could not find anything.
Any input/suggestion would be appreciated.

<!-- gh-comment-id:1680281477 --> @balu100 commented on GitHub (Aug 16, 2023): Same issue here, shared memory slows down the training extremely. I've tried to search for a way to disable this but could not find anything. Any input/suggestion would be appreciated.

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@Vagenda commented on GitHub (Sep 25, 2023):

I'm having performance issues as well. AMD Threadripper 3970X, nVidia RTX 4080 (16GB VRAM), 128GB RAM, 4TB Gen4 M.2. Got to the Step 2b Feature extraction and gave it 5 days - it never completed. I got 1.5GB of source audio all together for it. Throughout, GPU and VRAM were at 100%, and CPU was about 60% for the first few days, down to 40% later. Temps were ok. Just not sure what to do to fix this. I wish the process has a PAUSE/RESUME option.

<!-- gh-comment-id:1733783896 --> @Vagenda commented on GitHub (Sep 25, 2023): I'm having performance issues as well. AMD Threadripper 3970X, nVidia RTX 4080 (16GB VRAM), 128GB RAM, 4TB Gen4 M.2. Got to the Step 2b Feature extraction and gave it 5 days - it never completed. I got 1.5GB of source audio all together for it. Throughout, GPU and VRAM were at 100%, and CPU was about 60% for the first few days, down to 40% later. Temps were ok. Just not sure what to do to fix this. I wish the process has a PAUSE/RESUME option.

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@KCGD commented on GitHub (Sep 25, 2023):

Currently running with an i7-10750H and RTX 2060 (6gb vram) on Linux.

Running with ~50 minutes of audio, batch size of 3, for 100 epochs. Getting ~2:40 per epoch. Vram sitting around 4.9 gigs.

Not sure what to do to speed it up but I hope this info helps

<!-- gh-comment-id:1734457973 --> @KCGD commented on GitHub (Sep 25, 2023): Currently running with an i7-10750H and RTX 2060 (6gb vram) on Linux. Running with ~50 minutes of audio, batch size of 3, for 100 epochs. Getting ~2:40 per epoch. Vram sitting around 4.9 gigs. Not sure what to do to speed it up but I hope this info helps

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@ShuFengYingt commented on GitHub (Oct 5, 2023):

Currently running with an RTX4090,2870 around 5~10 s wav files for training. Almost 2 minutes per epoch.Batch size is 12. It only uses 9GB for training and the average usage of the GPU is only nearly 60%. Don't know how to speed up it. Maybe increase the batch size?

<!-- gh-comment-id:1748805745 --> @ShuFengYingt commented on GitHub (Oct 5, 2023): Currently running with an RTX4090,2870 around 5~10 s wav files for training. Almost 2 minutes per epoch.Batch size is 12. It only uses 9GB for training and the average usage of the GPU is only nearly 60%. Don't know how to speed up it. Maybe increase the batch size?

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@Jatts-Art commented on GitHub (Oct 17, 2023):

I too would also like to know how to disable the shared memory for pytorch. Can't find out any means to do so.

<!-- gh-comment-id:1765678881 --> @Jatts-Art commented on GitHub (Oct 17, 2023): I too would also like to know how to disable the shared memory for pytorch. Can't find out any means to do so.

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@AIhasArrived commented on GitHub (Nov 8, 2023):

Currently running with an RTX4090,2870 around 5~10 s wav files for training. Almost 2 minutes per epoch.Batch size is 12. It only uses 9GB for training and the average usage of the GPU is only nearly 60%. Don't know how to speed up it. Maybe increase the batch size?

5-10 sec wav files, but how many files? I dont have your speed, show me a screenshot of your configuraion if you dont mind?

<!-- gh-comment-id:1802543506 --> @AIhasArrived commented on GitHub (Nov 8, 2023): > Currently running with an RTX4090,2870 around 5~10 s wav files for training. Almost 2 minutes per epoch.Batch size is 12. It only uses 9GB for training and the average usage of the GPU is only nearly 60%. Don't know how to speed up it. Maybe increase the batch size? 5-10 sec wav files, but how many files? I dont have your speed, show me a screenshot of your configuraion if you dont mind?

bsakaguchi commented 2026-10-20 10:00:25 +00:00

Author

Owner

Copy link

@JD-2006 commented on GitHub (Nov 10, 2023):

This may not be related but, I had iterations jump from 17 seconds to 1:30 recently and couldn't figure out what happened.
I checked config.json and saw fp16 was set to false. When I 'reset' it to true, my iterations went back to 17 sec each again...
¯\(ツ)/¯

<!-- gh-comment-id:1806382787 --> @JD-2006 commented on GitHub (Nov 10, 2023): This may not be related but, I had iterations jump from 17 seconds to 1:30 recently and couldn't figure out what happened. I checked config.json and saw fp16 was set to false. When I 'reset' it to true, my iterations went back to 17 sec each again... ¯\\_(ツ)_/¯

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

20240604

v1

2.3.260718

2.2.231006

2.1.230814

2.0.230618

2.0.230528

1.2.230428

1.1.230416

1.0.230410

Labels

Clear labels   

AMD/Intel

  

bug

  

bug

  

bug

  

colab

  

dependencies

  

documentation

  

duplicate

  

enhancement

  

following up

  

good first issue

  

help wanted

  

invalid

  

question

  

stale

  

stale

  

stale

  

todo list

  

wontfix

No labels

AMD/Intel

bug

bug

bug

colab

dependencies

documentation

duplicate

enhancement

following up

good first issue

help wanted

invalid

question

stale

stale

stale

todo list

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

RVC-Project/Retrieval-based-Voice-Conversion-WebUI#536

No description provided.