[GH-ISSUE #1547] MultiGPU training seems slower than single #1021

New issue

Closed

opened 2026-10-20 10:05:11 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-20 10:05:11 +00:00

Owner

Originally created by @blakejrobinson on GitHub (Nov 15, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1547

I recently moved an extra GPU into my machine from another (mainly to help with other ML tasks). I figured I could also use it to speed up the training of RVC voices. I have a 4090 and a 3090.

However, when using multiple GPUs, something weird happens -

The step numbers seem different. For example with the single GPU, at epoch 20 it's reached step 115 of this dataset, but with double GPU it's reached step 95.
Looking at TensorBoard, the 2x GPU instance seems to be trained at half the speed of the 1xGPU training - for example:

The image above shows the exact same point of time in the training on both runs. The purple is the single GPU setup whereas the grey is the double GPU run. The curve itself looks similar; it just seems that the double GPU setup is taking over twice as long to get there, which seems counter-intuitive.

Same settings for both runs, just with the second one, gpu to use was changed from 0-1 to 0. Both are the same dataset (45 wavs), 48k, v2.

Is multi-gpu just not really supported with this kind of setup?

Originally created by @blakejrobinson on GitHub (Nov 15, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/1547 I recently moved an extra GPU into my machine from another (mainly to help with other ML tasks). I figured I could also use it to speed up the training of RVC voices. I have a 4090 and a 3090. However, when using multiple GPUs, something weird happens - 1) The step numbers seem different. For example with the single GPU, at epoch 20 it's reached step 115 of this dataset, but with double GPU it's reached step 95. 2) Looking at TensorBoard, the 2x GPU instance seems to be trained at half the speed of the 1xGPU training - for example: ![image](https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/assets/9593019/e4482435-0f05-475a-a639-be27628c080a) The image above shows the exact same point of time in the training on both runs. The purple is the single GPU setup whereas the grey is the double GPU run. The curve itself looks similar; it just seems that the double GPU setup is taking over twice as long to get there, which seems counter-intuitive. Same settings for both runs, just with the second one, gpu to use was changed from 0-1 to 0. Both are the same dataset (45 wavs), 48k, v2. Is multi-gpu just not really supported with this kind of setup?

bsakaguchi

2026-10-20 10:05:11 +00:00

closed this issue
added the
AMD/Intel

stale
labels

bsakaguchi commented

2026-10-20 10:05:12 +00:00

Author

Owner

@treya-lin commented on GitHub (Nov 15, 2023):

Hi, I also tried multi-gpu training once but didn't work out. May I ask: when you train with two cards, how did the required virtual memory change compared to single-gpu mode? Would each GPU require more memory in that case?

@treya-lin commented on GitHub (Nov 15, 2023): Hi, I also tried multi-gpu training once but didn't work out. May I ask: when you train with two cards, how did the required virtual memory change compared to single-gpu mode? Would each GPU require more memory in that case?

bsakaguchi commented

2026-10-20 10:05:12 +00:00

Author

Owner

@blakejrobinson commented on GitHub (Nov 15, 2023):

I had a quick check on vram usage.

dual-gpu mode, each card used ~14.5gb (with a batch size per gpu of 12).
single-gpu mode, the single card uses 8gb (with a batch size per gpu of 12).

This again doesn't make sense to me - I'd expect memory use to be dictated by batch size, so be close, or the same. Would there really be 13gb overhead (6.5/card)?

@blakejrobinson commented on GitHub (Nov 15, 2023): I had a quick check on vram usage. - dual-gpu mode, each card used ~14.5gb (with a batch size per gpu of 12). - single-gpu mode, the single card uses 8gb (with a batch size per gpu of 12). This again doesn't make sense to me - I'd expect memory use to be dictated by batch size, so be close, or the same. Would there really be 13gb overhead (6.5/card)?

bsakaguchi commented

2026-10-20 10:05:12 +00:00

Author

Owner

@treya-lin commented on GitHub (Nov 17, 2023):

I had a quick check on vram usage.

dual-gpu mode, each card used ~14.5gb (with a batch size per gpu of 12).

single-gpu mode, the single card uses 8gb (with a batch size per gpu of 12).

This again doesn't make sense to me - I'd expect memory use to be dictated by batch size, so be close, or the same. Would there really be 13gb overhead (6.5/card)?

Hi thanks for your reply! Yes the VRAM increase looks very strange, and it does not make sense that it slows down the training.
Let's see if the maintainers could figure out what might have been going wrong.

@treya-lin commented on GitHub (Nov 17, 2023): > I had a quick check on vram usage. > > * dual-gpu mode, each card used ~14.5gb (with a batch size per gpu of 12). > * single-gpu mode, the single card uses 8gb (with a batch size per gpu of 12). > > This again doesn't make sense to me - I'd expect memory use to be dictated by batch size, so be close, or the same. Would there really be 13gb overhead (6.5/card)? Hi thanks for your reply! Yes the VRAM increase looks very strange, and it does not make sense that it slows down the training. Let's see if the maintainers could figure out what might have been going wrong.

bsakaguchi commented

2026-10-20 10:05:12 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 28, 2024):

This issue was closed because it has been inactive for 15 days since being marked as stale.

@github-actions[bot] commented on GitHub (Apr 28, 2024): This issue was closed because it has been inactive for 15 days since being marked as stale.