[GH-ISSUE #244] Slower Multi-GPU training with 2x the number of GPUs and 4x the amount of VRAM #117

New issue

Open

opened 2026-10-20 09:55:27 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-20 09:55:27 +00:00

Owner

Originally created by @scf4 on GitHub (May 7, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/244

I have two systems training on identical datasets

System A has 4 x NVIDIA RTX A5000 (24GB VRAM per GPU), and a batch size of 12 per GPU.

System B has 7 x NVIDIA RTX A6000 (48GB VRAM per GPU), and a batch size of 18 per GPU.

I would expect System B to train much faster. However...

System A (96GB total VRAM, batch size 12) takes 11 seconds per epoch.
System B (336GB total VRAM, batch size 18) takes 13 seconds per epoch.

I'm wondering if this is down to the overhead of multi-GPU training, or if there's something I'm missing here?

Thank you

Originally created by @scf4 on GitHub (May 7, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/244 I have two systems training on identical datasets System A has 4 x NVIDIA RTX A5000 (24GB VRAM per GPU), and a batch size of 12 per GPU. System B has 7 x NVIDIA RTX A6000 (48GB VRAM per GPU), and a batch size of 18 per GPU. I would expect System B to train much faster. However... - System A (96GB total VRAM, batch size 12) takes 11 seconds per epoch. - System B (336GB total VRAM, batch size 18) takes 13 seconds per epoch. I'm wondering if this is down to the overhead of multi-GPU training, or if there's something I'm missing here? Thank you

bsakaguchi added the

help wanted

question

labels

2026-10-20 09:55:27 +00:00

bsakaguchi commented

2026-10-20 09:55:27 +00:00

Author

Owner

@BillieFrano commented on GitHub (Oct 3, 2023):

The vram does not accumulate, it does not matter if you have 1 or 100 GPUs. Your first item is 24 and the second item is 48. I think you could increase the batch size to 40, but this does not answer the question. We would have to see what other components are in the machines, ram, processor, mother...

@BillieFrano commented on GitHub (Oct 3, 2023): The vram does not accumulate, it does not matter if you have 1 or 100 GPUs. Your first item is 24 and the second item is 48. I think you could increase the batch size to 40, but this does not answer the question. We would have to see what other components are in the machines, ram, processor, mother...