[GH-ISSUE #2269] Hubert (latent variables) to VITS? #1552

New issue

Open

opened 2026-10-20 10:10:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-20 10:10:08 +00:00

Owner

Originally created by @Lukysoon on GitHub (Aug 28, 2024).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/2269

The VITS project contains posterior encoder which converts audio to latent space variables.
But HuBERT does the same.

Does RVC work by generating latent space variables with HuBert and than use it for voice conversion in VITS?

Thank you for answer.

Originally created by @Lukysoon on GitHub (Aug 28, 2024). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/2269 The VITS project contains posterior encoder which converts audio to latent space variables. But HuBERT does the same. Does RVC work by generating latent space variables with HuBert and than use it for voice conversion in VITS? Thank you for answer.

bsakaguchi commented

2026-10-20 10:10:08 +00:00

Author

Owner

@Nian-Ci commented on GitHub (Oct 3, 2024):

Yes, RVC is inspired by the well-known So-VITS-SVC, which in turn is inspired by Soft-VC (repository: https://github.com/bshall/soft-vc, paper: https://arxiv.org/abs/2111.02392). The author replaced the acoustic model and vocoder in Soft-VC with VITS.

@Nian-Ci commented on GitHub (Oct 3, 2024): Yes, RVC is inspired by the well-known So-VITS-SVC, which in turn is inspired by Soft-VC (repository: https://github.com/bshall/soft-vc, paper: https://arxiv.org/abs/2111.02392). The author replaced the acoustic model and vocoder in Soft-VC with VITS.

bsakaguchi commented

2026-10-20 10:10:09 +00:00

Author

Owner

@Lukysoon commented on GitHub (Oct 3, 2024):

Thank you!

@Lukysoon commented on GitHub (Oct 3, 2024): Thank you!

bsakaguchi commented

2026-10-20 10:10:09 +00:00

Author

Owner

@Lukysoon commented on GitHub (Oct 3, 2024):

Do you know about a code which could fine-tune VITS model? It seems that RVC trains entire model. We would like to experiment with just fine-tuning.
Thank you.

@Lukysoon commented on GitHub (Oct 3, 2024): Do you know about a code which could fine-tune VITS model? It seems that RVC trains entire model. We would like to experiment with just fine-tuning. Thank you.