mirror of
https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git
synced 2026-11-05 00:27:30 +00:00
[GH-ISSUE #662] Multi-speaker training failing #388
Labels
No labels
AMD/Intel
bug
bug
bug
colab
dependencies
documentation
duplicate
enhancement
following up
good first issue
help wanted
invalid
question
stale
stale
stale
todo list
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
RVC-Project/Retrieval-based-Voice-Conversion-WebUI#388
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Rolun on GitHub (Jul 1, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/662
Originally assigned to: @Tps-F on GitHub.
Hello,
I just trained on two speakers at the same time.
The filelist looks like this:
I have 184 samples of the first speaker, and 169 of the second, so fairly balanced.
When running Feature Extraction, I used Crepe with a lower hop for speaker 0 than for speaker 1.
I have double-checked that the audio files and the speaker id go in correctly into the forward pass of the model.
The speaker embeddings for 0 and 1 also come out different after training is finished, so something worked.
However, when inferencing on new audio, the output when using speaker ID 0 and speaker ID 1 both sound identical (and mostly like speaker 0, but sometimes they both sound like speaker 1 - for example when inferencing on training data for speaker 1).
They also look almost identical - only tiny differences that are not audible to my ear when comparing them in Audacity.
This is true when comparing the trained model after 1 epoch, 125 epochs, or 300 epochs (and in between).
It's almost like something went wrong with the speaker separation/disentangling.
When training the two speakers by themselves (one model each), the difference is very noticeable.
Any tips on where I might have gone wrong in training on multiple speakers?
Many thanks in advance!
@suzhenghang commented on GitHub (Jul 3, 2023):
@Rolun Is it possible to generate a corresponding faiss index file for two speaker models? May both two speakers use the index for speaker 0?
@Rolun commented on GitHub (Jul 3, 2023):
@suzhenghang Great point!
I ran some tests to see the impact of the faiss index on the different models.
Seems like you are right, and I found some other interesting things.
Here are the results:
What's interesting is that the faiss index does not seem to have much impact on the individual models, but means a lot for the mixed model.
And when turning off the faiss index (setting it to 0 in the interface), we come back to the speaker id not having any real impact on the output.
So although the faiss index has a large impact on the mixed model, I'm still curious as to why the speaker id doesn't have any impact in comparison.
Maybe that's just a property of this model (?), but it seems like something is still off.
Is the weights of the base model available somewhere so I can try speaker id there? I found this: https://huggingface.co/lj1995/VoiceConversionWebUI/blob/main/weights/%E7%99%BD%E8%8F%9C357k.pt but not sure if it's correct (gives a crash and multiple missing/unexpected keys when loaded in get_vc)
@Rolun commented on GitHub (Jul 3, 2023):
Update:
I re-initialized the emb_g layer and set it to only have 2 speakers, redid the fine-tuning and now it works great :)
I also misspoke in my last comment (sorry @suzhenghang ), the faiss index does contain both speaker voices, as seen from it using speaker 1 in some cases and speaker 0 in others. From the testing, if it only got samples from speaker 0 it likely would have sounded like speaker 0 for all tests.
It's still a mystery to me how the faiss index had such an influence on the outcome though - will keep experimenting with that.
@bochch commented on GitHub (Aug 13, 2023):
Hi. Is there any guide of how to train a model with several speakers? I tried a couple times experimenting with this but with no luck. @RVC-Boss I would appreciate any help
@yaodada1 commented on GitHub (Sep 11, 2023):
@Rolun Hi，I just tried training multi speakers model，my filelist.txt like this，after train the model with 500 epoch and index training，the vc results is wrong。No matter which index and sid I use to test, the results are only similar to speaker 0。
@Rolun commented on GitHub (Sep 11, 2023):
@yaodada1 RVC does not support multi-training natively. What worked for me was to set
hps.model.spk_embed_dimto number of speakers (4 in your case) and then replace theemb_g.weightwith a new one (so it starts training the speaker embedding layer from scratch) - I did this by a simple line like this:tmp["emb_g.weight"] = nn.Embedding(NUMBER_OF_SPEAKERS, 256).weight.data.You want to make both of these changes in train_nsf_sim_cache_sid_load_pretrain.py
@yaodada1 commented on GitHub (Sep 11, 2023):
thank you very much, i'll try.
@yaodada1 commented on GitHub (Sep 25, 2023):
It works，thank you again！
@Rolun commented on GitHub (Sep 25, 2023):
@yaodada1 Great to hear! :)
@Alidaling commented on GitHub (Jan 10, 2024):
"I didn't find the script train_nsf_sim_cache_sid_load_pretrain. Where can I find it? Looking forward to your reply."Thank you！
@kunibald413 commented on GitHub (Mar 17, 2024):
it was renamed to
infer/modules/train/train.pygithub.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI@b3d7075ba4@Frederieke93 commented on GitHub (Apr 4, 2024):
Whenever I am training multi-speaker, I notice that the voice characteristics are a bit similar.. It's not working as good as when I am doing a single-speaker training. I have re-initialized the embedding layer emb_g and set the new number of speakers, but still not the best output.
Anyone else has seen this problem and have found a solution for this?
Thank you!
@github-actions[bot] commented on GitHub (May 21, 2024):
This issue was closed because it has been inactive for 15 days since being marked as stale.
@Me210400 commented on GitHub (Jun 25, 2024):
Hi, how you get this filelist.txt?