SillyTavern/SillyTavern-Extras
1
0
Fork 0
mirror of https://github.com/SillyTavern/SillyTavern-Extras.git synced 2026-10-16 02:21:52 +00:00
Code Issues 38 Projects Packages Wiki Activity

[GH-ISSUE #178] Attempting to do TTS with Coqui and Tortoise results in "missing keys in state dict" #103

New issue
Closed
opened 2026-10-15 18:04:27 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-15 18:04:27 +00:00
Owner
Copy link

Originally created by @InconsolableCellist on GitHub (Oct 24, 2023).
Original GitHub issue: https://github.com/SillyTavern/SillyTavern-Extras/issues/178

I'm trying to generate voices with TTS using Tortoise and it's resulting in an error.

Steps to reproduce:

  1. Go to SillyTavern. Go to the Extensions tab. Go to TTS and set the following options:
  • TTS Provider: Coqui
  • Enabled: Checked
  • Auto Generation: Checked
  • Models: Coqui API (Experimental)
  • Model language: English
  • Mode: tortoise-v2 (multi-dataset dataset)

  1. Click "Add Voice" and name it "new"

  2. Select "new" as the Default Voice

  3. Go back to a story and click the megaphone icon to narrate the text

Observed Behavior:

No audio is output. The console output is:

2023-10-24 11:33:57 | INFO | werkzeug | xx.xx.xx.xx - - [24/Oct/2023 11:33:57] "OPTIONS /api/text-to-speech/coqui/generate-tts HTTP/1.1" 200 -
<Coqui-TTS module> Received TTS request for  {'text': "Test", 'model_id': 'tts_models/en/multi-dataset/tortoise-v2', 'language_id': None, 'speaker_id': None}
<Coqui-TTS module> Loading tts 
- model tts_models/en/multi-dataset/tortoise-v2 
 - speaker_id:  None 
 - language_id:  None 
 - using GPU
 > Downloading model to /home/user/.local/share/tts/tts_models--en--multi-dataset--tortoise-v2
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.72G/1.72G [02:03<00:00, 13.9MiB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 976M/976M [01:10<00:00, 13.8MiB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151M/151M [00:11<00:00, 13.0MiB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.17G/1.17G [01:14<00:00, 15.7MiB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.2M/25.2M [00:06<00:00, 4.06MiB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 101M/101M [00:07<00:00, 13.5MiB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 391M/391M [00:27<00:00, 14.4MiB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.07k/1.07k [00:01<00:00, 920iB/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.40k/4.40k [00:00<00:00, 873kiB/s]
 > Model's license - apache 2.0
 > Check https://choosealicense.com/licenses/apache-2.0/ for more info.
 > Using model: tortoise
Error(s) in loading state_dict for UnifiedVoice:
        Missing key(s) in state_dict: "gpt.h.0.attn.masked_bias", "gpt.h.1.attn.masked_bias", "gpt.h.2.attn.masked_bias", "gpt.h.3.attn.masked_bias", "gpt.h.4.attn.masked_bias", "gpt.h.5.attn.masked_bias", "gpt.h.6.attn.masked_bias", "gpt.h.7.attn.masked_bias", "gpt.h.8.attn.masked_bias", "gpt.h.9.attn.masked_bias", "gpt.h.10.attn.masked_bias", "gpt.h.11.attn.masked_bias", "gpt.h.12.attn.masked_bias", "gpt.h.13.attn.masked_bias", "gpt.h.14.attn.masked_bias", "gpt.h.15.attn.masked_bias", "gpt.h.16.attn.masked_bias", "gpt.h.17.attn.masked_bias", "gpt.h.18.attn.masked_bias", "gpt.h.19.attn.masked_bias", "gpt.h.20.attn.masked_bias", "gpt.h.21.attn.masked_bias", "gpt.h.22.attn.masked_bias", "gpt.h.23.attn.masked_bias", "gpt.h.24.attn.masked_bias", "gpt.h.25.attn.masked_bias", "gpt.h.26.attn.masked_bias", "gpt.h.27.attn.masked_bias", "gpt.h.28.attn.masked_bias", "gpt.h.29.attn.masked_bias".

Expected behavior:

Tortoise TTS works and generates a voice

Environment & More Information:

My launch options:

python3 server.py --enable-modules=caption,summarize,classify,sd,coqui-tts,rvc --listen --sd-remote --sd-remote-host xx.xx.xx.xx --classification-model joeddav/distilbert-base-uncased-go-emotions-student --coqui-gpu

On branch main, commit 2830a5ff

requirements-complete.txt lists the TTS dependency version as 0.15.6, which I have:

$ python
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import TTS
>>> print(TTS.__version__)
0.15.6

My transformers version:

$ pip list | grep -i transformers
...
transformers              4.28.1

The requirements file specifies that version.

Attempted fix

I found this issue on the tortoise-tts project that says to downgrade (upgrade in our case) to 4.29.2. I tried that and it didn't work, unfortunately.

I'm really surprised that solution above didn't work, so I tried twice. I'm not sure what else could be going on. I'm using micromamba and installed transformers and SillyTavern-extras in the same environment, and it doesn't look like server.py is attempting to change to some other venv.

Originally created by @InconsolableCellist on GitHub (Oct 24, 2023). Original GitHub issue: https://github.com/SillyTavern/SillyTavern-Extras/issues/178 I'm trying to generate voices with TTS using Tortoise and it's resulting in an error. ## Steps to reproduce: 1. Go to SillyTavern. Go to the Extensions tab. Go to TTS and set the following options: * TTS Provider: Coqui * Enabled: Checked * Auto Generation: Checked * Models: Coqui API (Experimental) * Model language: English * Mode: tortoise-v2 (multi-dataset dataset) 2. Click "Add Voice" and name it "new" 3. Select "new" as the Default Voice 4. Go back to a story and click the megaphone icon to narrate the text ## Observed Behavior: No audio is output. The console output is: ``` 2023-10-24 11:33:57 | INFO | werkzeug | xx.xx.xx.xx - - [24/Oct/2023 11:33:57] "OPTIONS /api/text-to-speech/coqui/generate-tts HTTP/1.1" 200 - <Coqui-TTS module> Received TTS request for {'text': "Test", 'model_id': 'tts_models/en/multi-dataset/tortoise-v2', 'language_id': None, 'speaker_id': None} <Coqui-TTS module> Loading tts - model tts_models/en/multi-dataset/tortoise-v2 - speaker_id: None - language_id: None - using GPU > Downloading model to /home/user/.local/share/tts/tts_models--en--multi-dataset--tortoise-v2 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.72G/1.72G [02:03<00:00, 13.9MiB/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 976M/976M [01:10<00:00, 13.8MiB/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151M/151M [00:11<00:00, 13.0MiB/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.17G/1.17G [01:14<00:00, 15.7MiB/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.2M/25.2M [00:06<00:00, 4.06MiB/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 101M/101M [00:07<00:00, 13.5MiB/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 391M/391M [00:27<00:00, 14.4MiB/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.07k/1.07k [00:01<00:00, 920iB/s] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.40k/4.40k [00:00<00:00, 873kiB/s] > Model's license - apache 2.0 > Check https://choosealicense.com/licenses/apache-2.0/ for more info. > Using model: tortoise Error(s) in loading state_dict for UnifiedVoice: Missing key(s) in state_dict: "gpt.h.0.attn.masked_bias", "gpt.h.1.attn.masked_bias", "gpt.h.2.attn.masked_bias", "gpt.h.3.attn.masked_bias", "gpt.h.4.attn.masked_bias", "gpt.h.5.attn.masked_bias", "gpt.h.6.attn.masked_bias", "gpt.h.7.attn.masked_bias", "gpt.h.8.attn.masked_bias", "gpt.h.9.attn.masked_bias", "gpt.h.10.attn.masked_bias", "gpt.h.11.attn.masked_bias", "gpt.h.12.attn.masked_bias", "gpt.h.13.attn.masked_bias", "gpt.h.14.attn.masked_bias", "gpt.h.15.attn.masked_bias", "gpt.h.16.attn.masked_bias", "gpt.h.17.attn.masked_bias", "gpt.h.18.attn.masked_bias", "gpt.h.19.attn.masked_bias", "gpt.h.20.attn.masked_bias", "gpt.h.21.attn.masked_bias", "gpt.h.22.attn.masked_bias", "gpt.h.23.attn.masked_bias", "gpt.h.24.attn.masked_bias", "gpt.h.25.attn.masked_bias", "gpt.h.26.attn.masked_bias", "gpt.h.27.attn.masked_bias", "gpt.h.28.attn.masked_bias", "gpt.h.29.attn.masked_bias". ``` ## Expected behavior: Tortoise TTS works and generates a voice ## Environment & More Information: My launch options: `python3 server.py --enable-modules=caption,summarize,classify,sd,coqui-tts,rvc --listen --sd-remote --sd-remote-host xx.xx.xx.xx --classification-model joeddav/distilbert-base-uncased-go-emotions-student --coqui-gpu` On branch `main`, commit 2830a5ff `requirements-complete.txt` lists the TTS dependency version as `0.15.6`, which I have: ``` $ python Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import TTS >>> print(TTS.__version__) 0.15.6 ``` My transformers version: ``` $ pip list | grep -i transformers ... transformers 4.28.1 ``` The requirements file specifies that version. ## Attempted fix I found [this issue](https://github.com/neonbjb/tortoise-tts/issues/472) on the tortoise-tts project that says to downgrade (upgrade in our case) to `4.29.2`. I tried that and it didn't work, unfortunately. I'm really surprised that solution above didn't work, so I tried twice. I'm not sure what else could be going on. I'm using `micromamba` and installed transformers and SillyTavern-extras in the same environment, and it doesn't look like `server.py` is attempting to change to some other venv.
bsakaguchi commented 2026-10-15 18:04:28 +00:00
Author
Owner
Copy link

@Ph0rk0z commented on GitHub (Oct 28, 2023):

With a newer version, even my custom tortoise model worked no problem. The coqui in requirements is relatively old.

Just upgrade coqui. They also released XTTS which is much faster tortoise with clone but this repo doesn't provide the required wav file. Using it from the command line the inference was much much faster.

<!-- gh-comment-id:1783915785 --> @Ph0rk0z commented on GitHub (Oct 28, 2023): With a newer version, even my custom tortoise model worked no problem. The coqui in requirements is relatively old. Just upgrade coqui. They also released XTTS which is much faster tortoise with clone but this repo doesn't provide the required wav file. Using it from the command line the inference was much much faster.
bsakaguchi commented 2026-10-15 18:04:28 +00:00
Author
Owner
Copy link

@Cohee1207 commented on GitHub (Oct 28, 2023):

I'd like to update the coqui TTS version, however the newer version of coqui conflicts with chromadb package. It's possible to just upgrade it separately by editing out chroma from requirements file.

<!-- gh-comment-id:1783918751 --> @Cohee1207 commented on GitHub (Oct 28, 2023): I'd like to update the coqui TTS version, however the newer version of coqui conflicts with chromadb package. It's possible to just upgrade it separately by editing out chroma from requirements file.
bsakaguchi commented 2026-10-15 18:04:28 +00:00
Author
Owner
Copy link

@Ph0rk0z commented on GitHub (Oct 29, 2023):

I haven't tried to use chromadb since you've implemented the transformers version of the same thing. Does chromadb crash when you install it first and install new COQUI over it? Things complain of conflicts and still run 90% of the time.

<!-- gh-comment-id:1784101280 --> @Ph0rk0z commented on GitHub (Oct 29, 2023): I haven't tried to use chromadb since you've implemented the transformers version of the same thing. Does chromadb crash when you install it first and install new COQUI over it? Things complain of conflicts and still run 90% of the time.
bsakaguchi commented 2026-10-15 18:04:29 +00:00
Author
Owner
Copy link

@Cohee1207 commented on GitHub (Oct 29, 2023):

The problem comes not from torch or transformers, but from tqdm this time. Coqui has it locked on 4.64.*, chroma requires 4.65 and above.

<!-- gh-comment-id:1784134047 --> @Cohee1207 commented on GitHub (Oct 29, 2023): The problem comes not from torch or transformers, but from tqdm this time. Coqui has it locked on 4.64.*, chroma requires 4.65 and above.
bsakaguchi commented 2026-10-15 18:04:29 +00:00
Author
Owner
Copy link

@Ph0rk0z commented on GitHub (Oct 29, 2023):

Does either one not work with the "wrong" TQDM? I distinctly remember running newer coqui than in the req and using chromadb. I'll give it another go as I have it now and see what happens.

edit: I tested it and chromadb loaded normally for me, but I had it previously installed.

<!-- gh-comment-id:1784155323 --> @Ph0rk0z commented on GitHub (Oct 29, 2023): Does either one not work with the "wrong" TQDM? I distinctly remember running newer coqui than in the req and using chromadb. I'll give it another go as I have it now and see what happens. edit: I tested it and chromadb loaded normally for me, but I had it previously installed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
bug

   
duplicate

   
planned

   
pull-request
Mirrored from GitHub Pull Request

  
under consideration
No labels
bug
duplicate
planned
pull-request
under consideration
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
SillyTavern/SillyTavern-Extras#103
No description provided.