[PR #207] [MERGED] Talkinghead performance improvements and refactoring #234

New issue

Closed

opened 2026-10-15 18:05:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:05:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/SillyTavern/SillyTavern-Extras/pull/207
Author: @Technologicat
Created: 12/20/2023
Status: ✅ Merged
Merged: 12/21/2023
Merged by: @Cohee1207

Base: main ← Head: appfixes

📝 Commits (10+)

23823fd talkinghead: fix and improve THA3 manual poser
e6f3380 server.py: no, don't yet use fp16 for talkinghead
e02c1d9 talkinghead: remove wxPython dependency from live mode
6c7ff71 comment out unused functions
bc0a832 add TODO list
dd741b6 coding style
0114634 remove unused import
2e01125 add TODO marker
15ce843 message wordings
6d3f32c fix typos in variable names

📊 Changes

22 files changed (+532 additions, -2370 deletions)

View changed files

📝 .gitignore (+1 -0)
📝 requirements-rocm.txt (+1 -0)
📝 requirements-silicon.txt (+1 -0)
📝 requirements.txt (+1 -1)
📝 server.py (+36 -7)
➖ talkinghead/start_standalone_app.sh (+0 -20)
📝 talkinghead/tha3/app/app.py (+286 -928)
📝 talkinghead/tha3/app/manual_poser.py (+12 -159)
➕ talkinghead/tha3/app/util.py (+194 -0)
➖ talkinghead/tha3/mocap/__init__.py (+0 -0)
➖ talkinghead/tha3/mocap/ifacialmocap_constants.py (+0 -239)
➖ talkinghead/tha3/mocap/ifacialmocap_pose.py (+0 -27)
➖ talkinghead/tha3/mocap/ifacialmocap_pose_converter.py (+0 -12)
➖ talkinghead/tha3/mocap/ifacialmocap_poser_converter_25.py (+0 -491)
➖ talkinghead/tha3/mocap/ifacialmocap_v2.py (+0 -89)
➖ talkinghead/tha3/models/LICENSE.txt (+0 -396)
➖ talkinghead/tha3/models/placeholder.txt (+0 -1)
➖ talkinghead/tha3/models/separable_float/editor.pt (+0 -0)
➖ talkinghead/tha3/models/separable_float/eyebrow_decomposer.pt (+0 -0)
➖ talkinghead/tha3/models/separable_float/eyebrow_morphing_combiner.pt (+0 -0)

...and 2 more files

📄 Description

Here's the next PR for talkinghead.

Live mode now runs at 30 FPS on an RTX 3070 Ti mobile.
The SillyTavern-extras server no longer crashes on exit when the talkinghead module is enabled, and also exits faster.
Live mode no longer needs wxPython. It's now just Torch, PIL, Flask, and THA3.
- Note that the separate manual poser GUI app does use wxPython.
The code for live mode has been shortened and simplified. The plugin itself now fits into ~400 lines.
- The plugin module talkinghead/tha3/app/app.py now only provides the live mode.
- All the remnants of IFacialMocap have been purged.
- The animation logic has been revised and rewritten.
Code shared between the live mode and the manual poser has been moved to a new module talkinghead/tha3/app/util.py.

NOTE: In this version, server.py always loads the live mode at float16 precision, as this saves VRAM with no visible difference in output, and should give a performance boost according to the original author (now that the live mode actually uses enough GPU compute for its inference performance to matter).

~~Thus, this version requires the separable_half model files to run.~~ EDIT: Autodownloader added. If the talkinghead/tha3/models/ directory is missing, the models will be automatically pulled in from HuggingFace. The default repo to install from is OktayAlpk/talking-head-anime-3, but for future-proofing, there's an option --talkinghead-models=somehfuser/somehfrepo to download from another repo. Note the plural in the option name, models.

~~I intend to later add an option to choose which talkinghead model to use.~~ EDIT: Added, e.g. --talkinghead-model=separable_half. The default is 'auto', which picks separable_half on GPU and separable_float on CPU.

New options TL;DR: generally, just use --talkinghead-gpu as before, and don't worry. But before the first run, delete (or rename) your talkinghead/tha3/models/ directory to trigger a one-time download of all four available THA3 models.

@Cohee1207: Opinion, please?

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/SillyTavern/SillyTavern-Extras/pull/207 **Author:** [@Technologicat](https://github.com/Technologicat) **Created:** 12/20/2023 **Status:** ✅ Merged **Merged:** 12/21/2023 **Merged by:** [@Cohee1207](https://github.com/Cohee1207) **Base:** `main` ← **Head:** `appfixes` --- ### 📝 Commits (10+) - [`23823fd`](https://github.com/SillyTavern/SillyTavern-Extras/commit/23823fdec032c7d896ecb010d6d79eccfb7220dc) talkinghead: fix and improve THA3 manual poser - [`e6f3380`](https://github.com/SillyTavern/SillyTavern-Extras/commit/e6f338064685ebf0d987fbc0fbc03bbba35a306e) server.py: no, don't yet use fp16 for talkinghead - [`e02c1d9`](https://github.com/SillyTavern/SillyTavern-Extras/commit/e02c1d98f273e71b3c7f07b79ac774d64a8e376b) talkinghead: remove wxPython dependency from live mode - [`6c7ff71`](https://github.com/SillyTavern/SillyTavern-Extras/commit/6c7ff7112f3655daeac2dceb107d4fa954c1fa93) comment out unused functions - [`bc0a832`](https://github.com/SillyTavern/SillyTavern-Extras/commit/bc0a832e88e00d0a62105545e96261e1bf94d2fb) add TODO list - [`dd741b6`](https://github.com/SillyTavern/SillyTavern-Extras/commit/dd741b6030b4355d9ba27f103e85fe8643536926) coding style - [`0114634`](https://github.com/SillyTavern/SillyTavern-Extras/commit/0114634a35ed37127a4bdbc4ef1f174856ad2d9b) remove unused import - [`2e01125`](https://github.com/SillyTavern/SillyTavern-Extras/commit/2e01125cb19511911c52e93e1abb4b7ed935ead9) add TODO marker - [`15ce843`](https://github.com/SillyTavern/SillyTavern-Extras/commit/15ce843ff2908955029033cc6ac77f00d588810e) message wordings - [`6d3f32c`](https://github.com/SillyTavern/SillyTavern-Extras/commit/6d3f32c41e005a0e7617a01a12928f5c3b3f9b88) fix typos in variable names ### 📊 Changes **22 files changed** (+532 additions, -2370 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+1 -0) 📝 `requirements-rocm.txt` (+1 -0) 📝 `requirements-silicon.txt` (+1 -0) 📝 `requirements.txt` (+1 -1) 📝 `server.py` (+36 -7) ➖ `talkinghead/start_standalone_app.sh` (+0 -20) 📝 `talkinghead/tha3/app/app.py` (+286 -928) 📝 `talkinghead/tha3/app/manual_poser.py` (+12 -159) ➕ `talkinghead/tha3/app/util.py` (+194 -0) ➖ `talkinghead/tha3/mocap/__init__.py` (+0 -0) ➖ `talkinghead/tha3/mocap/ifacialmocap_constants.py` (+0 -239) ➖ `talkinghead/tha3/mocap/ifacialmocap_pose.py` (+0 -27) ➖ `talkinghead/tha3/mocap/ifacialmocap_pose_converter.py` (+0 -12) ➖ `talkinghead/tha3/mocap/ifacialmocap_poser_converter_25.py` (+0 -491) ➖ `talkinghead/tha3/mocap/ifacialmocap_v2.py` (+0 -89) ➖ `talkinghead/tha3/models/LICENSE.txt` (+0 -396) ➖ `talkinghead/tha3/models/placeholder.txt` (+0 -1) ➖ `talkinghead/tha3/models/separable_float/editor.pt` (+0 -0) ➖ `talkinghead/tha3/models/separable_float/eyebrow_decomposer.pt` (+0 -0) ➖ `talkinghead/tha3/models/separable_float/eyebrow_morphing_combiner.pt` (+0 -0) _...and 2 more files_ </details> ### 📄 Description Here's the next PR for `talkinghead`. - Live mode now runs at **30 FPS** on an RTX 3070 Ti mobile. - The *SillyTavern-extras* server no longer crashes on exit when the `talkinghead` module is enabled, and also exits faster. - **Live mode no longer needs wxPython.** It's now just Torch, PIL, Flask, and THA3. - Note that the separate manual poser GUI app **does** use wxPython. - The code for live mode has been shortened and simplified. The plugin itself now fits into ~400 lines. - The plugin module `talkinghead/tha3/app/app.py` now only provides the live mode. - All the remnants of IFacialMocap have been purged. - The animation logic has been revised and rewritten. - Code shared between the live mode and the manual poser has been moved to a new module `talkinghead/tha3/app/util.py`. **NOTE**: In this version, `server.py` always loads the live mode at `float16` precision, as this saves VRAM with no visible difference in output, and should give a performance boost according to the original author (now that the live mode actually uses enough GPU compute for its inference performance to matter). ~**Thus, this version requires the `separable_half` model files to run.**~ **EDIT**: Autodownloader added. If the `talkinghead/tha3/models/` directory is missing, the models will be automatically pulled in from HuggingFace. The default repo to install from is [OktayAlpk/talking-head-anime-3](https://huggingface.co/OktayAlpk/talking-head-anime-3), but for future-proofing, there's an option `--talkinghead-models=somehfuser/somehfrepo` to download from another repo. Note the plural in the option name, **models**. ~I intend to later add an option to choose which `talkinghead` model to use.~ **EDIT**: Added, e.g. `--talkinghead-model=separable_half`. The default is 'auto', which picks `separable_half` on GPU and `separable_float` on CPU. **New options TL;DR**: generally, just use `--talkinghead-gpu` as before, and don't worry. But before the first run, delete (or rename) your `talkinghead/tha3/models/` directory to trigger a one-time download of all four available THA3 models. @Cohee1207: Opinion, please? --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 18:05:48 +00:00