[PR #209] [MERGED] talkinghead new features #232

New issue

Closed

opened 2026-10-15 18:05:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:05:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/SillyTavern/SillyTavern-Extras/pull/209
Author: @Technologicat
Created: 12/22/2023
Status: ✅ Merged
Merged: 12/24/2023
Merged by: @Cohee1207

Base: main ← Head: talkinghead-next-pr

📝 Commits (3)

ebbdc6c talkinghead-next new features
b1489ae merge TODO lists into the one in app.py, remove done TODO item.
ba28f6e update docstring

📊 Changes

3 files changed (+373 additions, -190 deletions)

View changed files

📝 talkinghead/tha3/app/app.py (+367 -180)
📝 talkinghead/tha3/app/manual_poser.py (+3 -7)
📝 talkinghead/tha3/app/util.py (+3 -3)

📄 Description

Here's the first batch of new features for talkinghead:

Rewrite sway animation (much more natural now).
- Use all non-morph axes: head, neck, body.
- New history-free, rate-based formulation.
- Every few seconds, randomize a new deviation from the target emotion pose for all sway axes.
- Assign this as the target pose, and let the pose interpolator perform the actual animation.
  - This gives us a non-linear, saturating exponential animation trajectory for free.
- Micro-sway: add small dynamic noise (re-generated every frame) on top of sway target pose.
  - This makes the motion look more natural, especially once we are near the target pose.
Add breathing animation.
- How noticeable this is, depends on the character.
When entering "confusion" emotion, allow blinking quickly in succession (for a short time).
Add framerate limiter, currently hardcoded to send ~25 FPS.
- In practice, the output FPS is a bit under this, because we don't account for the time it takes to pack and send the frame. This could be improved in the future.
- Reduces average GPU usage if the GPU renders faster than this.
Separate the render and network code.
- Only generate as many frames as are actually consumed.
- Add separate FPS counters for render and network send.

Also, some more code cleanup.

On an RTX 3070 Ti mobile, available render performance (with separable_half) is now ~50 FPS at factory clock rate (1700 MHz), and ~40 FPS when underclocked to 1100 MHz. Actual network send performance, after the rate-limiting to at most 25 FPS, is around 21 FPS.

Remaining TODOs are marked into talkinghead/tha3/app/app.py, there's a TODO list at the beginning.

I'll have to look at improving the network send next (I'd like it to produce a smooth 25 FPS, since there is definitely enough render performance for that), but I thought I'd post the new features now.

There are also some client-side bugs and missing features I might look into.

@Cohee1207: Comments?

EDIT: The plugin is now ~600 lines, up from the ~400 of the previous PR, due to the improved animations and the framerate limiter. I squash-merged all my talkinghead-next commits into this PR and rebased it onto the latest upstream/main, so it's clean as well as easy to merge. All important changes are summarized above. The extra two commits below fix minor omissions. From there on, anything new will go into a new branch to avoid messing this one up. :)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/SillyTavern/SillyTavern-Extras/pull/209 **Author:** [@Technologicat](https://github.com/Technologicat) **Created:** 12/22/2023 **Status:** ✅ Merged **Merged:** 12/24/2023 **Merged by:** [@Cohee1207](https://github.com/Cohee1207) **Base:** `main` ← **Head:** `talkinghead-next-pr` --- ### 📝 Commits (3) - [`ebbdc6c`](https://github.com/SillyTavern/SillyTavern-Extras/commit/ebbdc6cc88ad8a92a34ab74ecad3d374accea782) talkinghead-next new features - [`b1489ae`](https://github.com/SillyTavern/SillyTavern-Extras/commit/b1489ae6686fe487c7501d049a01dd513bed5af3) merge TODO lists into the one in `app.py`, remove done TODO item. - [`ba28f6e`](https://github.com/SillyTavern/SillyTavern-Extras/commit/ba28f6e5ee2ccffdb27e54e80950d86699f8a8d5) update docstring ### 📊 Changes **3 files changed** (+373 additions, -190 deletions) <details> <summary>View changed files</summary> 📝 `talkinghead/tha3/app/app.py` (+367 -180) 📝 `talkinghead/tha3/app/manual_poser.py` (+3 -7) 📝 `talkinghead/tha3/app/util.py` (+3 -3) </details> ### 📄 Description Here's the first batch of new features for `talkinghead`: - Rewrite sway animation (much more natural now). - Use all non-morph axes: head, neck, body. - New history-free, rate-based formulation. - Every few seconds, randomize a new deviation from the target emotion pose for all sway axes. - Assign this as the target pose, and let the pose interpolator perform the actual animation. - This gives us a non-linear, saturating exponential animation trajectory for free. - Micro-sway: add small dynamic noise (re-generated every frame) on top of sway target pose. - This makes the motion look more natural, especially once we are near the target pose. - Add breathing animation. - How noticeable this is, depends on the character. - When entering "confusion" emotion, allow blinking quickly in succession (for a short time). - Add framerate limiter, currently hardcoded to send ~25 FPS. - In practice, the output FPS is a bit under this, because we don't account for the time it takes to pack and send the frame. This could be improved in the future. - Reduces average GPU usage if the GPU renders faster than this. - Separate the render and network code. - Only generate as many frames as are actually consumed. - Add separate FPS counters for render and network send. Also, some more code cleanup. On an RTX 3070 Ti mobile, available render performance (with `separable_half`) is now ~50 FPS at factory clock rate (1700 MHz), and ~40 FPS when underclocked to 1100 MHz. Actual network send performance, after the rate-limiting to at most 25 FPS, is around 21 FPS. Remaining TODOs are marked into `talkinghead/tha3/app/app.py`, there's a TODO list at the beginning. I'll have to look at improving the network send next (I'd like it to produce a smooth 25 FPS, since there is definitely enough render performance for that), but I thought I'd post the new features now. There are also some client-side bugs and missing features I might look into. @Cohee1207: Comments? **EDIT**: The plugin is now ~600 lines, up from the ~400 of the previous PR, due to the improved animations and the framerate limiter. I squash-merged all my `talkinghead-next` commits into this PR and rebased it onto the latest `upstream/main`, so it's clean as well as easy to merge. All important changes are summarized above. The extra two commits below fix minor omissions. From there on, anything new will go into a new branch to avoid messing this one up. :) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 18:05:48 +00:00