rehan-remade/universal-modder
1
0
Fork 0
mirror of https://github.com/rehan-remade/universal-modder.git synced 2026-11-05 01:37:40 +00:00
Code Issues 15 Projects Packages Wiki Activity

[PR #276] [MERGED] Read tool output as UTF-8, not the locale codec (um win ps dies on CJK window titles) #275

New issue
Closed
opened 2026-10-15 20:55:04 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-15 20:55:04 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/rehan-remade/universal-modder/pull/276
Author: @23318yy-afk
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/14/2026
Merged by: @rehan-remade

Base: main ← Head: fix/utf8-tool-output

📝 Commits (4)

  • 9b4ad8d win: read tool output as UTF-8, not the locale codec
  • a47b3d1 win: UTF-8 for scan's PowerShell and WinDrive too, console code page for taskkill/reg (review fixes)
  • 76b770d renormalize line endings to match .gitattributes (as on main since #304)
  • 1710669 Merge remote-tracking branch 'origin/main' into land/pr-276

📊 Changes

8 files changed (+152 additions, -34 deletions)

View changed files

➕ tests/test_encoding.py (+78 -0)
📝 um/common.py (+32 -2)
📝 um/kb.py (+4 -4)
📝 um/ps1/WinDrive.ps1 (+10 -5)
📝 um/render3d.py (+2 -2)
📝 um/scan.py (+4 -4)
📝 um/video.py (+4 -4)
📝 um/win.py (+18 -13)

📄 Description

What broke

um win ps dies on a Windows machine whose window titles aren't ASCII:

Exception in thread Thread-1 (_readerthread):
UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf in position 85: illegal multibyte sequence
  File ".../subprocess.py", line 1599, in _readerthread

um/win.py:powershell() called subprocess.run(..., text=True) with no encoding=, so Python decoded the
pipe with locale.getpreferredencoding(): cp936 on a Chinese Windows, cp1252 on a Western one.
PowerShell 5.1 writes UTF-8 to a pipe — byte for byte:

>>> subprocess.run(["powershell", "-NoProfile", "-Command", "'\u4e2d\u6587\u6d4b\u8bd5'"], capture_output=True).stdout
b'\xe4\xb8\xad\xe6\x96\x87\xe6\xb5\x8b\xe8\xaf\x95\r\n'   # UTF-8; gbk can't decode it

ffmpeg does the same with the path it echoes, so a CJK game folder or save path takes um video down too:

b"Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\\DFY\\.um-smoke\\\xe4\xb8\xad\xe6\x96\x87 \xe6\xb5\x8b\xe8\xaf\x95.mp4'"

Linux CI is UTF-8 and never sees this, which is why it shipped.

The fix

  • um.common.text_utf8() — the one set of subprocess kwargs for reading a tool's text output:
    text=True, encoding="utf-8", errors="replace". Every subprocess.run/Popen in um/ that decodes a
    tool's output now goes through it (20 call sites across 6 modules: win, scan, video, render3d,
    kb, common).
  • win.powershell() also asks PowerShell for UTF-8 ([Console]::OutputEncoding) so the bytes and the
    decoder agree. The prefix is best-effort — a console-less process makes it throw, so it's wrapped.
  • errors="replace" means a codec we guessed wrong costs one replacement character instead of a traceback
    in a reader thread.

Also: the gfxcapture self-check was lying

um win setup decided with "gfxcapture" in out over the stdout of -h filter=gfxcapture. A build that
lacks the filter answers Unknown filter 'gfxcapture'. — on stdout — which contains the name, so the
check passed on exactly the builds it was meant to catch:

OLD (no gfxcapture): 'gfxcapture' in stdout -> True     <-- "ok"
NEW (has it):        'gfxcapture' in stdout -> True

It now matches the name as a word in -filters, which is a plain success/failure query:

OLD: old-check=True   new-check=False
NEW: old-check=True   new-check=True

Tests

tests/test_encoding.py (new, cross-platform — it fails on Windows, not on the Ubuntu runner):

  • a child that writes UTF-8 is read back correctly with PYTHONIOENCODING=cp1252 on the host;
  • undecodable bytes give replacement characters, not an exception;
  • an AST guard: no subprocess.* call in um/ may pass text=True without encoding=;
  • um video probe on 中文 测试.mp4.

Verified

On Windows 11, cp936, Python 3.12.15 (uv tool install):

before after
um win ps UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf 14 windows, CJK + emoji titles intact
um video probe "中文 测试.mp4" would raise {"duration": 3.0, "width": 640, ...}

145 passed, 1 skipped (the cygpath/Git Bash one) on main @ 5ba8150 — rebased onto it, so the count
includes upstream's own suite, which grew while this branch was open. Plus um kb check --index
(PASS, 182 notes) and um publish check . (0 failures) — the three steps this repo's CI runs.
The AST guard in tests/test_encoding.py is the interesting one: it is what proves no call site in um/,
upstream's new ones included, still decodes with the locale codec.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/rehan-remade/universal-modder/pull/276 **Author:** [@23318yy-afk](https://github.com/23318yy-afk) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/14/2026 **Merged by:** [@rehan-remade](https://github.com/rehan-remade) **Base:** `main` ← **Head:** `fix/utf8-tool-output` --- ### 📝 Commits (4) - [`9b4ad8d`](https://github.com/rehan-remade/universal-modder/commit/9b4ad8dfe4847467881402d26cde2037b26c10e1) win: read tool output as UTF-8, not the locale codec - [`a47b3d1`](https://github.com/rehan-remade/universal-modder/commit/a47b3d165f464f61321fb24e9af743f5516b4e45) win: UTF-8 for scan's PowerShell and WinDrive too, console code page for taskkill/reg (review fixes) - [`76b770d`](https://github.com/rehan-remade/universal-modder/commit/76b770d6f213836c700d851ce5ae1e30799f7c3e) renormalize line endings to match .gitattributes (as on main since #304) - [`1710669`](https://github.com/rehan-remade/universal-modder/commit/17106692f38e7b11807528293b6716b60a01be78) Merge remote-tracking branch 'origin/main' into land/pr-276 ### 📊 Changes **8 files changed** (+152 additions, -34 deletions) <details> <summary>View changed files</summary> ➕ `tests/test_encoding.py` (+78 -0) 📝 `um/common.py` (+32 -2) 📝 `um/kb.py` (+4 -4) 📝 `um/ps1/WinDrive.ps1` (+10 -5) 📝 `um/render3d.py` (+2 -2) 📝 `um/scan.py` (+4 -4) 📝 `um/video.py` (+4 -4) 📝 `um/win.py` (+18 -13) </details> ### 📄 Description ## What broke `um win ps` dies on a Windows machine whose window titles aren't ASCII: ``` Exception in thread Thread-1 (_readerthread): UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf in position 85: illegal multibyte sequence File ".../subprocess.py", line 1599, in _readerthread ``` `um/win.py:powershell()` called `subprocess.run(..., text=True)` with no `encoding=`, so Python decoded the pipe with `locale.getpreferredencoding()`: **cp936** on a Chinese Windows, cp1252 on a Western one. PowerShell 5.1 writes **UTF-8** to a pipe — byte for byte: ```python >>> subprocess.run(["powershell", "-NoProfile", "-Command", "'\u4e2d\u6587\u6d4b\u8bd5'"], capture_output=True).stdout b'\xe4\xb8\xad\xe6\x96\x87\xe6\xb5\x8b\xe8\xaf\x95\r\n' # UTF-8; gbk can't decode it ``` ffmpeg does the same with the path it echoes, so a CJK game folder or save path takes `um video` down too: ```python b"Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\\DFY\\.um-smoke\\\xe4\xb8\xad\xe6\x96\x87 \xe6\xb5\x8b\xe8\xaf\x95.mp4'" ``` Linux CI is UTF-8 and never sees this, which is why it shipped. ## The fix - **`um.common.text_utf8()`** — the one set of subprocess kwargs for reading a tool's text output: `text=True, encoding="utf-8", errors="replace"`. Every `subprocess.run`/`Popen` in `um/` that decodes a tool's output now goes through it (20 call sites across 6 modules: `win`, `scan`, `video`, `render3d`, `kb`, `common`). - **`win.powershell()`** also asks PowerShell for UTF-8 (`[Console]::OutputEncoding`) so the bytes and the decoder agree. The prefix is best-effort — a console-less process makes it throw, so it's wrapped. - `errors="replace"` means a codec we guessed wrong costs one replacement character instead of a traceback in a reader thread. ## Also: the `gfxcapture` self-check was lying `um win setup` decided with `"gfxcapture" in out` over the stdout of `-h filter=gfxcapture`. A build that *lacks* the filter answers `Unknown filter 'gfxcapture'.` — on **stdout** — which contains the name, so the check passed on exactly the builds it was meant to catch: ``` OLD (no gfxcapture): 'gfxcapture' in stdout -> True <-- "ok" NEW (has it): 'gfxcapture' in stdout -> True ``` It now matches the name as a word in `-filters`, which is a plain success/failure query: ``` OLD: old-check=True new-check=False NEW: old-check=True new-check=True ``` ## Tests `tests/test_encoding.py` (new, cross-platform — it fails on Windows, not on the Ubuntu runner): - a child that writes UTF-8 is read back correctly with `PYTHONIOENCODING=cp1252` on the host; - undecodable bytes give replacement characters, not an exception; - an AST guard: no `subprocess.*` call in `um/` may pass `text=True` without `encoding=`; - `um video probe` on `中文 测试.mp4`. ## Verified On Windows 11, cp936, Python 3.12.15 (`uv tool install`): | | before | after | |---|---|---| | `um win ps` | `UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf` | 14 windows, CJK + emoji titles intact | | `um video probe "中文 测试.mp4"` | would raise | `{"duration": 3.0, "width": 640, ...}` | `145 passed, 1 skipped` (the cygpath/Git Bash one) on `main` @ 5ba8150 — rebased onto it, so the count includes upstream's own suite, which grew while this branch was open. Plus `um kb check --index` (PASS, 182 notes) and `um publish check .` (0 failures) — the three steps this repo's CI runs. The AST guard in `tests/test_encoding.py` is the interesting one: it is what proves no call site in `um/`, upstream's new ones included, still decodes with the locale codec. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-15 20:55:04 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
star-chart
docs/discord-in-readme
docs/delisted-game-pointers
fix/star-chart-on-push
fix/gta-build-vcvars
docs/sources-and-refs
docs/star-chart
docs/branded-teaser
docs/brand-readme
docs/leaks-rule
fix/no-symlinks
feat/comfy
docs/re-archives
docs/gemini-windows-install
feat/opencode-config
docs/every-game-welcome
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
rehan-remade/universal-modder#275
No description provided.