[PR #184] [CLOSED] cli: print as UTF-8 so a Windows cp1252 console cannot crash a command #186

New issue

Closed

opened 2026-10-15 20:54:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 20:54:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/rehan-remade/universal-modder/pull/184
Author: @Ramnath0521
Created: 10/10/2026
Status: ❌ Closed

Base: main ← Head: fix/issue-152

📝 Commits (1)

9a5ecb8 cli: print as UTF-8 so a Windows cp1252 console cannot crash a command

📊 Changes

2 files changed (+21 additions, -0 deletions)

View changed files

📝 tests/test_um.py (+16 -0)
📝 um/cli.py (+5 -0)

📄 Description

Fixes #152

Cause

Python picks the console's legacy encoding (cp1252) for stdout on Windows. um kb search prints raw note text (the hit snippets), and the first character cp1252 cannot encode, such as ≥ or →, raises UnicodeEncodeError and aborts the command. Other subcommands that print note or repo text can hit the same thing, which the issue also points out.

Reproduced on Windows 11 / Python 3.12 with a note containing The boon pool ≥ 3 → cached:

UnicodeEncodeError: 'charmap' codec can't encode character '\u2265' in position 21: character maps to <undefined>

Change

um.cli.main() reconfigures sys.stdout and sys.stderr to UTF-8 before anything is printed, which is what PYTHONUTF8=1 (the workaround in the issue) already does, so every subcommand is covered and not only kb. Streams that cannot be reconfigured (no reconfigure(), or a closed/odd stream) are left alone.

Test

test_cli_prints_text_a_legacy_console_cannot_encode runs cli.main(["kb", "search", ...]) against a note holding ≥ and →, with sys.stdout replaced by a cp1252 text wrapper over a byte buffer, and checks the output arrives as UTF-8.

Before the change: UnicodeEncodeError: 'charmap' codec can't encode character '\u2265'.
After: passes. The real python -m um.cli kb search boon --root ... with PYTHONIOENCODING=cp1252 -X utf8=0 also crashes before and prints the snippet after.

I ran the suite on Windows 11 with python -m pytest -q tests: 78 passed (77 before, plus this test). I also ran um <group> --help for every group and um kb check --index, the other commands from test.yml; all exit 0. I did not run um publish check . or the Linux CI job.

One thing to be aware of: output from a legacy (non-UTF-8) console window will now be UTF-8 bytes, so non-ASCII text can look garbled there, where before it crashed. Agents and pipes, the main readers of this tool, decode UTF-8. If you would rather keep the console's own encoding and substitute ? for what it cannot show, reconfigure(errors="replace") is a one-word change; I went with UTF-8 to match the PYTHONUTF8=1 workaround.

Written with AI assistance (Claude Code). I reproduced the failure on my own Windows machine, read the fix, and verified the added test fails before it and passes after. Errors are mine.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/rehan-remade/universal-modder/pull/184 **Author:** [@Ramnath0521](https://github.com/Ramnath0521) **Created:** 10/10/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/issue-152` --- ### 📝 Commits (1) - [`9a5ecb8`](https://github.com/rehan-remade/universal-modder/commit/9a5ecb8354d11ef0a1a48d8f654638089ff303e7) cli: print as UTF-8 so a Windows cp1252 console cannot crash a command ### 📊 Changes **2 files changed** (+21 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `tests/test_um.py` (+16 -0) 📝 `um/cli.py` (+5 -0) </details> ### 📄 Description Fixes #152 ## Cause Python picks the console's legacy encoding (cp1252) for stdout on Windows. `um kb search` prints raw note text (the hit snippets), and the first character cp1252 cannot encode, such as `≥` or `→`, raises `UnicodeEncodeError` and aborts the command. Other subcommands that print note or repo text can hit the same thing, which the issue also points out. Reproduced on Windows 11 / Python 3.12 with a note containing `The boon pool ≥ 3 → cached`: ``` UnicodeEncodeError: 'charmap' codec can't encode character '\u2265' in position 21: character maps to <undefined> ``` ## Change `um.cli.main()` reconfigures `sys.stdout` and `sys.stderr` to UTF-8 before anything is printed, which is what `PYTHONUTF8=1` (the workaround in the issue) already does, so every subcommand is covered and not only `kb`. Streams that cannot be reconfigured (no `reconfigure()`, or a closed/odd stream) are left alone. ## Test `test_cli_prints_text_a_legacy_console_cannot_encode` runs `cli.main(["kb", "search", ...])` against a note holding `≥` and `→`, with `sys.stdout` replaced by a cp1252 text wrapper over a byte buffer, and checks the output arrives as UTF-8. - Before the change: `UnicodeEncodeError: 'charmap' codec can't encode character '\u2265'`. - After: passes. The real `python -m um.cli kb search boon --root ...` with `PYTHONIOENCODING=cp1252 -X utf8=0` also crashes before and prints the snippet after. I ran the suite on Windows 11 with `python -m pytest -q tests`: 78 passed (77 before, plus this test). I also ran `um <group> --help` for every group and `um kb check --index`, the other commands from `test.yml`; all exit 0. I did not run `um publish check .` or the Linux CI job. One thing to be aware of: output from a legacy (non-UTF-8) console window will now be UTF-8 bytes, so non-ASCII text can look garbled there, where before it crashed. Agents and pipes, the main readers of this tool, decode UTF-8. If you would rather keep the console's own encoding and substitute `?` for what it cannot show, `reconfigure(errors="replace")` is a one-word change; I went with UTF-8 to match the `PYTHONUTF8=1` workaround. --- Written with AI assistance (Claude Code). I reproduced the failure on my own Windows machine, read the fix, and verified the added test fails before it and passes after. Errors are mine. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-15 20:54:07 +00:00