morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #6492] [MERGED] fix(python): keep pyc string previews on Unicode boundaries #7366

New issue

Closed

opened 2026-11-01 21:47:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:47:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6492
Author: @xiajiahang
Created: 10/23/2026
Status: ✅ Merged
Merged: 10/23/2026
Merged by: @gensokyo-dev

Base: main ← Head: fix/python-pyc-unicode-preview

📝 Commits (5)

3903535 fix(python): keep pyc string previews on Unicode boundaries
990368b merge: sync main for pyc preview follow-up
361d052 fix(python): retain source code-point boundaries in previews
e11b8de fix(python): retain source preview boundaries in shared marshal decoder
7c0f7ab chore: sync pyc preview fix with main

📊 Changes

6 files changed (+235 additions, -21 deletions)

View changed files

📝 src/artifacts/pyinstaller/MarshalSubset.ts (+1 -1)
📝 src/domain/python/marshalUtf8.test.ts (+45 -1)
📝 src/domain/python/marshalUtf8.ts (+45 -6)
📝 src/python/PythonPycParser.test.ts (+18 -0)
📝 src/python/PythonPycParser.ts (+6 -13)
📝 tests/acceptance/pythonPycCli.test.ts (+120 -0)

📄 Description

Summary

Fixes #6476. PYC string previews preserve source code-point boundaries within the existing 80 UTF-16 unit budget: omit a four-byte scalar crossing the cut, while retaining a separately encoded high surrogate that fits under the existing surrogatepass policy.

The shared marshal decoder records the endpoint before JavaScript loses that source distinction. PYC consumes the endpoint; PyInstaller consumes the same decoded text. Native fatal TextDecoder spans remain, all admitted bytes are validated after the prefix closes, and the superseded downstream cutoff is removed. No schema, byte-count/digest, truncation, coverage, authority or CLI/MCP contract change beyond the corrected preview. Current main's negative-zero/decode-limit fixes are preserved.

Validation

Head 7c0f7abc6b, base ba9c5bc68; CI checkout c9758427dd has those exact parents and the head's identical source tree.

Decoder/PYC/PyInstaller: 207 source tests passed. Filesystem/compiled CLI/real stdio MCP: 23 passed, one existing int64 case timed out in the initial three-file one-CPU run. That unchanged case then passed separately on both head and main with its original 30-second timeout; the initial failure remains recorded.
Build, npm run check (six tasks plus architecture guards), and diff checks passed.
Linux CPython 3.11.2 compiled seven inert assignments covering scalar/ASCII/source-surrogate boundaries and a 2 MB string; no generated bytecode executed. Independent marshal inspection verified previews and file/bytecode digests. Those seven artifacts plus actual stdlib argparse (194 code objects) passed compiled CLI and real stdio MCP, with identical normalized reports. Scalar-only JSON passed jq; the CLI regression also checks TOON without introduced replacement characters. Exact-base replay reproduces the separately encoded-surrogate omission.
Codex reviewed this exact head without major findings; the prior source-boundary thread is addressed and resolved. Greptile's earlier review applies to the old head.

Cost check

Three alternating pairs per input; fresh Node/isolated home/temp, compile cache disabled, OS caches warm/uncontrolled. Main → head medians:

Input	Wall s	Family CPU s	Family peak MiB	Estimated tokens
short	2.651 → 2.740	2.466 → 2.521	341.9 → 339.7	897 → 897
argparse	2.842 → 2.945	2.638 → 2.599	346.1 → 345.6	48,843 → 48,843
large	2.842 → 2.742	2.427 → 2.413	348.4 → 336.8	913 → 913

Wall includes Python/jq verification. Reply sizes are unchanged (2,862 / 157,950 / 2,880 bytes). Tokens use tiktoken 0.14.0/cl100k_base over all CLI arguments plus full JSON replies, with no catalog emitted; estimates, not model usage or an MCP comparison. No speedup/token reduction is claimed: short/stdlib wall medians rose slightly and the large-input median fell.

All local runs verified payload cleanup and no OOM/resource stops under CPUQuota=100%, 3 GiB early stop/4 GiB hard cap. Family counters are observed at payload exit; final whole-scope counters are unavailable. Other Python producers, Windows and the full repository suite were not replayed locally.

CI attribution

Run 40032312240: static/build/docs/readiness passed. All 101 source failures reproduce on the exact tested main, with matching error messages after removing color and two known temporary fixture-root suffixes; stack equality is not claimed. Seven cancelled test shards had started and are not counted as passing. All eight shard logs were inspected; their 179 failures also reproduce on the exact main, with matching file/title multiplicities, including duplicate test titles. Interrupted logs lack full assertion text, so full-message equality is not claimed for those cases. The affected compiled CLI/MCP/filesystem files completed successfully (24 tests) even though their overall jobs failed or were cancelled. These 280 observed failures are baseline failures. Under the task’s explicit merge authorization, the failing checks and stale earlier changes request may be bypassed; no reviewer approval is implied. Cancelled/unexecuted remainder stays unverified.

Postmerge 1081008c9: fresh build and all eight real-artifact CLI/stdio oracle replays passed, with verified payload cleanup and no resource stops.

Author readiness

Human owner: @xiajiahang

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6492 **Author:** [@xiajiahang](https://github.com/xiajiahang) **Created:** 10/23/2026 **Status:** ✅ Merged **Merged:** 10/23/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `fix/python-pyc-unicode-preview` --- ### 📝 Commits (5) - [`3903535`](https://github.com/morluto/rea/commit/3903535bc3ef683497b86cafb6ec15a4eba73b30) fix(python): keep pyc string previews on Unicode boundaries - [`990368b`](https://github.com/morluto/rea/commit/990368b571460f8c25668cf6df6c3430b39a282d) merge: sync main for pyc preview follow-up - [`361d052`](https://github.com/morluto/rea/commit/361d052b92b001d8ea7bcb5a91168af5e988a420) fix(python): retain source code-point boundaries in previews - [`e11b8de`](https://github.com/morluto/rea/commit/e11b8de21ee23c49fe3f66728d04b151e71d2bc2) fix(python): retain source preview boundaries in shared marshal decoder - [`7c0f7ab`](https://github.com/morluto/rea/commit/7c0f7abc6bcbecd5577dc81fff8cf407a7d99c29) chore: sync pyc preview fix with main ### 📊 Changes **6 files changed** (+235 additions, -21 deletions) <details> <summary>View changed files</summary> 📝 `src/artifacts/pyinstaller/MarshalSubset.ts` (+1 -1) 📝 `src/domain/python/marshalUtf8.test.ts` (+45 -1) 📝 `src/domain/python/marshalUtf8.ts` (+45 -6) 📝 `src/python/PythonPycParser.test.ts` (+18 -0) 📝 `src/python/PythonPycParser.ts` (+6 -13) 📝 `tests/acceptance/pythonPycCli.test.ts` (+120 -0) </details> ### 📄 Description ## Summary Fixes #6476. PYC string previews preserve source code-point boundaries within the existing 80 UTF-16 unit budget: omit a four-byte scalar crossing the cut, while retaining a separately encoded high surrogate that fits under the existing surrogatepass policy. The shared marshal decoder records the endpoint before JavaScript loses that source distinction. PYC consumes the endpoint; PyInstaller consumes the same decoded text. Native fatal TextDecoder spans remain, all admitted bytes are validated after the prefix closes, and the superseded downstream cutoff is removed. No schema, byte-count/digest, truncation, coverage, authority or CLI/MCP contract change beyond the corrected preview. Current main's negative-zero/decode-limit fixes are preserved. ## Validation Head `7c0f7abc6b`, base `ba9c5bc68`; CI checkout `c9758427dd` has those exact parents and the head's identical source tree. - Decoder/PYC/PyInstaller: **207 source tests passed**. Filesystem/compiled CLI/real stdio MCP: **23 passed, one existing int64 case timed out** in the initial three-file one-CPU run. That unchanged case then passed separately on both head and main with its original 30-second timeout; the initial failure remains recorded. - Build, `npm run check` (six tasks plus architecture guards), and diff checks passed. - Linux CPython 3.11.2 compiled seven inert assignments covering scalar/ASCII/source-surrogate boundaries and a 2 MB string; no generated bytecode executed. Independent marshal inspection verified previews and file/bytecode digests. Those seven artifacts plus actual stdlib argparse (194 code objects) passed compiled CLI and real stdio MCP, with identical normalized reports. Scalar-only JSON passed jq; the CLI regression also checks TOON without introduced replacement characters. Exact-base replay reproduces the separately encoded-surrogate omission. - Codex reviewed this exact head without major findings; the prior source-boundary thread is addressed and resolved. Greptile's earlier review applies to the old head. ## Cost check Three alternating pairs per input; fresh Node/isolated home/temp, compile cache disabled, OS caches warm/uncontrolled. Main → head medians: | Input | Wall s | Family CPU s | Family peak MiB | Estimated tokens | | --- | ---: | ---: | ---: | ---: | | short | 2.651 → 2.740 | 2.466 → 2.521 | 341.9 → 339.7 | 897 → 897 | | argparse | 2.842 → 2.945 | 2.638 → 2.599 | 346.1 → 345.6 | 48,843 → 48,843 | | large | 2.842 → 2.742 | 2.427 → 2.413 | 348.4 → 336.8 | 913 → 913 | Wall includes Python/jq verification. Reply sizes are unchanged (2,862 / 157,950 / 2,880 bytes). Tokens use tiktoken 0.14.0/cl100k_base over all CLI arguments plus full JSON replies, with no catalog emitted; estimates, not model usage or an MCP comparison. No speedup/token reduction is claimed: short/stdlib wall medians rose slightly and the large-input median fell. All local runs verified payload cleanup and no OOM/resource stops under CPUQuota=100%, 3 GiB early stop/4 GiB hard cap. Family counters are observed at payload exit; final whole-scope counters are unavailable. Other Python producers, Windows and the full repository suite were not replayed locally. ## CI attribution Run [40032312240](https://github.com/morluto/rea/actions/runs/40032312240): static/build/docs/readiness passed. All **101 source failures reproduce on the exact tested main**, with matching error messages after removing color and two known temporary fixture-root suffixes; stack equality is not claimed. Seven cancelled test shards had started and are not counted as passing. All eight shard logs were inspected; their **179 failures also reproduce on the exact main**, with matching file/title multiplicities, including duplicate test titles. Interrupted logs lack full assertion text, so full-message equality is not claimed for those cases. The affected compiled CLI/MCP/filesystem files completed successfully (**24 tests**) even though their overall jobs failed or were cancelled. These 280 observed failures are baseline failures. Under the task’s explicit merge authorization, the failing checks and stale earlier changes request may be bypassed; no reviewer approval is implied. Cancelled/unexecuted remainder stays unverified. Postmerge `1081008c9`: fresh build and all eight real-artifact CLI/stdio oracle replays passed, with verified payload cleanup and no resource stops. ## Author readiness Human owner: @xiajiahang - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:47:21 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-05 01:48:00 +00:00

[PR #7366] [CLOSED] test: align fixtures with admitted sources, inventory rows and Java property framing #7835

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7366

No description provided.

Rows
Columns