morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4978] [Bug] XML plist data bytes diverge from native CFPropertyList decoding #3079

New issue

Closed

opened 2026-10-31 00:36:12 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:36:12 +00:00

Owner

Originally created by @xu91102 on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4978

Before submitting

I searched open and closed issues and inspected related production diffs.
This report uses only reduced synthetic inputs, with no credentials or private target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

On main b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf, the built public inspect-keyed-archive CLI interprets XML <data> differently from Apple's native property-list reader:

Data text in the same reduced archive	Native `plutil` binary conversion, read with Python `plistlib`	REA public CLI
`AQID`	`010203`	exit 0, data `AQID` (correct)
`A Q I D` plus newline	`010203`	exit 0, data `AQID` (correct)
`AQI`	empty data	exit 0, observed data `AQI=` (`0102`)
`AQ!ID`	`010203`	exit 1, `artifact_operation_failed`

Apple accepts the last two spellings; this is not a claim that they are canonical RFC Base64. REA currently reports bytes the native reader does not produce, or refuses a natively readable archive. Each listed call was executed once against an independent immutable input. All original input SHA-256 values stayed unchanged.

Reproduction

Save the following XML as selected.plist, changing only the text inside <data> for the four rows above:

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0"><dict>
  <key>$archiver</key><string>NSKeyedArchiver</string>
  <key>$version</key><integer>100000</integer>
  <key>$objects</key><array><string>$null</string><data>AQI</data></array>
  <key>$top</key><dict><key>root</key><dict><key>CF$UID</key><integer>1</integer></dict></dict>
</dict></plist>

Run node scripts/rea.mjs inspect-keyed-archive selected.plist --json from the built checkout. Independently run plutil -convert binary1 -o native.bplist -- selected.plist, then read plistlib.load(open('native.bplist', 'rb'))['$objects'][1] in Python. Native conversion wrote a separate output file; it did not rewrite the selected source.

Expected behavior

The XML data value and its native binary conversion should retain the same bytes. The existing keyed archive contract promises original serialized fields and typed data values. A decoder discrepancy must not become an incorrect observed value.

Impact and scope

Observed in two distinct data spellings, with two successful positive controls. This affects the shared XML value interpretation used by keyed archives; other consumers are source-derived scope and have not yet been exercised for this issue. No target code, provider engine, agent or model was run. MCP, Windows/Linux, full application workflows and performance are NOT RUN. Last known good version: NOT_ENOUGH_INFO.

REA version

Source main b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf, freshly compiled locally.

Runtime and platform

Node 24.18.0, macOS arm64; native /usr/bin/plutil, independent Python plistlib binary readback.

Target and artifact context

Synthetic XML Foundation keyed archive, no executable target.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction: XML property-list value decoding.

Analysis

The shared data branch uses WHATWG atob. Apple's CoreFoundation data parser skips non-alphabet characters and emits only completed quartets; that explains both measured differences. The native conversions above are the runtime oracle; the historical Apple source is supporting analysis, not a claim about this macOS build's exact implementation.

Actual related production diffs were read: #4047/#4054 concern real character data, #4908 concerns non-finite real identity, and #841/#899 concern typed scalar/graph projection. They do not repair the data Base64 owner. This is separate from #4852's deferred mixed numeric fallback contract; no numeric behavior change is proposed. Additional rejected CDATA/comment probes are preserved locally as negative diagnostics and are not counted as natively accepted data spellings or a separate contribution.

This initial report was filed before the local implementation experiment. No commit, fork push or upstream PR has been created for this issue.

Follow-up: local experiment held after same-core PR appeared (2026-10-18)

A subsequent uncommitted local experiment on the original b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf baseline completed 237 distinct focused/README tests, 32 actual installed-package CLI calls and six feature MCP tool requests. Those results apply only to that frozen experiment. Independent Standards/Spec review was interrupted and did not produce reports; this is not a reviewed or delivered fix. Complete agent/model workflows, Windows/Linux and performance remain NOT RUN.

A fresh duplicate check then found #4981 by @avichal-08. I read its complete three-file production/test diff: it replaces atob in the shared XML data value owner with CoreFoundation-style skip-nonalphabet/completed-quartet decoding for the same AQI and AQ!ID observations. This is the same root cause despite a different implementation, so my six-file local experiment is now read-only HOLD, with no commit, fork push, upstream PR or CI. It will not be repackaged around entity text, CDATA/comments, newline normalization or lifecycle remainder scope.

At this readback, #4981 is OPEN and not merged, and this issue is still OPEN. I have not executed its head and do not claim that it covers all local experimental cases. Adoption of my local source or author credit is NOT_ESTABLISHED. The earlier reproduction and all RED/HARNESS records remain preserved; no maintainer or human readiness is inferred.

Main follow-up (2026-10-19)

Maintainer @morluto fixed this root on main in cee95b2ae6a19122de95103b2d23bc6f8c9c4428, adapting #4981 and explicitly crediting its author @avichal-08. This issue was closed on 2026-10-18 at 18:28:39 UTC. #4981 was subsequently closed without merge at 18:46:20 UTC. I read the actual eight-file main commit patch; the shared XML source owner now binds raw data spans to validated DOM nodes before entity/line normalization, while a shared quartet grammar sizes and materializes the output.

My uncommitted six-file local experiment remains read-only HOLD. I have no upstream PR, commit, fork push, CI or established source-adoption/author-credit claim for this issue, and will not repackage remaining cases. I have not executed this merged main commit's plist-specific acceptance; the maintainer's reported checks and measurements are not my runs, and the earlier local 237 tests/32 installed CLI calls/six MCP requests do not transfer to main. The interrupted independent reviews remain NOT_RUN. No issue-close call or reply to another contributor was made by me.

Originally created by @xu91102 on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4978 ### Before submitting - [x] I searched open and closed issues and inspected related production diffs. - [x] This report uses only reduced synthetic inputs, with no credentials or private target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior On main `b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf`, the built public `inspect-keyed-archive` CLI interprets XML `<data>` differently from Apple's native property-list reader: | Data text in the same reduced archive | Native `plutil` binary conversion, read with Python `plistlib` | REA public CLI | | --- | --- | --- | | `AQID` | `010203` | exit 0, data `AQID` (correct) | | ` A Q I D` plus newline | `010203` | exit 0, data `AQID` (correct) | | `AQI` | empty data | exit 0, **observed** data `AQI=` (`0102`) | | `AQ!ID` | `010203` | exit 1, `artifact_operation_failed` | Apple accepts the last two spellings; this is not a claim that they are canonical RFC Base64. REA currently reports bytes the native reader does not produce, or refuses a natively readable archive. Each listed call was executed once against an independent immutable input. All original input SHA-256 values stayed unchanged. ### Reproduction Save the following XML as `selected.plist`, changing only the text inside `<data>` for the four rows above: ```xml <?xml version="1.0" encoding="UTF-8"?> <plist version="1.0"><dict> <key>$archiver</key><string>NSKeyedArchiver</string> <key>$version</key><integer>100000</integer> <key>$objects</key><array><string>$null</string><data>AQI</data></array> <key>$top</key><dict><key>root</key><dict><key>CF$UID</key><integer>1</integer></dict></dict> </dict></plist> ``` Run `node scripts/rea.mjs inspect-keyed-archive selected.plist --json` from the built checkout. Independently run `plutil -convert binary1 -o native.bplist -- selected.plist`, then read `plistlib.load(open('native.bplist', 'rb'))['$objects'][1]` in Python. Native conversion wrote a separate output file; it did not rewrite the selected source. ### Expected behavior The XML data value and its native binary conversion should retain the same bytes. The existing [keyed archive contract](https://github.com/morluto/rea/blob/b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf/docs/native-investigation.md#L331) promises original serialized fields and typed data values. A decoder discrepancy must not become an incorrect `observed` value. ### Impact and scope Observed in two distinct data spellings, with two successful positive controls. This affects the shared XML value interpretation used by keyed archives; other consumers are source-derived scope and have not yet been exercised for this issue. No target code, provider engine, agent or model was run. MCP, Windows/Linux, full application workflows and performance are NOT RUN. Last known good version: NOT_ENOUGH_INFO. ### REA version Source main `b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf`, freshly compiled locally. ### Runtime and platform Node 24.18.0, macOS arm64; native `/usr/bin/plutil`, independent Python `plistlib` binary readback. ### Target and artifact context Synthetic XML Foundation keyed archive, no executable target. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction: XML property-list value decoding. ### Analysis The [shared data branch](https://github.com/morluto/rea/blob/b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf/src/domain/propertyListXml.ts#L135) uses WHATWG `atob`. Apple's CoreFoundation data parser skips non-alphabet characters and emits only completed quartets; that explains both measured differences. The native conversions above are the runtime oracle; the historical Apple source is supporting analysis, not a claim about this macOS build's exact implementation. Actual related production diffs were read: #4047/#4054 concern real character data, #4908 concerns non-finite real identity, and #841/#899 concern typed scalar/graph projection. They do not repair the data Base64 owner. This is separate from #4852's deferred mixed numeric fallback contract; no numeric behavior change is proposed. Additional rejected CDATA/comment probes are preserved locally as negative diagnostics and are not counted as natively accepted data spellings or a separate contribution. This initial report was filed before the local implementation experiment. No commit, fork push or upstream PR has been created for this issue. ### Follow-up: local experiment held after same-core PR appeared (2026-10-18) A subsequent uncommitted local experiment on the original `b83b9caf4a5f0241a5fd370792f1aed1bffbb2cf` baseline completed 237 distinct focused/README tests, 32 actual installed-package CLI calls and six feature MCP tool requests. Those results apply only to that frozen experiment. Independent Standards/Spec review was interrupted and did not produce reports; this is not a reviewed or delivered fix. Complete agent/model workflows, Windows/Linux and performance remain NOT RUN. A fresh duplicate check then found #4981 by @avichal-08. I read its complete three-file production/test diff: it replaces `atob` in the shared XML data value owner with CoreFoundation-style skip-nonalphabet/completed-quartet decoding for the same `AQI` and `AQ!ID` observations. This is the same root cause despite a different implementation, so my six-file local experiment is now read-only HOLD, with no commit, fork push, upstream PR or CI. It will not be repackaged around entity text, CDATA/comments, newline normalization or lifecycle remainder scope. At this readback, #4981 is OPEN and not merged, and this issue is still OPEN. I have not executed its head and do not claim that it covers all local experimental cases. Adoption of my local source or author credit is NOT_ESTABLISHED. The earlier reproduction and all RED/HARNESS records remain preserved; no maintainer or human readiness is inferred. ### Main follow-up (2026-10-19) Maintainer @morluto fixed this root on main in [cee95b2ae6a19122de95103b2d23bc6f8c9c4428](https://github.com/morluto/rea/commit/cee95b2ae6a19122de95103b2d23bc6f8c9c4428), adapting #4981 and explicitly crediting its author @avichal-08. This issue was closed on 2026-10-18 at 18:28:39 UTC. #4981 was subsequently closed without merge at 18:46:20 UTC. I read the actual eight-file main commit patch; the shared XML source owner now binds raw data spans to validated DOM nodes before entity/line normalization, while a shared quartet grammar sizes and materializes the output. My uncommitted six-file local experiment remains read-only HOLD. I have no upstream PR, commit, fork push, CI or established source-adoption/author-credit claim for this issue, and will not repackage remaining cases. I have not executed this merged main commit's plist-specific acceptance; the maintainer's reported checks and measurements are not my runs, and the earlier local 237 tests/32 installed CLI calls/six MCP requests do not transfer to main. The interrupted independent reviews remain NOT_RUN. No issue-close call or reply to another contributor was made by me.

bsakaguchi closed this issue

2026-10-31 00:36:12 +00:00

bsakaguchi commented

2026-10-31 00:36:13 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

Confirmed — the atob-vs-CoreFoundation divergence is real and this report nails it: atob tolerates "AQI" by padding it to "AQI=" while CoreFoundation only emits complete quartets, and "AQ!ID" is rejected by atob but accepted by CF's skip-non-alphabet behavior. Two details worth keeping in the #4981 fix: (1) CoreFoundation's skips are not limited to whitespace — any non-alphabet character (like "!") is dropped, so the decoder should implement drop-then-quartet rather than approximate atob's permissiveness; (2) keep the raw text visible in the Evidence alongside the decoded bytes for auditing, since skip-based decoding is inherently lossy about what was originally in the file. Also +1 on treating the native reader as the runtime oracle — behavior parity with plutil beats RFC purity when the whole ecosystem treats it as ground truth.

@XinwangTao commented on GitHub (Oct 18, 2026): Confirmed — the atob-vs-CoreFoundation divergence is real and this report nails it: atob tolerates "AQI" by padding it to "AQI=" while CoreFoundation only emits complete quartets, and "AQ!ID" is rejected by atob but accepted by CF's skip-non-alphabet behavior. Two details worth keeping in the #4981 fix: (1) CoreFoundation's skips are not limited to whitespace — any non-alphabet character (like "!") is dropped, so the decoder should implement drop-then-quartet rather than approximate atob's permissiveness; (2) keep the raw <data> text visible in the Evidence alongside the decoded bytes for auditing, since skip-based decoding is inherently lossy about what was originally in the file. Also +1 on treating the native reader as the runtime oracle — behavior parity with plutil beats RFC purity when the whole ecosystem treats it as ground truth.

bsakaguchi commented

2026-10-31 00:36:14 +00:00

Author

Owner

@morluto commented on GitHub (Oct 18, 2026):

Fixed on main in cee95b2ae. Reviewed and adapted PR #4981 at 4cf6cac14a656c37c20dcf4a0baa0e3e75be510d, crediting @avichal-08. The quartet decoder fixes the two reported spellings, but a DOM-only value has already lost raw entity spelling and some line endings. Native controls also show that non-ASCII and child markup must fail, and <data/> differs from explicit empty <data></data>. A candidate that simply replaces atob leaves those owner-boundary discrepancies.

The structural fix binds validated raw spans to the exact DOM nodes at the XML source owner. The existing depth walk records spans and error reasons; it finishes before data output allocation. Value decoding asks that owner for the data fact, without serializing XML or reinterpreting expanded text. One quartet grammar handles exact output sizing and materialization, including padding resets, so malformed-but-accepted ASCII cannot cause a slack buffer plus copy. Strict XML well-formedness remains the admission policy. Other scalar compatibility policies, including the separately reported #4852 behavior, are not claimed fixed.

Executed evidence under Node 24.18.0/macOS: original source selection 2 failed / 98 passed; expanded unchanged-main selection 29 failed / 98 passed; the PR production patch applied to frozen current main on that expanded set 23 failed / 104 passed. Public unchanged-main CLI/MCP selection: original 2 failed / 7 passed, expanded 5 failed / 7 passed. Candidate: 178 source tests in nine files and 54 compiled public/provider tests in five files passed, including actual macOS plist reads and typed CLI/stdio-MCP malformed-data errors. Original archive bytes/digests remain checked. Full npm run check, npm run docs:check (five checks), and independent final source reviews passed. Small authored native plutil controls and pinned CoreFoundation source establish the literal semantics; no target or deep-provider execution. Linux/Windows and complete model/agent workflows remain unverified.

Twelve completed public CLI tasks compared frozen main with the candidate. Canonical complete normalized result/source digest and reply estimate (588 cl100k tokens) agree; warm medians: elapsed 0.4341 → 0.4614 s, CPU 0.6450 → 0.6637 s, sampled family RSS 353.6 → 353.9 MB. Incomplete data is corrected and its complete reply estimate changes 589 → 592 tokens. Two alternating warm repeats/build plus separate first processes, no true cold-cache or statistical speed claim; whole-agent/catalog/MCP, large-input and failure costs remain unmeasured. No performance improvement claimed.

No directly related source-representation cleanup remains identified by this review. Broader scalar validity and native-format parity remain tracked separately. DeepWiki/Context7 findings were checked against installed xmldom 0.9.12, the actual native controls, and CoreFoundation commit 384ff680be7c7445fee6ce0c55e9ae11a46e07c5 rather than treated as runtime evidence.

@morluto commented on GitHub (Oct 18, 2026): Fixed on main in [cee95b2ae](https://github.com/morluto/rea/commit/cee95b2ae6a19122de95103b2d23bc6f8c9c4428). Reviewed and adapted PR #4981 at `4cf6cac14a656c37c20dcf4a0baa0e3e75be510d`, crediting @avichal-08. The quartet decoder fixes the two reported spellings, but a DOM-only value has already lost raw entity spelling and some line endings. Native controls also show that non-ASCII and child markup must fail, and `<data/>` differs from explicit empty `<data></data>`. A candidate that simply replaces `atob` leaves those owner-boundary discrepancies. The structural fix binds validated raw spans to the exact DOM nodes at the XML source owner. The existing depth walk records spans and error reasons; it finishes before data output allocation. Value decoding asks that owner for the data fact, without serializing XML or reinterpreting expanded text. One quartet grammar handles exact output sizing and materialization, including padding resets, so malformed-but-accepted ASCII cannot cause a slack buffer plus copy. Strict XML well-formedness remains the admission policy. Other scalar compatibility policies, including the separately reported #4852 behavior, are not claimed fixed. Executed evidence under Node 24.18.0/macOS: original source selection 2 failed / 98 passed; expanded unchanged-main selection 29 failed / 98 passed; the PR production patch applied to frozen current main on that expanded set 23 failed / 104 passed. Public unchanged-main CLI/MCP selection: original 2 failed / 7 passed, expanded 5 failed / 7 passed. Candidate: **178 source tests in nine files and 54 compiled public/provider tests in five files passed**, including actual macOS plist reads and typed CLI/stdio-MCP malformed-data errors. Original archive bytes/digests remain checked. Full `npm run check`, `npm run docs:check` (five checks), and independent final source reviews passed. Small authored native `plutil` controls and pinned CoreFoundation source establish the literal semantics; no target or deep-provider execution. Linux/Windows and complete model/agent workflows remain unverified. Twelve completed public CLI tasks compared frozen main with the candidate. Canonical complete normalized result/source digest and reply estimate (588 cl100k tokens) agree; warm medians: elapsed 0.4341 → 0.4614 s, CPU 0.6450 → 0.6637 s, sampled family RSS 353.6 → 353.9 MB. Incomplete data is corrected and its complete reply estimate changes 589 → 592 tokens. Two alternating warm repeats/build plus separate first processes, no true cold-cache or statistical speed claim; whole-agent/catalog/MCP, large-input and failure costs remain unmeasured. No performance improvement claimed. No directly related source-representation cleanup remains identified by this review. Broader scalar validity and native-format parity remain tracked separately. DeepWiki/Context7 findings were checked against installed xmldom 0.9.12, the actual native controls, and CoreFoundation commit `384ff680be7c7445fee6ce0c55e9ae11a46e07c5` rather than treated as runtime evidence.

bsakaguchi referenced this issue

2026-10-31 00:38:46 +00:00

[GH-ISSUE #5412] [Bug] Bare-name managed type selection omits a nested type with a namespace #3282

bsakaguchi referenced this issue

2026-10-31 01:07:52 +00:00

[PR #3079] [MERGED] fix(managed): pair same-named nested types by their enclosing types #5597