[PR #6990] [MERGED] fix(apple): refuse unpaired surrogates in IPS JSON strings #7553

Closed
opened 2026-11-03 13:38:40 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6990
Author: @DeryFerd
Created: 10/24/2026
Status: ✅ Merged
Merged: 10/24/2026
Merged by: @morluto

Base: main ← Head: fix/apple-ips-unpaired-surrogates


📝 Commits (2)

  • 3e67c75 fix(apple): refuse unpaired surrogates in IPS JSON strings
  • f92ec17 fix(ci): merge main to restore the setup lint gate

📊 Changes

2 files changed (+63 additions, -7 deletions)

View changed files

📝 src/native/apple/losslessJson.test.ts (+29 -0)
📝 src/native/apple/losslessJson.ts (+34 -7)

📄 Description

Summary

The Apple IPS JSON reader now refuses strings that decode to text containing an unpaired surrogate. A crash report's strings come from the crashed process, so an escaped lone surrogate in procName, an exception message, or a symbol name previously flowed into Evidence as-is: malformed text under a coverage verdict of complete, with no refusal and no uncertainty marker. The reader now treats that like any other malformed input, and the report is refused with a typed format failure naming the reason.

No issue is linked because none exists. The report comes from reading this new parser against the way the rest of REA's text parsers already treat the same input, and the comparison is the argument for the change.

Problem and expected behavior

Smallest trigger: an .ips report whose JSON contains "procName": "test\uD800". The escape is valid JSON syntax, JSON.parse accepts it, and the decoded string carries U+D800 with no following low surrogate.

Before this change, parseString returned that text unchanged through two paths. The escaped path went through JSON.parse, which preserves lone surrogates in its result. The unescaped path returned the raw slice verbatim. Nothing in src/native/apple/ inspected the decoded text, so the report projected with complete coverage while carrying text that no Unicode consumer can render.

Expected behavior: text that is not valid Unicode is malformed input, the same way an invalid escape or an unterminated string is. The reader refuses it with a typed failure naming the problem, and the existing projection turns that into the usual format error for the report instead of Evidence with invalid strings.

Change and scope

parseString now validates the decoded value through requirePairedSurrogates before returning. The check walks the string once: a high surrogate must be followed by its low surrogate, and a low surrogate must follow a high one. Anything else is refused with "JSON string contains an unpaired surrogate".

Placing the check on the decoded value rather than inside the escape handler matters for two reasons. A valid emoji arrives as two separate escapes, so validating each escape in isolation would refuse well-formed text. And the unescaped path never calls the escape handler at all, so an escape-local check would miss raw text entirely. One check after decoding covers both.

consumeJson needs no change. A LosslessJsonFailure with the default reason already projects as an AppleIpsFormatFailure for the report, so the refusal travels through the existing path with the parser's own message.

Out of scope: a normalization mode that replaces lone surrogates with the replacement character while keeping the rest of the report. Refusing matches the nearest precedent in the codebase, the safetensors header reader, which rejects a model whose JSON contains an unpaired surrogate. Keeping malformed text out of Evidence is also the narrower change; a replacement mode would need a decision about where the substitution is recorded.

Contract and boundary impact

  • Semantic owner and earliest changed stage: src/native/apple/losslessJson.ts owns IPS JSON text decoding; the check runs when a string is decoded, before shape projection or Evidence construction.
  • Contribution scope fit and linked maintainer decision: routine fix inside an existing parser boundary, no target execution, injection, mutation, or protection recovery; not applicable.
  • CLI and MCP/tool-catalog contract: none. No tool schema, description, or example change. A previously-succeeding report containing an escaped lone surrogate now returns a typed format failure.
  • Provider, bridge, target-format, or platform compatibility: Apple IPS only, all platforms. Reports without unpaired surrogates, including ones using paired surrogate escapes, parse identically.
  • Evidence, artifact, provenance, or reconstruction contract: the change removes one way for Evidence to carry invalid text under complete coverage. Artifact identity, provenance, and failed attempts are untouched.
  • Target selection, authority, upstream behavior, and cleanup details: not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none. No dependency, lockfile, catalog, or skill bundle change.

Expected effects (check known effects; use documentation-only on its own).

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: refuses JSON string escapes for unpaired surrogates in src/native/apple/losslessJson.test.ts. It asserts that a lone \ud800 escape and a trailing \udfff escape are refused with the named message, and that a paired \ud83d\ude00 escape still parses, which is the case an escape-local check would have broken.
  • Base reproduction or other evidence: on the base revision the refusal case fails because the parser accepted the escape and returned the decoded string, while the paired case passes. That pair of outcomes is the defect and the non-regression in one run.
  • User-visible CLI/MCP output (if applicable): a report containing an escaped lone surrogate now fails with the typed format error carrying "JSON string contains an unpaired surrogate" instead of returning Evidence whose strings are invalid Unicode.
  • Remaining proof gaps: no real macOS crash report was decoded for this change; coverage is the reader boundary and its suite, which is where the defect lives. Whether Apple's own reporter ever emits such an escape is unknown, and the refusal does not depend on that.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

CI follow-up at f92ec17f4 (2026-10-25), Node 24.22.1 / npm 11.16.0 on macOS:

  • The failing Static checks job reported setupRegistrationPrewarm.test.ts: a 193-line function exceeded the 150-line test limit. This was inherited from main, and main fixed it in 2acdd5413.
  • Merged current main c7ae27807 to include that fix. The PR's two-file IPS parser diff is unchanged.
  • npm run check:fast (typecheck/lint) and npm run format:check: passed.
  • Source tests for src/native/apple/losslessJson.test.ts and AppleIpsParser.test.ts: 58 passed.
  • npm run test:focused -- tests/boundary/filesystem/setupRegistrationPrewarm.test.ts tests/acceptance/appleIpsCli.test.ts: 9 passed, including the public CLI/stdio MCP IPS coverage.
  • Fresh GitHub CI is pending; these local checks do not establish a completed hosted run.

Earlier author validation:

  • npx vitest run src/native/apple — 311 passed across 12 files, including the 44 AppleIpsParser cases, so no existing fixture or projection changes behavior.
  • npx tsc -p .tmp-tsconfig.json --noEmit over losslessJson.ts, its test, and the shape consumer — exit 0.
  • npx oxlint src/native/apple/losslessJson.ts src/native/apple/losslessJson.test.ts — 0 warnings, 0 errors.
  • npx oxfmt --check on both files — all matched files use the correct format.
  • npm run verify:module-boundaries — pass.

Compatibility, safety, and release

  • Breaking changes or migration steps: none. Reports without unpaired surrogates parse identically. A report that contains one previously produced Evidence with invalid text and now returns a typed format failure naming the reason, which is the corrected behavior rather than a removal. No ! marker is needed.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable. The change is inside the IPS text reader; no provider session, bridge, or OS process is involved.
  • Package or release metadata impact: none. No dependency, lockfile, catalog, or skill bundle change.
  • Security, privacy, process, or containment review: the change narrows what reaches Evidence, adds no input path, and does not alter token or credential handling.

Author readiness

Human owner: @DeryFerd

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed. No owning document describes this reader's string validation, and docs/apple-ips.md documents tool behavior that does not change.
  • User-visible CLI/MCP changes include representative output, quoted under Evidence and regression coverage.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6990 **Author:** [@DeryFerd](https://github.com/DeryFerd) **Created:** 10/24/2026 **Status:** ✅ Merged **Merged:** 10/24/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/apple-ips-unpaired-surrogates` --- ### 📝 Commits (2) - [`3e67c75`](https://github.com/morluto/rea/commit/3e67c751020e6f530e3adeb6747ef33ffcc4453e) fix(apple): refuse unpaired surrogates in IPS JSON strings - [`f92ec17`](https://github.com/morluto/rea/commit/f92ec17f48ce5a679d5bc7af1966b0f3d7c1b080) fix(ci): merge main to restore the setup lint gate ### 📊 Changes **2 files changed** (+63 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `src/native/apple/losslessJson.test.ts` (+29 -0) 📝 `src/native/apple/losslessJson.ts` (+34 -7) </details> ### 📄 Description ## Summary The Apple IPS JSON reader now refuses strings that decode to text containing an unpaired surrogate. A crash report's strings come from the crashed process, so an escaped lone surrogate in `procName`, an exception message, or a symbol name previously flowed into Evidence as-is: malformed text under a coverage verdict of complete, with no refusal and no uncertainty marker. The reader now treats that like any other malformed input, and the report is refused with a typed format failure naming the reason. No issue is linked because none exists. The report comes from reading this new parser against the way the rest of REA's text parsers already treat the same input, and the comparison is the argument for the change. ## Problem and expected behavior Smallest trigger: an `.ips` report whose JSON contains `"procName": "test\uD800"`. The escape is valid JSON syntax, `JSON.parse` accepts it, and the decoded string carries U+D800 with no following low surrogate. Before this change, `parseString` returned that text unchanged through two paths. The escaped path went through `JSON.parse`, which preserves lone surrogates in its result. The unescaped path returned the raw slice verbatim. Nothing in `src/native/apple/` inspected the decoded text, so the report projected with complete coverage while carrying text that no Unicode consumer can render. Expected behavior: text that is not valid Unicode is malformed input, the same way an invalid escape or an unterminated string is. The reader refuses it with a typed failure naming the problem, and the existing projection turns that into the usual format error for the report instead of Evidence with invalid strings. ## Change and scope `parseString` now validates the decoded value through `requirePairedSurrogates` before returning. The check walks the string once: a high surrogate must be followed by its low surrogate, and a low surrogate must follow a high one. Anything else is refused with "JSON string contains an unpaired surrogate". Placing the check on the decoded value rather than inside the escape handler matters for two reasons. A valid emoji arrives as two separate escapes, so validating each escape in isolation would refuse well-formed text. And the unescaped path never calls the escape handler at all, so an escape-local check would miss raw text entirely. One check after decoding covers both. `consumeJson` needs no change. A `LosslessJsonFailure` with the default reason already projects as an `AppleIpsFormatFailure` for the report, so the refusal travels through the existing path with the parser's own message. Out of scope: a normalization mode that replaces lone surrogates with the replacement character while keeping the rest of the report. Refusing matches the nearest precedent in the codebase, the safetensors header reader, which rejects a model whose JSON contains an unpaired surrogate. Keeping malformed text out of Evidence is also the narrower change; a replacement mode would need a decision about where the substitution is recorded. ## Contract and boundary impact - Semantic owner and earliest changed stage: `src/native/apple/losslessJson.ts` owns IPS JSON text decoding; the check runs when a string is decoded, before shape projection or Evidence construction. - Contribution scope fit and linked maintainer decision: routine fix inside an existing parser boundary, no target execution, injection, mutation, or protection recovery; not applicable. - CLI and MCP/tool-catalog contract: none. No tool schema, description, or example change. A previously-succeeding report containing an escaped lone surrogate now returns a typed format failure. - Provider, bridge, target-format, or platform compatibility: Apple IPS only, all platforms. Reports without unpaired surrogates, including ones using paired surrogate escapes, parse identically. - Evidence, artifact, provenance, or reconstruction contract: the change removes one way for Evidence to carry invalid text under complete coverage. Artifact identity, provenance, and failed attempts are untouched. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. No dependency, lockfile, catalog, or skill bundle change. Expected effects (check known effects; use documentation-only on its own). - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: `refuses JSON string escapes for unpaired surrogates` in `src/native/apple/losslessJson.test.ts`. It asserts that a lone `\ud800` escape and a trailing `\udfff` escape are refused with the named message, and that a paired `\ud83d\ude00` escape still parses, which is the case an escape-local check would have broken. - Base reproduction or other evidence: on the base revision the refusal case fails because the parser accepted the escape and returned the decoded string, while the paired case passes. That pair of outcomes is the defect and the non-regression in one run. - User-visible CLI/MCP output (if applicable): a report containing an escaped lone surrogate now fails with the typed format error carrying "JSON string contains an unpaired surrogate" instead of returning Evidence whose strings are invalid Unicode. - Remaining proof gaps: no real macOS crash report was decoded for this change; coverage is the reader boundary and its suite, which is where the defect lives. Whether Apple's own reporter ever emits such an escape is unknown, and the refusal does not depend on that. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed CI follow-up at `f92ec17f4` (2026-10-25), Node 24.22.1 / npm 11.16.0 on macOS: - The failing Static checks job reported `setupRegistrationPrewarm.test.ts`: a 193-line function exceeded the 150-line test limit. This was inherited from main, and main fixed it in `2acdd5413`. - Merged current main `c7ae27807` to include that fix. The PR's two-file IPS parser diff is unchanged. - `npm run check:fast` (typecheck/lint) and `npm run format:check`: passed. - Source tests for `src/native/apple/losslessJson.test.ts` and `AppleIpsParser.test.ts`: **58 passed**. - `npm run test:focused -- tests/boundary/filesystem/setupRegistrationPrewarm.test.ts tests/acceptance/appleIpsCli.test.ts`: **9 passed**, including the public CLI/stdio MCP IPS coverage. - Fresh GitHub CI is pending; these local checks do not establish a completed hosted run. Earlier author validation: - `npx vitest run src/native/apple` — 311 passed across 12 files, including the 44 `AppleIpsParser` cases, so no existing fixture or projection changes behavior. - `npx tsc -p .tmp-tsconfig.json --noEmit` over `losslessJson.ts`, its test, and the shape consumer — exit 0. - `npx oxlint src/native/apple/losslessJson.ts src/native/apple/losslessJson.test.ts` — 0 warnings, 0 errors. - `npx oxfmt --check` on both files — all matched files use the correct format. - `npm run verify:module-boundaries` — pass. ## Compatibility, safety, and release - Breaking changes or migration steps: none. Reports without unpaired surrogates parse identically. A report that contains one previously produced Evidence with invalid text and now returns a typed format failure naming the reason, which is the corrected behavior rather than a removal. No `!` marker is needed. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. The change is inside the IPS text reader; no provider session, bridge, or OS process is involved. - Package or release metadata impact: none. No dependency, lockfile, catalog, or skill bundle change. - Security, privacy, process, or containment review: the change narrows what reaches Evidence, adds no input path, and does not alter token or credential handling. ## Author readiness Human owner: @DeryFerd - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. No owning document describes this reader's string validation, and `docs/apple-ips.md` documents tool behavior that does not change. - [x] User-visible CLI/MCP changes include representative output, quoted under Evidence and regression coverage. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-11-03 13:38:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#7553
No description provided.