[PR #2863] [CLOSED] fix(saz): accept U+FFFD in well-formed metadata #5447

Closed
opened 2026-10-31 01:06:42 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2863
Author: @Dragolone
Created: 10/13/2026
Status: ❌ Closed

Base: main ← Head: fix/saz-metadata-replacement-character


📝 Commits (7)

  • cda4251 fix(saz): accept U+FFFD in well-formed metadata
  • c6d8672 fix(comparison): recheck historical calls-only changes (#2223)
  • 50c5cd5 fix(ci): isolate Linux Hopper process fixture inventory
  • bfc44c2 fix(android): make DEX descriptor pattern portable
  • 7d68747 Merge upstream main into fix/saz-metadata-replacement-character
  • d2d4f8a Merge branch 'main' into fix/saz-metadata-replacement-character
  • da429f3 Merge branch 'main' into fix/saz-metadata-replacement-character

📊 Changes

2 files changed (+33 additions, -1 deletions)

View changed files

➕ src/browser/history/SazMetadata.test.ts (+24 -0)
📝 src/browser/history/SazMetadata.ts (+9 -1)

📄 Description

Summary

A SAZ capture is rejected as invalid input when any raw/N_m.xml metadata member contains U+FFFD (REPLACEMENT CHARACTER), even though the XML is well-formed.

xmldom 0.9.12 emits a warning ("Unicode replacement character detected, source encoding issues?") whenever the source text contains U+FFFD (lib/sax.js, start of parse). decodeSazMetadata treats every onError report, warnings included, as a well-formedness failure. U+FFFD is a legal XML Char ([#xE000-#xFFFD]). The decoder already uses a fatal TextDecoder, so a U+FFFD that reaches the parser was encoded in the source. It cannot be a decoding artifact, which is the case the warning is about.

Related: #2601 asks for the opposite direction in the same handler, rejecting &, NUL and surrogate references that are currently accepted. This PR does not change that behavior and does not conflict with it; it only stops rejecting one legal character. No issue covers this case: I searched for "SAZ metadata", "replacement character", "U+FFFD", "xmldom warning" and "Fiddler".

Problem and expected behavior

Smallest trigger: metadata <Session SID="1"><SessionFlags><SessionFlag N="ui-comments" V="caf�"/></SessionFlags></Session> (with a real U+FFFD) in an otherwise ordinary archive.

  • Before: rea inspect-web-network-capture fffd.saz saz --json reports {"category": "invalid_input", "code": "invalid_request"}, and decodeSazMetadata throws "SAZ metadata is not well-formed XML."
  • After: total_records: 1, with the attribute value retained as caf�.

Expected per docs/web-network-captures.md: malformed metadata fails validation; well-formed metadata is projected.

Change and scope

The onError handler ignores only the warning whose message starts with "Unicode replacement character detected". Every other warning, error or fatal error still sets parseFailed, so malformed XML is still rejected. That includes documents that also contain U+FFFD, and xmldom reports some of them only as warnings: unquoted attributes, attributes without separating whitespace, mismatched elements.

Not included: the stricter checks requested in #2601.

Contribution scope: an ordinary format-inspection correctness fix within existing boundaries; a maintainer scope decision is not applicable.

Contract and boundary impact

  • Semantic owner and earliest changed stage: decodeSazMetadata in src/browser/history/SazMetadata.ts.
  • Contribution scope fit and linked maintainer decision: in scope (format inspection); not applicable.
  • CLI and MCP/tool-catalog contract: no schema change; affected captures now succeed instead of failing as invalid input.
  • Provider, bridge, target-format, or platform compatibility: depends on xmldom's warning text; the dependency is pinned (0.9.12), and the new test fails if that message changes.
  • Evidence, artifact, provenance, or reconstruction contract: none; the character is retained verbatim.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: new src/browser/history/SazMetadata.test.ts:
    • "retains U+FFFD in well-formed metadata as an ordinary XML character" (an attribute and a text node);
    • "still rejects malformed metadata that also contains U+FFFD" for an unquoted attribute, attributes without separating whitespace, and a mismatched element.
  • Base reproduction: on 94072364c the first test fails with "SAZ metadata is not well-formed XML."; the rejection cases pass on both base and fix.
  • User-visible CLI output (rea inspect-web-network-capture fffd.saz saz --json): before {"category": "invalid_input", "code": "invalid_request"}, after normalized_result.total_records = 1.
  • Remaining proof gaps: I have not seen a Fiddler-produced SAZ with U+FFFD in metadata; the fixture is synthetic.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

macOS (Apple silicon), Node 24.19.0.

  • npx vitest run src/browser/history/SazMetadata.test.ts on the base — the retention test fails, the three rejection tests pass.
  • npx vitest run src/browser/history/ with the fix — 6 files, 103 tests passed.
  • npm run build:cached && npx vitest run tests/boundary/browser/sazWebCapture.test.ts — 13 passed, 1 skipped.
  • rea inspect-web-network-capture before/after as above.
  • npm run verify:test-discovery — "Discovered all 1089 test files exactly once."
  • npm run typecheck — passed.
  • npx oxfmt --check and npx oxlint on both changed files — passed, no errors.
  • npm run verify:module-boundaries — passed.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: no change to DTD or entity handling; malformed XML stays rejected.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2863 **Author:** [@Dragolone](https://github.com/Dragolone) **Created:** 10/13/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/saz-metadata-replacement-character` --- ### 📝 Commits (7) - [`cda4251`](https://github.com/morluto/rea/commit/cda4251ad6fbb13d5090664655ed4b7fccdad2d2) fix(saz): accept U+FFFD in well-formed metadata - [`c6d8672`](https://github.com/morluto/rea/commit/c6d86721f64c2fbbed4cdf30a6f4bc8eb7779bde) fix(comparison): recheck historical calls-only changes (#2223) - [`50c5cd5`](https://github.com/morluto/rea/commit/50c5cd57c345c172a00d87103f6488435d28fe0a) fix(ci): isolate Linux Hopper process fixture inventory - [`bfc44c2`](https://github.com/morluto/rea/commit/bfc44c228ecc9a32a508f09c093d734051ad4647) fix(android): make DEX descriptor pattern portable - [`7d68747`](https://github.com/morluto/rea/commit/7d68747bce26b6963250bd1d982b6e73f42dd1bb) Merge upstream main into fix/saz-metadata-replacement-character - [`d2d4f8a`](https://github.com/morluto/rea/commit/d2d4f8acd2b67e34b7076287fda9f1c8e96c5b56) Merge branch 'main' into fix/saz-metadata-replacement-character - [`da429f3`](https://github.com/morluto/rea/commit/da429f35c24e1182fb0fedfaff20393f52c52d62) Merge branch 'main' into fix/saz-metadata-replacement-character ### 📊 Changes **2 files changed** (+33 additions, -1 deletions) <details> <summary>View changed files</summary> ➕ `src/browser/history/SazMetadata.test.ts` (+24 -0) 📝 `src/browser/history/SazMetadata.ts` (+9 -1) </details> ### 📄 Description ## Summary A SAZ capture is rejected as invalid input when any `raw/N_m.xml` metadata member contains U+FFFD (REPLACEMENT CHARACTER), even though the XML is well-formed. xmldom 0.9.12 emits a `warning` ("Unicode replacement character detected, source encoding issues?") whenever the source text contains U+FFFD (`lib/sax.js`, start of `parse`). `decodeSazMetadata` treats every `onError` report, warnings included, as a well-formedness failure. U+FFFD is a legal XML `Char` (`[#xE000-#xFFFD]`). The decoder already uses a fatal `TextDecoder`, so a U+FFFD that reaches the parser was encoded in the source. It cannot be a decoding artifact, which is the case the warning is about. Related: #2601 asks for the opposite direction in the same handler, rejecting `&`, NUL and surrogate references that are currently accepted. This PR does not change that behavior and does not conflict with it; it only stops rejecting one legal character. No issue covers this case: I searched for "SAZ metadata", "replacement character", "U+FFFD", "xmldom warning" and "Fiddler". ## Problem and expected behavior Smallest trigger: metadata `<Session SID="1"><SessionFlags><SessionFlag N="ui-comments" V="caf�"/></SessionFlags></Session>` (with a real U+FFFD) in an otherwise ordinary archive. - Before: `rea inspect-web-network-capture fffd.saz saz --json` reports `{"category": "invalid_input", "code": "invalid_request"}`, and `decodeSazMetadata` throws "SAZ metadata is not well-formed XML." - After: `total_records: 1`, with the attribute value retained as `caf�`. Expected per `docs/web-network-captures.md`: malformed metadata fails validation; well-formed metadata is projected. ## Change and scope The `onError` handler ignores only the `warning` whose message starts with "Unicode replacement character detected". Every other warning, error or fatal error still sets `parseFailed`, so malformed XML is still rejected. That includes documents that also contain U+FFFD, and xmldom reports some of them only as warnings: unquoted attributes, attributes without separating whitespace, mismatched elements. Not included: the stricter checks requested in #2601. Contribution scope: an ordinary format-inspection correctness fix within existing boundaries; a maintainer scope decision is not applicable. ## Contract and boundary impact - Semantic owner and earliest changed stage: `decodeSazMetadata` in `src/browser/history/SazMetadata.ts`. - Contribution scope fit and linked maintainer decision: in scope (format inspection); not applicable. - CLI and MCP/tool-catalog contract: no schema change; affected captures now succeed instead of failing as invalid input. - Provider, bridge, target-format, or platform compatibility: depends on xmldom's warning text; the dependency is pinned (0.9.12), and the new test fails if that message changes. - Evidence, artifact, provenance, or reconstruction contract: none; the character is retained verbatim. - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: new `src/browser/history/SazMetadata.test.ts`: - "retains U+FFFD in well-formed metadata as an ordinary XML character" (an attribute and a text node); - "still rejects malformed metadata that also contains U+FFFD" for an unquoted attribute, attributes without separating whitespace, and a mismatched element. - Base reproduction: on `94072364c` the first test fails with "SAZ metadata is not well-formed XML."; the rejection cases pass on both base and fix. - User-visible CLI output (`rea inspect-web-network-capture fffd.saz saz --json`): before `{"category": "invalid_input", "code": "invalid_request"}`, after `normalized_result.total_records = 1`. - Remaining proof gaps: I have not seen a Fiddler-produced SAZ with U+FFFD in metadata; the fixture is synthetic. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed macOS (Apple silicon), Node 24.19.0. - `npx vitest run src/browser/history/SazMetadata.test.ts` on the base — the retention test fails, the three rejection tests pass. - `npx vitest run src/browser/history/` with the fix — 6 files, 103 tests passed. - `npm run build:cached && npx vitest run tests/boundary/browser/sazWebCapture.test.ts` — 13 passed, 1 skipped. - `rea inspect-web-network-capture` before/after as above. - `npm run verify:test-discovery` — "Discovered all 1089 test files exactly once." - `npm run typecheck` — passed. - `npx oxfmt --check` and `npx oxlint` on both changed files — passed, no errors. - `npm run verify:module-boundaries` — passed. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. - Package or release metadata impact: none. - Security, privacy, process, or containment review: no change to DTD or entity handling; malformed XML stays rejected. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:06:42 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5447
No description provided.