[PR #5127] [CLOSED] fix(artifacts): preserve original Unicode spelling when NFC normalization changes a logical path #6779

Closed
opened 2026-11-01 21:42:58 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5127
Author: @Varshith-Kali
Created: 10/18/2026
Status: ❌ Closed

Base: main ← Head: fix/unicode-5108-fork


📝 Commits (6)

  • cc5f2e9 fix(artifacts): preserve original Unicode spelling when NFC normalization changes a logical path
  • e16e2ec fix(artifacts): name the entry kind in the unicode-normalization limitation
  • c32fe2c test(boundary): cover NFD folder and ZIP directory entries in unicode-normalization inventory
  • dc4e970 fix(artifacts): carry pre-NFC PyInstaller member spelling into inventory
  • 6401bde fix(artifacts): carry pre-NFC Node SEA member spelling into inventory
  • 616dda5 fix(artifacts): keep trailing slash in NFC spelling for trailing-slash file members

📊 Changes

8 files changed (+420 additions, -7 deletions)

View changed files

📝 src/artifacts/ArtifactPaths.test.ts (+86 -0)
📝 src/artifacts/ArtifactPaths.ts (+38 -0)
📝 src/artifacts/NodeSeaArtifactReader.test.ts (+43 -0)
📝 src/artifacts/NodeSeaArtifactReader.ts (+27 -1)
📝 src/artifacts/PyInstallerArtifactReader.ts (+22 -3)
📝 src/artifacts/inventory/scanReader.ts (+13 -3)
📝 src/artifacts/pyinstaller/PyInstallerArtifactReader.test.ts (+82 -0)
➕ tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts (+109 -0)

📄 Description

Summary

Fixes #5108. inspect-artifact on a ZIP whose member name is not NFC-normalized (for example caf\u0065\u0301.txt in NFD, e + U+0301) reported the member only under its NFC logical path (caf\u00e9.txt, U+00E9). The Evidence contained no original decoded spelling, no raw result, and no limitation describing the change.

Problem and expected behavior

Smallest trigger: a ZIP with one member named caf\u0065\u0301.txt (NFD). Before, the inventoried occurrence had logical_path caf\u00e9.txt (NFC) and empty limitations. Expected: the NFC logical path is kept for identity, but the original decoded spelling is preserved in the occurrence's limitations so the normalization is visible in the Evidence.

Change and scope

  • src/artifacts/ArtifactPaths.ts: new unicodeNormalizationLimitation(original, normalized) helper. Returns undefined when the spelling is already NFC (a trailing-slash trim is structural, not a spelling change, and is not reported).
  • src/artifacts/inventory/scanReader.ts: visitArtifactEntries now keeps the pre-normalization joined path and, when normalization changed the spelling, records the original decoded spelling in the occurrence's limitations. This follows the existing pattern used for English Unicode case-fold collisions (annotateEnglishUnicodeCaseCollisions).
  • Scope: ordinary fix within existing boundaries (artifact inventory evidence fidelity). Maintainer scope decision: not applicable. Intentionally not included: traversal omission paths (different shape, no limitations field) and any change to NFC identity itself, which stays as the shared inventory spelling.

Contract and boundary impact

  • Semantic owner and earliest changed stage: artifact inventory (scanReader.visitArtifactEntries), the producing boundary for logical paths.
  • Contribution scope fit and linked maintainer decision: ordinary fix, not applicable.
  • CLI and MCP/tool-catalog contract: no schema change. artifactOccurrenceSchema.limitations is an existing string array; occurrences whose names were NFC-normalized now carry one additional limitation entry.
  • Provider, bridge, target-format, or platform compatibility: none.
  • Evidence, artifact, provenance, or reconstruction contract: Evidence now preserves the original decoded spelling where it was previously dropped. No identity values change (occurrence IDs, logical paths, digests unchanged).
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Expected effects (check known effects; use documentation-only on its own).

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated:
    • src/artifacts/ArtifactPaths.test.ts: 3 unit tests for unicodeNormalizationLimitation (NFD spelling preserved, already-NFC silent, trailing-slash trim silent).
    • tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts: 2 integration tests driving inventoryArtifact on a real ZIP with an NFD member name (asserts the NFC logical_path plus a limitation containing the original spelling; asserts no limitation for an NFC name).
  • Base reproduction or other evidence: reproduced on main at 32f0e51 with the issue's method (Python-created ZIP, NFD member). Before the fix the occurrence showed the NFC logical_path with limitations: []; after the fix the same logical_path plus a limitation naming the original NFD spelling. The 4 new behavior tests fail on the unfixed base (red) and pass with the fix (green).
  • User-visible CLI/MCP output (if applicable): an affected occurrence's limitations gains one entry, e.g. Archive member name "caf\u0065\u0301.txt" was Unicode-normalized to NFC "caf\u00e9.txt" for its logical path; the original decoded spelling is preserved in this limitation.
  • Remaining proof gaps: none for the reported case. Nested archives and non-ZIP readers share the same visitArtifactEntries path, so they get the same behavior; not separately exercised.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npx vitest run tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts src/artifacts/ArtifactPaths.test.ts: 12 passed with the fix; the 4 new behavior tests fail on the unfixed base.
  • npx vitest run src/artifacts/ src/artifacts/inventory/: 20 files, 197 tests passed.
  • npx vitest run tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts tests/boundary/filesystem/artifactDirectoryIdentity.test.ts: 29 passed.
  • npx tsc --noEmit: clean.
  • npm run lint: clean (only pre-existing warnings).
  • npm run format:check: clean.

Compatibility, safety, and release

  • Breaking changes or migration steps: none. Additive limitation entries only; no schema or identity change.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable (pure inventory path logic, exercised on Linux / Node 24).
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: the change preserves more source evidence; no new data collection.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

Author readiness

Human owner: @Varshith-Kali

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5127 **Author:** [@Varshith-Kali](https://github.com/Varshith-Kali) **Created:** 10/18/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/unicode-5108-fork` --- ### 📝 Commits (6) - [`cc5f2e9`](https://github.com/morluto/rea/commit/cc5f2e9e45bfefedf7a5cafadefa231c1eb5d58a) fix(artifacts): preserve original Unicode spelling when NFC normalization changes a logical path - [`e16e2ec`](https://github.com/morluto/rea/commit/e16e2ec67360e207e09cc19c59d2b428efe2658c) fix(artifacts): name the entry kind in the unicode-normalization limitation - [`c32fe2c`](https://github.com/morluto/rea/commit/c32fe2c77d2bd6bd0182ce35699f2fc3b07ee3c1) test(boundary): cover NFD folder and ZIP directory entries in unicode-normalization inventory - [`dc4e970`](https://github.com/morluto/rea/commit/dc4e9701bf709a3700a270b349598bb77a448381) fix(artifacts): carry pre-NFC PyInstaller member spelling into inventory - [`6401bde`](https://github.com/morluto/rea/commit/6401bdeb80c362b69fa62d9b843016a4b9153fb6) fix(artifacts): carry pre-NFC Node SEA member spelling into inventory - [`616dda5`](https://github.com/morluto/rea/commit/616dda593026a40fa2c57ffdc02a5880b6174ed2) fix(artifacts): keep trailing slash in NFC spelling for trailing-slash file members ### 📊 Changes **8 files changed** (+420 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `src/artifacts/ArtifactPaths.test.ts` (+86 -0) 📝 `src/artifacts/ArtifactPaths.ts` (+38 -0) 📝 `src/artifacts/NodeSeaArtifactReader.test.ts` (+43 -0) 📝 `src/artifacts/NodeSeaArtifactReader.ts` (+27 -1) 📝 `src/artifacts/PyInstallerArtifactReader.ts` (+22 -3) 📝 `src/artifacts/inventory/scanReader.ts` (+13 -3) 📝 `src/artifacts/pyinstaller/PyInstallerArtifactReader.test.ts` (+82 -0) ➕ `tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts` (+109 -0) </details> ### 📄 Description ## Summary Fixes #5108. `inspect-artifact` on a ZIP whose member name is not NFC-normalized (for example `caf\u0065\u0301.txt` in NFD, e + U+0301) reported the member only under its NFC logical path (`caf\u00e9.txt`, U+00E9). The Evidence contained no original decoded spelling, no raw result, and no limitation describing the change. ## Problem and expected behavior Smallest trigger: a ZIP with one member named `caf\u0065\u0301.txt` (NFD). Before, the inventoried occurrence had `logical_path` `caf\u00e9.txt` (NFC) and empty `limitations`. Expected: the NFC logical path is kept for identity, but the original decoded spelling is preserved in the occurrence's limitations so the normalization is visible in the Evidence. ## Change and scope - `src/artifacts/ArtifactPaths.ts`: new `unicodeNormalizationLimitation(original, normalized)` helper. Returns undefined when the spelling is already NFC (a trailing-slash trim is structural, not a spelling change, and is not reported). - `src/artifacts/inventory/scanReader.ts`: `visitArtifactEntries` now keeps the pre-normalization joined path and, when normalization changed the spelling, records the original decoded spelling in the occurrence's limitations. This follows the existing pattern used for English Unicode case-fold collisions (`annotateEnglishUnicodeCaseCollisions`). - Scope: ordinary fix within existing boundaries (artifact inventory evidence fidelity). Maintainer scope decision: not applicable. Intentionally not included: traversal omission paths (different shape, no limitations field) and any change to NFC identity itself, which stays as the shared inventory spelling. ## Contract and boundary impact - Semantic owner and earliest changed stage: artifact inventory (`scanReader.visitArtifactEntries`), the producing boundary for logical paths. - Contribution scope fit and linked maintainer decision: ordinary fix, not applicable. - CLI and MCP/tool-catalog contract: no schema change. `artifactOccurrenceSchema.limitations` is an existing string array; occurrences whose names were NFC-normalized now carry one additional limitation entry. - Provider, bridge, target-format, or platform compatibility: none. - Evidence, artifact, provenance, or reconstruction contract: Evidence now preserves the original decoded spelling where it was previously dropped. No identity values change (occurrence IDs, logical paths, digests unchanged). - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. Expected effects (check known effects; use documentation-only on its own). - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `src/artifacts/ArtifactPaths.test.ts`: 3 unit tests for `unicodeNormalizationLimitation` (NFD spelling preserved, already-NFC silent, trailing-slash trim silent). - `tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts`: 2 integration tests driving `inventoryArtifact` on a real ZIP with an NFD member name (asserts the NFC `logical_path` plus a limitation containing the original spelling; asserts no limitation for an NFC name). - Base reproduction or other evidence: reproduced on main at 32f0e51 with the issue's method (Python-created ZIP, NFD member). Before the fix the occurrence showed the NFC `logical_path` with `limitations: []`; after the fix the same `logical_path` plus a limitation naming the original NFD spelling. The 4 new behavior tests fail on the unfixed base (red) and pass with the fix (green). - User-visible CLI/MCP output (if applicable): an affected occurrence's `limitations` gains one entry, e.g. `Archive member name "caf\u0065\u0301.txt" was Unicode-normalized to NFC "caf\u00e9.txt" for its logical path; the original decoded spelling is preserved in this limitation.` - Remaining proof gaps: none for the reported case. Nested archives and non-ZIP readers share the same `visitArtifactEntries` path, so they get the same behavior; not separately exercised. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npx vitest run tests/boundary/filesystem/artifactInventoryUnicodeNormalization.test.ts src/artifacts/ArtifactPaths.test.ts`: 12 passed with the fix; the 4 new behavior tests fail on the unfixed base. - `npx vitest run src/artifacts/ src/artifacts/inventory/`: 20 files, 197 tests passed. - `npx vitest run tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts tests/boundary/filesystem/artifactDirectoryIdentity.test.ts`: 29 passed. - `npx tsc --noEmit`: clean. - `npm run lint`: clean (only pre-existing warnings). - `npm run format:check`: clean. ## Compatibility, safety, and release - Breaking changes or migration steps: none. Additive limitation entries only; no schema or identity change. - Real Hopper/Ghidra, browser, or OS coverage: not applicable (pure inventory path logic, exercised on Linux / Node 24). - Package or release metadata impact: none. - Security, privacy, process, or containment review: the change preserves more source evidence; no new data collection. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. ## Author readiness Human owner: @Varshith-Kali - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-11-01 21:42:58 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#6779
No description provided.