morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5108] [Bug] inspect-artifact loses original decomposed Unicode member names #3145

New issue

Closed

opened 2026-10-31 00:37:10 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 00:37:10 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5108

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A ZIP member named cafe\u0301.txt (decomposed e plus combining acute) appears in inspect-artifact Evidence only as caf\u00e9.txt. The complete returned Evidence contains no original decoded spelling, no raw result, and no limitation describing the change.

This was reproduced through the actual source CLI command registry, createCli(...).serve(['inspect-artifact', path, '--json']), with real ZIP parsing and hashing. The command exits 0 and reports complete container-entry coverage. No provider was mocked. The compiled executable launcher and published package were not run.

Reproduction

From a source checkout of the revision below with its locked dependencies installed, run the following. Python creates and independently checks a harmless stored ZIP. The source CLI is loaded through the repository's locked tsx dependency; no build is required.

WORK=$(mktemp -d)
mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/tmp"
python3 - "$WORK" <<'PYFIXTURE'
import pathlib, sys, zipfile
archive = pathlib.Path(sys.argv[1]) / 'single-nfd.zip'
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_STORED) as z:
    z.writestr('cafe\u0301.txt', b'alpha\n')
with zipfile.ZipFile(archive) as z:
    member = z.infolist()[0]
    assert z.testzip() is None
    assert member.flag_bits & 0x800
    assert member.filename == 'cafe\u0301.txt'
    assert z.read(member) == b'alpha\n'
    print('source code points:', [hex(ord(c)) for c in member.filename])
PYFIXTURE
cat > "$WORK/probe.mts" <<'JSPROBE'
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
const { createCli } = await import(
  pathToFileURL(join(process.cwd(), 'src/cli.ts')).href
);
const root = process.argv[2];
let output = '';
let exitCode = 0;
await createCli(process.env).serve(
  ['inspect-artifact', join(root, 'single-nfd.zip'), '--json'],
  { stdout: chunk => { output += chunk; },
    exit: code => { exitCode = code; } }
);
const evidence = JSON.parse(output);
const inventory = evidence.normalized_result.substeps.find(
  s => s.operation === 'inventory_artifact'
).evidence;
const occurrence = inventory.normalized_result.occurrences.find(
  o => o.logical_path !== '.'
);
console.log(JSON.stringify({
  exitCode,
  path: occurrence.logical_path,
  codepoints: [...occurrence.logical_path].map(c => c.codePointAt(0).toString(16)),
  raw_result: inventory.raw_result,
  source_location: occurrence.source_location,
  limitations: occurrence.limitations,
  coverage: inventory.normalized_result.coverage,
  retains_combining_acute: JSON.stringify(evidence).includes('\u0301')
}, null, 2));
JSPROBE
env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" \
  XDG_CACHE_HOME="$WORK/cache" TMPDIR="$WORK/tmp" \
  node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$WORK"

The exact recipe above was also executed successfully and returned the output below. Additional real-ZIP runs through the shared analysis entry point checked an NFC/NFD pair and an ASCII control: both Unicode occurrences survived with separate occurrence identities, but the NFD spelling was absent; plain.txt remained unchanged.

Expected behavior

Keep the intentional NFC logical_path for lookup while retaining the decoded source member spelling when normalization changes it, so an analyst can recover that spelling from the returned Evidence.

The repository preservation contract requires preserving paths and source values when normalization loses information. The ZIP reader already retains the original name in a limitation when removing a leading current-directory prefix. The equivalent provenance is absent for the NFC change.

Impact and scope

The returned complete Evidence cannot distinguish the original composed spelling from the original decomposed spelling. This is a bounded filename-provenance loss, reproduced for a single ordinary UTF-8 ZIP member without any path collision. The source ZIP and its content bytes remain intact, and canonical logical identity is intentional.

A verified workaround is to read and retain the source names separately with Python zipfile, as the fixture check does. This does not restore the missing spelling in existing REA Evidence. NOT_ENOUGH_INFO: prevalence in real archives and behavior on other platforms or artifact formats were not measured.

REA version

6.7.0 source, main revision 3537e58c1ab29309f06eabf50bc52adba9817a0c. The relevant source files are unchanged from tag rea-agents-6.7.0; that published package's runtime was not tested.

Runtime and platform

Node.js v26.12.0; macOS 27.0.1, build 26A434; arm64. Locked source dependencies and Python standard-library zipfile. No native analysis engine or external service was used.

Target and artifact context

Self-authored stored ZIP containing one regular UTF-8 member, cafe\u0301.txt, with bytes alpha\n. Python verified its filename, UTF-8 flag, content and CRC. All REA home, configuration, cache and temporary locations were isolated for the subprocess.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The source member's relevant code points are 65 301; the CLI returns e9 instead. Observed source CLI output:

{
  "exitCode": 0,
  "path": "café.txt",
  "codepoints": ["63", "61", "66", "e9", "2e", "74", "78", "74"],
  "raw_result": null,
  "source_location": null,
  "limitations": [],
  "coverage": {"scope": "container-entries", "status": "complete"},
  "retains_combining_acute": false
}

The entire parsed response was searched for U+0301, rather than checking only the displayed path. No source-name field or other retained occurrence fact preserved it.

Typed outcome or recovery expected

The single-member inspection should continue to succeed with NFC logical identity and verified content, while preserving the original decomposed source spelling alongside that identity. Canonically colliding layouts may continue to receive a typed path refusal during extraction.

Analysis or suspected cause

Source-derived explanation of the observed loss: the inventory scan passes the NFC-normalized path to occurrence construction. That projection stores only logical_path and existing entry limitations; the occurrence schema has no original-name field. The inspection Evidence has no raw result to recover that source spelling.

Additional context

Issue search distinguished #1488, which aligns joins with intended NFC lookup; #3152, which preserves duplicate physical occurrences; #4271, which corrects the diagnosis for extraction collisions; and #3237, which fixes legacy filename decoding. This report concerns source-spelling retention after correct UTF-8 decoding and reproduces without a collision.

NOT_ENOUGH_INFO: first affected release, published-package runtime reproduction, compiled-launcher behavior, MCP transport and other platforms were not tested. No extraction failure or target execution is claimed.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5108 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior A ZIP member named `cafe\u0301.txt` (decomposed `e` plus combining acute) appears in `inspect-artifact` Evidence only as `caf\u00e9.txt`. The complete returned Evidence contains no original decoded spelling, no raw result, and no limitation describing the change. This was reproduced through the actual source CLI command registry, `createCli(...).serve(['inspect-artifact', path, '--json'])`, with real ZIP parsing and hashing. The command exits 0 and reports complete container-entry coverage. No provider was mocked. The compiled executable launcher and published package were not run. ## Reproduction From a source checkout of the revision below with its locked dependencies installed, run the following. Python creates and independently checks a harmless stored ZIP. The source CLI is loaded through the repository's locked `tsx` dependency; no build is required. ```sh WORK=$(mktemp -d) mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/tmp" python3 - "$WORK" <<'PYFIXTURE' import pathlib, sys, zipfile archive = pathlib.Path(sys.argv[1]) / 'single-nfd.zip' with zipfile.ZipFile(archive, 'w', zipfile.ZIP_STORED) as z: z.writestr('cafe\u0301.txt', b'alpha\n') with zipfile.ZipFile(archive) as z: member = z.infolist()[0] assert z.testzip() is None assert member.flag_bits & 0x800 assert member.filename == 'cafe\u0301.txt' assert z.read(member) == b'alpha\n' print('source code points:', [hex(ord(c)) for c in member.filename]) PYFIXTURE cat > "$WORK/probe.mts" <<'JSPROBE' import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; const { createCli } = await import( pathToFileURL(join(process.cwd(), 'src/cli.ts')).href ); const root = process.argv[2]; let output = ''; let exitCode = 0; await createCli(process.env).serve( ['inspect-artifact', join(root, 'single-nfd.zip'), '--json'], { stdout: chunk => { output += chunk; }, exit: code => { exitCode = code; } } ); const evidence = JSON.parse(output); const inventory = evidence.normalized_result.substeps.find( s => s.operation === 'inventory_artifact' ).evidence; const occurrence = inventory.normalized_result.occurrences.find( o => o.logical_path !== '.' ); console.log(JSON.stringify({ exitCode, path: occurrence.logical_path, codepoints: [...occurrence.logical_path].map(c => c.codePointAt(0).toString(16)), raw_result: inventory.raw_result, source_location: occurrence.source_location, limitations: occurrence.limitations, coverage: inventory.normalized_result.coverage, retains_combining_acute: JSON.stringify(evidence).includes('\u0301') }, null, 2)); JSPROBE env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" \ XDG_CACHE_HOME="$WORK/cache" TMPDIR="$WORK/tmp" \ node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$WORK" ``` The exact recipe above was also executed successfully and returned the output below. Additional real-ZIP runs through the shared analysis entry point checked an NFC/NFD pair and an ASCII control: both Unicode occurrences survived with separate occurrence identities, but the NFD spelling was absent; `plain.txt` remained unchanged. ## Expected behavior Keep the intentional NFC `logical_path` for lookup while retaining the decoded source member spelling when normalization changes it, so an analyst can recover that spelling from the returned Evidence. The repository [preservation contract](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/AGENTS.md#L13-L31) requires preserving paths and source values when normalization loses information. The [ZIP reader already retains the original name in a limitation when removing a leading current-directory prefix](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/artifacts/ZipArtifactReader.ts#L279-L300). The equivalent provenance is absent for the NFC change. ## Impact and scope The returned complete Evidence cannot distinguish the original composed spelling from the original decomposed spelling. This is a bounded filename-provenance loss, reproduced for a single ordinary UTF-8 ZIP member without any path collision. The source ZIP and its content bytes remain intact, and canonical logical identity is intentional. A verified workaround is to read and retain the source names separately with Python `zipfile`, as the fixture check does. This does not restore the missing spelling in existing REA Evidence. NOT_ENOUGH_INFO: prevalence in real archives and behavior on other platforms or artifact formats were not measured. ## REA version 6.7.0 source, main revision `3537e58c1ab29309f06eabf50bc52adba9817a0c`. The relevant source files are unchanged from tag `rea-agents-6.7.0`; that published package's runtime was not tested. ## Runtime and platform Node.js v26.12.0; macOS 27.0.1, build 26A434; arm64. Locked source dependencies and Python standard-library `zipfile`. No native analysis engine or external service was used. ## Target and artifact context Self-authored stored ZIP containing one regular UTF-8 member, `cafe\u0301.txt`, with bytes `alpha\n`. Python verified its filename, UTF-8 flag, content and CRC. All REA home, configuration, cache and temporary locations were isolated for the subprocess. ## Interface CLI ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence The source member's relevant code points are `65 301`; the CLI returns `e9` instead. Observed source CLI output: ```json { "exitCode": 0, "path": "café.txt", "codepoints": ["63", "61", "66", "e9", "2e", "74", "78", "74"], "raw_result": null, "source_location": null, "limitations": [], "coverage": {"scope": "container-entries", "status": "complete"}, "retains_combining_acute": false } ``` The entire parsed response was searched for U+0301, rather than checking only the displayed path. No source-name field or other retained occurrence fact preserved it. ## Typed outcome or recovery expected The single-member inspection should continue to succeed with NFC logical identity and verified content, while preserving the original decomposed source spelling alongside that identity. Canonically colliding layouts may continue to receive a typed path refusal during extraction. ## Analysis or suspected cause Source-derived explanation of the observed loss: the [inventory scan](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/artifacts/inventory/scanReader.ts#L178-L197) passes the [NFC-normalized path](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/artifacts/ArtifactPaths.ts#L23-L41) to [occurrence construction](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/artifacts/inventory/ArtifactGraphConstruction.ts#L89-L112). That projection stores only `logical_path` and existing entry limitations; the [occurrence schema](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/domain/artifactGraph.ts#L114-L140) has no original-name field. The [inspection Evidence](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/artifacts/ArtifactProvider.ts#L360-L367) has no raw result to recover that source spelling. ## Additional context Issue search distinguished [#1488](https://github.com/morluto/rea/issues/1488), which aligns joins with intended NFC lookup; [#3152](https://github.com/morluto/rea/issues/3152), which preserves duplicate physical occurrences; [#4271](https://github.com/morluto/rea/issues/4271), which corrects the diagnosis for extraction collisions; and [#3237](https://github.com/morluto/rea/issues/3237), which fixes legacy filename decoding. This report concerns source-spelling retention after correct UTF-8 decoding and reproduces without a collision. NOT_ENOUGH_INFO: first affected release, published-package runtime reproduction, compiled-launcher behavior, MCP transport and other platforms were not tested. No extraction failure or target execution is claimed.

bsakaguchi closed this issue

2026-10-31 00:37:10 +00:00

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@Varshith-Kali commented on GitHub (Oct 18, 2026):

Hi, I'd like to take this on.

I reproduced it on main (32f0e51): a ZIP member named café.txt in NFD (e + U+0301) is inventoried with logical_path in NFC (U+00E9), and the Evidence carries no trace of the original decoded spelling. The NFC step itself happens in normalizeArtifactPath via logicalPathNfc, which is the right place for identity, but the original spelling is dropped at that producing boundary with nothing recorded.

My plan: keep the NFC logical path for identity, and when normalization changes a member's spelling, record the original decoded spelling in that occurrence's limitations, following the existing pattern used for case-fold collisions. I'll add a regression test driving inspect-artifact on an NFD-named ZIP member.

@Varshith-Kali commented on GitHub (Oct 18, 2026): Hi, I'd like to take this on. I reproduced it on main (32f0e51): a ZIP member named `café.txt` in NFD (e + U+0301) is inventoried with logical_path in NFC (U+00E9), and the Evidence carries no trace of the original decoded spelling. The NFC step itself happens in `normalizeArtifactPath` via `logicalPathNfc`, which is the right place for identity, but the original spelling is dropped at that producing boundary with nothing recorded. My plan: keep the NFC logical path for identity, and when normalization changes a member's spelling, record the original decoded spelling in that occurrence's limitations, following the existing pattern used for case-fold collisions. I'll add a regression test driving `inspect-artifact` on an NFD-named ZIP member.

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Confirmed — silently NFC-normalizing a member name breaks name fidelity for evidence: an artifact with cafe\u0301.txt is reported as cafe\u00e9.txt with no raw value and no limitation, so the change is indistinguishable from the original spelling to downstream consumers. For forensic/artifact workflows that's a real integrity problem, not just cosmetic.

I support the direction of PR #5125/#5127: keep the member's own name when its logical path gets NFC-normalized, and I'd suggest recording a limitation (or an original/normalized pair) whenever normalization actually changes bytes, so evidence remains self-describing and reversible.

@XinwangTao commented on GitHub (Oct 19, 2026): Confirmed — silently NFC-normalizing a member name breaks name fidelity for evidence: an artifact with cafe\u0301.txt is reported as cafe\u00e9.txt with no raw value and no limitation, so the change is indistinguishable from the original spelling to downstream consumers. For forensic/artifact workflows that's a real integrity problem, not just cosmetic. I support the direction of PR #5125/#5127: keep the member's own name when its logical path gets NFC-normalized, and I'd suggest recording a limitation (or an original/normalized pair) whenever normalization actually changes bytes, so evidence remains self-describing and reversible.

bsakaguchi commented

2026-10-31 00:37:11 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

ArtifactGraphConstruction emits source_path whenever entry sourcePath exists or original entry spelling differs from logicalPathNfc, while normalized logical_path remains the lookup identity. ZIP regression inventories an NFD member and asserts original decomposed source_path, normalized logical key and verified hash; it also covers leading ./ and an unchanged name. No archive runtime run here.

Implementation: ArtifactGraphConstruction.ts, artifactGraph.ts.

The relevant regression checks in artifactZipIntegrity.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. ArtifactGraphConstruction emits source_path whenever entry sourcePath exists or original entry spelling differs from logicalPathNfc, while normalized logical_path remains the lookup identity. ZIP regression inventories an NFD member and asserts original decomposed source_path, normalized logical key and verified hash; it also covers leading ./ and an unchanged name. No archive runtime run here. Implementation: [ArtifactGraphConstruction.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/artifacts/inventory/ArtifactGraphConstruction.ts#L108), [artifactGraph.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/artifactGraph.ts#L154). The relevant regression checks in [artifactZipIntegrity.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/filesystem/artifactZipIntegrity.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 00:38:58 +00:00

[GH-ISSUE #5444] [Docs] Capture guide incorrectly says unsafe standard HAR numbers fail validation #3305

bsakaguchi referenced this issue

2026-10-31 01:08:15 +00:00

[PR #3159] [MERGED] fix(captures): admit schema-valid HAR numbers that evidence cannot publish exactly #5647