morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5320] [Bug] inspect-artifact reports CP949 ZIP member names only as CP437 text, with no limitation #3237

New issue

Closed

opened 2026-10-31 00:38:10 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 00:38:10 +00:00

Owner

Originally created by @eastminnn on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5320

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A ZIP member whose name bytes are CP949 (Korean), stored without the UTF-8 flag (bit 11) and without a Unicode Path extra field, appears in inspect-artifact Evidence only as its CP437 reading. 한글.txt (bytes c7 d1 b1 db 2e 74 78 74) is reported as ╟╤▒█.txt. The occurrence has no limitation, the inventory reports complete coverage, and the returned Evidence contains no Hangul text or other hint that the name may be in a different legacy code page.

Decoding unflagged names as CP437 follows APPNOTE and the decision in #3237. This report is not asking to change that. It is about the Evidence presenting a likely lossy reading as complete with nothing a caller can act on.

Reproduction

From a source checkout of the revision below with locked dependencies installed (npm ci), run the following from the repository root. Python creates the ZIP and checks it independently. The source CLI is loaded through the repository's locked tsx; no build is required.

WORK=$(mktemp -d)
mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/tmp"
python3 - "$WORK/names.zip" <<'PYFIXTURE'
import sys, zipfile
out = sys.argv[1]
korean = "한글.txt".encode("cp949")
placeholder = b"X" * (len(korean) - 4) + b".txt"
with zipfile.ZipFile(out, "w", zipfile.ZIP_STORED) as z:
    z.writestr(placeholder.decode("ascii"), b"korean\n")
    z.writestr("plain.txt", b"ascii\n")
data = open(out, "rb").read()
assert data.count(placeholder) == 2
open(out, "wb").write(data.replace(placeholder, korean))
with zipfile.ZipFile(out) as z:
    assert z.testzip() is None
    for info in z.infolist():
        raw = info.filename.encode("cp437")
        print(bool(info.flag_bits & 0x800), raw.hex(" "), repr(info.filename),
              repr(raw.decode("cp949", "replace")))
PYFIXTURE
cat > "$WORK/probe.mts" <<'JSPROBE'
import { join } from "node:path";
import { pathToFileURL } from "node:url";
const [repo, zipPath] = process.argv.slice(2);
const { createCli } = await import(pathToFileURL(join(repo, "src/cli.ts")).href);
let output = "";
let exitCode = 0;
await createCli(process.env).serve(["inspect-artifact", zipPath, "--json"], {
  stdout: (chunk) => { output += chunk; },
  exit: (code) => { exitCode = code; },
});
const evidence = JSON.parse(output);
const inventory = evidence.normalized_result.substeps.find(
  (s) => s.operation === "inventory_artifact",
).evidence;
const text = JSON.stringify(evidence);
console.log(JSON.stringify({
  exitCode,
  occurrences: inventory.normalized_result.occurrences
    .filter((o) => o.logical_path !== ".")
    .map((o) => ({ logical_path: o.logical_path, limitations: o.limitations ?? [] })),
  coverage: inventory.normalized_result.coverage,
  evidence_contains_hangul: /[가-힣]/.test(text),
}, null, 2));
JSPROBE
env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" \
  XDG_CACHE_HOME="$WORK/cache" TMPDIR="$WORK/tmp" \
  node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$PWD" "$WORK/names.zip"

Reproduced on every run (3 runs).

Expected behavior

Keep the CP437 logical_path and the current flag and Unicode Path precedence. When an unflagged name contains non-ASCII bytes, the Evidence should let a caller know the name came from a legacy code page whose reading is uncertain, so an agent does not treat ╟╤▒█.txt as a confident name.

One possible form, following the existing pattern for the omitted ./ prefix (ZipArtifactReader.ts#L294-L300): an occurrence limitation that keeps the raw name bytes and, when they are valid CP949, the alternative reading. This is a proposal, not a verified requirement.

Impact and scope

Callers cannot tell from the Evidence that the reported name may be a misreading, and cannot recover the Korean name without re-reading the archive. Content bytes, CRC checks and member identity are unaffected; this is a filename-provenance gap.

Other ZIP readers treat this as a known case. Python's standard-library zipfile also decodes unflagged names as CP437 by default, but offers metadata_encoding for archives written in another code page; with metadata_encoding="cp949" it reads this fixture's member as 한글.txt.

Observed only with this self-authored fixture. I have not verified which current Korean Windows archivers (Explorer, Bandizip, ALZip) still write CP949 names without the UTF-8 flag; this varies by tool, version and settings. Other legacy East Asian code pages such as Shift_JIS likely behave the same way but were not tested.

REA version

6.7.0 source, main revision 833136507dd79d959340b09862a53af344e4f1b3. The published package was not tested.

Runtime and platform

Node.js v24.15.0 (the repository pins 24.18.0 for development; 24.15.0 is within the supported ^24.11.0 range); macOS 15.5; arm64; Python 3.14.2 standard-library zipfile. No native analysis engine or external service was used.

Target and artifact context

Self-authored stored ZIP with two unflagged members: 한글.txt as CP949 bytes and plain.txt as ASCII. Python verified the names, flags, contents and CRCs. All REA home, configuration, cache and temporary locations were isolated.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Fixture check (Python):

False c7 d1 b1 db 2e 74 78 74 '╟╤▒█.txt' '한글.txt'
False 70 6c 61 69 6e 2e 74 78 74 'plain.txt' 'plain.txt'

Observed source CLI output:

{
  "exitCode": 0,
  "occurrences": [
    { "logical_path": "plain.txt", "limitations": [] },
    { "logical_path": "╟╤▒█.txt", "limitations": [] }
  ],
  "coverage": { "scope": "container-entries", "status": "complete" },
  "evidence_contains_hangul": false
}

Typed outcome or recovery expected

Inspection should still succeed with the same logical_path and complete coverage, plus a limitation on the affected occurrence. ASCII-only and UTF-8 flagged names should be unchanged.

Analysis or suspected cause

Source-derived: ZipArtifactReader.ts#L184 sets filenameEncoding: "cp437", and decodeZipEntryText decodes unflagged names only with decodeCp437. Nothing records that the name came from a legacy code page.

Design caveat for any fix: CP949 validity is not proof. Some valid CP437 names are also valid CP949, for example naïve.txt (6e 61 8b 76 65 ...) reads as na땦e.txt. An alternative reading should therefore be labeled as a candidate, never replace the CP437 name, and its wording or which code pages to try may need a maintainer decision.

Additional context

Related but distinct: #3237 (unflagged names were guessed as UTF-8; fixed in 7777abf4 by decoding them as CP437, which this report keeps) and #5108 (original NFD spelling is lost after correct UTF-8 decoding). #2365 handled ASCII control bytes in legacy names. Issue and PR searches for cp949, euc-kr, shift_jis, legacy zip name, code page, filename encoding and korean filename found no report for this case; the only cp949 match, #3707, concerns the GRF archive format rather than ZIP names.

Originally created by @eastminnn on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5320 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior A ZIP member whose name bytes are CP949 (Korean), stored without the UTF-8 flag (bit 11) and without a Unicode Path extra field, appears in `inspect-artifact` Evidence only as its CP437 reading. `한글.txt` (bytes `c7 d1 b1 db 2e 74 78 74`) is reported as `╟╤▒█.txt`. The occurrence has no limitation, the inventory reports `complete` coverage, and the returned Evidence contains no Hangul text or other hint that the name may be in a different legacy code page. Decoding unflagged names as CP437 follows APPNOTE and the decision in #3237. This report is not asking to change that. It is about the Evidence presenting a likely lossy reading as complete with nothing a caller can act on. ## Reproduction From a source checkout of the revision below with locked dependencies installed (`npm ci`), run the following from the repository root. Python creates the ZIP and checks it independently. The source CLI is loaded through the repository's locked `tsx`; no build is required. ```sh WORK=$(mktemp -d) mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/tmp" python3 - "$WORK/names.zip" <<'PYFIXTURE' import sys, zipfile out = sys.argv[1] korean = "한글.txt".encode("cp949") placeholder = b"X" * (len(korean) - 4) + b".txt" with zipfile.ZipFile(out, "w", zipfile.ZIP_STORED) as z: z.writestr(placeholder.decode("ascii"), b"korean\n") z.writestr("plain.txt", b"ascii\n") data = open(out, "rb").read() assert data.count(placeholder) == 2 open(out, "wb").write(data.replace(placeholder, korean)) with zipfile.ZipFile(out) as z: assert z.testzip() is None for info in z.infolist(): raw = info.filename.encode("cp437") print(bool(info.flag_bits & 0x800), raw.hex(" "), repr(info.filename), repr(raw.decode("cp949", "replace"))) PYFIXTURE cat > "$WORK/probe.mts" <<'JSPROBE' import { join } from "node:path"; import { pathToFileURL } from "node:url"; const [repo, zipPath] = process.argv.slice(2); const { createCli } = await import(pathToFileURL(join(repo, "src/cli.ts")).href); let output = ""; let exitCode = 0; await createCli(process.env).serve(["inspect-artifact", zipPath, "--json"], { stdout: (chunk) => { output += chunk; }, exit: (code) => { exitCode = code; }, }); const evidence = JSON.parse(output); const inventory = evidence.normalized_result.substeps.find( (s) => s.operation === "inventory_artifact", ).evidence; const text = JSON.stringify(evidence); console.log(JSON.stringify({ exitCode, occurrences: inventory.normalized_result.occurrences .filter((o) => o.logical_path !== ".") .map((o) => ({ logical_path: o.logical_path, limitations: o.limitations ?? [] })), coverage: inventory.normalized_result.coverage, evidence_contains_hangul: /[가-힣]/.test(text), }, null, 2)); JSPROBE env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" \ XDG_CACHE_HOME="$WORK/cache" TMPDIR="$WORK/tmp" \ node node_modules/tsx/dist/cli.mjs "$WORK/probe.mts" "$PWD" "$WORK/names.zip" ``` Reproduced on every run (3 runs). ## Expected behavior Keep the CP437 `logical_path` and the current flag and Unicode Path precedence. When an unflagged name contains non-ASCII bytes, the Evidence should let a caller know the name came from a legacy code page whose reading is uncertain, so an agent does not treat `╟╤▒█.txt` as a confident name. One possible form, following the existing pattern for the omitted `./` prefix ([ZipArtifactReader.ts#L294-L300](https://github.com/morluto/rea/blob/833136507dd79d959340b09862a53af344e4f1b3/src/artifacts/ZipArtifactReader.ts#L294-L300)): an occurrence limitation that keeps the raw name bytes and, when they are valid CP949, the alternative reading. This is a proposal, not a verified requirement. ## Impact and scope Callers cannot tell from the Evidence that the reported name may be a misreading, and cannot recover the Korean name without re-reading the archive. Content bytes, CRC checks and member identity are unaffected; this is a filename-provenance gap. Other ZIP readers treat this as a known case. Python's standard-library `zipfile` also decodes unflagged names as CP437 by default, but offers `metadata_encoding` for archives written in another code page; with `metadata_encoding="cp949"` it reads this fixture's member as `한글.txt`. Observed only with this self-authored fixture. I have not verified which current Korean Windows archivers (Explorer, Bandizip, ALZip) still write CP949 names without the UTF-8 flag; this varies by tool, version and settings. Other legacy East Asian code pages such as Shift_JIS likely behave the same way but were not tested. ## REA version 6.7.0 source, main revision `833136507dd79d959340b09862a53af344e4f1b3`. The published package was not tested. ## Runtime and platform Node.js v24.15.0 (the repository pins 24.18.0 for development; 24.15.0 is within the supported `^24.11.0` range); macOS 15.5; arm64; Python 3.14.2 standard-library `zipfile`. No native analysis engine or external service was used. ## Target and artifact context Self-authored stored ZIP with two unflagged members: `한글.txt` as CP949 bytes and `plain.txt` as ASCII. Python verified the names, flags, contents and CRCs. All REA home, configuration, cache and temporary locations were isolated. ## Interface CLI ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Fixture check (Python): ```text False c7 d1 b1 db 2e 74 78 74 '╟╤▒█.txt' '한글.txt' False 70 6c 61 69 6e 2e 74 78 74 'plain.txt' 'plain.txt' ``` Observed source CLI output: ```json { "exitCode": 0, "occurrences": [ { "logical_path": "plain.txt", "limitations": [] }, { "logical_path": "╟╤▒█.txt", "limitations": [] } ], "coverage": { "scope": "container-entries", "status": "complete" }, "evidence_contains_hangul": false } ``` ## Typed outcome or recovery expected Inspection should still succeed with the same `logical_path` and `complete` coverage, plus a limitation on the affected occurrence. ASCII-only and UTF-8 flagged names should be unchanged. ## Analysis or suspected cause Source-derived: [ZipArtifactReader.ts#L184](https://github.com/morluto/rea/blob/833136507dd79d959340b09862a53af344e4f1b3/src/artifacts/ZipArtifactReader.ts#L184) sets `filenameEncoding: "cp437"`, and [`decodeZipEntryText`](https://github.com/morluto/rea/blob/833136507dd79d959340b09862a53af344e4f1b3/src/artifacts/ZipArtifactReader.ts#L306-L315) decodes unflagged names only with [`decodeCp437`](https://github.com/morluto/rea/blob/833136507dd79d959340b09862a53af344e4f1b3/src/artifacts/ZipArtifactReader.ts#L327-L332). Nothing records that the name came from a legacy code page. Design caveat for any fix: CP949 validity is not proof. Some valid CP437 names are also valid CP949, for example `naïve.txt` (`6e 61 8b 76 65 ...`) reads as `na땦e.txt`. An alternative reading should therefore be labeled as a candidate, never replace the CP437 name, and its wording or which code pages to try may need a maintainer decision. ## Additional context Related but distinct: #3237 (unflagged names were guessed as UTF-8; fixed in 7777abf4 by decoding them as CP437, which this report keeps) and #5108 (original NFD spelling is lost after correct UTF-8 decoding). #2365 handled ASCII control bytes in legacy names. Issue and PR searches for `cp949`, `euc-kr`, `shift_jis`, `legacy zip name`, `code page`, `filename encoding` and `korean filename` found no report for this case; the only `cp949` match, #3707, concerns the GRF archive format rather than ZIP names.

bsakaguchi closed this issue

2026-10-31 00:38:10 +00:00

bsakaguchi commented

2026-10-31 00:38:11 +00:00

Author

Owner

@darshjoshi commented on GitHub (Oct 20, 2026):

I'd like to work on this. I'll keep the CP437 logical_path and the current flag and Unicode Path precedence, and add an occurrence limitation when an unflagged name has non-ASCII bytes, carrying the raw name bytes. PR to follow.

@darshjoshi commented on GitHub (Oct 20, 2026): I'd like to work on this. I'll keep the CP437 `logical_path` and the current flag and Unicode Path precedence, and add an occurrence limitation when an unflagged name has non-ASCII bytes, carrying the raw name bytes. PR to follow.

bsakaguchi commented

2026-10-31 00:38:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 20, 2026):

Agreed with the framing: keeping the APPNOTE-mandated CP437 decoding while surfacing a limitation when an unflagged name contains non-ASCII bytes is the right trade-off. Without a marker, an agent has no way to distinguish a confidently decoded path from a lossy legacy code-page reading, so ╟╤▒█.txt would silently be treated as a real filename. The #5321 approach of noting legacy code page readings gives downstream callers exactly that signal.

@XinwangTao commented on GitHub (Oct 20, 2026): Agreed with the framing: keeping the APPNOTE-mandated CP437 decoding while surfacing a limitation when an unflagged name contains non-ASCII bytes is the right trade-off. Without a marker, an agent has no way to distinguish a confidently decoded path from a lossy legacy code-page reading, so ╟╤▒█.txt would silently be treated as a real filename. The #5321 approach of noting legacy code page readings gives downstream callers exactly that signal.

bsakaguchi commented

2026-10-31 00:38:12 +00:00

Author

Owner

@eastminnn commented on GitHub (Oct 21, 2026):

Fixed by 2e03251b6 (on main, thanks @morluto): an unflagged name with non-ASCII bytes now carries ZIP name uses CP437 fallback; source encoding unverified; raw hex: ….

I re-ran the reproduction above on 60956c8c: ╟╤▒█.txt gets ZIP name uses CP437 fallback; source encoding unverified; raw hex: c7d1b1db2e747874., plain.txt has no limitation, and coverage stays complete. Closing.

@eastminnn commented on GitHub (Oct 21, 2026): Fixed by 2e03251b6 (on main, thanks @morluto): an unflagged name with non-ASCII bytes now carries `ZIP name uses CP437 fallback; source encoding unverified; raw hex: …`. I re-ran the reproduction above on `60956c8c`: `╟╤▒█.txt` gets `ZIP name uses CP437 fallback; source encoding unverified; raw hex: c7d1b1db2e747874.`, `plain.txt` has no limitation, and coverage stays `complete`. Closing.

bsakaguchi commented

2026-10-31 00:38:12 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

Verified on my side too — the re-run on 60956c8c now emits ZIP name uses CP437 fallback; source encoding unverified; raw hex: c7d1b1db2e747874. with coverage still complete, exactly the signal we agreed on. Closing as completed is appropriate; downstream callers can now distinguish a confident path from a legacy code-page reading.

@XinwangTao commented on GitHub (Oct 21, 2026): Verified on my side too — the re-run on 60956c8c now emits `ZIP name uses CP437 fallback; source encoding unverified; raw hex: c7d1b1db2e747874.` with coverage still complete, exactly the signal we agreed on. Closing as completed is appropriate; downstream callers can now distinguish a confident path from a legacy code-page reading.

bsakaguchi referenced this issue

2026-10-31 01:08:27 +00:00

[PR #3241] [CLOSED] fix(artifacts): decode unflagged ZIP names as the legacy character set #5667

bsakaguchi referenced this issue

2026-10-31 01:08:30 +00:00

[PR #3245] [CLOSED] fix(artifacts): respect ZIP legacy filename encoding #5673

bsakaguchi referenced this issue

2026-11-01 21:43:45 +00:00

[PR #5321] [CLOSED] fix(artifacts): note legacy code page readings of unflagged ZIP names #6878