morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6653] [Bug] Reference reader substitutes another file after lossy POSIX filename decoding #4032

New issue

Closed

opened 2026-10-31 00:46:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:12 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6653

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Executed at the production readReferenceSource filesystem boundary: two distinct valid POSIX byte filenames collapse to the same decoded pathname. The reader reports success and reads the replacement-character file twice, omitting the other file's bytes.

The files are 626164ff2e6a73 (bad<FF>.js, containing RAW_FILE) and 626164efbfbd2e6a73 (bad�.js, containing UNICODE_FILE). Raw-byte readdir confirms both entries, and direct raw-path readFile returns RAW_FILE. The reader instead returns two bad�.js file entries containing UNICODE_FILE, with bytesRead: 24.

Reproduction

The successful reader probe ran once on Linux with the three directory cases below plus a supplementary symlink-target control. This reduced reproduction retains the three directory cases and uses unchanged production source. It requires the checkout's locked dependencies.

Save this source import resolver as /tmp/reference-source-loader.mjs:

import { registerHooks } from 'node:module';
import { existsSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
registerHooks({ resolve(specifier, context, next) {
  try { return next(specifier, context); } catch (error) {
    if (error.code === 'ERR_MODULE_NOT_FOUND' && /^\.{1,2}\//.test(specifier) && specifier.endsWith('.js')) {
      const source = new URL(specifier.slice(0, -3) + '.ts', context.parentURL);
      if (existsSync(fileURLToPath(source))) return next(source.href, context);
    }
    throw error;
  }
} });

Save as /tmp/reference-filenames.mjs:

import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const repo = resolve(process.argv[2]);
const load = (name) => import(pathToFileURL(join(repo, 'src', name + '.ts')).href);
const { readReferenceSource } = await load('reference/ReferenceSourceReader');
const { ArtifactResourceScope } = await load('artifacts/ArtifactResourceScope');
const base = await mkdtemp('/tmp/reference-filenames-');
const raw = Buffer.from([0x62, 0x61, 0x64, 0xff, 0x2e, 0x6a, 0x73]);
const text = raw.toString('utf8');
const rawPath = (root) => Buffer.concat([Buffer.from(root + '/'), raw]);
try {
  for (const name of ['unicode-control', 'raw-only', 'raw-and-replacement']) {
    const root = join(base, name);
    await mkdir(root);
    if (name !== 'unicode-control') await writeFile(rawPath(root), 'RAW_FILE');
    if (name !== 'raw-only') await writeFile(join(root, text), 'UNICODE_FILE');
    console.log(name, (await readdir(root, { encoding: 'buffer' })).map(x => x.toString('hex')));
    if (name !== 'unicode-control') console.log('raw file:', (await readFile(rawPath(root))).toString());
    const resources = new ArtifactResourceScope();
    try {
      const result = await readReferenceSource(root, resources);
      if (!result.ok) throw new Error(JSON.stringify(result.error));
      console.log(JSON.stringify({ bytesRead: result.value.bytesRead, entries: result.value.entries.map(e =>
        e.status === 'read' && e.kind === 'file'
          ? { path: e.path, size: e.size, text: Buffer.from(e.bytes).toString() }
          : { status: e.status, kind: e.kind, path: e.path, code: e.code }) }));
    } finally { await resources.close(); }
  }
} finally { await rm(base, { recursive: true, force: true }); }

From the checkout root:

node --experimental-transform-types --import /tmp/reference-source-loader.mjs   /tmp/reference-filenames.mjs "$PWD"

The import hook only maps unresolved relative .js source imports to their existing .ts siblings; Node transforms their TypeScript. No analyzed source is executed. No reader/importer probe was rerun after this draft was written.

Expected behavior

Retain distinct filesystem identities. If the graph's text path representation cannot encode a byte filename, report that limitation without accessing a different file under a lossy replacement name. Do not return another file's bytes as a successful observation of the original entry.

The inventory path contract states that reference inventories retain filesystem spelling. Repository guidance requires preserving selected paths and evidence, and keeping identity separate from display normalization.

Impact and scope

The observed reader invents a duplicate successful observation, omits RAW_FILE, and inflates bytes read from distinct members. Raw-only input instead produces an unknown-kind I/O entry under the substituted name; the Unicode-only control reads correctly.

This is bounded wrong-file evidence on a Linux filesystem accepting non-UTF-8 names. No application graph, CLI/MCP result, source mutation, or downstream comparison was executed. Other platforms and real-world frequency are unknown. Renaming source entries would change the input identity; no result-preserving workaround was verified.

REA version

Package metadata 6.7.0; source revision 23def04096869b85c4d3c14a73ed09168483c9f6 with locked dependencies. Published tarball execution was not tested.

Runtime and platform

Node.js v24.22.1, Linux x64; source loading with --experimental-transform-types and the resolver above. The successful probe exited 0. No native provider was involved.

Target and artifact context

Self-authored local files with distinct raw basename bytes and ASCII contents. The principal failure requires no symlink, ignore policy, malformed source contents, or concurrent filesystem change.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

Observed collision case:

{"actualBasenameHex":["626164efbfbd2e6a73","626164ff2e6a73"],"actualRawFileText":"RAW_FILE","reader":{"ok":true,"bytesRead":24,"entries":[{"status":"read","kind":"file","path":"bad�.js","size":12,"text":"UNICODE_FILE"},{"status":"read","kind":"file","path":"bad�.js","size":12,"text":"UNICODE_FILE"}]}}

The Unicode-only control returned one correct 12-byte file. The raw-only case returned zero bytes and a failed unknown-kind bad�.js entry. Evidence provenance: executed production reader, with raw-byte Node filesystem controls. Application import and public adapters were not executed; duplicate-path graph rejection is a source-derived consequence, not a captured result.

Typed outcome or recovery expected

Preserve each member's identity and bytes, or retain a truthful unsupported-name observation for the unrepresentable entry. Keep the valid Unicode filename readable. Never substitute its contents for the raw filename's entry.

Analysis or suspected cause

Source-derived: directory enumeration uses the default string decoding and passes child.name into processing. Path reconstruction then joins and accesses that decoded name. Metadata checks validate the replacement-character file after the original byte identity has already been lost.

Additional context

Bounded open/closed issue and PR searches found no matching filename-decoding report; the duplicate search was refreshed immediately before submission. PR #1311 concerns malformed source contents, not directory-entry names. PR #3562 concerns roots/symlink targets with parent components; this fixture has none. Last known good version: NOT_ENOUGH_INFO; no regression range was established.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A stable local fixture directly produces wrong-file observations and omitted bytes at the reference reader; observed harm is bounded evidence incorrectness, without demonstrated severe downstream consequences.
Workaround status: unverified
Workaround basis: Raw-byte filesystem access confirms the intended bytes, but no REA workflow preserving those member identities and restoring reader output was tested.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6653 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Executed at the production `readReferenceSource` filesystem boundary: two distinct valid POSIX byte filenames collapse to the same decoded pathname. The reader reports success and reads the replacement-character file twice, omitting the other file's bytes. The files are `626164ff2e6a73` (`bad<FF>.js`, containing `RAW_FILE`) and `626164efbfbd2e6a73` (`bad�.js`, containing `UNICODE_FILE`). Raw-byte `readdir` confirms both entries, and direct raw-path `readFile` returns `RAW_FILE`. The reader instead returns two `bad�.js` file entries containing `UNICODE_FILE`, with `bytesRead: 24`. ## Reproduction The successful reader probe ran once on Linux with the three directory cases below plus a supplementary symlink-target control. This reduced reproduction retains the three directory cases and uses unchanged production source. It requires the checkout's locked dependencies. Save this source import resolver as `/tmp/reference-source-loader.mjs`: ```javascript import { registerHooks } from 'node:module'; import { existsSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; registerHooks({ resolve(specifier, context, next) { try { return next(specifier, context); } catch (error) { if (error.code === 'ERR_MODULE_NOT_FOUND' && /^\.{1,2}\//.test(specifier) && specifier.endsWith('.js')) { const source = new URL(specifier.slice(0, -3) + '.ts', context.parentURL); if (existsSync(fileURLToPath(source))) return next(source.href, context); } throw error; } } }); ``` Save as `/tmp/reference-filenames.mjs`: ```javascript import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises'; import { join, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; const repo = resolve(process.argv[2]); const load = (name) => import(pathToFileURL(join(repo, 'src', name + '.ts')).href); const { readReferenceSource } = await load('reference/ReferenceSourceReader'); const { ArtifactResourceScope } = await load('artifacts/ArtifactResourceScope'); const base = await mkdtemp('/tmp/reference-filenames-'); const raw = Buffer.from([0x62, 0x61, 0x64, 0xff, 0x2e, 0x6a, 0x73]); const text = raw.toString('utf8'); const rawPath = (root) => Buffer.concat([Buffer.from(root + '/'), raw]); try { for (const name of ['unicode-control', 'raw-only', 'raw-and-replacement']) { const root = join(base, name); await mkdir(root); if (name !== 'unicode-control') await writeFile(rawPath(root), 'RAW_FILE'); if (name !== 'raw-only') await writeFile(join(root, text), 'UNICODE_FILE'); console.log(name, (await readdir(root, { encoding: 'buffer' })).map(x => x.toString('hex'))); if (name !== 'unicode-control') console.log('raw file:', (await readFile(rawPath(root))).toString()); const resources = new ArtifactResourceScope(); try { const result = await readReferenceSource(root, resources); if (!result.ok) throw new Error(JSON.stringify(result.error)); console.log(JSON.stringify({ bytesRead: result.value.bytesRead, entries: result.value.entries.map(e => e.status === 'read' && e.kind === 'file' ? { path: e.path, size: e.size, text: Buffer.from(e.bytes).toString() } : { status: e.status, kind: e.kind, path: e.path, code: e.code }) })); } finally { await resources.close(); } } } finally { await rm(base, { recursive: true, force: true }); } ``` From the checkout root: ```sh node --experimental-transform-types --import /tmp/reference-source-loader.mjs /tmp/reference-filenames.mjs "$PWD" ``` The import hook only maps unresolved relative `.js` source imports to their existing `.ts` siblings; Node transforms their TypeScript. No analyzed source is executed. No reader/importer probe was rerun after this draft was written. ## Expected behavior Retain distinct filesystem identities. If the graph's text path representation cannot encode a byte filename, report that limitation without accessing a different file under a lossy replacement name. Do not return another file's bytes as a successful observation of the original entry. The [inventory path contract](https://github.com/morluto/rea/blob/23def04096869b85c4d3c14a73ed09168483c9f6/src/domain/referenceSourcePathIdentity.ts#L12) states that reference inventories retain filesystem spelling. [Repository guidance](https://github.com/morluto/rea/blob/23def04096869b85c4d3c14a73ed09168483c9f6/AGENTS.md#L13) requires preserving selected paths and evidence, and keeping identity separate from display normalization. ## Impact and scope The observed reader invents a duplicate successful observation, omits `RAW_FILE`, and inflates bytes read from distinct members. Raw-only input instead produces an unknown-kind I/O entry under the substituted name; the Unicode-only control reads correctly. This is bounded wrong-file evidence on a Linux filesystem accepting non-UTF-8 names. No application graph, CLI/MCP result, source mutation, or downstream comparison was executed. Other platforms and real-world frequency are unknown. Renaming source entries would change the input identity; no result-preserving workaround was verified. ## REA version Package metadata 6.7.0; source revision `23def04096869b85c4d3c14a73ed09168483c9f6` with locked dependencies. Published tarball execution was not tested. ## Runtime and platform Node.js v24.22.1, Linux x64; source loading with `--experimental-transform-types` and the resolver above. The successful probe exited 0. No native provider was involved. ## Target and artifact context Self-authored local files with distinct raw basename bytes and ASCII contents. The principal failure requires no symlink, ignore policy, malformed source contents, or concurrent filesystem change. ## Interface Other ## Earliest failure stage Capture or observation ## Relevant output or evidence Observed collision case: ```json {"actualBasenameHex":["626164efbfbd2e6a73","626164ff2e6a73"],"actualRawFileText":"RAW_FILE","reader":{"ok":true,"bytesRead":24,"entries":[{"status":"read","kind":"file","path":"bad�.js","size":12,"text":"UNICODE_FILE"},{"status":"read","kind":"file","path":"bad�.js","size":12,"text":"UNICODE_FILE"}]}} ``` The Unicode-only control returned one correct 12-byte file. The raw-only case returned zero bytes and a failed unknown-kind `bad�.js` entry. Evidence provenance: executed production reader, with raw-byte Node filesystem controls. Application import and public adapters were not executed; duplicate-path graph rejection is a source-derived consequence, not a captured result. ## Typed outcome or recovery expected Preserve each member's identity and bytes, or retain a truthful unsupported-name observation for the unrepresentable entry. Keep the valid Unicode filename readable. Never substitute its contents for the raw filename's entry. ## Analysis or suspected cause Source-derived: [directory enumeration](https://github.com/morluto/rea/blob/23def04096869b85c4d3c14a73ed09168483c9f6/src/reference/ReferenceSourceReaderEntries.ts#L71) uses the default string decoding and passes `child.name` into processing. [Path reconstruction](https://github.com/morluto/rea/blob/23def04096869b85c4d3c14a73ed09168483c9f6/src/reference/ReferenceSourceReaderEntries.ts#L197) then joins and accesses that decoded name. Metadata checks validate the replacement-character file after the original byte identity has already been lost. ## Additional context Bounded open/closed issue and PR searches found no matching filename-decoding report; the duplicate search was refreshed immediately before submission. [PR #1311](https://github.com/morluto/rea/pull/1311) concerns malformed source contents, not directory-entry names. [PR #3562](https://github.com/morluto/rea/pull/3562) concerns roots/symlink targets with parent components; this fixture has none. Last known good version: NOT_ENOUGH_INFO; no regression range was established. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A stable local fixture directly produces wrong-file observations and omitted bytes at the reference reader; observed harm is bounded evidence incorrectness, without demonstrated severe downstream consequences. - Workaround status: unverified - Workaround basis: Raw-byte filesystem access confirms the intended bytes, but no REA workflow preserving those member identities and restoring reader output was tested.

bsakaguchi closed this issue

2026-10-31 00:46:12 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:06 +00:00

[PR #4032] [MERGED] fix(browser): preserve source-map media types #6004