[GH-ISSUE #7593] [Bug] Rendered Markdown replaces archive-name NUL characters with U+FFFD #4524

New issue

Closed

opened 2026-10-31 00:50:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:50:29 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7593

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

node scripts/rea.mjs inspect-keyed-archive "$WORK/nul.bplist" --format md exits 0 and emits a class-name cell containing a raw U+0000. Rendering the result with the checkout's locked VitePress renderer changes My\x00Class to My�Class. A second archive containing an actual U+FFFD produces the same rendered class name. The paired JSON output preserves the NUL, and decoding paired TOON output also preserves it.

Reproduction

Use the built baseline checkout below with its locked dependencies and Node 24.22.1 available as node. The supplied verification checkout was already built; a fresh checkout needs npm ci && npm run build:cached first.
From the repository root, run this inline fixture and public-CLI comparison. It isolates REA state and deletes its generated files afterward:

export WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config"
export XDG_DATA_HOME="$WORK/data" XDG_CACHE_HOME="$WORK/cache"
export XDG_STATE_HOME="$WORK/state"
mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_DATA_HOME" "$XDG_CACHE_HOME" "$XDG_STATE_HOME"
node --input-type=module <<'JS'
import { writeFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { buildBinary } from 'plist';
import { decode } from '@toon-format/toon';
import { createMarkdownRenderer, disposeMdItInstance } from 'vitepress';
import { parseFragment } from 'parse5';
const renderer = await createMarkdownRenderer(process.env.WORK, { linkify: false });
const text = n => n.nodeName === '#text' ? n.value : (n.childNodes ?? []).map(text).join('');
const renderedNames = [];
for (const [label, name] of [['nul', 'My\x00Class'], ['replacement', 'My\ufffdClass']]) {
  const path = `${process.env.WORK}/${label}.bplist`;
  writeFileSync(path, buildBinary({
    $version: 100000, $archiver: 'NSKeyedArchiver', $top: { root: { UID: 1 } },
    $objects: ['$null', { $class: { UID: 2 } },
      { $classname: name, $classes: [name, 'NSObject'] }, name, [name, 'NSObject'], 'NSObject'],
  }));
  const out = {};
  for (const format of ['json', 'toon', 'md']) {
    const result = spawnSync(process.execPath,
      ['scripts/rea.mjs', 'inspect-keyed-archive', path, '--format', format],
      { encoding: 'utf8', timeout: 30000 });
    if (result.status !== 0) throw new Error(result.stderr || String(result.error));
    out[format] = result.stdout;
  }
  const rows = [];
  (function visit(n) {
    if (n.tagName === 'tr') rows.push((n.childNodes ?? []).filter(c => c.tagName === 'td').map(text));
    else (n.childNodes ?? []).forEach(visit);
  })(parseFragment(renderer.render(out.md)));
  const rendered = rows.find(r => r[0] === '1')[3];
  renderedNames.push(rendered);
  const jsonName = JSON.parse(out.json).normalized_result.objects[1].class_name;
  const toonName = decode(out.toon).normalized_result.objects[1].class_name;
  console.log(JSON.stringify({ label, jsonName, toonName, rendered,
    markdownNuls: [...out.md].filter(c => c === '\x00').length,
    markdownReplacementCharacters: [...out.md].filter(c => c === '\ufffd').length }));
  if (jsonName !== name || toonName !== name) throw new Error('control lost source text');
}
console.log('rendered names collide:', renderedNames[0] === renderedNames[1]);
disposeMdItInstance();
JS

The NUL failure reproduced in both the initial independent comparison and a replay of this reduced reproduction. All six commands in the reduced reproduction exited 0.

Expected behavior

A rendered name containing a NUL should remain distinguishable from a name containing U+FFFD. If the rendered text format cannot display the source character literally, its representation should identify the original code unit rather than silently substitute another character. Formatting should preserve the selected result's identity; JSON should retain its original value.

Impact and scope

P2: a successful rendered report collapses two distinct class identifiers to the same displayed text for the tested binary archives. This is bounded to the rendered Markdown projection; the executed JSON command preserves the exact name and provides a practical workaround. TOON also preserves this NUL input. The raw Markdown retains the NUL, so loss of stored Evidence or input data is not established. Headers, headings, other renderers, other platforms, and occurrence in real-world archives were not tested.

REA version

Exact node scripts/rea.mjs --version output: 6.8.0.
Source commit: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1; arm64. Used the checkout's locked plist, TOON, VitePress and parse5 dependencies. No Hopper, Ghidra, IDA or GUI application was used.

Target and artifact context

Two inline-generated binary Foundation keyed archives with one archived object: one class name is My\x00Class, and the control class name is My\ufffdClass. The object table also contains those strings as scalar values. Python plistlib independently decoded the first binary fixture and retained its NUL-bearing class name.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The reduced public-CLI/renderer comparison prints:

{"label":"nul","jsonName":"My\x00Class","toonName":"My\x00Class","rendered":"My�Class","markdownNuls":2,"markdownReplacementCharacters":0}
{"label":"replacement","jsonName":"My�Class","toonName":"My�Class","rendered":"My�Class","markdownNuls":0,"markdownReplacementCharacters":5}
rendered names collide: true

Note: in the real output the NUL is the JSON escape for U+0000 (backslash, u, four zeros); it is shown as \x00 here because GitHub rewrites that literal six-character text into ^@ in issue bodies.

The initial byte inspection independently counted two raw NUL bytes and zero U+FFFD characters in the NUL fixture's Markdown output. Its JSON output had zero NUL bytes and five NUL escapes. Thus the NUL is present in the emitted Markdown and changes during rendering.

Contract sources at the stated baseline:

docs/cli.md:965-973 promises literal source text in table headers/cells and describes formatting as presentation.
docs/native-investigation.md:611-620 documents binary keyed-archive inspection, including preserved class descriptors and unknown classes.
tests/boundary/cli/markdownLiteralCells.test.ts:33-105 already verifies public CLI literal cell text with locked VitePress and parse5, the same rendering oracle used here.

Source-derived path: src/cli/artifactCommands.ts:29-71 registers the public inspection command. src/cli/cliOutputFormatter.ts:61-68 escapes table syntax and line breaks but leaves U+0000 unchanged; src/cli/cliResultOutput.ts:525-544 formats and writes successful Markdown. These locators explain reachability; the executed comparison establishes the failure.

Typed outcome or recovery expected

The same rendering comparison should keep NUL-bearing names distinguishable from actual U+FFFD names, while JSON and TOON retain their exact source strings. A visible code-unit escape is one possible representation. This is a proposed restoration check; no fix was implemented.

Analysis or suspected cause

The shared Markdown escaping pass permits U+0000 in ordinary cell text. The executed renderer replaces that character while parsing the Markdown. Nested JSON strings are already escaped and do not have the same raw-NUL representation. The likely correction belongs at Markdown text representation rather than the archive decoder, which the JSON control and independent plist decoder both show preserving the source name.

Additional context

Duplicate search covered open and closed issues, open and merged PRs, and issue comments. Closed issue #7432 and merged PR #7506 concern edge-space trimming; the PR encodes spaces at cell edges and does not address NUL. Issue #7325 concerns brace-delimited provider objects disappearing through VitePress attributes. Closed issue #6755 concerns table pipes. Their closure checks differ from retaining a distinguishable representation of U+0000.

This also differs from unpaired-surrogate output replacement: an independently executed surrogate control already contained U+FFFD in its emitted TOON/Markdown UTF-8 bytes. Here NUL survives the byte write, TOON round-trips it, and the failure first appears during Markdown rendering. The first affected release and last known good version are unknown.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Two distinct class identifiers render as the same text despite successful public CLI completion in the paired synthetic archive comparison.
Workaround status: available
Workaround basis: The paired JSON command retained the exact NUL-bearing class name; decoding the paired TOON output also retained it.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7593 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior `node scripts/rea.mjs inspect-keyed-archive "$WORK/nul.bplist" --format md` exits 0 and emits a class-name cell containing a raw U+0000. Rendering the result with the checkout's locked VitePress renderer changes `My\x00Class` to `My�Class`. A second archive containing an actual U+FFFD produces the same rendered class name. The paired JSON output preserves the NUL, and decoding paired TOON output also preserves it. ### Reproduction 1. Use the built baseline checkout below with its locked dependencies and Node 24.22.1 available as `node`. The supplied verification checkout was already built; a fresh checkout needs `npm ci && npm run build:cached` first. 2. From the repository root, run this inline fixture and public-CLI comparison. It isolates REA state and deletes its generated files afterward: ```sh export WORK="$(mktemp -d)" trap 'rm -rf "$WORK"' EXIT export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" export XDG_DATA_HOME="$WORK/data" XDG_CACHE_HOME="$WORK/cache" export XDG_STATE_HOME="$WORK/state" mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_DATA_HOME" "$XDG_CACHE_HOME" "$XDG_STATE_HOME" node --input-type=module <<'JS' import { writeFileSync } from 'node:fs'; import { spawnSync } from 'node:child_process'; import { buildBinary } from 'plist'; import { decode } from '@toon-format/toon'; import { createMarkdownRenderer, disposeMdItInstance } from 'vitepress'; import { parseFragment } from 'parse5'; const renderer = await createMarkdownRenderer(process.env.WORK, { linkify: false }); const text = n => n.nodeName === '#text' ? n.value : (n.childNodes ?? []).map(text).join(''); const renderedNames = []; for (const [label, name] of [['nul', 'My\x00Class'], ['replacement', 'My\ufffdClass']]) { const path = `${process.env.WORK}/${label}.bplist`; writeFileSync(path, buildBinary({ $version: 100000, $archiver: 'NSKeyedArchiver', $top: { root: { UID: 1 } }, $objects: ['$null', { $class: { UID: 2 } }, { $classname: name, $classes: [name, 'NSObject'] }, name, [name, 'NSObject'], 'NSObject'], })); const out = {}; for (const format of ['json', 'toon', 'md']) { const result = spawnSync(process.execPath, ['scripts/rea.mjs', 'inspect-keyed-archive', path, '--format', format], { encoding: 'utf8', timeout: 30000 }); if (result.status !== 0) throw new Error(result.stderr || String(result.error)); out[format] = result.stdout; } const rows = []; (function visit(n) { if (n.tagName === 'tr') rows.push((n.childNodes ?? []).filter(c => c.tagName === 'td').map(text)); else (n.childNodes ?? []).forEach(visit); })(parseFragment(renderer.render(out.md))); const rendered = rows.find(r => r[0] === '1')[3]; renderedNames.push(rendered); const jsonName = JSON.parse(out.json).normalized_result.objects[1].class_name; const toonName = decode(out.toon).normalized_result.objects[1].class_name; console.log(JSON.stringify({ label, jsonName, toonName, rendered, markdownNuls: [...out.md].filter(c => c === '\x00').length, markdownReplacementCharacters: [...out.md].filter(c => c === '\ufffd').length })); if (jsonName !== name || toonName !== name) throw new Error('control lost source text'); } console.log('rendered names collide:', renderedNames[0] === renderedNames[1]); disposeMdItInstance(); JS ``` The NUL failure reproduced in both the initial independent comparison and a replay of this reduced reproduction. All six commands in the reduced reproduction exited 0. ### Expected behavior A rendered name containing a NUL should remain distinguishable from a name containing U+FFFD. If the rendered text format cannot display the source character literally, its representation should identify the original code unit rather than silently substitute another character. Formatting should preserve the selected result's identity; JSON should retain its original value. ### Impact and scope P2: a successful rendered report collapses two distinct class identifiers to the same displayed text for the tested binary archives. This is bounded to the rendered Markdown projection; the executed JSON command preserves the exact name and provides a practical workaround. TOON also preserves this NUL input. The raw Markdown retains the NUL, so loss of stored Evidence or input data is not established. Headers, headings, other renderers, other platforms, and occurrence in real-world archives were not tested. ### REA version Exact `node scripts/rea.mjs --version` output: `6.8.0`. Source commit: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; macOS 27.0.1; arm64. Used the checkout's locked plist, TOON, VitePress and parse5 dependencies. No Hopper, Ghidra, IDA or GUI application was used. ### Target and artifact context Two inline-generated binary Foundation keyed archives with one archived object: one class name is `My\x00Class`, and the control class name is `My\ufffdClass`. The object table also contains those strings as scalar values. Python plistlib independently decoded the first binary fixture and retained its NUL-bearing class name. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The reduced public-CLI/renderer comparison prints: ```text {"label":"nul","jsonName":"My\x00Class","toonName":"My\x00Class","rendered":"My�Class","markdownNuls":2,"markdownReplacementCharacters":0} {"label":"replacement","jsonName":"My�Class","toonName":"My�Class","rendered":"My�Class","markdownNuls":0,"markdownReplacementCharacters":5} rendered names collide: true ``` Note: in the real output the NUL is the JSON escape for U+0000 (backslash, `u`, four zeros); it is shown as `\x00` here because GitHub rewrites that literal six-character text into `^@` in issue bodies. The initial byte inspection independently counted two raw NUL bytes and zero U+FFFD characters in the NUL fixture's Markdown output. Its JSON output had zero NUL bytes and five NUL escapes. Thus the NUL is present in the emitted Markdown and changes during rendering. Contract sources at the stated baseline: - `docs/cli.md:965-973` promises literal source text in table headers/cells and describes formatting as presentation. - `docs/native-investigation.md:611-620` documents binary keyed-archive inspection, including preserved class descriptors and unknown classes. - `tests/boundary/cli/markdownLiteralCells.test.ts:33-105` already verifies public CLI literal cell text with locked VitePress and parse5, the same rendering oracle used here. Source-derived path: `src/cli/artifactCommands.ts:29-71` registers the public inspection command. `src/cli/cliOutputFormatter.ts:61-68` escapes table syntax and line breaks but leaves U+0000 unchanged; `src/cli/cliResultOutput.ts:525-544` formats and writes successful Markdown. These locators explain reachability; the executed comparison establishes the failure. ### Typed outcome or recovery expected The same rendering comparison should keep NUL-bearing names distinguishable from actual U+FFFD names, while JSON and TOON retain their exact source strings. A visible code-unit escape is one possible representation. This is a proposed restoration check; no fix was implemented. ### Analysis or suspected cause The shared Markdown escaping pass permits U+0000 in ordinary cell text. The executed renderer replaces that character while parsing the Markdown. Nested JSON strings are already escaped and do not have the same raw-NUL representation. The likely correction belongs at Markdown text representation rather than the archive decoder, which the JSON control and independent plist decoder both show preserving the source name. ### Additional context Duplicate search covered open and closed issues, open and merged PRs, and issue comments. Closed [issue #7432](https://github.com/morluto/rea/issues/7432) and merged [PR #7506](https://github.com/morluto/rea/pull/7506) concern edge-space trimming; the PR encodes spaces at cell edges and does not address NUL. [Issue #7325](https://github.com/morluto/rea/issues/7325) concerns brace-delimited provider objects disappearing through VitePress attributes. Closed [issue #6755](https://github.com/morluto/rea/issues/6755) concerns table pipes. Their closure checks differ from retaining a distinguishable representation of U+0000. This also differs from unpaired-surrogate output replacement: an independently executed surrogate control already contained U+FFFD in its emitted TOON/Markdown UTF-8 bytes. Here NUL survives the byte write, TOON round-trips it, and the failure first appears during Markdown rendering. The first affected release and last known good version are unknown. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Two distinct class identifiers render as the same text despite successful public CLI completion in the paired synthetic archive comparison. - Workaround status: available - Workaround basis: The paired JSON command retained the exact NUL-bearing class name; decoding the paired TOON output also retained it.

bsakaguchi closed this issue

2026-10-31 00:50:29 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:17 +00:00

[PR #4524] [MERGED] fix(browser): reject invalid UTF-8 in HTTP CDP discovery #6285