morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7592] [Bug] Successful keyed-archive text output substitutes lone surrogates with U+FFFD #4522

New issue

Closed

opened 2026-10-31 00:50:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:50:29 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7592

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

The built CLI accepts a binary keyed archive whose root scalar contains an unpaired UTF-16 code unit. For the same archive, node scripts/rea.mjs inspect-keyed-archive "$WORK/scalar.bplist" --format json preserves U+D800, while --format toon, the default format, and --format md emit U+FFFD in its place. Every command exits 0 with empty stderr and no encoding-loss diagnostic.

For the scalar "probe:\ud800:end", JSON and YAML decode back to that exact code-unit sequence. Decoding emitted TOON bytes returns "probe:�:end". Markdown also contains the replacement character in the scalar value cell. The alteration happens in the emitted bytes, without requiring a Markdown renderer.

Reproduction

Use the baseline below with its pinned Node 24.22.1, locked dependencies and built runtime available. Verification used the supplied built checkout; dependency installation and rebuilding were not part of this replay.

Set up an isolated environment from the checkout root:

export WORK="$(mktemp -d)"
export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config"
export XDG_CACHE_HOME="$WORK/cache" XDG_DATA_HOME="$WORK/data"
export XDG_STATE_HOME="$WORK/state" TMPDIR="$WORK/tmp"
mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_CACHE_HOME" "$XDG_DATA_HOME" "$XDG_STATE_HOME" "$TMPDIR"

Generate a small inert binary keyed archive using the locked plist dependency. The root references a scalar string; no class is instantiated:

node --input-type=module <<'JS'
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { buildBinary } from 'plist';

const archive = {
  $version: 100000,
  $archiver: 'NSKeyedArchiver',
  $top: { root: { UID: 1 } },
  $objects: ['$null', 'probe:\ud800:end'],
};
writeFileSync(join(process.env.WORK, 'scalar.bplist'), buildBinary(archive));
JS

Execute the supported CLI in each format and inspect the captured UTF-8 output:

node --input-type=module <<'JS'
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { join } from 'node:path';
import { decode } from '@toon-format/toon';
import { parse as parseYaml } from 'yaml';

const input = 'probe:\ud800:end';
for (const format of ['json', 'yaml', 'toon', 'md', 'default']) {
  const args = [
    'scripts/rea.mjs', 'inspect-keyed-archive',
    join(process.env.WORK, 'scalar.bplist'),
    ...(format === 'default' ? [] : ['--format', format]),
  ];
  const result = spawnSync(process.execPath, args, { env: process.env });
  assert.equal(result.status, 0, result.stderr.toString());
  assert.equal(result.stderr.length, 0);
  const text = result.stdout.toString('utf8');
  const document = format === 'json' ? JSON.parse(text)
    : format === 'yaml' ? parseYaml(text)
    : format === 'md' ? undefined : decode(text);
  console.log(JSON.stringify({
    format,
    exit: result.status,
    replacements: (text.match(/\ufffd/g) ?? []).length,
    ...(document ? {
      value: document.normalized_result.objects[1].value,
      equal: document.normalized_result.objects[1].value === input,
    } : { replacementCell: text.includes('probe:\ufffd:end') }),
  }));
}
JS

The reduced high-surrogate default-format case reproduced in four fresh processes, including a replay of these report commands. A separate matrix also reproduced the loss for a lone low surrogate; valid surrogate-pair and authored-U+FFFD controls retained their values.

Remove the generated fixtures and isolated environment:

rm -rf "$WORK"

Expected behavior

The repository's evidence-preservation invariant requires preserving source identity and explicitly identifying uncertainty when normalization loses information. Its keyed-archive contract retains raw serialized fields, and its Markdown contract promises literal source text.

An admitted scalar should retain an unambiguous representation of its source code units in successful output. If the input or selected text format cannot represent that value, report the limitation or refuse it explicitly. Returning success with an undisclosed substitution makes a source lone surrogate indistinguishable from an authored U+FFFD. This expectation does not assume that an unpaired surrogate is valid Unicode.

Impact and scope

P2: this is a bounded evidence-presentation defect with a directly verified workaround. Select --format json or --format yaml to retain the code-unit sequence. The source artifact was not modified.

Observed inputs include lone U+D800 and U+DC00 in scalar-root keyed archives, plus an archive with the same kind of unit in a class name. TOON and default output replace the unit; Markdown replaces directly displayed scalar cells. A valid U+1F600 pair survived in all tested formats. An actual U+FFFD scalar produced the same displayed value as the substituted cases.

These synthetic fixtures do not establish occurrence frequency in real archives. Other commands, platforms, terminal rendering and MCP were not tested.

REA version

Exact output of node scripts/rea.mjs --version:

6.8.0

Source baseline: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1, arm64. Locked dependencies: plist 5.0.0, incur 0.6.1, @toon-format/toon 2.3.1.

Target and artifact context

An inline-generated binary NSKeyedArchiver property list with two objects, a native UID root reference, and a scalar containing U+D800. Direct inspection confirmed that the binary fixture contains that raw UTF-16 code unit.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The reproduction prints these format-dependent results for the same scalar:

{"format":"json","exit":0,"replacements":0,"value":"probe:\ud800:end","equal":true}
{"format":"yaml","exit":0,"replacements":0,"value":"probe:\ud800:end","equal":true}
{"format":"toon","exit":0,"replacements":1,"value":"probe:�:end","equal":false}
{"format":"md","exit":0,"replacements":1,"replacementCell":true}
{"format":"default","exit":0,"replacements":1,"value":"probe:�:end","equal":false}

Contract sources at the baseline: AGENTS.md:13,19,33, docs/native-investigation.md:611-617, and docs/cli.md:954,965-970. The JSON equivalence cases in tests/boundary/cli/jsonOutput.test.ts:93-106 include a lone high surrogate.

For ordinary successful unfiltered TOON, src/cli/cliResultOutput.ts:357-367 delegates to Incur. scripts/rea.mjs:141-150,254-255 forwards the formatted string to stdout. Markdown scalar cells use String(value) in src/cli/cliOutputFormatter.ts:112-118; the escaping at :61-68 leaves lone surrogates intact, and src/cli/cliResultOutput.ts:525-544 writes the formatted text.

Analysis or suspected cause

An additional executed boundary check found that Incur's TOON encoding decodes back to the original code unit while it remains a JavaScript string. Converting that string through UTF-8 bytes changes the unit to U+FFFD. This supports a serialization-boundary explanation for TOON. The Markdown source path similarly passes the raw code unit to a text write.

Additional context

Duplicate search covered open and closed issues, open/closed/merged PRs, and issue comments with surrogate, Unicode, TOON, Markdown and format-specific queries.

The closest preview report, #6476, was closed by preserving valid scalar boundaries while explicitly retaining source-encoded lone surrogates. #5530 concerns token windows; this reproduction uses none. #5097 concerns Frida byte-budget accounting. #7432 and PR #7506 concern Markdown edge-space rendering, and #7271 concerns token counts. PR #6990 refuses malformed strings in the separate Apple IPS reader. None has the same input contract and closure criterion as this successful keyed-archive output substitution.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7592 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior The built CLI accepts a binary keyed archive whose root scalar contains an unpaired UTF-16 code unit. For the same archive, `node scripts/rea.mjs inspect-keyed-archive "$WORK/scalar.bplist" --format json` preserves U+D800, while `--format toon`, the default format, and `--format md` emit U+FFFD in its place. Every command exits 0 with empty stderr and no encoding-loss diagnostic. For the scalar `"probe:\ud800:end"`, JSON and YAML decode back to that exact code-unit sequence. Decoding emitted TOON bytes returns `"probe:�:end"`. Markdown also contains the replacement character in the scalar value cell. The alteration happens in the emitted bytes, without requiring a Markdown renderer. ### Reproduction Use the baseline below with its pinned Node 24.22.1, locked dependencies and built runtime available. Verification used the supplied built checkout; dependency installation and rebuilding were not part of this replay. 1. Set up an isolated environment from the checkout root: ~~~sh export WORK="$(mktemp -d)" export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" export XDG_CACHE_HOME="$WORK/cache" XDG_DATA_HOME="$WORK/data" export XDG_STATE_HOME="$WORK/state" TMPDIR="$WORK/tmp" mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_CACHE_HOME" "$XDG_DATA_HOME" "$XDG_STATE_HOME" "$TMPDIR" ~~~ 2. Generate a small inert binary keyed archive using the locked plist dependency. The root references a scalar string; no class is instantiated: ~~~sh node --input-type=module <<'JS' import { writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { buildBinary } from 'plist'; const archive = { $version: 100000, $archiver: 'NSKeyedArchiver', $top: { root: { UID: 1 } }, $objects: ['$null', 'probe:\ud800:end'], }; writeFileSync(join(process.env.WORK, 'scalar.bplist'), buildBinary(archive)); JS ~~~ 3. Execute the supported CLI in each format and inspect the captured UTF-8 output: ~~~sh node --input-type=module <<'JS' import assert from 'node:assert/strict'; import { spawnSync } from 'node:child_process'; import { join } from 'node:path'; import { decode } from '@toon-format/toon'; import { parse as parseYaml } from 'yaml'; const input = 'probe:\ud800:end'; for (const format of ['json', 'yaml', 'toon', 'md', 'default']) { const args = [ 'scripts/rea.mjs', 'inspect-keyed-archive', join(process.env.WORK, 'scalar.bplist'), ...(format === 'default' ? [] : ['--format', format]), ]; const result = spawnSync(process.execPath, args, { env: process.env }); assert.equal(result.status, 0, result.stderr.toString()); assert.equal(result.stderr.length, 0); const text = result.stdout.toString('utf8'); const document = format === 'json' ? JSON.parse(text) : format === 'yaml' ? parseYaml(text) : format === 'md' ? undefined : decode(text); console.log(JSON.stringify({ format, exit: result.status, replacements: (text.match(/\ufffd/g) ?? []).length, ...(document ? { value: document.normalized_result.objects[1].value, equal: document.normalized_result.objects[1].value === input, } : { replacementCell: text.includes('probe:\ufffd:end') }), })); } JS ~~~ The reduced high-surrogate default-format case reproduced in four fresh processes, including a replay of these report commands. A separate matrix also reproduced the loss for a lone low surrogate; valid surrogate-pair and authored-U+FFFD controls retained their values. 4. Remove the generated fixtures and isolated environment: ~~~sh rm -rf "$WORK" ~~~ ### Expected behavior The repository's evidence-preservation invariant requires preserving source identity and explicitly identifying uncertainty when normalization loses information. Its keyed-archive contract retains raw serialized fields, and its Markdown contract promises literal source text. An admitted scalar should retain an unambiguous representation of its source code units in successful output. If the input or selected text format cannot represent that value, report the limitation or refuse it explicitly. Returning success with an undisclosed substitution makes a source lone surrogate indistinguishable from an authored U+FFFD. This expectation does not assume that an unpaired surrogate is valid Unicode. ### Impact and scope P2: this is a bounded evidence-presentation defect with a directly verified workaround. Select `--format json` or `--format yaml` to retain the code-unit sequence. The source artifact was not modified. Observed inputs include lone U+D800 and U+DC00 in scalar-root keyed archives, plus an archive with the same kind of unit in a class name. TOON and default output replace the unit; Markdown replaces directly displayed scalar cells. A valid U+1F600 pair survived in all tested formats. An actual U+FFFD scalar produced the same displayed value as the substituted cases. These synthetic fixtures do not establish occurrence frequency in real archives. Other commands, platforms, terminal rendering and MCP were not tested. ### REA version Exact output of `node scripts/rea.mjs --version`: ~~~text 6.8.0 ~~~ Source baseline: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; macOS 27.0.1, arm64. Locked dependencies: plist 5.0.0, incur 0.6.1, @toon-format/toon 2.3.1. ### Target and artifact context An inline-generated binary NSKeyedArchiver property list with two objects, a native UID root reference, and a scalar containing U+D800. Direct inspection confirmed that the binary fixture contains that raw UTF-16 code unit. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The reproduction prints these format-dependent results for the same scalar: ~~~text {"format":"json","exit":0,"replacements":0,"value":"probe:\ud800:end","equal":true} {"format":"yaml","exit":0,"replacements":0,"value":"probe:\ud800:end","equal":true} {"format":"toon","exit":0,"replacements":1,"value":"probe:�:end","equal":false} {"format":"md","exit":0,"replacements":1,"replacementCell":true} {"format":"default","exit":0,"replacements":1,"value":"probe:�:end","equal":false} ~~~ Contract sources at the baseline: `AGENTS.md:13,19,33`, `docs/native-investigation.md:611-617`, and `docs/cli.md:954,965-970`. The JSON equivalence cases in `tests/boundary/cli/jsonOutput.test.ts:93-106` include a lone high surrogate. For ordinary successful unfiltered TOON, `src/cli/cliResultOutput.ts:357-367` delegates to Incur. `scripts/rea.mjs:141-150,254-255` forwards the formatted string to stdout. Markdown scalar cells use `String(value)` in `src/cli/cliOutputFormatter.ts:112-118`; the escaping at `:61-68` leaves lone surrogates intact, and `src/cli/cliResultOutput.ts:525-544` writes the formatted text. ### Analysis or suspected cause An additional executed boundary check found that Incur's TOON encoding decodes back to the original code unit while it remains a JavaScript string. Converting that string through UTF-8 bytes changes the unit to U+FFFD. This supports a serialization-boundary explanation for TOON. The Markdown source path similarly passes the raw code unit to a text write. ### Additional context Duplicate search covered open and closed issues, open/closed/merged PRs, and issue comments with surrogate, Unicode, TOON, Markdown and format-specific queries. The closest preview report, [#6476](https://github.com/morluto/rea/issues/6476), was closed by preserving valid scalar boundaries while explicitly retaining source-encoded lone surrogates. [#5530](https://github.com/morluto/rea/issues/5530) concerns token windows; this reproduction uses none. [#5097](https://github.com/morluto/rea/issues/5097) concerns Frida byte-budget accounting. [#7432](https://github.com/morluto/rea/issues/7432) and [PR #7506](https://github.com/morluto/rea/pull/7506) concern Markdown edge-space rendering, and [#7271](https://github.com/morluto/rea/issues/7271) concerns token counts. [PR #6990](https://github.com/morluto/rea/pull/6990) refuses malformed strings in the separate Apple IPS reader. None has the same input contract and closure criterion as this successful keyed-archive output substitution.

bsakaguchi closed this issue

2026-10-31 00:50:29 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4522

No description provided.

Rows
Columns