morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5530] [Bug] CLI token windows duplicate digest characters and corrupt emoji #3366

New issue

Closed

opened 2026-10-31 00:39:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:39:35 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5530

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

Valid --token-limit/--token-offset windows silently change successful TOON output from inspect-plist. Joining consecutive filtered digest pages inserts a duplicate character, turning a 64-character SHA-256 into 65 characters. A filtered emoji page introduces U+FFFD although its complete output contains no replacement character. All six commands in the reproduction exited 0 with empty stderr.

Reproduction

From a built checkout of the revision below, run this independently executed six-call reproduction. It creates a local XML plist and isolated HOME/XDG directories, then removes them. Output filters make the token positions independent of absolute fixture paths.

python3 - <<'PY'
import json
import os
import pathlib
import plistlib
import re
import subprocess
import tempfile

with tempfile.TemporaryDirectory(prefix='token-window-') as directory:
    work = pathlib.Path(directory)
    env = dict(os.environ)
    for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME'):
        location = work / key.lower()
        location.mkdir()
        env[key] = str(location)
    fixture = work / 'paging.plist'
    fixture.write_bytes(plistlib.dumps({'Long': 'abcdefghij' * 300, 'Emoji': '😀é' * 300}))

    def run(name, *flags):
        result = subprocess.run(['node', 'scripts/rea.mjs', 'inspect-plist', str(fixture), *flags], env=env, capture_output=True)
        (work / (name + '.stdout')).write_bytes(result.stdout)
        (work / (name + '.stderr')).write_bytes(result.stderr)
        assert result.returncode == 0, result.stderr.decode()
        assert result.stderr == b''
        return result.stdout

    def body(page):
        return re.sub(rb'\n\[truncated: showing tokens [^\n]+\]\n$', b'', page)

    digest_flags = ('--format', 'toon', '--filter-output', 'subject.digest.sha256')
    full = run('digest-full', *digest_flags)
    first = run('digest-first', *digest_flags, '--token-limit', '10', '--token-offset', '0')
    second = run('digest-second', *digest_flags, '--token-limit', '10', '--token-offset', '10')
    joined = body(first) + body(second)
    digest = lambda text: re.search(rb'sha256: ([0-9a-f]+)', text).group(1).decode()
    print('full digest:', digest(full))
    print('paged digest:', digest(joined))
    print('digest lengths:', len(digest(full)), len(digest(joined)))
    print('first page is source prefix:', full.startswith(body(first)))
    print('joined digest bodies match:', full.rstrip(b'\n') == joined)

    emoji_flags = ('--format', 'toon', '--filter-output', 'normalized_result.value.Emoji')
    emoji_full = run('emoji-full', *emoji_flags)
    emoji_page = run('emoji-page', *emoji_flags, '--token-limit', '7', '--token-offset', '21')
    print('replacement present (full, page):', b'\xef\xbf\xbd' in emoji_full, b'\xef\xbf\xbd' in emoji_page)
    print(emoji_page.decode(), end='')
    complete = json.loads(run('complete-json', '--format', 'json'))
    print('unwindowed JSON preserves fixture:', complete['normalized_result']['value'] == {'Long': 'abcdefghij' * 300, 'Emoji': '😀é' * 300})
PY

Expected behavior

The advertised token controls limit output and skip its first tokens. Consecutive windows should preserve the selected characters when reassembled after removing truncation markers, and a window should not introduce U+FFFD by dividing valid Unicode. This concerns text fidelity, not parseability of each page or exact model-token counts.

Contract references: AGENTS.md:13 requires preserving output, digests and metadata; inspect-plist --help advertises both token controls; docs/cli.md:437-443 documents TOON and output controls.

Impact and scope

Callers paging evidence can receive incorrect identifiers or metadata text. The independently reproduced scope is successful TOON output with the valid windows above. Unwindowed JSON preserved both fixture values and the digest, providing a practical workaround; unwindowed filtered output also retains the original text. No input-file or stored-evidence mutation was demonstrated. Markdown and failure-envelope behavior were not reproduced, and broader frequency is unknown.

REA version

6.7.0; source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8; locked Incur 0.6.1 and tokenx 1.3.0.

Runtime and platform

Node.js v26.12.0; macOS, darwin arm64. No optional analysis provider or external API was used. NOT_ENOUGH_INFO: the verifier did not record the exact macOS release.

Target and artifact context

Synthetic local XML property list containing ASCII text and repeated emoji/accent pairs, generated by the reproduction above.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The independently executed script printed:

full digest: 6764c190e641f1c59c638f9bf32fee05f66992811c5605ebd25056156f21e221
paged digest: 6764c190e641f1c59cc638f9bf32fee05f66992811c5605ebd25056156f21e221
digest lengths: 64 65
first page is source prefix: True
joined digest bodies match: False
replacement present (full, page): False True
😀é😀é😀é😀é😀é😀é😀é�
[truncated: showing tokens 21–28 of 309]
unwindowed JSON preserves fixture: True

The first digest page ends in ...1c59c; the next starts with c638..., duplicating c. The full emoji output has no U+FFFD, but its 21–28 page does. The script also checks a successful unwindowed JSON round trip.

Typed outcome or recovery expected

Return text windows whose consecutive bodies preserve the original digest and valid Unicode. Unwindowed JSON is an observed recovery path.

Analysis or suspected cause

Source-derived: src/cliOutput.ts:182-190 accepts TOON windows, and src/cli/cliResultOutput.ts:244-245 delegates successful non-JSON rendering to Incur. Bundled node_modules/incur/dist/Cli.js:1047-1061 applies sliceByTokens from tokenx. In node_modules/tokenx/dist/index.mjs:123-132, proportional UTF-16 ranges floor the start and ceil the end. Adjacent ranges can overlap, and a boundary can divide a surrogate pair. This explains the observed duplicated digest character and replacement character. The fix choice is left to REA's maintainers.

The shared helper also appears in REA's full-failure formatter at src/cli/cliResultOutput.ts:353-372; that scope is source-derived and was not reproduced here.

Additional context

Prepared with AI assistance and independently reproduced through the built CLI. Latest published REA release remains 6.7.0. A subsequent source check through main commit c374c2fe7290b914d73d9e27af38b498753ec593 found no fix to successful TOON token slicing or the locked dependencies; the later CLI change concerns typed failures through filters. The runtime reproduction reported here is on the stated baseline, not a rerun on that later main commit.

Related #5110 and PR #5119 concern reversed ranges for exhausted failure windows, rather than text corruption inside valid successful-result windows. PR #4807 explicitly leaves token-window rendering with Incur. NOT_ENOUGH_INFO: first affected release and last known good version.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5530 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior Valid `--token-limit`/`--token-offset` windows silently change successful TOON output from `inspect-plist`. Joining consecutive filtered digest pages inserts a duplicate character, turning a 64-character SHA-256 into 65 characters. A filtered emoji page introduces U+FFFD although its complete output contains no replacement character. All six commands in the reproduction exited 0 with empty stderr. ### Reproduction From a built checkout of the revision below, run this independently executed six-call reproduction. It creates a local XML plist and isolated HOME/XDG directories, then removes them. Output filters make the token positions independent of absolute fixture paths. ```sh python3 - <<'PY' import json import os import pathlib import plistlib import re import subprocess import tempfile with tempfile.TemporaryDirectory(prefix='token-window-') as directory: work = pathlib.Path(directory) env = dict(os.environ) for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME'): location = work / key.lower() location.mkdir() env[key] = str(location) fixture = work / 'paging.plist' fixture.write_bytes(plistlib.dumps({'Long': 'abcdefghij' * 300, 'Emoji': '😀é' * 300})) def run(name, *flags): result = subprocess.run(['node', 'scripts/rea.mjs', 'inspect-plist', str(fixture), *flags], env=env, capture_output=True) (work / (name + '.stdout')).write_bytes(result.stdout) (work / (name + '.stderr')).write_bytes(result.stderr) assert result.returncode == 0, result.stderr.decode() assert result.stderr == b'' return result.stdout def body(page): return re.sub(rb'\n\[truncated: showing tokens [^\n]+\]\n$', b'', page) digest_flags = ('--format', 'toon', '--filter-output', 'subject.digest.sha256') full = run('digest-full', *digest_flags) first = run('digest-first', *digest_flags, '--token-limit', '10', '--token-offset', '0') second = run('digest-second', *digest_flags, '--token-limit', '10', '--token-offset', '10') joined = body(first) + body(second) digest = lambda text: re.search(rb'sha256: ([0-9a-f]+)', text).group(1).decode() print('full digest:', digest(full)) print('paged digest:', digest(joined)) print('digest lengths:', len(digest(full)), len(digest(joined))) print('first page is source prefix:', full.startswith(body(first))) print('joined digest bodies match:', full.rstrip(b'\n') == joined) emoji_flags = ('--format', 'toon', '--filter-output', 'normalized_result.value.Emoji') emoji_full = run('emoji-full', *emoji_flags) emoji_page = run('emoji-page', *emoji_flags, '--token-limit', '7', '--token-offset', '21') print('replacement present (full, page):', b'\xef\xbf\xbd' in emoji_full, b'\xef\xbf\xbd' in emoji_page) print(emoji_page.decode(), end='') complete = json.loads(run('complete-json', '--format', 'json')) print('unwindowed JSON preserves fixture:', complete['normalized_result']['value'] == {'Long': 'abcdefghij' * 300, 'Emoji': '😀é' * 300}) PY ``` ### Expected behavior The advertised token controls limit output and skip its first tokens. Consecutive windows should preserve the selected characters when reassembled after removing truncation markers, and a window should not introduce U+FFFD by dividing valid Unicode. This concerns text fidelity, not parseability of each page or exact model-token counts. Contract references: `AGENTS.md:13` requires preserving output, digests and metadata; `inspect-plist --help` advertises both token controls; `docs/cli.md:437-443` documents TOON and output controls. ### Impact and scope Callers paging evidence can receive incorrect identifiers or metadata text. The independently reproduced scope is successful TOON output with the valid windows above. Unwindowed JSON preserved both fixture values and the digest, providing a practical workaround; unwindowed filtered output also retains the original text. No input-file or stored-evidence mutation was demonstrated. Markdown and failure-envelope behavior were not reproduced, and broader frequency is unknown. ### REA version 6.7.0; source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`; locked Incur 0.6.1 and tokenx 1.3.0. ### Runtime and platform Node.js v26.12.0; macOS, darwin arm64. No optional analysis provider or external API was used. NOT_ENOUGH_INFO: the verifier did not record the exact macOS release. ### Target and artifact context Synthetic local XML property list containing ASCII text and repeated emoji/accent pairs, generated by the reproduction above. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The independently executed script printed: ```text full digest: 6764c190e641f1c59c638f9bf32fee05f66992811c5605ebd25056156f21e221 paged digest: 6764c190e641f1c59cc638f9bf32fee05f66992811c5605ebd25056156f21e221 digest lengths: 64 65 first page is source prefix: True joined digest bodies match: False replacement present (full, page): False True 😀é😀é😀é😀é😀é😀é😀é� [truncated: showing tokens 21–28 of 309] unwindowed JSON preserves fixture: True ``` The first digest page ends in `...1c59c`; the next starts with `c638...`, duplicating `c`. The full emoji output has no U+FFFD, but its 21–28 page does. The script also checks a successful unwindowed JSON round trip. ### Typed outcome or recovery expected Return text windows whose consecutive bodies preserve the original digest and valid Unicode. Unwindowed JSON is an observed recovery path. ### Analysis or suspected cause Source-derived: `src/cliOutput.ts:182-190` accepts TOON windows, and `src/cli/cliResultOutput.ts:244-245` delegates successful non-JSON rendering to Incur. Bundled `node_modules/incur/dist/Cli.js:1047-1061` applies `sliceByTokens` from tokenx. In `node_modules/tokenx/dist/index.mjs:123-132`, proportional UTF-16 ranges floor the start and ceil the end. Adjacent ranges can overlap, and a boundary can divide a surrogate pair. This explains the observed duplicated digest character and replacement character. The fix choice is left to REA's maintainers. The shared helper also appears in REA's full-failure formatter at `src/cli/cliResultOutput.ts:353-372`; that scope is source-derived and was not reproduced here. ### Additional context Prepared with AI assistance and independently reproduced through the built CLI. Latest published REA release remains 6.7.0. A subsequent source check through main commit `c374c2fe7290b914d73d9e27af38b498753ec593` found no fix to successful TOON token slicing or the locked dependencies; the later CLI change concerns typed failures through filters. The runtime reproduction reported here is on the stated baseline, not a rerun on that later main commit. Related #5110 and PR #5119 concern reversed ranges for exhausted failure windows, rather than text corruption inside valid successful-result windows. PR #4807 explicitly leaves token-window rendering with Incur. NOT_ENOUGH_INFO: first affected release and last known good version.

bsakaguchi closed this issue

2026-10-31 00:39:35 +00:00

bsakaguchi commented

2026-10-31 00:39:36 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with canonical issue #5110 ([Bug] Exhausted CLI failure token windows display a reversed range) and siblings #5448, #5531, and #5532. Addresses CLI token window slicing anomalies, unicode/emoji corruption, and boundary duplication. Cataloged under Cluster AQ in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with canonical issue #5110 (`[Bug] Exhausted CLI failure token windows display a reversed range`) and siblings #5448, #5531, and #5532. Addresses CLI token window slicing anomalies, unicode/emoji corruption, and boundary duplication. Cataloged under Cluster AQ in consolidation tracker #4330.

bsakaguchi referenced this issue

2026-10-31 01:09:30 +00:00

[PR #3516] [MERGED] fix(managed): report ILONLY-clear assemblies as mixed in boundary summaries #5796

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3366

No description provided.

Rows
Columns