morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7663] [Bug] A covering Markdown token window drops the missing-selection document #4587

New issue

Closed

opened 2026-10-31 00:51:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:51:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7663

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

For a successful inspection with an absent output selector, node scripts/rea.mjs inspect-byte-entropy "$file" --format md --filter-output nope emits an empty two-column Markdown table. Adding --token-count prints 13. Adding --token-offset 0 instead emits zero bytes, despite requesting the whole output without skipping any tokens. A covering --token-limit 1000 also emits zero bytes. Offset 5 prints an empty page with [truncated: showing tokens 0–0 of 0], contradicting the unwindowed document's count of 13.

Every invocation exits 0 with empty stderr. The discrepancy also occurred with a reduced valid Apple IPS report through inspect-apple-ips.

Reproduction

Use a built checkout at the revision below, its existing locked dependencies, and Node.js v24.22.1 on PATH. No dependency installation or rebuild was performed for this verification.
Run this inline script from the checkout root. It creates the seven-byte input, isolates HOME/XDG state, blocks package-manager commands, captures each public CLI call, and deletes the generated files afterward:

python3 -I <<'PY'
import json
import os
from pathlib import Path
import shutil
import subprocess
import tempfile

node = shutil.which('node')
with tempfile.TemporaryDirectory(prefix='rea-md-window-') as directory:
    work = Path(directory)
    env = dict(os.environ)
    for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME',
                'XDG_CACHE_HOME', 'XDG_STATE_HOME'):
        location = work / key.lower()
        location.mkdir()
        env[key] = str(location)
    tools = work / 'bin'
    tools.mkdir()
    for name in ('npx', 'npm', 'bun'):
        shim = tools / name
        shim.write_text('#!/bin/sh\nexit 127\n')
        shim.chmod(0o755)
    env['PATH'] = str(tools) + os.pathsep + env['PATH']
    fixture = work / 'target.bin'
    fixture.write_bytes(b'fixture')
    base = [node, 'scripts/rea.mjs', 'inspect-byte-entropy', str(fixture),
            '--format', 'md', '--filter-output', 'nope']
    for flags in ([], ['--token-count'], ['--token-offset', '0'],
                  ['--token-limit', '1000'], ['--token-offset', '5']):
        result = subprocess.run(base + flags, env=env, capture_output=True,
                                timeout=30)
        print(json.dumps({'flags': flags, 'exit': result.returncode,
                          'stdout_bytes': len(result.stdout),
                          'stdout': result.stdout.decode(),
                          'stderr': result.stderr.decode()}, ensure_ascii=False))
PY

Compare the ordinary output and its count with the three windows. The ordinary output is 32 bytes, its count is 13, both covering windows have zero stdout bytes, and offset 5 reports a total of zero.

The byte-entropy matrix reproduced in four independent rounds, including two runs of the exact script above. The five-case Apple IPS matrix reproduced once.

Expected behavior

Token counting and token windowing should describe the same selected emitted document. If the ordinary Markdown result is the empty-document table and its estimate is 13, an offset of zero without a limit, or a limit of 1000, should preserve that complete table. Offset 5 should select from that document and report a total of 13.

This expectation uses the documented emitted-document count and exhausted-window total contracts, the result owner's existing fallback for absent selections, and the fitting-window preservation behavior. It does not require exact model-token counts or a new error for an absent selector. Contract locators and their scope are recorded in the evidence below.

Impact and scope

P2: callers using supported Markdown output controls receive inconsistent document and token-total information for a missing field. The observed result is a 13-token document disappearing under a covering window, with another window reporting a total of zero. No downstream pager failure, model-budget incident, stored-Evidence change, target mutation, crash, or security boundary crossing was observed.

Removing the token-window controls is a demonstrated workaround: it returns the complete fallback table. Observed scope is successful missing-selection Markdown output for the two local inspection commands on macOS arm64. TOON controls count the absent selection as zero tokens and do not exhibit the 13-versus-zero discrepancy; JSON token windows are explicitly rejected. Other platforms and commands remain untested.

REA version

Exact node scripts/rea.mjs --version stdout:

6.8.0

Source baseline: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; Darwin 27.0.0, arm64. Existing checkout build and locked dependencies. No optional analysis provider, remote target, or external API was used.

Target and artifact context

A regular file containing the seven ASCII bytes fixture. The corroborating IPS fixture contained two consecutive JSON objects: {"bug_type":"309","app_name":"fixture"} and {"procName":"fixture","threads":[],"usedImages":[]}.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The reproduction captures these results. Flags are appended to inspect-byte-entropy <file> --format md --filter-output nope; every row exits 0 with empty stderr.

Added flags	Stdout	UTF-8 bytes
none	Two-line table shown below, with final newline	32
`--token-count`	`13` and newline	3
`--token-offset 0`	Empty	0
`--token-limit 1000`	Empty	0
`--token-offset 5`	Leading newline, marker below, final newline	40

Ordinary output:

| Key | Value |
|-----|-------|

Offset-5 marker:

[truncated: showing tokens 0–0 of 0]

The same five variants with inspect-apple-ips <report.ips> produced the same table, count, empty outputs and marker. That command also returned exit 0 with empty stderr in each case.

Contract and source checks at the baseline:

docs/cli.md:975-978 defines filtering over successful command data. docs/cli.md:1093-1101 defines counting over the selected emitted document and exhausted ranges at its total.
src/cli/cliResultOutput.ts:402-415 constructs an empty-object document for a selection without a JSON value. Its counting branch at :416-454 and ordinary Markdown emission at :523-550 use that document.
tests/boundary/cli/cliTokenWindowOutput.test.ts:70, :77, and :99 exercise fitting-window preservation and ownership for TOON. They are supporting text-window coverage, not direct tests of the missing-selection Markdown case. Merged PR #5814 explicitly includes TOON/Markdown text-window fidelity in its scope.
src/cli/cliResultOutput.ts:476-486 instead formats the raw filtered value for successful windows. src/cli/cliOutputFormatter.ts:148-152 formats an absent top-level value as empty text; src/cli/tokenWindow.ts:36-58 computes that text's total as zero.

The prescribed local source-test command was attempted through its direct script entry point, node scripts/run-development-tests.mjs local tests/boundary/cli/cliTokenWindowOutput.test.ts. It exited 1 with Use test:focused for built test projects: tests/boundary/cli/cliTokenWindowOutput.test.ts; no test assertions ran. Runtime evidence here comes from the public CLI matrices, not a passing test-suite claim.

Typed outcome or recovery expected

Preserve the ordinary selected Markdown representation for a covering text window and use that representation's token total. Omitting token-window controls currently recovers the complete fallback.

Analysis or suspected cause

Source-derived explanation: the ordinary renderer and count path use the constructed fallback document, while the successful-window branch formats the raw selection first. A missing selection therefore becomes an empty string only on the window path. The empty-string presentation guard delegates a fitting window back to Incur, which emits nothing; a nonzero offset formats an exhausted zero-token marker. This accounts for the observed discrepancy without requiring any change to the inspection result itself.

Additional context

Open and closed issues, issue comments, and open/closed/merged PRs were searched. Related #6447 concerns JSON counts differing from emitted JSON; #7271 and PR #7278 concern full-output TOON/YAML counts; #7581 concerns manifests and schema requests bypassing token controls. None has this Markdown window/fallback restoration as its reported outcome.

#5530 and merged PR #5814 address duplicated characters and Unicode boundaries; #5110 concerns exhausted failure ranges; #5532 and PR #5544 concern invalid numeric bounds. This input uses valid integer controls and an ordinary successful result.

Merged PR #2333 explicitly scoped selected-null preservation to streamed JSON/JSONL and left other formats unchanged. It is not used as an independent promise about Markdown token windows. The present report concerns a missing selection, not selected-null presentation.

NOT_ENOUGH_INFO: the first affected release, last known good revision, behavior on other platforms, and restored behavior after a fix were not established. No checkout file or hunter draft was edited, and nothing was published on GitHub.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Valid Markdown window requests replace the selected 13-token fallback with empty output and report a contradictory zero total. No severe downstream incident or stored-data damage was observed.
Workaround status: available
Workaround basis: Removing the token-window controls returned the complete 32-byte fallback table in every byte-entropy round and in the Apple IPS control.

Originally created by @coygeek on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7663 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior For a successful inspection with an absent output selector, `node scripts/rea.mjs inspect-byte-entropy "$file" --format md --filter-output nope` emits an empty two-column Markdown table. Adding `--token-count` prints `13`. Adding `--token-offset 0` instead emits zero bytes, despite requesting the whole output without skipping any tokens. A covering `--token-limit 1000` also emits zero bytes. Offset 5 prints an empty page with `[truncated: showing tokens 0–0 of 0]`, contradicting the unwindowed document's count of 13. Every invocation exits 0 with empty stderr. The discrepancy also occurred with a reduced valid Apple IPS report through `inspect-apple-ips`. ### Reproduction 1. Use a built checkout at the revision below, its existing locked dependencies, and Node.js v24.22.1 on PATH. No dependency installation or rebuild was performed for this verification. 2. Run this inline script from the checkout root. It creates the seven-byte input, isolates HOME/XDG state, blocks package-manager commands, captures each public CLI call, and deletes the generated files afterward: ```sh python3 -I <<'PY' import json import os from pathlib import Path import shutil import subprocess import tempfile node = shutil.which('node') with tempfile.TemporaryDirectory(prefix='rea-md-window-') as directory: work = Path(directory) env = dict(os.environ) for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME'): location = work / key.lower() location.mkdir() env[key] = str(location) tools = work / 'bin' tools.mkdir() for name in ('npx', 'npm', 'bun'): shim = tools / name shim.write_text('#!/bin/sh\nexit 127\n') shim.chmod(0o755) env['PATH'] = str(tools) + os.pathsep + env['PATH'] fixture = work / 'target.bin' fixture.write_bytes(b'fixture') base = [node, 'scripts/rea.mjs', 'inspect-byte-entropy', str(fixture), '--format', 'md', '--filter-output', 'nope'] for flags in ([], ['--token-count'], ['--token-offset', '0'], ['--token-limit', '1000'], ['--token-offset', '5']): result = subprocess.run(base + flags, env=env, capture_output=True, timeout=30) print(json.dumps({'flags': flags, 'exit': result.returncode, 'stdout_bytes': len(result.stdout), 'stdout': result.stdout.decode(), 'stderr': result.stderr.decode()}, ensure_ascii=False)) PY ``` 3. Compare the ordinary output and its count with the three windows. The ordinary output is 32 bytes, its count is 13, both covering windows have zero stdout bytes, and offset 5 reports a total of zero. The byte-entropy matrix reproduced in four independent rounds, including two runs of the exact script above. The five-case Apple IPS matrix reproduced once. ### Expected behavior Token counting and token windowing should describe the same selected emitted document. If the ordinary Markdown result is the empty-document table and its estimate is 13, an offset of zero without a limit, or a limit of 1000, should preserve that complete table. Offset 5 should select from that document and report a total of 13. This expectation uses the documented emitted-document count and exhausted-window total contracts, the result owner's existing fallback for absent selections, and the fitting-window preservation behavior. It does not require exact model-token counts or a new error for an absent selector. Contract locators and their scope are recorded in the evidence below. ### Impact and scope P2: callers using supported Markdown output controls receive inconsistent document and token-total information for a missing field. The observed result is a 13-token document disappearing under a covering window, with another window reporting a total of zero. No downstream pager failure, model-budget incident, stored-Evidence change, target mutation, crash, or security boundary crossing was observed. Removing the token-window controls is a demonstrated workaround: it returns the complete fallback table. Observed scope is successful missing-selection Markdown output for the two local inspection commands on macOS arm64. TOON controls count the absent selection as zero tokens and do not exhibit the 13-versus-zero discrepancy; JSON token windows are explicitly rejected. Other platforms and commands remain untested. ### REA version Exact `node scripts/rea.mjs --version` stdout: ```text 6.8.0 ``` Source baseline: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; Darwin 27.0.0, arm64. Existing checkout build and locked dependencies. No optional analysis provider, remote target, or external API was used. ### Target and artifact context A regular file containing the seven ASCII bytes `fixture`. The corroborating IPS fixture contained two consecutive JSON objects: `{"bug_type":"309","app_name":"fixture"}` and `{"procName":"fixture","threads":[],"usedImages":[]}`. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The reproduction captures these results. Flags are appended to `inspect-byte-entropy <file> --format md --filter-output nope`; every row exits 0 with empty stderr. | Added flags | Stdout | UTF-8 bytes | | --- | --- | ---: | | none | Two-line table shown below, with final newline | 32 | | `--token-count` | `13` and newline | 3 | | `--token-offset 0` | Empty | 0 | | `--token-limit 1000` | Empty | 0 | | `--token-offset 5` | Leading newline, marker below, final newline | 40 | Ordinary output: ```text | Key | Value | |-----|-------| ``` Offset-5 marker: ```text [truncated: showing tokens 0–0 of 0] ``` The same five variants with `inspect-apple-ips <report.ips>` produced the same table, count, empty outputs and marker. That command also returned exit 0 with empty stderr in each case. Contract and source checks at the baseline: - `docs/cli.md:975-978` defines filtering over successful command data. `docs/cli.md:1093-1101` defines counting over the selected emitted document and exhausted ranges at its total. - `src/cli/cliResultOutput.ts:402-415` constructs an empty-object document for a selection without a JSON value. Its counting branch at `:416-454` and ordinary Markdown emission at `:523-550` use that document. - `tests/boundary/cli/cliTokenWindowOutput.test.ts:70`, `:77`, and `:99` exercise fitting-window preservation and ownership for TOON. They are supporting text-window coverage, not direct tests of the missing-selection Markdown case. Merged [PR #5814](https://github.com/morluto/rea/pull/5814) explicitly includes TOON/Markdown text-window fidelity in its scope. - `src/cli/cliResultOutput.ts:476-486` instead formats the raw filtered value for successful windows. `src/cli/cliOutputFormatter.ts:148-152` formats an absent top-level value as empty text; `src/cli/tokenWindow.ts:36-58` computes that text's total as zero. The prescribed local source-test command was attempted through its direct script entry point, `node scripts/run-development-tests.mjs local tests/boundary/cli/cliTokenWindowOutput.test.ts`. It exited 1 with `Use test:focused for built test projects: tests/boundary/cli/cliTokenWindowOutput.test.ts`; no test assertions ran. Runtime evidence here comes from the public CLI matrices, not a passing test-suite claim. ### Typed outcome or recovery expected Preserve the ordinary selected Markdown representation for a covering text window and use that representation's token total. Omitting token-window controls currently recovers the complete fallback. ### Analysis or suspected cause Source-derived explanation: the ordinary renderer and count path use the constructed fallback document, while the successful-window branch formats the raw selection first. A missing selection therefore becomes an empty string only on the window path. The empty-string presentation guard delegates a fitting window back to Incur, which emits nothing; a nonzero offset formats an exhausted zero-token marker. This accounts for the observed discrepancy without requiring any change to the inspection result itself. ### Additional context Open and closed issues, issue comments, and open/closed/merged PRs were searched. Related [#6447](https://github.com/morluto/rea/issues/6447) concerns JSON counts differing from emitted JSON; [#7271](https://github.com/morluto/rea/issues/7271) and [PR #7278](https://github.com/morluto/rea/pull/7278) concern full-output TOON/YAML counts; [#7581](https://github.com/morluto/rea/issues/7581) concerns manifests and schema requests bypassing token controls. None has this Markdown window/fallback restoration as its reported outcome. [#5530](https://github.com/morluto/rea/issues/5530) and merged PR #5814 address duplicated characters and Unicode boundaries; [#5110](https://github.com/morluto/rea/issues/5110) concerns exhausted failure ranges; [#5532](https://github.com/morluto/rea/issues/5532) and [PR #5544](https://github.com/morluto/rea/pull/5544) concern invalid numeric bounds. This input uses valid integer controls and an ordinary successful result. Merged [PR #2333](https://github.com/morluto/rea/pull/2333) explicitly scoped selected-null preservation to streamed JSON/JSONL and left other formats unchanged. It is not used as an independent promise about Markdown token windows. The present report concerns a missing selection, not selected-null presentation. NOT_ENOUGH_INFO: the first affected release, last known good revision, behavior on other platforms, and restored behavior after a fix were not established. No checkout file or hunter draft was edited, and nothing was published on GitHub. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Valid Markdown window requests replace the selected 13-token fallback with empty output and report a contradictory zero total. No severe downstream incident or stored-data damage was observed. - Workaround status: available - Workaround basis: Removing the token-window controls returned the complete 32-byte fallback table in every byte-entropy round and in the Apple IPS control.

bsakaguchi closed this issue

2026-10-31 00:51:01 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:34 +00:00

[PR #4587] [CLOSED] fix(process): handle disappearing symlinks during filesystem snapshot (#4575) #6324