morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7690] [Bug] JSONL token counting replaces parser diagnostics with numeric results #4614

New issue

Closed

opened 2026-10-31 00:51:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:51:21 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7690

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

node scripts/rea.mjs inspect-artifact --format jsonl --token-count --zzz-bogus exits 1 and emits a JSONL error record with result:18. Its diagnostic code and message are absent, and stderr is empty. Removing --token-count returns the original UNKNOWN diagnostic identifying the unsupported flag.

Omitting the required path has the same problem: node scripts/rea.mjs inspect-artifact --format jsonl --token-count emits an error record with result:85. Without counting, the record contains VALIDATION_ERROR, argument guidance and fieldErrors naming path. Adding --full-output to the counted unsupported-flag case still returns result:18.

A missing-file control reaches an application failure instead of parser rejection. With the same JSONL/counting options, it retains its complete target_unavailable diagnostic object. All failing cases exited 1 with empty stderr in both independent rounds.

Reproduction

Start at the built source baseline below, using its locked dependencies and Node.js v24.22.1 available as node. No target fixture or provider is needed for the parser cases. The missing-file control deliberately names an absent path.
Run this reduced replay from the checkout root. It isolates HOME/XDG state, places refusing package-manager fixtures first on PATH, captures each invocation and removes generated directories afterward:

python3 -I <<'PY'
import json
import os
from pathlib import Path
import shutil
import subprocess
import tempfile

node = shutil.which('node')
with tempfile.TemporaryDirectory(prefix='rea-parser-count-') as directory:
    work = Path(directory)
    env = {k: v for k, v in os.environ.items()
           if not k.startswith(('REA_', 'HOPPER_', 'GHIDRA_', 'XDG_'))}
    for key, name in [('HOME', 'home'), ('XDG_CONFIG_HOME', 'config'),
                      ('XDG_DATA_HOME', 'data'), ('XDG_CACHE_HOME', 'cache'),
                      ('XDG_STATE_HOME', 'state')]:
        path = work / name
        path.mkdir()
        env[key] = str(path)
    tools = work / 'bin'
    tools.mkdir()
    for name in ('npx', 'npm', 'bun'):
        shim = tools / name
        shim.write_text('#!/bin/sh\nprintf "fixture package runner: disabled\\n" >&2\nexit 97\n')
        shim.chmod(0o755)
    env['PATH'] = str(tools) + os.pathsep + env['PATH']
    env['npm_config_prefix'] = str(work / 'npm-global')
    cases = [
        ['inspect-artifact', '--format', 'jsonl', '--token-count', '--zzz-bogus'],
        ['inspect-artifact', '--format', 'jsonl', '--zzz-bogus'],
        ['inspect-artifact', '--format', 'jsonl', '--token-count'],
        ['inspect-artifact', '--format', 'jsonl'],
        ['inspect-artifact', str(work / 'missing-artifact'),
         '--format', 'jsonl', '--token-count'],
        ['inspect-artifact', '--json', '--token-count', '--zzz-bogus'],
        ['inspect-artifact', '--format', 'jsonl', '--token-count',
         '--full-output', '--zzz-bogus'],
        ['inspect-artifact', '--format', 'jsonl', '--full-output', '--zzz-bogus'],
    ]
    for args in cases:
        result = subprocess.run([node, 'scripts/rea.mjs', *args], env=env,
                                capture_output=True, text=True, timeout=60)
        value = json.loads(result.stdout)
        payload = value.get('result') if isinstance(value, dict) else value
        print(json.dumps({'args': args, 'exit': result.returncode,
                          'result_type': type(payload).__name__,
                          'output': value, 'stderr': result.stderr}))
PY

Compare the counted parser failures with their uncounted controls. Each case was executed in two initial independent rounds and once more by replaying the embedded script above. All three attempts reproduced the corresponding result. The script runs each case once.

Expected behavior

JSONL error records should retain their original diagnostic payload when token counting is enabled, including errors produced by argument parsing. The output should preserve UNKNOWN with its unsupported-flag message, or VALIDATION_ERROR with its argument details, rather than replace that payload with a number. A full-output request should retain its error envelope.

The JSONL contract expressly retains original diagnostics and includes SDK validation/sentinel failures, whose failure status comes from the SDK. It also preserves errors under counting. The exact contract locators are listed in Evidence below.

Impact and scope

P2: callers requesting a token estimate lose the only diagnostic explaining these rejected inputs. The JSONL error framing and nonzero exit status survive, but callers cannot inspect the missing error code, message or argument details. This reproduced in both rounds on the stated baseline.

Omitting --token-count is a demonstrated workaround that restores those diagnostics. No data loss, successful-analysis corruption or downstream consumer failure was observed. Other commands, platforms, sentinel failures under counting, output filters and token windows were not exercised.

REA version

Exact node scripts/rea.mjs --version stdout:

6.8.0

Source baseline: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; Darwin, arm64. Existing locked Incur dependency: 0.6.1. No dependency installation, runtime rebuild or external analysis provider was used.

Target and artifact context

The unsupported-flag and missing-required-path cases fail before any target is opened. The application control uses a nonexistent file within the isolated invocation directory.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Captured public CLI stdout, each with exit 1 and empty stderr:

$ node scripts/rea.mjs inspect-artifact --format jsonl --token-count --zzz-bogus
{"$rea":"jsonl/v1","type":"error","ok":false,"result":18}

$ node scripts/rea.mjs inspect-artifact --format jsonl --zzz-bogus
{"$rea":"jsonl/v1","type":"error","ok":false,"result":{"code":"UNKNOWN","message":"Unknown flag: --zzz-bogus"}}

$ node scripts/rea.mjs inspect-artifact --format jsonl --token-count
{"$rea":"jsonl/v1","type":"error","ok":false,"result":85}

The uncounted missing-path control returns an object with code:VALIDATION_ERROR, the message REA could not read the command arguments. Check the named argument and its expected type, then try again., and a fieldErrors entry with path:path, expected:string, and missing:true.

The counted missing-file control returns an object with code:target_unavailable, category:unavailable, message, retryable:false, remediation.action and details.path/reason. These are selected fields from the captured response; its generated absolute path is omitted here. The counted full-output parser case remains exactly the first error record above; its uncounted control has an object containing ok:false, error.code:UNKNOWN, error.message and meta.command/duration.

node scripts/rea.mjs inspect-artifact --json --token-count --zzz-bogus prints only 18 and exits 1. This is a control for the framework counting path; the diagnostic-retention defect reported here is the JSONL contract violation.

Expected-source locators at the baseline: docs/cli.md:128-130 requires retaining the original diagnostic payload and explicitly addresses SDK validation/sentinel errors; docs/cli.md:136-137 retains errors with token counting. tests/acceptance/jsonLinesCli.test.ts:24-35 and tests/boundary/cli/jsonLinesOutput.test.ts:486-518 cover SDK validation records without counting. Classified-failure counting coverage is at tests/boundary/cli/jsonLinesOutput.test.ts:369-399.

Typed outcome or recovery expected

Preserve the parser diagnostic as the error record's object-valued result, including its code, message and available argument correction fields. Preserve the requested failure envelope when full output is enabled. The input should continue to fail with a nonzero exit status.

Analysis or suspected cause

Source-derived explanation: the pinned framework counts failed parser output before REA frames it. Incur's command parser handles failures before calling the registered command callback (node_modules/incur/src/internal/command.ts:80-108,234-255). Its CLI error path invokes the writer (node_modules/incur/src/Cli.ts:1964-2004), whose token-count branch counts output.error before normal error formatting (:1653-1657). REA's SDK adapter then parses that rendered count and wraps it as a JSONL error using the SDK exit status (src/cli/cliResultOutput.ts:233-265, wired by scripts/rea.mjs:133-190).

These parser errors bypass the ordinary command-result wrapper (src/cli/cliResultOutput.ts:80-112). The classified-failure inference in src/cliLogging.ts:72-90 therefore does not explain the parser loss. The application control instead reaches the error-record branch in src/cli/jsonLinesOutput.ts:46-75, which precedes counting.

REA owns the JSONL diagnostic-retention guarantee and SDK adaptation. Incur supplies the count, but its general counting policy does not satisfy that stronger REA contract. No patch or instrumented internal execution trace is claimed.

Additional context

Open and closed issues, all PR states and related comments were searched. #7558 loses a committed Evidence receipt after a JSONL selection refusal; #7581 ignores token controls for built-in documents; #7663 loses a successful Markdown missing-selection document under token windows; #7441 concerns parser JSON indentation. Their inputs and closure checks differ from retaining parser diagnostics under JSONL counting.

Closed #6447 and open #7271/PR #7278 concern estimates of successful documents. Closed #5531 concerns malformed output-flag values selecting the wrong format; its closure addresses that separate path. Merged PR #5629 introduced SDK error framing, but the retained-payload failure with counting remains reproducible at this baseline. No exact duplicate was found. NOT_ENOUGH_INFO: first affected release and last known good revision were not established.

Originally created by @coygeek on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7690 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior `node scripts/rea.mjs inspect-artifact --format jsonl --token-count --zzz-bogus` exits 1 and emits a JSONL error record with `result:18`. Its diagnostic code and message are absent, and stderr is empty. Removing `--token-count` returns the original `UNKNOWN` diagnostic identifying the unsupported flag. Omitting the required path has the same problem: `node scripts/rea.mjs inspect-artifact --format jsonl --token-count` emits an error record with `result:85`. Without counting, the record contains `VALIDATION_ERROR`, argument guidance and `fieldErrors` naming `path`. Adding `--full-output` to the counted unsupported-flag case still returns `result:18`. A missing-file control reaches an application failure instead of parser rejection. With the same JSONL/counting options, it retains its complete `target_unavailable` diagnostic object. All failing cases exited 1 with empty stderr in both independent rounds. ### Reproduction 1. Start at the built source baseline below, using its locked dependencies and Node.js v24.22.1 available as `node`. No target fixture or provider is needed for the parser cases. The missing-file control deliberately names an absent path. 2. Run this reduced replay from the checkout root. It isolates HOME/XDG state, places refusing package-manager fixtures first on PATH, captures each invocation and removes generated directories afterward: ```sh python3 -I <<'PY' import json import os from pathlib import Path import shutil import subprocess import tempfile node = shutil.which('node') with tempfile.TemporaryDirectory(prefix='rea-parser-count-') as directory: work = Path(directory) env = {k: v for k, v in os.environ.items() if not k.startswith(('REA_', 'HOPPER_', 'GHIDRA_', 'XDG_'))} for key, name in [('HOME', 'home'), ('XDG_CONFIG_HOME', 'config'), ('XDG_DATA_HOME', 'data'), ('XDG_CACHE_HOME', 'cache'), ('XDG_STATE_HOME', 'state')]: path = work / name path.mkdir() env[key] = str(path) tools = work / 'bin' tools.mkdir() for name in ('npx', 'npm', 'bun'): shim = tools / name shim.write_text('#!/bin/sh\nprintf "fixture package runner: disabled\\n" >&2\nexit 97\n') shim.chmod(0o755) env['PATH'] = str(tools) + os.pathsep + env['PATH'] env['npm_config_prefix'] = str(work / 'npm-global') cases = [ ['inspect-artifact', '--format', 'jsonl', '--token-count', '--zzz-bogus'], ['inspect-artifact', '--format', 'jsonl', '--zzz-bogus'], ['inspect-artifact', '--format', 'jsonl', '--token-count'], ['inspect-artifact', '--format', 'jsonl'], ['inspect-artifact', str(work / 'missing-artifact'), '--format', 'jsonl', '--token-count'], ['inspect-artifact', '--json', '--token-count', '--zzz-bogus'], ['inspect-artifact', '--format', 'jsonl', '--token-count', '--full-output', '--zzz-bogus'], ['inspect-artifact', '--format', 'jsonl', '--full-output', '--zzz-bogus'], ] for args in cases: result = subprocess.run([node, 'scripts/rea.mjs', *args], env=env, capture_output=True, text=True, timeout=60) value = json.loads(result.stdout) payload = value.get('result') if isinstance(value, dict) else value print(json.dumps({'args': args, 'exit': result.returncode, 'result_type': type(payload).__name__, 'output': value, 'stderr': result.stderr})) PY ``` 3. Compare the counted parser failures with their uncounted controls. Each case was executed in two initial independent rounds and once more by replaying the embedded script above. All three attempts reproduced the corresponding result. The script runs each case once. ### Expected behavior JSONL error records should retain their original diagnostic payload when token counting is enabled, including errors produced by argument parsing. The output should preserve `UNKNOWN` with its unsupported-flag message, or `VALIDATION_ERROR` with its argument details, rather than replace that payload with a number. A full-output request should retain its error envelope. The JSONL contract expressly retains original diagnostics and includes SDK validation/sentinel failures, whose failure status comes from the SDK. It also preserves errors under counting. The exact contract locators are listed in Evidence below. ### Impact and scope P2: callers requesting a token estimate lose the only diagnostic explaining these rejected inputs. The JSONL error framing and nonzero exit status survive, but callers cannot inspect the missing error code, message or argument details. This reproduced in both rounds on the stated baseline. Omitting `--token-count` is a demonstrated workaround that restores those diagnostics. No data loss, successful-analysis corruption or downstream consumer failure was observed. Other commands, platforms, sentinel failures under counting, output filters and token windows were not exercised. ### REA version Exact `node scripts/rea.mjs --version` stdout: ```text 6.8.0 ``` Source baseline: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js `v24.22.1`; Darwin, arm64. Existing locked Incur dependency: `0.6.1`. No dependency installation, runtime rebuild or external analysis provider was used. ### Target and artifact context The unsupported-flag and missing-required-path cases fail before any target is opened. The application control uses a nonexistent file within the isolated invocation directory. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Captured public CLI stdout, each with exit 1 and empty stderr: ```text $ node scripts/rea.mjs inspect-artifact --format jsonl --token-count --zzz-bogus {"$rea":"jsonl/v1","type":"error","ok":false,"result":18} $ node scripts/rea.mjs inspect-artifact --format jsonl --zzz-bogus {"$rea":"jsonl/v1","type":"error","ok":false,"result":{"code":"UNKNOWN","message":"Unknown flag: --zzz-bogus"}} $ node scripts/rea.mjs inspect-artifact --format jsonl --token-count {"$rea":"jsonl/v1","type":"error","ok":false,"result":85} ``` The uncounted missing-path control returns an object with `code:VALIDATION_ERROR`, the message `REA could not read the command arguments. Check the named argument and its expected type, then try again.`, and a `fieldErrors` entry with `path:path`, `expected:string`, and `missing:true`. The counted missing-file control returns an object with `code:target_unavailable`, `category:unavailable`, `message`, `retryable:false`, `remediation.action` and `details.path/reason`. These are selected fields from the captured response; its generated absolute path is omitted here. The counted full-output parser case remains exactly the first error record above; its uncounted control has an object containing `ok:false`, `error.code:UNKNOWN`, `error.message` and `meta.command/duration`. `node scripts/rea.mjs inspect-artifact --json --token-count --zzz-bogus` prints only `18` and exits 1. This is a control for the framework counting path; the diagnostic-retention defect reported here is the JSONL contract violation. Expected-source locators at the baseline: `docs/cli.md:128-130` requires retaining the original diagnostic payload and explicitly addresses SDK validation/sentinel errors; `docs/cli.md:136-137` retains errors with token counting. `tests/acceptance/jsonLinesCli.test.ts:24-35` and `tests/boundary/cli/jsonLinesOutput.test.ts:486-518` cover SDK validation records without counting. Classified-failure counting coverage is at `tests/boundary/cli/jsonLinesOutput.test.ts:369-399`. ### Typed outcome or recovery expected Preserve the parser diagnostic as the error record's object-valued `result`, including its code, message and available argument correction fields. Preserve the requested failure envelope when full output is enabled. The input should continue to fail with a nonzero exit status. ### Analysis or suspected cause Source-derived explanation: the pinned framework counts failed parser output before REA frames it. Incur's command parser handles failures before calling the registered command callback (`node_modules/incur/src/internal/command.ts:80-108,234-255`). Its CLI error path invokes the writer (`node_modules/incur/src/Cli.ts:1964-2004`), whose token-count branch counts `output.error` before normal error formatting (`:1653-1657`). REA's SDK adapter then parses that rendered count and wraps it as a JSONL error using the SDK exit status (`src/cli/cliResultOutput.ts:233-265`, wired by `scripts/rea.mjs:133-190`). These parser errors bypass the ordinary command-result wrapper (`src/cli/cliResultOutput.ts:80-112`). The classified-failure inference in `src/cliLogging.ts:72-90` therefore does not explain the parser loss. The application control instead reaches the error-record branch in `src/cli/jsonLinesOutput.ts:46-75`, which precedes counting. REA owns the JSONL diagnostic-retention guarantee and SDK adaptation. Incur supplies the count, but its general counting policy does not satisfy that stronger REA contract. No patch or instrumented internal execution trace is claimed. ### Additional context Open and closed issues, all PR states and related comments were searched. [#7558](https://github.com/morluto/rea/issues/7558) loses a committed Evidence receipt after a JSONL selection refusal; [#7581](https://github.com/morluto/rea/issues/7581) ignores token controls for built-in documents; [#7663](https://github.com/morluto/rea/issues/7663) loses a successful Markdown missing-selection document under token windows; [#7441](https://github.com/morluto/rea/issues/7441) concerns parser JSON indentation. Their inputs and closure checks differ from retaining parser diagnostics under JSONL counting. Closed [#6447](https://github.com/morluto/rea/issues/6447) and open [#7271](https://github.com/morluto/rea/issues/7271)/[PR #7278](https://github.com/morluto/rea/pull/7278) concern estimates of successful documents. Closed [#5531](https://github.com/morluto/rea/issues/5531) concerns malformed output-flag values selecting the wrong format; its closure addresses that separate path. Merged [PR #5629](https://github.com/morluto/rea/pull/5629) introduced SDK error framing, but the retained-payload failure with counting remains reproducible at this baseline. No exact duplicate was found. NOT_ENOUGH_INFO: first affected release and last known good revision were not established.

bsakaguchi closed this issue

2026-10-31 00:51:22 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4614

No description provided.

Rows
Columns