[GH-ISSUE #6447] [Bug] CLI --token-count measures a different document from emitted JSON #3860

New issue

Closed

opened 2026-10-31 00:44:42 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:44:42 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6447

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

providers --json --token-count reports an estimate for indented JSON even though the corresponding piped command emits compact JSON. A missing-field selection reports 0 although its output is {}. Adding --full-output also leaves the envelope out of the counted value.

These are comparisons using REA's own locked tokenx estimator, rather than a claim about exact tokens for a particular model.

Reproduction

From a fresh checkout at the source revision below, install and build with npm ci && npm run build:cached. Then run this from the checkout root. The workload is the built-in provider inventory; no target fixture or external service is needed. The script creates an isolated HOME inline and captures stdout through a pipe.

WORK=$(mktemp -d)
export WORK
mkdir -p "$WORK/home"
HOME="$WORK/home" node --input-type=module <<'NODE'
import { spawnSync } from 'node:child_process';
import { estimateTokenCount } from 'tokenx';
import { Formatter } from 'incur';

const cases = [
  ['json', ['--json']],
  ['missing', ['--json', '--filter-output', 'nope']],
  ['toon control', []],
  ['full json', ['--json', '--full-output']],
];
for (const [name, flags] of cases) {
  const run = (extra) => spawnSync(process.execPath,
    ['scripts/rea.mjs', 'providers', ...flags, ...extra],
    { encoding: 'utf8', env: process.env, maxBuffer: 1000000 });
  const output = run([]);
  const count = run(['--token-count']);
  if (output.status !== 0 || count.status !== 0)
    throw new Error(output.stderr + count.stderr);
  console.log(name, {
    reported: Number(count.stdout.trim()),
    outputEstimate: estimateTokenCount(output.stdout),
    ...(name === 'json' ? {
      indentedEstimate: estimateTokenCount(
        Formatter.format(JSON.parse(output.stdout), 'json')),
    } : {}),
    ...(name === 'missing' ? { document: output.stdout } : {}),
  });
}
NODE

The CLI comparisons reproduced in both of two rounds with the already built checkout and installed locked dependencies. The install/build commands above are preparation for a fresh checkout, not additional verification performed for these observations.

Expected behavior

providers --help describes --token-count as “Print token count of output (instead of output).” The count should use the same selected document and destination-specific formatting that the invocation would emit without that flag, including an empty-selection document or requested envelope.

The CLI output contract specifies compact JSON for pipes/redirection and envelope selection with --full-output. The JSON writer implements that destination-dependent formatting.

Impact and scope

P2: callers using the option for output budgeting get an estimate for a different representation, including zero for a nonempty JSON document. Observed scope is successful providers output with the flags above. Capturing the actual output and applying tokenx directly is a demonstrated workaround. No command-result loss, security impact, or downstream budgeting failure was observed. Other commands and failure results were not tested.

REA version

6.7.0 (source 9eb829aca3)

Runtime and platform

Node.js v26.12.1; macOS 27.0.1; arm64. No Hopper or Ghidra provider was used.

Target and artifact context

Built-in provider inventory with an isolated HOME; no binary target or external artifact.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

All compared CLI invocations exited 0 with empty stderr. Observed estimates in two rounds:

Flags before adding --token-count	Reported	Estimate of actual stdout
--json	10647 both times	10362 both times
--json --filter-output nope	0 both times	1 both times (`{}` plus newline)
no flags (TOON control)	9849 both times	9849 both times
--json --full-output	10647 both times	10382 / 10383

Reformatting the captured plain JSON with Formatter.format(value, 'json') estimated 10647 both times, matching the reported count. Provider inventory values depend on the environment; the mismatch is the assertion, rather than these specific totals. Full-output metadata varies between calls.

Typed outcome or recovery expected

Return an estimate for the emitted document. Capturing the output and estimating that string recovers the correct representation today.

Analysis or suspected cause

Source-derived: the token-count branch formats failure ?? filtered directly and returns. It precedes document construction, which adds the envelope and substitutes {} for an undefined selection. It also bypasses the JSON writer's terminal-versus-pipe formatting choice. These source paths explain the observed differences.

Additional context

Related reports address different output failures: #912 concerns absent JSON for an empty selection, #4938 concerns omitted array entries rejected during serialization, and #1371 concerns string-capacity crashes. None of the related reports examined describes these successful-call count discrepancies.

NOT_ENOUGH_INFO: first affected release and last known good version were not established.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6447 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior `providers --json --token-count` reports an estimate for indented JSON even though the corresponding piped command emits compact JSON. A missing-field selection reports 0 although its output is `{}`. Adding `--full-output` also leaves the envelope out of the counted value. These are comparisons using REA's own locked tokenx estimator, rather than a claim about exact tokens for a particular model. ### Reproduction From a fresh checkout at the source revision below, install and build with `npm ci && npm run build:cached`. Then run this from the checkout root. The workload is the built-in provider inventory; no target fixture or external service is needed. The script creates an isolated HOME inline and captures stdout through a pipe. ```sh WORK=$(mktemp -d) export WORK mkdir -p "$WORK/home" HOME="$WORK/home" node --input-type=module <<'NODE' import { spawnSync } from 'node:child_process'; import { estimateTokenCount } from 'tokenx'; import { Formatter } from 'incur'; const cases = [ ['json', ['--json']], ['missing', ['--json', '--filter-output', 'nope']], ['toon control', []], ['full json', ['--json', '--full-output']], ]; for (const [name, flags] of cases) { const run = (extra) => spawnSync(process.execPath, ['scripts/rea.mjs', 'providers', ...flags, ...extra], { encoding: 'utf8', env: process.env, maxBuffer: 1000000 }); const output = run([]); const count = run(['--token-count']); if (output.status !== 0 || count.status !== 0) throw new Error(output.stderr + count.stderr); console.log(name, { reported: Number(count.stdout.trim()), outputEstimate: estimateTokenCount(output.stdout), ...(name === 'json' ? { indentedEstimate: estimateTokenCount( Formatter.format(JSON.parse(output.stdout), 'json')), } : {}), ...(name === 'missing' ? { document: output.stdout } : {}), }); } NODE ``` The CLI comparisons reproduced in both of two rounds with the already built checkout and installed locked dependencies. The install/build commands above are preparation for a fresh checkout, not additional verification performed for these observations. ### Expected behavior `providers --help` describes `--token-count` as “Print token count of output (instead of output).” The count should use the same selected document and destination-specific formatting that the invocation would emit without that flag, including an empty-selection document or requested envelope. The [CLI output contract](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/cli.md#L543-L554) specifies compact JSON for pipes/redirection and envelope selection with `--full-output`. The [JSON writer](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/cli/jsonOutput.ts#L14-L26) implements that destination-dependent formatting. ### Impact and scope P2: callers using the option for output budgeting get an estimate for a different representation, including zero for a nonempty JSON document. Observed scope is successful `providers` output with the flags above. Capturing the actual output and applying tokenx directly is a demonstrated workaround. No command-result loss, security impact, or downstream budgeting failure was observed. Other commands and failure results were not tested. ### REA version 6.7.0 (source 9eb829aca34be36ff112826337996f6139f82eff) ### Runtime and platform Node.js v26.12.1; macOS 27.0.1; arm64. No Hopper or Ghidra provider was used. ### Target and artifact context Built-in provider inventory with an isolated HOME; no binary target or external artifact. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence All compared CLI invocations exited 0 with empty stderr. Observed estimates in two rounds: | Flags before adding --token-count | Reported | Estimate of actual stdout | | --- | --- | --- | | --json | 10647 both times | 10362 both times | | --json --filter-output nope | 0 both times | 1 both times (`{}` plus newline) | | no flags (TOON control) | 9849 both times | 9849 both times | | --json --full-output | 10647 both times | 10382 / 10383 | Reformatting the captured plain JSON with `Formatter.format(value, 'json')` estimated 10647 both times, matching the reported count. Provider inventory values depend on the environment; the mismatch is the assertion, rather than these specific totals. Full-output metadata varies between calls. ### Typed outcome or recovery expected Return an estimate for the emitted document. Capturing the output and estimating that string recovers the correct representation today. ### Analysis or suspected cause Source-derived: the [token-count branch](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/cli/cliResultOutput.ts#L256-L283) formats `failure ?? filtered` directly and returns. It precedes [document construction](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/cli/cliResultOutput.ts#L305-L331), which adds the envelope and substitutes `{}` for an undefined selection. It also bypasses the JSON writer's terminal-versus-pipe formatting choice. These source paths explain the observed differences. ### Additional context Related reports address different output failures: #912 concerns absent JSON for an empty selection, #4938 concerns omitted array entries rejected during serialization, and #1371 concerns string-capacity crashes. None of the related reports examined describes these successful-call count discrepancies. NOT_ENOUGH_INFO: first affected release and last known good version were not established.

bsakaguchi closed this issue

2026-10-31 00:44:43 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:42 +00:00

[PR #3559] [CLOSED] fix(electron): block and record shell.trashItem #5823

bsakaguchi referenced this issue

2026-10-31 01:10:42 +00:00

[PR #3860] [MERGED] fix(javascript): reject Webpack getters replaced through any identity write #5955