morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7271] [Bug] evidence-import --token-count --full-output undercounts TOON and YAML #4316

New issue

Open

opened 2026-10-31 00:48:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:48:51 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7271

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

The registered evidence-import command reports 13 with --format toon --full-output --token-count, while the document emitted without the counting flag estimates to 31 using the same locked tokenx library. YAML behaves the same way. This was executed through the source CLI registry and the executable's output-routing policy; the packaged executable was not built or run.

Reproduction

Use an existing installation of the checkout's locked dependencies. For a fresh checkout, npm ci is preparation; it was not rerun for this verification. From the checkout root, run the source CLI replay below. It creates an isolated HOME and an empty local bundle, dispatches the registered evidence-import command, and routes output with the executable's result-owner policy. It does not invoke the packaged executable.

export REA_VERIFY_ROOT=$(mktemp -d)
export HOME="$REA_VERIFY_ROOT/home"
mkdir -p "$HOME"
node --import tsx --input-type=module <<'NODE'
import { Writable } from 'node:stream';
import { writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
import { estimateTokenCount } from 'tokenx';
const load = (path) => import(pathToFileURL(join(process.cwd(), path)).href);
const { createCli, closeCliJavaScriptResources } = await load('src/cli.ts');
const { createCliResultOutput } = await load('src/cli/cliResultOutput.ts');
const { parseCliOutputArguments } = await load('src/cliOutput.ts');
const fixture = join(process.env.REA_VERIFY_ROOT, 'bundle.json');
await writeFile(fixture, JSON.stringify({artifacts:[],captures:[],environments:[],providers:[],records:[],scenarios:[],unknowns:[]}));
async function run(format, full, count) {
  const args = ['evidence-import', fixture, '--format', format,
    ...(full ? ['--full-output'] : []), ...(count ? ['--token-count'] : [])];
  let text = '', status = 0;
  const destination = new Writable({write(chunk, encoding, done) {text += chunk.toString(); done();}});
  const output = createCliResultOutput(parseCliOutputArguments(args), destination);
  await createCli({HOME:process.env.HOME,REA_LOG_LEVEL:'silent'}, output).serve(args, {
    stdout(value) {if (!output.handled && !output.captureSdkOutput(value)) text += value;},
    exit(code) {status = code;}
  });
  await output.finishSdkOutput(status);
  if (status) throw new Error(`CLI failed: ${status}: ${text}`);
  return text;
}
try {
  for (const format of ['toon','yaml','json','md']) {
    for (const full of [false,true]) {
      const normal = await run(format,full,false);
      const reported = Number((await run(format,full,true)).trim());
      console.log(JSON.stringify({format,full,reported,emittedEstimate:estimateTokenCount(normal)}));
    }
  }
} finally {await closeCliJavaScriptResources();}
NODE

Read each row's reported and emittedEstimate values. The replay reproduced once per format/flag combination. An independent focused integration test also reproduced the full-output failure three times per affected format and recorded the result-owner call from the registered command.

Expected behavior

When --full-output is selected, the token estimate must account for the envelope and command metadata that the same invocation would emit without --token-count. Bare output should continue to estimate just the selected result. This compares REA's own tokenx estimates, without asserting a model-specific exact token count. The applicable contract is docs/cli.md:1085–1089, which explicitly includes the requested full-output envelope and metadata.

Impact and scope

P2: output-budgeting callers get a smaller estimate than the selected document for the tested flag combinations. The focused test reproduced three of three times per affected format, and the separate source replay reproduced once more. Capturing the emitted text and applying tokenx directly is a demonstrated workaround. Full JSON and Markdown controls also match. No analysis failure, Evidence loss or downstream budget incident was observed; other command/error/filter/window combinations remain untested.

REA version

615fbbbdd7 (package.json version 6.8.0)

Runtime and platform

Node.js v26.12.1, macOS arm64, existing locked dependencies: incur 0.6.1 and tokenx 1.3.0. Focused integration test: Vitest 5.0.3. Source replay: locked tsx loader. No Hopper, Ghidra, IDA or device was used.

Target and artifact context

A valid empty local Evidence bundle with empty artifacts, captures, environments, providers, records, scenarios and unknowns arrays. No binary target or external analysis service.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Both affected full-output invocations complete with captured exit status 0 and print 13 when counting. Removing the counting flag produces an envelope whose captured text estimates to 31. All three integration attempts per affected format had this mismatch; the standalone source replay confirmed it again. Bare output and the JSON/Markdown controls matched their emitted estimates.

Format	Full output	Reported count	Estimate of emitted document
toon	no	13	13
toon	yes	13	31
yaml	no	13	13
yaml	yes	13	31
json	yes	40	40
md	yes	79	79

One captured full-output document was:

ok: true
data:
  imported: 0
  unknowns_added: 0
  total: 0
meta:
  command: evidence-import
  duration: 3ms

This sample is eight lines and 109 characters including its final newline. Duration varies, so restoration should compare the estimator with the selected representation rather than hard-code a universal token total.

The focused integration test traced the real command's accepted result to the output owner for both formats. Production connection: scripts/rea.mjs:135–160 passes the owner into the CLI; src/cli.ts:168 wraps its registered commands; src/cliEvidenceCommands.ts:24–33 registers this import. That launcher connection is source-derived, while the registry/output comparison is executed evidence.

Typed outcome or recovery expected

Return the estimate for the selected full document. Retain bare-output counts and the matching JSON/Markdown behavior; compare variable metadata rather than requiring one fixed count for every invocation.

Analysis or suspected cause

Source-derived: src/cli/cliResultOutput.ts:321–331 returns control to Incur for these successful unfiltered TOON/YAML results before the envelope-aware count branch. Locked Incur's node_modules/incur/dist/Cli.js:1082–1085 counts output.data before its full-output selection. This explains the observed bare-result count; no repair strategy is prescribed.

Additional context

Closed issue #6447 is related JSON count/representation history. The present TOON/YAML discrepancy still reproduces. Open and closed issue searches paired token-count with full-output, yaml and toon; no open duplicate was found. The first affected release and last known good release were not established.

Originally created by @coygeek on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7271 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior The registered `evidence-import` command reports `13` with `--format toon --full-output --token-count`, while the document emitted without the counting flag estimates to `31` using the same locked tokenx library. YAML behaves the same way. This was executed through the source CLI registry and the executable's output-routing policy; the packaged executable was not built or run. ### Reproduction Use an existing installation of the checkout's locked dependencies. For a fresh checkout, `npm ci` is preparation; it was not rerun for this verification. From the checkout root, run the source CLI replay below. It creates an isolated HOME and an empty local bundle, dispatches the registered `evidence-import` command, and routes output with the executable's result-owner policy. It does not invoke the packaged executable. ```sh export REA_VERIFY_ROOT=$(mktemp -d) export HOME="$REA_VERIFY_ROOT/home" mkdir -p "$HOME" node --import tsx --input-type=module <<'NODE' import { Writable } from 'node:stream'; import { writeFile } from 'node:fs/promises'; import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; import { estimateTokenCount } from 'tokenx'; const load = (path) => import(pathToFileURL(join(process.cwd(), path)).href); const { createCli, closeCliJavaScriptResources } = await load('src/cli.ts'); const { createCliResultOutput } = await load('src/cli/cliResultOutput.ts'); const { parseCliOutputArguments } = await load('src/cliOutput.ts'); const fixture = join(process.env.REA_VERIFY_ROOT, 'bundle.json'); await writeFile(fixture, JSON.stringify({artifacts:[],captures:[],environments:[],providers:[],records:[],scenarios:[],unknowns:[]})); async function run(format, full, count) { const args = ['evidence-import', fixture, '--format', format, ...(full ? ['--full-output'] : []), ...(count ? ['--token-count'] : [])]; let text = '', status = 0; const destination = new Writable({write(chunk, encoding, done) {text += chunk.toString(); done();}}); const output = createCliResultOutput(parseCliOutputArguments(args), destination); await createCli({HOME:process.env.HOME,REA_LOG_LEVEL:'silent'}, output).serve(args, { stdout(value) {if (!output.handled && !output.captureSdkOutput(value)) text += value;}, exit(code) {status = code;} }); await output.finishSdkOutput(status); if (status) throw new Error(`CLI failed: ${status}: ${text}`); return text; } try { for (const format of ['toon','yaml','json','md']) { for (const full of [false,true]) { const normal = await run(format,full,false); const reported = Number((await run(format,full,true)).trim()); console.log(JSON.stringify({format,full,reported,emittedEstimate:estimateTokenCount(normal)})); } } } finally {await closeCliJavaScriptResources();} NODE ``` Read each row's `reported` and `emittedEstimate` values. The replay reproduced once per format/flag combination. An independent focused integration test also reproduced the full-output failure three times per affected format and recorded the result-owner call from the registered command. ### Expected behavior When `--full-output` is selected, the token estimate must account for the envelope and command metadata that the same invocation would emit without `--token-count`. Bare output should continue to estimate just the selected result. This compares REA's own tokenx estimates, without asserting a model-specific exact token count. The applicable contract is `docs/cli.md:1085–1089`, which explicitly includes the requested full-output envelope and metadata. ### Impact and scope P2: output-budgeting callers get a smaller estimate than the selected document for the tested flag combinations. The focused test reproduced three of three times per affected format, and the separate source replay reproduced once more. Capturing the emitted text and applying tokenx directly is a demonstrated workaround. Full JSON and Markdown controls also match. No analysis failure, Evidence loss or downstream budget incident was observed; other command/error/filter/window combinations remain untested. ### REA version 615fbbbdd750de2df5d92b9827d0c7c4422c1d03 (package.json version 6.8.0) ### Runtime and platform Node.js v26.12.1, macOS arm64, existing locked dependencies: incur 0.6.1 and tokenx 1.3.0. Focused integration test: Vitest 5.0.3. Source replay: locked tsx loader. No Hopper, Ghidra, IDA or device was used. ### Target and artifact context A valid empty local Evidence bundle with empty artifacts, captures, environments, providers, records, scenarios and unknowns arrays. No binary target or external analysis service. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Both affected full-output invocations complete with captured exit status 0 and print `13` when counting. Removing the counting flag produces an envelope whose captured text estimates to `31`. All three integration attempts per affected format had this mismatch; the standalone source replay confirmed it again. Bare output and the JSON/Markdown controls matched their emitted estimates. | Format | Full output | Reported count | Estimate of emitted document | | --- | --- | --- | --- | | toon | no | 13 | 13 | | toon | yes | 13 | 31 | | yaml | no | 13 | 13 | | yaml | yes | 13 | 31 | | json | yes | 40 | 40 | | md | yes | 79 | 79 | One captured full-output document was: ```text ok: true data: imported: 0 unknowns_added: 0 total: 0 meta: command: evidence-import duration: 3ms ``` This sample is eight lines and 109 characters including its final newline. Duration varies, so restoration should compare the estimator with the selected representation rather than hard-code a universal token total. The focused integration test traced the real command's accepted result to the output owner for both formats. Production connection: `scripts/rea.mjs:135–160` passes the owner into the CLI; `src/cli.ts:168` wraps its registered commands; `src/cliEvidenceCommands.ts:24–33` registers this import. That launcher connection is source-derived, while the registry/output comparison is executed evidence. ### Typed outcome or recovery expected Return the estimate for the selected full document. Retain bare-output counts and the matching JSON/Markdown behavior; compare variable metadata rather than requiring one fixed count for every invocation. ### Analysis or suspected cause Source-derived: `src/cli/cliResultOutput.ts:321–331` returns control to Incur for these successful unfiltered TOON/YAML results before the envelope-aware count branch. Locked Incur's `node_modules/incur/dist/Cli.js:1082–1085` counts `output.data` before its full-output selection. This explains the observed bare-result count; no repair strategy is prescribed. ### Additional context Closed [issue #6447](https://github.com/morluto/rea/issues/6447) is related JSON count/representation history. The present TOON/YAML discrepancy still reproduces. Open and closed issue searches paired token-count with full-output, yaml and toon; no open duplicate was found. The first affected release and last known good release were not established.

bsakaguchi referenced this issue

2026-10-31 01:12:21 +00:00

[PR #4316] [CLOSED] perf(mcp): defer managed inspectors until first use #6165

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4316

No description provided.

Rows
Columns