morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5107] [Bug] CLI Evidence comparison cannot accept explicit observation pairs #3143

New issue

Closed

opened 2026-10-31 00:37:07 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:37:07 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5107

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

The source CLI compare accepts two canonical Evidence bundle paths but has no input for explicit record pairs. With two valid one-record bundles and known correspondence, it returns unknown, records_changed: 0, unresolved: 2. Attempting --record-pairs returns Unknown flag: --record-pairs.

The same files and explicit pair return changed, records_changed: 1, unresolved: 0 through production MCP registration and SDK in-memory transport. This MCP check used a fixture session: a mocked session boundary, without provider execution. The unpaired MCP control agrees with the CLI's conservative unknown result.

Reproduction

Verification exercised source adapters, not the compiled package entrypoint. From a checkout at the revision below with lockfile dependencies already installed, run the following source-boundary probe. It creates authenticated synthetic records, invokes the production CLI parser/handler and compares the explicit-pair domain control. It neither analyzes nor executes an artifact.

WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT
mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/data" "$WORK/temp"
env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" XDG_CACHE_HOME="$WORK/cache" XDG_DATA_HOME="$WORK/data" TMPDIR="$WORK/temp" WORK="$WORK" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS'
import { writeFile } from 'node:fs/promises';
import { createCli } from './src/cli.ts';
import { createEvidence } from './src/domain/evidence.ts';
import { createEvidenceBundle } from './src/domain/evidenceBundle.ts';
import { compareBundles } from './src/domain/bundleComparison.ts';
const work = process.env.WORK;
const record = value => createEvidence(undefined,
  { id: 'fixture', name: 'Fixture', version: '1' }, {
    operation: 'observe', parameters: {}, result: { value },
    confidence: 'derived', authority: 'analyst-inference'
  });
const oldRecord = record(1), newRecord = record(2);
const left = createEvidenceBundle([oldRecord]);
const right = createEvidenceBundle([newRecord]);
const pairs = [{ left_evidence_id: oldRecord.evidence_id,
                 right_evidence_id: newRecord.evidence_id }];
await writeFile(`${work}/left.json`, JSON.stringify(left));
await writeFile(`${work}/right.json`, JSON.stringify(right));
const serve = async args => {
  let output = '', exitCode = 0;
  await createCli({}).serve(args, {
    env: {}, stdout: text => { output += text; },
    exit: code => { exitCode = code; }
  });
  return { exitCode, result: JSON.parse(output) };
};
const paths = [`${work}/left.json`, `${work}/right.json`];
const result = await serve(['compare', ...paths, '--json']);
console.log('CLI:', JSON.stringify({ exitCode: result.exitCode,
  status: result.result.status, summary: result.result.summary }));
const flag = await serve(['compare', ...paths, '--record-pairs',
  JSON.stringify(pairs), '--json']);
console.log('pair input:', JSON.stringify(flag));
const paired = compareBundles(left, right, pairs);
console.log('paired domain:', JSON.stringify({ status: paired.status,
  summary: paired.summary }));
JS

The independent MCP check used createApplicationMcpHarness() from the repository's test fixtures and client.callTool({name: 'compare_bundles', arguments: {left_bundle_path, right_bundle_path, record_pairs: pairs}}) on the same generated files. It authenticated the returned Evidence with parseEvidence. No target was opened.

Expected behavior

Provide a CLI input that preserves caller-selected one-to-one observation pairs for the same bundle-comparison workflow. The CLI documentation promises the same application workflows and Evidence contracts as MCP. The comparison contract expressly includes explicit pairs.

Unpaired different IDs should continue to remain unresolved; correspondence must not be guessed. The missing capability is supplying a known pair from the CLI.

Impact and scope

CLI users cannot classify explicitly corresponding changed records through compare or its compare-bundles alias. The observed effect is bounded to this comparison capability. MCP explicit pairing is a demonstrated alternative at the tested SDK boundary. No data loss, Evidence corruption, provider failure or security-boundary crossing was observed. Real-world frequency is NOT_ENOUGH_INFO.

REA version

Source revision 3537e58c1ab29309f06eabf50bc52adba9817a0c, package version 6.7.0. Relevant CLI, application, MCP and domain comparison sources are unchanged from stable tag rea-agents-6.7.0; the published stable package was not executed.

Runtime and platform

Node.js 26.12.0; macOS 27.0.1, arm64. Source execution through the checkout's installed tsx loader. No provider engine was involved. The repository-pinned Node 24.18.0 runtime was not tested.

Target and artifact context

Two canonical synthetic Evidence bundles, each containing one record from the same fixture provider, operation and parameters. Their authenticated normalized results are {value: 1} and {value: 2}. No artifact bytes or live runtime observations are claimed.

Interface

CLI

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Observed source-adapter results:

CLI: exitCode=0, status=unknown, records_changed=0, unresolved=2
--record-pairs: exitCode=1, message="Unknown flag: --record-pairs"
paired domain: status=changed, records_changed=1, unresolved=0
MCP SDK in-memory, unpaired: isError=false, status=unknown, records_changed=0, unresolved=2
MCP SDK in-memory, paired: isError=false, status=changed, records_changed=1, unresolved=0
identical-input CLI control: exitCode=0, status=unchanged, records_unchanged=1, unresolved=0

CLI registration exposes only left/right arguments. Its application workflow validates both files and calls comparison without pairs. MCP forwards the selected pairs.

Typed outcome or recovery expected

A supported CLI input for explicit correspondence should reach the shared comparer and produce the paired change report. Keep unknown for unpaired membership and preserve both bundle digests and Evidence IDs.

Analysis or suspected cause

Source-derived: the CLI application input contains only paths and invokes the empty-pair default. Neither the CLI schema nor this application boundary can carry the MCP contract's record_pairs value.

Additional context

The closed display-only bundle comparison report has a different trigger and repair. Its semantic controls explicitly preserve the distinction between unpaired unknown membership and paired changed observations. General CLI parity work in #30 and #565 does not identify this omitted comparison input.

Compiled entrypoint behavior, production stdio transport, stable-release runtime execution and first affected release are NOT_ENOUGH_INFO. The source CLI and production MCP registration boundaries above were executed; no real provider behavior was tested.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5107 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area CLI ## Observed behavior The source CLI `compare` accepts two canonical Evidence bundle paths but has no input for explicit record pairs. With two valid one-record bundles and known correspondence, it returns `unknown`, `records_changed: 0`, `unresolved: 2`. Attempting `--record-pairs` returns `Unknown flag: --record-pairs`. The same files and explicit pair return `changed`, `records_changed: 1`, `unresolved: 0` through production MCP registration and SDK in-memory transport. This MCP check used a fixture session: a mocked session boundary, without provider execution. The unpaired MCP control agrees with the CLI's conservative `unknown` result. ## Reproduction Verification exercised source adapters, not the compiled package entrypoint. From a checkout at the revision below with lockfile dependencies already installed, run the following source-boundary probe. It creates authenticated synthetic records, invokes the production CLI parser/handler and compares the explicit-pair domain control. It neither analyzes nor executes an artifact. ```sh WORK=$(mktemp -d) trap 'rm -rf "$WORK"' EXIT mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/data" "$WORK/temp" env -i PATH="$PATH" HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" XDG_CACHE_HOME="$WORK/cache" XDG_DATA_HOME="$WORK/data" TMPDIR="$WORK/temp" WORK="$WORK" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS' import { writeFile } from 'node:fs/promises'; import { createCli } from './src/cli.ts'; import { createEvidence } from './src/domain/evidence.ts'; import { createEvidenceBundle } from './src/domain/evidenceBundle.ts'; import { compareBundles } from './src/domain/bundleComparison.ts'; const work = process.env.WORK; const record = value => createEvidence(undefined, { id: 'fixture', name: 'Fixture', version: '1' }, { operation: 'observe', parameters: {}, result: { value }, confidence: 'derived', authority: 'analyst-inference' }); const oldRecord = record(1), newRecord = record(2); const left = createEvidenceBundle([oldRecord]); const right = createEvidenceBundle([newRecord]); const pairs = [{ left_evidence_id: oldRecord.evidence_id, right_evidence_id: newRecord.evidence_id }]; await writeFile(`${work}/left.json`, JSON.stringify(left)); await writeFile(`${work}/right.json`, JSON.stringify(right)); const serve = async args => { let output = '', exitCode = 0; await createCli({}).serve(args, { env: {}, stdout: text => { output += text; }, exit: code => { exitCode = code; } }); return { exitCode, result: JSON.parse(output) }; }; const paths = [`${work}/left.json`, `${work}/right.json`]; const result = await serve(['compare', ...paths, '--json']); console.log('CLI:', JSON.stringify({ exitCode: result.exitCode, status: result.result.status, summary: result.result.summary })); const flag = await serve(['compare', ...paths, '--record-pairs', JSON.stringify(pairs), '--json']); console.log('pair input:', JSON.stringify(flag)); const paired = compareBundles(left, right, pairs); console.log('paired domain:', JSON.stringify({ status: paired.status, summary: paired.summary })); JS ``` The independent MCP check used `createApplicationMcpHarness()` from the repository's test fixtures and `client.callTool({name: 'compare_bundles', arguments: {left_bundle_path, right_bundle_path, record_pairs: pairs}})` on the same generated files. It authenticated the returned Evidence with `parseEvidence`. No target was opened. ## Expected behavior Provide a CLI input that preserves caller-selected one-to-one observation pairs for the same bundle-comparison workflow. The CLI documentation promises the [same application workflows and Evidence contracts as MCP](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/docs/cli.md#L3-L5). The [comparison contract](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/contracts/sessionToolContracts.ts#L115-L119) expressly includes explicit pairs. Unpaired different IDs should continue to remain unresolved; correspondence must not be guessed. The missing capability is supplying a known pair from the CLI. ## Impact and scope CLI users cannot classify explicitly corresponding changed records through `compare` or its `compare-bundles` alias. The observed effect is bounded to this comparison capability. MCP explicit pairing is a demonstrated alternative at the tested SDK boundary. No data loss, Evidence corruption, provider failure or security-boundary crossing was observed. Real-world frequency is NOT_ENOUGH_INFO. ## REA version Source revision `3537e58c1ab29309f06eabf50bc52adba9817a0c`, package version 6.7.0. Relevant CLI, application, MCP and domain comparison sources are unchanged from stable tag `rea-agents-6.7.0`; the published stable package was not executed. ## Runtime and platform Node.js 26.12.0; macOS 27.0.1, arm64. Source execution through the checkout's installed tsx loader. No provider engine was involved. The repository-pinned Node 24.18.0 runtime was not tested. ## Target and artifact context Two canonical synthetic Evidence bundles, each containing one record from the same fixture provider, operation and parameters. Their authenticated normalized results are `{value: 1}` and `{value: 2}`. No artifact bytes or live runtime observations are claimed. ## Interface CLI ## Earliest failure stage Request parsing or validation ## Relevant output or evidence Observed source-adapter results: ```text CLI: exitCode=0, status=unknown, records_changed=0, unresolved=2 --record-pairs: exitCode=1, message="Unknown flag: --record-pairs" paired domain: status=changed, records_changed=1, unresolved=0 MCP SDK in-memory, unpaired: isError=false, status=unknown, records_changed=0, unresolved=2 MCP SDK in-memory, paired: isError=false, status=changed, records_changed=1, unresolved=0 identical-input CLI control: exitCode=0, status=unchanged, records_unchanged=1, unresolved=0 ``` [CLI registration](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/cliEvidenceCommands.ts#L50-L65) exposes only left/right arguments. [Its application workflow](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/application/EvidenceBundleCommands.ts#L69-L91) validates both files and calls comparison without pairs. [MCP forwards the selected pairs](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/server/registerBundleComparisonTool.ts#L35-L54). ## Typed outcome or recovery expected A supported CLI input for explicit correspondence should reach the shared comparer and produce the paired change report. Keep `unknown` for unpaired membership and preserve both bundle digests and Evidence IDs. ## Analysis or suspected cause Source-derived: the CLI application input contains only paths and invokes the [empty-pair default](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/src/domain/bundleComparison.ts#L85-L96). Neither the CLI schema nor this application boundary can carry the MCP contract's `record_pairs` value. ## Additional context The closed [display-only bundle comparison report](https://github.com/morluto/rea/issues/3216) has a different trigger and repair. Its semantic controls explicitly preserve the distinction between unpaired unknown membership and paired changed observations. General CLI parity work in [#30](https://github.com/morluto/rea/issues/30) and [#565](https://github.com/morluto/rea/issues/565) does not identify this omitted comparison input. Compiled entrypoint behavior, production stdio transport, stable-release runtime execution and first affected release are NOT_ENOUGH_INFO. The source CLI and production MCP registration boundaries above were executed; no real provider behavior was tested.

bsakaguchi closed this issue

2026-10-31 00:37:08 +00:00

bsakaguchi commented

2026-10-31 00:37:08 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Confirmed — the CLI/MCP contract drift here is the real cost: the same two one-record bundles return a conservative unknown via the CLI while production MCP resolves them with records_changed: 1, which makes CLI-driven comparisons silently weaker than the documented Evidence contract.

Supporting Draft PR #5129's --record-pairs; I'd also suggest reusing the exact pairing/validation logic the MCP handler already has (same correspondence checks, same error strings) plus a CLI-vs-MCP parity test asserting identical results for the same inputs, so this drift can't come back unnoticed.

@XinwangTao commented on GitHub (Oct 19, 2026): Confirmed — the CLI/MCP contract drift here is the real cost: the same two one-record bundles return a conservative unknown via the CLI while production MCP resolves them with records_changed: 1, which makes CLI-driven comparisons silently weaker than the documented Evidence contract. Supporting Draft PR #5129's --record-pairs; I'd also suggest reusing the exact pairing/validation logic the MCP handler already has (same correspondence checks, same error strings) plus a CLI-vs-MCP parity test asserting identical results for the same inputs, so this drift can't come back unnoticed.

bsakaguchi commented

2026-10-31 00:37:08 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

The compare command and its compare-bundles alias now expose the documented --record_pairs JSON-array option. This is an underscore spelling, not the issue reproduction's attempted --record-pairs spelling; the expected outcome did not require that exact flag name. The command parses JSON, validates the same record-pair schema, and forwards selected pairs through compareEvidenceBundlesCommand into compareBundles.

Implementation: cliEvidenceCommands.ts, EvidenceBundleCommands.ts.

The relevant regression checks in bundleComparisonCli.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. The compare command and its compare-bundles alias now expose the documented --record_pairs JSON-array option. This is an underscore spelling, not the issue reproduction's attempted --record-pairs spelling; the expected outcome did not require that exact flag name. The command parses JSON, validates the same record-pair schema, and forwards selected pairs through compareEvidenceBundlesCommand into compareBundles. Implementation: [cliEvidenceCommands.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/cliEvidenceCommands.ts#L54), [EvidenceBundleCommands.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/EvidenceBundleCommands.ts#L216). The relevant regression checks in [bundleComparisonCli.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/cli/bundleComparisonCli.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:08:11 +00:00

[PR #3143] [CLOSED] fix(browser): explain attached service-worker policy refusals #5640

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3143

No description provided.

Rows
Columns