[GH-ISSUE #7384] [Bug] find_changed_behavior reports observed equality for a comparison with unknown divergence #4391

New issue

Open

opened 2026-10-31 00:49:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:27 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7384

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Calling MCP compare_process_captures on compatible captures with a network-scoped residual unknown returns five unchanged dimensions and first_divergence.status: "unknown". Passing that returned Evidence directly to find_changed_behavior reports behavior_status: "observed_unchanged", zero unresolved findings and no changed-behavior registry unknown. A generic coverage warning survives in limitations, but the aggregate presents equality despite the comparison explicitly withholding that conclusion. Environment and cleanup residuals produced the same result; a complete control correctly reported observed equality.

Reproduction

Use Node.js 24.22.1 and the source revision below. A fresh checkout needs npm ci && npm run build:cached; verification rebuilt with existing locked dependencies and did not rerun installation. Run this from the repository root. Each case in this public stdio MCP sequence was executed once. All observations are synthetic; no target or analysis engine runs.

export WORK="$(mktemp -d)"
export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" XDG_DATA_HOME="$WORK/data"
export XDG_CACHE_HOME="$WORK/cache" XDG_STATE_HOME="$WORK/state" TMPDIR="$WORK/tmp"
mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_DATA_HOME" "$XDG_CACHE_HOME" "$XDG_STATE_HOME" "$TMPDIR"
node --input-type=module <<'JS'
import assert from 'node:assert/strict';
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';
import { createEvidence } from './dist/domain/evidence.js';
import { EMPTY_PROCESS_CAPTURE_EXAMPLE as example } from './dist/domain/process/processCaptureExample.js';
import { PROCESS_PROVIDER } from './dist/domain/process/processEvidenceProvider.js';
const client = new Client({ name: 'comparison-check', version: '1' });
const transport = new StdioClientTransport({
  command: process.execPath, args: ['scripts/rea.mjs', 'mcp'],
  env: Object.fromEntries(['PATH', 'HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME', 'TMPDIR'].map(k => [k, process.env[k]])),
  stderr: 'pipe',
});
transport.stderr?.on('data', () => {});
const call = async (name, args) => {
  const reply = await client.callTool({ name, arguments: args });
  assert(!reply.isError, JSON.stringify(reply));
  return reply.structuredContent;
};
const source = (side, scope) => createEvidence(undefined, PROCESS_PROVIDER, {
  predicateType: 'rea.process-capture', operation: 'capture_process_scenario',
  parameters: { side, scope },
  result: { ...example, residual_unknowns: scope ? [{ scope, reason: 'Observation coverage was unavailable.' }] : [] },
  confidence: 'observed', authority: 'controlled-replay',
  environment: { id: 'fixture', platform: 'fixture', architecture: 'fixture', isolation: 'process' },
});
try {
  await client.connect(transport);
  for (const scope of [null, 'network', 'environment', 'cleanup']) {
    const comparison = await call('compare_process_captures', { left: source('left', scope), right: source('right', scope) });
    const aggregate = (await call('find_changed_behavior', { comparisons: [comparison] })).normalized_result;
    const unknowns = await call('list_unknowns', { domain: 'changed-behavior' });
    console.log(JSON.stringify({ scope, comparison: comparison.normalized_result, aggregate, changed_behavior_unknowns: unknowns.result.total }));
  }
} finally {
  await client.close();
  await transport.close();
}
JS

Expected behavior

The aggregate must preserve unresolved overall comparison evidence: return unknown, emit an unresolved finding citing the comparison and its unknown-divergence reason, and retain the workflow's changed-behavior unknown. The complete control must continue returning observed_unchanged. src/contracts/sessionToolContracts.ts:127-129 promises unresolved findings for incomplete comparisons. src/domain/process/processComparison.ts:488-497 supplies the reason that residual unknowns prevent proving no divergence; src/domain/reconstructionVerification.ts:288-295 already treats this exact comparison shape as unknown for an overall claim.

Impact and scope

P2: a bounded accepted-input workflow gives callers an incorrect overall equality conclusion and no unresolved finding or registry entry to revisit. A practical workaround is to inspect the retained comparison's first_divergence and withhold equality when it is unknown; the reproduction preserves that signal and the generic warning. The observed scope is network, environment and cleanup residuals in synthetic compatible captures. Live-capture frequency, other platforms and downstream incidents were not established. No security boundary crossing was observed.

REA version

Exact node scripts/rea.mjs --version output: 6.8.0. Verified source commit: 081f2e2559db4802ffdb8783f6bc8d6803eb8c3d.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1; Darwin kernel 27.0.0; arm64.

Target and artifact context

Synthetic process-capture Evidence derived from the public empty-capture example, with matching normalization and comparison commitments, no truncation and empty observed collections. Distinct left/right records share a fixture environment. A residual unknown is the only difference from the complete control. The records do not claim observation of an executed target.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

All public tool calls succeeded. Selected fields from the network case, with Evidence IDs and duplicate limitations omitted:

{
  "comparison": {
    "status": "unchanged",
    "terminal": "unchanged",
    "interaction": "unchanged",
    "exit": "unchanged",
    "filesystem": "unchanged",
    "process": "unchanged",
    "first_divergence": {
      "status": "unknown",
      "reason": "Residual unknowns prevent proving that no divergence occurred."
    }
  },
  "aggregate": {
    "behavior_status": "observed_unchanged",
    "summary": { "observed_changes": 0, "static_candidates": 0, "contradictions": 0, "unresolved": 0 },
    "findings": { "items": [] },
    "limitations": ["Artifact identity is unavailable for this observation.", "Observation coverage was unavailable."]
  },
  "changed_behavior_unknowns": 0
}

The clean control has first_divergence: {"status":"none"}, observed_unchanged, no findings and zero changed-behavior unknowns. The environment and cleanup cases match the network case above. An independent imported-bundle replay also reproduced the network failure once.

Analysis or suspected cause

Source-derived: src/domain/changedBehavior.ts:137-140,344-350 computes behavior status from each process comparison's aggregate status without checking first_divergence. Its findings builder at lines 222-251 skips all five unchanged dimensions and emits no comparison-level unresolved finding. src/server/registerInvestigationTools.ts:96-103 then uses the reported behavior status to decide whether to record a changed-behavior unknown. Generic Evidence limitations survive separately.

Additional context

Closed #3980 concerned the same producer shape passing an overall verify_reconstruction claim; its fix is present in this build, while this separate aggregate still reports equality. Open producer reports about raw-only traces or filesystem-effect uncertainty have different closure conditions; this fixture has no trace or filesystem effects. Proposed restoration checks, not executed against a fix: repeat this sequence and require unresolved aggregation for the three residual scopes, preserve the complete control, and verify that an actual observed change still produces observed_changed.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7384 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Calling MCP `compare_process_captures` on compatible captures with a network-scoped residual unknown returns five unchanged dimensions and `first_divergence.status: "unknown"`. Passing that returned Evidence directly to `find_changed_behavior` reports `behavior_status: "observed_unchanged"`, zero unresolved findings and no changed-behavior registry unknown. A generic coverage warning survives in `limitations`, but the aggregate presents equality despite the comparison explicitly withholding that conclusion. Environment and cleanup residuals produced the same result; a complete control correctly reported observed equality. ## Reproduction Use Node.js 24.22.1 and the source revision below. A fresh checkout needs `npm ci && npm run build:cached`; verification rebuilt with existing locked dependencies and did not rerun installation. Run this from the repository root. Each case in this public stdio MCP sequence was executed once. All observations are synthetic; no target or analysis engine runs. ```sh export WORK="$(mktemp -d)" export HOME="$WORK/home" XDG_CONFIG_HOME="$WORK/config" XDG_DATA_HOME="$WORK/data" export XDG_CACHE_HOME="$WORK/cache" XDG_STATE_HOME="$WORK/state" TMPDIR="$WORK/tmp" mkdir -p "$HOME" "$XDG_CONFIG_HOME" "$XDG_DATA_HOME" "$XDG_CACHE_HOME" "$XDG_STATE_HOME" "$TMPDIR" node --input-type=module <<'JS' import assert from 'node:assert/strict'; import { Client } from '@modelcontextprotocol/client'; import { StdioClientTransport } from '@modelcontextprotocol/client/stdio'; import { createEvidence } from './dist/domain/evidence.js'; import { EMPTY_PROCESS_CAPTURE_EXAMPLE as example } from './dist/domain/process/processCaptureExample.js'; import { PROCESS_PROVIDER } from './dist/domain/process/processEvidenceProvider.js'; const client = new Client({ name: 'comparison-check', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: ['scripts/rea.mjs', 'mcp'], env: Object.fromEntries(['PATH', 'HOME', 'XDG_CONFIG_HOME', 'XDG_DATA_HOME', 'XDG_CACHE_HOME', 'XDG_STATE_HOME', 'TMPDIR'].map(k => [k, process.env[k]])), stderr: 'pipe', }); transport.stderr?.on('data', () => {}); const call = async (name, args) => { const reply = await client.callTool({ name, arguments: args }); assert(!reply.isError, JSON.stringify(reply)); return reply.structuredContent; }; const source = (side, scope) => createEvidence(undefined, PROCESS_PROVIDER, { predicateType: 'rea.process-capture', operation: 'capture_process_scenario', parameters: { side, scope }, result: { ...example, residual_unknowns: scope ? [{ scope, reason: 'Observation coverage was unavailable.' }] : [] }, confidence: 'observed', authority: 'controlled-replay', environment: { id: 'fixture', platform: 'fixture', architecture: 'fixture', isolation: 'process' }, }); try { await client.connect(transport); for (const scope of [null, 'network', 'environment', 'cleanup']) { const comparison = await call('compare_process_captures', { left: source('left', scope), right: source('right', scope) }); const aggregate = (await call('find_changed_behavior', { comparisons: [comparison] })).normalized_result; const unknowns = await call('list_unknowns', { domain: 'changed-behavior' }); console.log(JSON.stringify({ scope, comparison: comparison.normalized_result, aggregate, changed_behavior_unknowns: unknowns.result.total })); } } finally { await client.close(); await transport.close(); } JS ``` ## Expected behavior The aggregate must preserve unresolved overall comparison evidence: return `unknown`, emit an unresolved finding citing the comparison and its unknown-divergence reason, and retain the workflow's changed-behavior unknown. The complete control must continue returning `observed_unchanged`. `src/contracts/sessionToolContracts.ts:127-129` promises unresolved findings for incomplete comparisons. `src/domain/process/processComparison.ts:488-497` supplies the reason that residual unknowns prevent proving no divergence; `src/domain/reconstructionVerification.ts:288-295` already treats this exact comparison shape as unknown for an overall claim. ## Impact and scope P2: a bounded accepted-input workflow gives callers an incorrect overall equality conclusion and no unresolved finding or registry entry to revisit. A practical workaround is to inspect the retained comparison's `first_divergence` and withhold equality when it is unknown; the reproduction preserves that signal and the generic warning. The observed scope is network, environment and cleanup residuals in synthetic compatible captures. Live-capture frequency, other platforms and downstream incidents were not established. No security boundary crossing was observed. ## REA version Exact `node scripts/rea.mjs --version` output: `6.8.0`. Verified source commit: `081f2e2559db4802ffdb8783f6bc8d6803eb8c3d`. ## Runtime and platform Node.js `v24.22.1`; macOS `27.0.1`; Darwin kernel `27.0.0`; `arm64`. ## Target and artifact context Synthetic process-capture Evidence derived from the public empty-capture example, with matching normalization and comparison commitments, no truncation and empty observed collections. Distinct left/right records share a fixture environment. A residual unknown is the only difference from the complete control. The records do not claim observation of an executed target. ## Interface MCP ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence All public tool calls succeeded. Selected fields from the network case, with Evidence IDs and duplicate limitations omitted: ```json { "comparison": { "status": "unchanged", "terminal": "unchanged", "interaction": "unchanged", "exit": "unchanged", "filesystem": "unchanged", "process": "unchanged", "first_divergence": { "status": "unknown", "reason": "Residual unknowns prevent proving that no divergence occurred." } }, "aggregate": { "behavior_status": "observed_unchanged", "summary": { "observed_changes": 0, "static_candidates": 0, "contradictions": 0, "unresolved": 0 }, "findings": { "items": [] }, "limitations": ["Artifact identity is unavailable for this observation.", "Observation coverage was unavailable."] }, "changed_behavior_unknowns": 0 } ``` The clean control has `first_divergence: {"status":"none"}`, `observed_unchanged`, no findings and zero changed-behavior unknowns. The environment and cleanup cases match the network case above. An independent imported-bundle replay also reproduced the network failure once. ## Analysis or suspected cause Source-derived: `src/domain/changedBehavior.ts:137-140,344-350` computes behavior status from each process comparison's aggregate `status` without checking `first_divergence`. Its findings builder at lines 222-251 skips all five unchanged dimensions and emits no comparison-level unresolved finding. `src/server/registerInvestigationTools.ts:96-103` then uses the reported behavior status to decide whether to record a changed-behavior unknown. Generic Evidence limitations survive separately. ## Additional context Closed [#3980](https://github.com/morluto/rea/issues/3980) concerned the same producer shape passing an overall `verify_reconstruction` claim; its fix is present in this build, while this separate aggregate still reports equality. Open producer reports about raw-only traces or filesystem-effect uncertainty have different closure conditions; this fixture has no trace or filesystem effects. Proposed restoration checks, not executed against a fix: repeat this sequence and require unresolved aggregation for the three residual scopes, preserve the complete control, and verify that an actual observed change still produces `observed_changed`.

bsakaguchi referenced this issue

2026-10-31 01:12:43 +00:00

[PR #4391] [CLOSED] fix(browser): bound observe_web_session windows to the Node.js timer limit #6211

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:44:39 +00:00

[PR #5639] [CLOSED] fix(browser): exclude replayed lifecycle milestones from session activity #7006