morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3931] [Bug] Trace comparison marks an unrelated truncated terminal dimension as changed #1539

New issue

Closed

opened 2026-10-16 21:06:51 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:06:51 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3931

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

Comparing two capture Evidence files with a trace specification can turn the terminal dimension from unknown into changed even when the retained terminal observations are identical and the trace diagnostic identifies only a process difference. The CLI returns the false terminal change inside derived comparison Evidence. The overall comparison correctly remains changed because the process observations differ.

All three CLI calls exited 0 and returned compare_process_captures Evidence. The script's assertions passed and produced these relevant fields:

withoutTrace: terminal=unknown, process=added, status=changed
withTrace:   terminal=changed, process=changed, status=changed
control:     terminal=unchanged, process=changed, status=changed

The traced comparison returned trace.verdict: "different" and a left-side cardinality diagnostic with event_ids: ["worker"], no locations, and message worker observed 0 times; expected 1..1. The production renderer returned no retained frames and retention { "budget_bytes": 5, "observed_bytes": 0, "retained_bytes": 0, "observed_frames": 1, "retained_frames": 0, "unknown_size_frames": 1 }. Thus the terminal result changes when adding trace validation even though the only failing event belongs to the process source.

Reproduction

Run this from the baseline checkout with dependencies and dist/ already present. It creates disposable inputs, generates an omitted rendered frame from a five-byte renderer budget, and compares the same pair with and without a trace specification. Its final comparison removes rendered truncation as a control. The frame, process sample, journal layout, and trace specification follow the repository's fixtures. These are fixture-backed comparison inputs rather than recordings of a real command.

repro_dir="$(mktemp -d)"
REA_VERIFY_DIR="$repro_dir" node --input-type=module <<'JS'
import { writeFileSync, mkdirSync } from 'node:fs';
import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
import assert from 'node:assert/strict';
const load = path => import(pathToFileURL(resolve(path)));
const { EMPTY_PROCESS_CAPTURE_EXAMPLE: base } = await load('dist/domain/process/processCaptureExample.js');
const { parseProcessCapture } = await load('dist/domain/process/processCaptureParsing.js');
const { createEvidence } = await load('dist/domain/evidence.js');
const { PROCESS_PROVIDER } = await load('dist/domain/process/processEvidenceProvider.js');
const { TerminalRenderer } = await load('dist/process/capture/TerminalRenderer.js');
const root = process.env.REA_VERIFY_DIR + '/';
const terminal = { sequence: 0, at_ms: 900, data: 'Ready' };
const worker = { at_ms: 1, pid: 1, parent_pid: 0, command: 'worker', process_group_id: 1, session_id: 1 };
const renderer = new TerminalRenderer({ columns: 5, rows: 1, scrollback: 0, maxBytes: 5,
  normalize: (value, maxBytes) => Buffer.byteLength(value) > maxBytes ? { state: 'over_budget' } : { state: 'normalized', value, utf8Bytes: Buffer.byteLength(value) } });
renderer.write('Ready', 900);
assert.deepEqual(await renderer.frames(), []);
const renderedRetention = renderer.retention();
await renderer.dispose();
console.log('renderer retention', JSON.stringify(renderedRetention));
function evidence(withWorker, truncated = true) {
  const capture = structuredClone(base);
  capture.frames = [terminal];
  capture.process_samples = withWorker ? [worker] : [];
  const entries = [{ collection: 'filesystem_checkpoints', index: 0 }, { collection: 'frames', index: 0 },
    ...(withWorker ? [{ collection: 'process_samples', index: 0 }] : []),
    { collection: 'lifecycle', index: 0 }, { collection: 'lifecycle', index: 1 }, { collection: 'filesystem_checkpoints', index: 1 }];
  capture.event_journal = entries.map((entry, capture_order) => ({ capture_order, ...entry }));
  capture.truncation_details.raw_terminal = { budget_bytes: 5, observed_bytes: 5, retained_bytes: 5, observed_frames: 1, retained_frames: 1 };
  capture.truncation_details.process.retained_samples = capture.process_samples.length;
  capture.truncated = truncated;
  if (truncated) capture.truncation_details.rendered_terminal = renderedRetention;
  return createEvidence(undefined, PROCESS_PROVIDER, { predicateType: 'rea.process-capture', operation: 'capture_process_scenario', parameters: {}, result: parseProcessCapture(capture), confidence: 'observed', authority: 'controlled-replay' });
}
const trace = { events: [{ id: 'ready', source: 'terminal_raw', exact: terminal, cardinality: { kind: 'required' } },
  { id: 'worker', source: 'process', exact: worker, cardinality: { kind: 'required' } }], language: {
  kind: 'partial_order', happens_before: [{ before: 'ready', after: 'worker' }], not_before: [], unordered_groups: [], prefix: ['ready'], suffix: ['worker'] } };
for (const [name, value] of Object.entries({ left: evidence(false), right: evidence(true), controlLeft: evidence(false, false), controlRight: evidence(true, false), trace })) {
  writeFileSync(root + name + '.json', JSON.stringify(value));
}
mkdirSync(root + 'home', { recursive: true });
function compare(name, left, right, useTrace) {
  const args = ['scripts/rea.mjs', 'compare-process-captures', root + left + '.json', root + right + '.json', ...(useTrace ? [root + 'trace.json'] : []), '--json'];
  const output = spawnSync(process.execPath, args, { encoding: 'utf8', env: { ...process.env, HOME: root + 'home' } });
  assert.equal(output.status, 0, output.stdout + output.stderr);
  writeFileSync(root + name + '.output.json', output.stdout);
  const envelope = JSON.parse(output.stdout);
  const result = envelope.normalized_result;
  console.log(name, JSON.stringify({ operation: envelope.operation, terminal: result.terminal, process: result.process, status: result.status, trace: result.trace }));
  return result;
}
const plain = compare('withoutTrace', 'left', 'right', false);
const traced = compare('withTrace', 'left', 'right', true);
const control = compare('control', 'controlLeft', 'controlRight', true);
assert.equal(plain.terminal, 'unknown');
assert.equal(traced.terminal, 'changed');
assert.equal(traced.trace.verdict, 'different');
assert.deepEqual(traced.trace.diagnostic.event_ids, ['worker']);
assert.equal(traced.process, 'changed');
assert.equal(traced.status, 'changed');
assert.equal(control.terminal, 'unchanged');
console.log('All CLI failure and control assertions passed.');
JS

Expected behavior

An incomplete comparison dimension remains unknown unless observations establish a difference in that dimension. The trace can evaluate complete raw terminal output despite omitted rendered snapshots, but a missing process event provides no evidence that the terminal changed. For this input, the traced result should retain terminal: "unknown", report the process difference, and keep aggregate status: "changed".

Impact and scope

A supported CLI comparison returns a false dimension-level change in derived Evidence when the trace covers terminal and process observations, rendered coverage is incomplete, and the diagnostic concerns only process observations. This is a bounded result-integrity defect; the demonstrated aggregate status remains correct. Population frequency and downstream session effects were not measured.

Workaround: The observed no-trace comparison preserves terminal uncertainty but removes the caller's requested trace validation. The control without truncation requires different observation coverage. Neither preserves both the existing incomplete capture inputs and the requested traced comparison.

REA version

rea 6.5.0 at source commit 4692bb3330

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Fixture-backed process-capture Evidence with identical complete raw terminal frames, omitted rendered snapshots, and a process-only trace failure. No real process capture ran.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Expected source: docs/process-capture.md:423-428 requires affected dimensions to remain unknown and allows trace assertions to use complete raw output independently of omitted rendered snapshots. src/domain/process/processComparison.ts:332-333 states that a different verdict is attributed to failing-event dimensions.
Failure source: The executed CLI reproduction above returns normalized_result.terminal: "changed". src/application/process/ProcessCli.ts:106-144 parses both capture Evidence files and the optional trace, then returns derived comparison Evidence. Its input guards at lines 151-203 accept the reproduction. src/domain/process/processComparison.ts:352-356 promotes a covered exact unknown result to changed despite the process-only diagnostic.

The source-derived explanation is that the trace projection accepts every exact status other than unchanged as a change, including unknown. The CLI result and controls above establish the violation directly. MCP persistence and real process capture behavior were not tested.

Typed outcome or recovery expected

For the same fixture Evidence and trace specification, the terminal dimension must remain unknown, the process difference must remain reported, the trace must retain its process cardinality diagnostic, and the aggregate status must remain changed. Preserve the no-trace result and the control without rendered truncation, where the terminal dimension is unchanged. These checks distinguish preserving uncertainty from suppressing a legitimate process difference or marking every terminal comparison unknown.

Analysis or suspected cause

Source-derived explanation and exact locators are in Relevant output or evidence. Provider and transport limits are stated there separately from reproduced results.

Additional context

Related closed issues #2616 and #3181 concern equal complete observations and finite-trace diagnostic attribution. Open PR #2929 retains the exact-non-unchanged promotion, which includes unknown; this report requires preserving incomplete terminal uncertainty for a process-only diagnostic. The aggregate changed status is correct because the process observations differ. The defect is the terminal dimension losing its unknown state.

Freshness check: the cited source and contract files have no changes between the tested source commit and upstream main at 22ec8bdb2f.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3931 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior Comparing two capture Evidence files with a trace specification can turn the terminal dimension from `unknown` into `changed` even when the retained terminal observations are identical and the trace diagnostic identifies only a process difference. The CLI returns the false terminal change inside derived comparison Evidence. The overall comparison correctly remains `changed` because the process observations differ. All three CLI calls exited 0 and returned `compare_process_captures` Evidence. The script's assertions passed and produced these relevant fields: ```text withoutTrace: terminal=unknown, process=added, status=changed withTrace: terminal=changed, process=changed, status=changed control: terminal=unchanged, process=changed, status=changed ``` The traced comparison returned `trace.verdict: "different"` and a left-side `cardinality` diagnostic with `event_ids: ["worker"]`, no locations, and message `worker observed 0 times; expected 1..1.` The production renderer returned no retained frames and retention `{ "budget_bytes": 5, "observed_bytes": 0, "retained_bytes": 0, "observed_frames": 1, "retained_frames": 0, "unknown_size_frames": 1 }`. Thus the terminal result changes when adding trace validation even though the only failing event belongs to the process source. ### Reproduction Run this from the baseline checkout with dependencies and `dist/` already present. It creates disposable inputs, generates an omitted rendered frame from a five-byte renderer budget, and compares the same pair with and without a trace specification. Its final comparison removes rendered truncation as a control. The frame, process sample, journal layout, and trace specification follow the repository's fixtures. These are fixture-backed comparison inputs rather than recordings of a real command. ```sh repro_dir="$(mktemp -d)" REA_VERIFY_DIR="$repro_dir" node --input-type=module <<'JS' import { writeFileSync, mkdirSync } from 'node:fs'; import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; import { spawnSync } from 'node:child_process'; import assert from 'node:assert/strict'; const load = path => import(pathToFileURL(resolve(path))); const { EMPTY_PROCESS_CAPTURE_EXAMPLE: base } = await load('dist/domain/process/processCaptureExample.js'); const { parseProcessCapture } = await load('dist/domain/process/processCaptureParsing.js'); const { createEvidence } = await load('dist/domain/evidence.js'); const { PROCESS_PROVIDER } = await load('dist/domain/process/processEvidenceProvider.js'); const { TerminalRenderer } = await load('dist/process/capture/TerminalRenderer.js'); const root = process.env.REA_VERIFY_DIR + '/'; const terminal = { sequence: 0, at_ms: 900, data: 'Ready' }; const worker = { at_ms: 1, pid: 1, parent_pid: 0, command: 'worker', process_group_id: 1, session_id: 1 }; const renderer = new TerminalRenderer({ columns: 5, rows: 1, scrollback: 0, maxBytes: 5, normalize: (value, maxBytes) => Buffer.byteLength(value) > maxBytes ? { state: 'over_budget' } : { state: 'normalized', value, utf8Bytes: Buffer.byteLength(value) } }); renderer.write('Ready', 900); assert.deepEqual(await renderer.frames(), []); const renderedRetention = renderer.retention(); await renderer.dispose(); console.log('renderer retention', JSON.stringify(renderedRetention)); function evidence(withWorker, truncated = true) { const capture = structuredClone(base); capture.frames = [terminal]; capture.process_samples = withWorker ? [worker] : []; const entries = [{ collection: 'filesystem_checkpoints', index: 0 }, { collection: 'frames', index: 0 }, ...(withWorker ? [{ collection: 'process_samples', index: 0 }] : []), { collection: 'lifecycle', index: 0 }, { collection: 'lifecycle', index: 1 }, { collection: 'filesystem_checkpoints', index: 1 }]; capture.event_journal = entries.map((entry, capture_order) => ({ capture_order, ...entry })); capture.truncation_details.raw_terminal = { budget_bytes: 5, observed_bytes: 5, retained_bytes: 5, observed_frames: 1, retained_frames: 1 }; capture.truncation_details.process.retained_samples = capture.process_samples.length; capture.truncated = truncated; if (truncated) capture.truncation_details.rendered_terminal = renderedRetention; return createEvidence(undefined, PROCESS_PROVIDER, { predicateType: 'rea.process-capture', operation: 'capture_process_scenario', parameters: {}, result: parseProcessCapture(capture), confidence: 'observed', authority: 'controlled-replay' }); } const trace = { events: [{ id: 'ready', source: 'terminal_raw', exact: terminal, cardinality: { kind: 'required' } }, { id: 'worker', source: 'process', exact: worker, cardinality: { kind: 'required' } }], language: { kind: 'partial_order', happens_before: [{ before: 'ready', after: 'worker' }], not_before: [], unordered_groups: [], prefix: ['ready'], suffix: ['worker'] } }; for (const [name, value] of Object.entries({ left: evidence(false), right: evidence(true), controlLeft: evidence(false, false), controlRight: evidence(true, false), trace })) { writeFileSync(root + name + '.json', JSON.stringify(value)); } mkdirSync(root + 'home', { recursive: true }); function compare(name, left, right, useTrace) { const args = ['scripts/rea.mjs', 'compare-process-captures', root + left + '.json', root + right + '.json', ...(useTrace ? [root + 'trace.json'] : []), '--json']; const output = spawnSync(process.execPath, args, { encoding: 'utf8', env: { ...process.env, HOME: root + 'home' } }); assert.equal(output.status, 0, output.stdout + output.stderr); writeFileSync(root + name + '.output.json', output.stdout); const envelope = JSON.parse(output.stdout); const result = envelope.normalized_result; console.log(name, JSON.stringify({ operation: envelope.operation, terminal: result.terminal, process: result.process, status: result.status, trace: result.trace })); return result; } const plain = compare('withoutTrace', 'left', 'right', false); const traced = compare('withTrace', 'left', 'right', true); const control = compare('control', 'controlLeft', 'controlRight', true); assert.equal(plain.terminal, 'unknown'); assert.equal(traced.terminal, 'changed'); assert.equal(traced.trace.verdict, 'different'); assert.deepEqual(traced.trace.diagnostic.event_ids, ['worker']); assert.equal(traced.process, 'changed'); assert.equal(traced.status, 'changed'); assert.equal(control.terminal, 'unchanged'); console.log('All CLI failure and control assertions passed.'); JS ``` ### Expected behavior An incomplete comparison dimension remains `unknown` unless observations establish a difference in that dimension. The trace can evaluate complete raw terminal output despite omitted rendered snapshots, but a missing process event provides no evidence that the terminal changed. For this input, the traced result should retain `terminal: "unknown"`, report the process difference, and keep aggregate `status: "changed"`. ### Impact and scope A supported CLI comparison returns a false dimension-level change in derived Evidence when the trace covers terminal and process observations, rendered coverage is incomplete, and the diagnostic concerns only process observations. This is a bounded result-integrity defect; the demonstrated aggregate status remains correct. Population frequency and downstream session effects were not measured. Workaround: The observed no-trace comparison preserves terminal uncertainty but removes the caller's requested trace validation. The control without truncation requires different observation coverage. Neither preserves both the existing incomplete capture inputs and the requested traced comparison. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Fixture-backed process-capture Evidence with identical complete raw terminal frames, omitted rendered snapshots, and a process-only trace failure. No real process capture ran. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence - Expected source: `docs/process-capture.md:423-428` requires affected dimensions to remain unknown and allows trace assertions to use complete raw output independently of omitted rendered snapshots. `src/domain/process/processComparison.ts:332-333` states that a different verdict is attributed to failing-event dimensions. - Failure source: The executed CLI reproduction above returns `normalized_result.terminal: "changed"`. `src/application/process/ProcessCli.ts:106-144` parses both capture Evidence files and the optional trace, then returns derived comparison Evidence. Its input guards at lines 151-203 accept the reproduction. `src/domain/process/processComparison.ts:352-356` promotes a covered exact `unknown` result to `changed` despite the process-only diagnostic. The source-derived explanation is that the trace projection accepts every exact status other than `unchanged` as a change, including `unknown`. The CLI result and controls above establish the violation directly. MCP persistence and real process capture behavior were not tested. ### Typed outcome or recovery expected For the same fixture Evidence and trace specification, the terminal dimension must remain `unknown`, the process difference must remain reported, the trace must retain its process cardinality diagnostic, and the aggregate status must remain `changed`. Preserve the no-trace result and the control without rendered truncation, where the terminal dimension is `unchanged`. These checks distinguish preserving uncertainty from suppressing a legitimate process difference or marking every terminal comparison unknown. ### Analysis or suspected cause Source-derived explanation and exact locators are in Relevant output or evidence. Provider and transport limits are stated there separately from reproduced results. ### Additional context Related closed issues #2616 and #3181 concern equal complete observations and finite-trace diagnostic attribution. Open PR #2929 retains the exact-non-unchanged promotion, which includes unknown; this report requires preserving incomplete terminal uncertainty for a process-only diagnostic. The aggregate changed status is correct because the process observations differ. The defect is the terminal dimension losing its unknown state. Freshness check: the cited source and contract files have no changes between the tested source commit and upstream main at 22ec8bdb2f7d4626587f23b1eed4a207df600386.

bsakaguchi closed this issue

2026-10-16 21:06:51 +00:00

bsakaguchi commented

2026-10-16 21:06:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Confirmed: adding a trace spec should not flip an unrelated dimension — the failing event (worker) belongs to the process source only, so terminal should stay unknown/unchanged instead of turning changed. Dimension-level comparison should attribute failures to their source (process vs. terminal) and leave the terminal verdict alone when no terminal observation or probe differs.

@XinwangTao commented on GitHub (Oct 15, 2026): Confirmed: adding a trace spec should not flip an unrelated dimension — the failing event (worker) belongs to the process source only, so terminal should stay unknown/unchanged instead of turning changed. Dimension-level comparison should attribute failures to their source (process vs. terminal) and leave the terminal verdict alone when no terminal observation or probe differs.

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:12:14 +00:00

[PR #1539] [MERGED] fix(javascript): avoid argument limits when expanding escaped receivers #2955