morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5923] [Bug] Trace comparisons report no first divergence for ordering failures and missing journals #3600

New issue

Closed

opened 2026-10-31 00:42:16 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:16 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5923

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

The built compare-process-captures CLI emits first_divergence: {"status":"none"} for two trace comparisons that cannot establish an absence of divergence:

Identical per-dimension observations with opposite cross-source journal order produce overall changed, terminal/process changed, and trace different. The trace diagnostic identifies the violated ready -> worker edge, while first divergence says none.
An otherwise identical capture with an empty event journal produces overall unknown, terminal/process unknown, and trace unknown. The journal diagnostic states that the cross-source journal is missing, while first divergence again says none.

Both inputs pass the production Evidence and capture parsers. The capture fixtures have matching comparison commitments and normalization, no truncation, and no residual unknown entries. All CLI calls exit 0 and return derived comparison Evidence.

Reproduction

Dependencies and the compiled runtime were already prepared at the revision below; no installation or build was rerun. This uses synthetic capture fixtures rather than recording a live target.

Save the following as src/domain/process/traceLocalization.repro.test.ts in a disposable checkout. The assertions deliberately record the current incorrect none result, together with two controls. This focused test was executed; the suggested filename differs from the temporary filename used for the run.

import { expect, it } from "vitest";
import { execFileSync } from "node:child_process";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { capture, values, partialSpecification } from "./processTraceComparison.fixture.js";
import { parseProcessCapture } from "./processCaptureParsing.js";
import { createEvidence } from "../evidence.js";
import { PROCESS_PROVIDER } from "./processEvidenceProvider.js";

it("publishes trace-only and missing-journal comparisons through the real CLI", () => {
  const directory = mkdtempSync(join(process.env.REA_REPRO_ROOT!, "localization-"));
  try {
  const left = capture(values(["terminal", "process"]));
  const right = capture(values(["process", "terminal"]));
  const missing = parseProcessCapture({ ...left, event_journal: [] });
  for (const [name, result] of Object.entries({ left, right, missing })) {
    const evidence = createEvidence(undefined, PROCESS_PROVIDER, {
      predicateType: "rea.process-capture", operation: "capture_process_scenario",
      parameters: {}, result, confidence: "observed", authority: "controlled-replay", limitations: [],
    });
    writeFileSync(join(directory, `${name}.json`), JSON.stringify(evidence));
  }
  writeFileSync(join(directory, "specification.json"), JSON.stringify(partialSpecification()));
  for (const [label, lhs, rhs, useTrace, expected] of [
    ["ordered-control", "left", "left", true, "unchanged"],
    ["exact-control", "left", "right", false, "unchanged"],
    ["ordering", "left", "right", true, "changed"],
    ["missing-journal", "missing", "left", true, "unknown"],
  ] as const) {
    const stdout = execFileSync(process.execPath, ["scripts/rea.mjs", "compare-process-captures",
      join(directory, `${lhs}.json`), join(directory, `${rhs}.json`),
      ...(useTrace ? [join(directory, "specification.json")] : []), "--json"], { encoding: "utf8", timeout: 30000 });
    writeFileSync(join(directory, `${label}.output.json`), stdout);
    const result = JSON.parse(stdout).normalized_result;
    console.log(JSON.stringify({ label, ...result }));
    expect(result.status).toBe(expected);
    expect(result.first_divergence).toEqual({ status: "none" });
  }
  } finally {
    rmSync(directory, { recursive: true, force: true });
  }
}, 120000);

Run it from the repository root with a disposable directory for the fixture files:

export REA_REPRO_ROOT="$(mktemp -d)"
node node_modules/vitest/vitest.mjs run src/domain/process/traceLocalization.repro.test.ts --maxWorkers=1
rmdir "$REA_REPRO_ROOT"

The test invokes four real CLI subprocesses and removes their fixture directories. Each command has the documented shape:

node scripts/rea.mjs compare-process-captures left.json right.json specification.json --json

The exact control omits the third positional argument. The missing-journal case uses the parsed capture whose event_journal is empty.

Expected behavior

For the ordering case, preserve the correct changed statuses and the violated-edge diagnostic, but report the first divergence as unknown with an explanation if the current location representation cannot express that cross-source ordering difference. For the missing-journal case, preserve the unknown trace result and report unknown localization with its missing-evidence reason. Neither case should claim none.

The documented comparison contract reports differences with locations, uses explicit ordering constraints, and returns unknown when a required complete journal is missing. The coverage contract prevents incomplete evidence from locating a first divergence across all dimensions. The existing ordering test establishes that this ordering change is a real difference in both participating dimensions.

Impact and scope

This is a bounded P2 result-integrity defect in first-divergence localization. Consumers using that field as a summary can report no divergence despite a detected ordering failure, or despite missing ordering evidence. The overall classification and detailed trace diagnostic remain correct in the executed cases, and the input Evidence is preserved.

A practical workaround is to inspect trace.verdict and trace.diagnostic alongside status instead of interpreting first_divergence.status: none as proof that no divergence occurred. The retained diagnostic identifies the violated edge or missing journal. Both controls show why simply dropping the trace specification would lose the requested ordering analysis.

The verified scope is these synthetic, valid comparison inputs. Frequency in live process captures and downstream effects were not measured. No data mutation or security boundary crossing was observed.

REA version

rea-agents 6.7.0, source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; reproduced with Node 26.12.0, Node 24.22.0, and the repository-pinned Node 24.18.0; Vitest 5.0.3.

Target and artifact context

Synthetic canonical process-capture Evidence built from the repository's terminal/process trace fixtures, with one terminal frame and one process sample per complete capture. No live target, PTY, debugger, or external analysis provider ran.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Each focused run passed one test and executed four successful CLI subprocesses. Relevant fields from the returned normalized_result:

ordered-control:
  status=unchanged; terminal=unchanged; process=unchanged
  first_divergence={"status":"none"}; trace.verdict=equivalent

exact-control, opposite journals without a trace specification:
  status=unchanged; terminal=unchanged; process=unchanged
  first_divergence={"status":"none"}

ordering, opposite journals with the partial-order specification:
  status=changed; terminal=changed; process=changed
  first_divergence={"status":"none"}; trace.verdict=different
  trace.diagnostic.kind=edge; side=right
  message="ready must happen before worker."
  event_ids=["ready","worker"]
  locations=[
    {"collection":"frames","index":0,"capture_order":2},
    {"collection":"process_samples","index":0,"capture_order":1}
  ]

missing-journal, with the same specification:
  status=unknown; terminal=unknown; process=unknown
  first_divergence={"status":"none"}; trace.verdict=unknown
  trace.diagnostic.kind=journal; side=left
  message="Capture has no cross-source event journal."

Interaction, exit, and filesystem remain unchanged in every case. The CLI registration and application workflow establish the supported route exercised by these calls.

Typed outcome or recovery expected

Return first_divergence: {"status":"unknown","reason":...} when this trace difference or journal uncertainty cannot be localized by the existing first-divergence representation. Keep the aggregate statuses, diagnostic evidence, and both unchanged controls intact.

Analysis or suspected cause

Source-derived: trace verdict projection updates the dimension statuses using journal-based trace evaluation. The localization calculation instead compares per-dimension observation collections and skips unknown dimensions. Those collections do not encode cross-source journal order. With identical collections, no candidate is found; the fallback checks truncation, normalization, and residual unknown arrays but misses the trace-only failure or uncertainty.

This explains the observed output without requiring the correct trace verdict to change. Adding a conservative localization fallback at this producing boundary would avoid claiming absence when the requested trace comparison did not establish it.

Additional context

Related reports concern different contracts: #2645 concerns ranking interaction differences by dispatch time; #2616, #3181, and #3931 concern dimension attribution; #3180 concerns ignored filesystem timestamps; #3980 concerns downstream reconstruction verification; #5916 concerns duplicate finite-trace admission. These do not cover the two localization results above.

After fetching upstream main at bcfcd8b6e2ee0c44c7ac656688db5ceb61a847d5, the comparison producer, trace evaluator, and CLI application workflow had no changes from the tested revision. This is a source freshness check; the upstream runtime was not rebuilt or executed. MCP transport, live capture, other platforms, and the full suite were not run.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5923 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior The built `compare-process-captures` CLI emits `first_divergence: {"status":"none"}` for two trace comparisons that cannot establish an absence of divergence: - Identical per-dimension observations with opposite cross-source journal order produce overall `changed`, terminal/process `changed`, and trace `different`. The trace diagnostic identifies the violated `ready -> worker` edge, while first divergence says `none`. - An otherwise identical capture with an empty event journal produces overall `unknown`, terminal/process `unknown`, and trace `unknown`. The journal diagnostic states that the cross-source journal is missing, while first divergence again says `none`. Both inputs pass the production Evidence and capture parsers. The capture fixtures have matching comparison commitments and normalization, no truncation, and no residual unknown entries. All CLI calls exit 0 and return derived comparison Evidence. ### Reproduction Dependencies and the compiled runtime were already prepared at the revision below; no installation or build was rerun. This uses synthetic capture fixtures rather than recording a live target. Save the following as `src/domain/process/traceLocalization.repro.test.ts` in a disposable checkout. The assertions deliberately record the current incorrect `none` result, together with two controls. This focused test was executed; the suggested filename differs from the temporary filename used for the run. ```ts import { expect, it } from "vitest"; import { execFileSync } from "node:child_process"; import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; import { join } from "node:path"; import { capture, values, partialSpecification } from "./processTraceComparison.fixture.js"; import { parseProcessCapture } from "./processCaptureParsing.js"; import { createEvidence } from "../evidence.js"; import { PROCESS_PROVIDER } from "./processEvidenceProvider.js"; it("publishes trace-only and missing-journal comparisons through the real CLI", () => { const directory = mkdtempSync(join(process.env.REA_REPRO_ROOT!, "localization-")); try { const left = capture(values(["terminal", "process"])); const right = capture(values(["process", "terminal"])); const missing = parseProcessCapture({ ...left, event_journal: [] }); for (const [name, result] of Object.entries({ left, right, missing })) { const evidence = createEvidence(undefined, PROCESS_PROVIDER, { predicateType: "rea.process-capture", operation: "capture_process_scenario", parameters: {}, result, confidence: "observed", authority: "controlled-replay", limitations: [], }); writeFileSync(join(directory, `${name}.json`), JSON.stringify(evidence)); } writeFileSync(join(directory, "specification.json"), JSON.stringify(partialSpecification())); for (const [label, lhs, rhs, useTrace, expected] of [ ["ordered-control", "left", "left", true, "unchanged"], ["exact-control", "left", "right", false, "unchanged"], ["ordering", "left", "right", true, "changed"], ["missing-journal", "missing", "left", true, "unknown"], ] as const) { const stdout = execFileSync(process.execPath, ["scripts/rea.mjs", "compare-process-captures", join(directory, `${lhs}.json`), join(directory, `${rhs}.json`), ...(useTrace ? [join(directory, "specification.json")] : []), "--json"], { encoding: "utf8", timeout: 30000 }); writeFileSync(join(directory, `${label}.output.json`), stdout); const result = JSON.parse(stdout).normalized_result; console.log(JSON.stringify({ label, ...result })); expect(result.status).toBe(expected); expect(result.first_divergence).toEqual({ status: "none" }); } } finally { rmSync(directory, { recursive: true, force: true }); } }, 120000); ``` Run it from the repository root with a disposable directory for the fixture files: ```sh export REA_REPRO_ROOT="$(mktemp -d)" node node_modules/vitest/vitest.mjs run src/domain/process/traceLocalization.repro.test.ts --maxWorkers=1 rmdir "$REA_REPRO_ROOT" ``` The test invokes four real CLI subprocesses and removes their fixture directories. Each command has the documented shape: ```sh node scripts/rea.mjs compare-process-captures left.json right.json specification.json --json ``` The exact control omits the third positional argument. The missing-journal case uses the parsed capture whose `event_journal` is empty. ### Expected behavior For the ordering case, preserve the correct changed statuses and the violated-edge diagnostic, but report the first divergence as `unknown` with an explanation if the current location representation cannot express that cross-source ordering difference. For the missing-journal case, preserve the unknown trace result and report unknown localization with its missing-evidence reason. Neither case should claim `none`. The [documented comparison contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/process-capture.md#L80-L91) reports differences with locations, uses explicit ordering constraints, and returns unknown when a required complete journal is missing. The [coverage contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/process-capture.md#L447-L453) prevents incomplete evidence from locating a first divergence across all dimensions. The [existing ordering test](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/process/processTraceComparison.test.ts#L654-L666) establishes that this ordering change is a real difference in both participating dimensions. ### Impact and scope This is a bounded P2 result-integrity defect in first-divergence localization. Consumers using that field as a summary can report no divergence despite a detected ordering failure, or despite missing ordering evidence. The overall classification and detailed trace diagnostic remain correct in the executed cases, and the input Evidence is preserved. A practical workaround is to inspect `trace.verdict` and `trace.diagnostic` alongside `status` instead of interpreting `first_divergence.status: none` as proof that no divergence occurred. The retained diagnostic identifies the violated edge or missing journal. Both controls show why simply dropping the trace specification would lose the requested ordering analysis. The verified scope is these synthetic, valid comparison inputs. Frequency in live process captures and downstream effects were not measured. No data mutation or security boundary crossing was observed. ### REA version rea-agents 6.7.0, source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; reproduced with Node 26.12.0, Node 24.22.0, and the repository-pinned Node 24.18.0; Vitest 5.0.3. ### Target and artifact context Synthetic canonical process-capture Evidence built from the repository's terminal/process trace fixtures, with one terminal frame and one process sample per complete capture. No live target, PTY, debugger, or external analysis provider ran. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Each focused run passed one test and executed four successful CLI subprocesses. Relevant fields from the returned `normalized_result`: ```text ordered-control: status=unchanged; terminal=unchanged; process=unchanged first_divergence={"status":"none"}; trace.verdict=equivalent exact-control, opposite journals without a trace specification: status=unchanged; terminal=unchanged; process=unchanged first_divergence={"status":"none"} ordering, opposite journals with the partial-order specification: status=changed; terminal=changed; process=changed first_divergence={"status":"none"}; trace.verdict=different trace.diagnostic.kind=edge; side=right message="ready must happen before worker." event_ids=["ready","worker"] locations=[ {"collection":"frames","index":0,"capture_order":2}, {"collection":"process_samples","index":0,"capture_order":1} ] missing-journal, with the same specification: status=unknown; terminal=unknown; process=unknown first_divergence={"status":"none"}; trace.verdict=unknown trace.diagnostic.kind=journal; side=left message="Capture has no cross-source event journal." ``` Interaction, exit, and filesystem remain unchanged in every case. The [CLI registration](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/cli/processCommands.ts#L32-L56) and [application workflow](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/process/ProcessCli.ts#L120-L166) establish the supported route exercised by these calls. ### Typed outcome or recovery expected Return `first_divergence: {"status":"unknown","reason":...}` when this trace difference or journal uncertainty cannot be localized by the existing first-divergence representation. Keep the aggregate statuses, diagnostic evidence, and both unchanged controls intact. ### Analysis or suspected cause Source-derived: [trace verdict projection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/process/processComparison.ts#L317-L378) updates the dimension statuses using journal-based trace evaluation. The [localization calculation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/process/processComparison.ts#L472-L514) instead compares per-dimension observation collections and skips unknown dimensions. Those collections do not encode cross-source journal order. With identical collections, no candidate is found; the fallback checks truncation, normalization, and residual unknown arrays but misses the trace-only failure or uncertainty. This explains the observed output without requiring the correct trace verdict to change. Adding a conservative localization fallback at this producing boundary would avoid claiming absence when the requested trace comparison did not establish it. ### Additional context Related reports concern different contracts: #2645 concerns ranking interaction differences by dispatch time; #2616, #3181, and #3931 concern dimension attribution; #3180 concerns ignored filesystem timestamps; #3980 concerns downstream reconstruction verification; #5916 concerns duplicate finite-trace admission. These do not cover the two localization results above. After fetching upstream main at `bcfcd8b6e2ee0c44c7ac656688db5ceb61a847d5`, the comparison producer, trace evaluator, and CLI application workflow had no changes from the tested revision. This is a source freshness check; the upstream runtime was not rebuilt or executed. MCP transport, live capture, other platforms, and the full suite were not run.

bsakaguchi closed this issue

2026-10-31 00:42:16 +00:00

bsakaguchi commented

2026-10-31 00:42:17 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

With no observed per-source first difference, non-equivalent or unknown trace results now yield first_divergence unknown rather than none. Separate regressions check genuine cross-source reorder, missing journal and an equivalent control.

Implementation: processComparison.ts, processTraceLifecycleFollowup.test.ts.

The relevant regression checks in processTraceLifecycleFollowup.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. With no observed per-source first difference, non-equivalent or unknown trace results now yield first_divergence unknown rather than none. Separate regressions check genuine cross-source reorder, missing journal and an equivalent control. Implementation: [processComparison.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/process/processComparison.ts#L490), [processTraceLifecycleFollowup.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/process/processTraceLifecycleFollowup.test.ts#L121). The relevant regression checks in [processTraceLifecycleFollowup.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/process/processTraceLifecycleFollowup.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:09:53 +00:00

[PR #3600] [MERGED] fix(contracts): reject NUL bytes in objdump and rizin local path inputs #5846

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425