morluto/rea
1
0
Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00
Code Issues 1.3k Projects Packages Wiki Activity

[GH-ISSUE #724] [Feature] Explain content, exit outcome, and temporal differences in process comparisons #322

New issue
Open
opened 2026-10-15 18:23:52 +00:00 by bsakaguchi · 12 comments
bsakaguchi commented 2026-10-15 18:23:52 +00:00
Owner
Copy link

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/724

Before submitting

  • I searched the open and closed issues and relevant pull requests for a related report.
  • I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Current limitation

Two captures of the exact same Node command had identical raw terminal text, final rendered lines, and exit objects {code: 0, signal: 0, reason: "exited"}. In a fresh connection the comparison nevertheless reported terminal=changed and exit=changed. first_divergence showed the same normalized command/PIDs at different sample times; process remained unknown.

A new identical pair with time_bucket_ms=1000 reported terminal=unchanged but exit=changed with identical exit objects. Settlement elapsed time differed (182 ms versus 86 ms), and first_divergence pointed to timestamps of empty filesystem checkpoints although filesystem observations were unknown.

Temporal capture differences are real. This request concerns explanation of comparison facets, not a claim that the entire traces are equivalent or that elapsed time may be discarded.

Use cases

A reconstruction analyst asks whether a command changed its printed result or exit outcome. The result should distinguish those facts from timing/settlement/sample variation and identify what remains unobserved.

Desired outcome and success criteria

  • Explain each changed dimension using the compared facet/path and observed left/right values.
  • Make unchanged output content and exit outcome visible independently of temporal/lifecycle changes.
  • Retain raw/normalized time observations and normalization commitments.
  • Do not use an unobserved filesystem/process facet as proof of a semantic change or equivalence.
  • Exercise an identical-output/identical-exit control, a true output difference, a true exit-code difference, and incomplete capture coverage through CLI/MCP.
  • Caller-selected normalization remains explicit and cannot hide meaningful differences without being reflected in the result.

Authority boundaries and non-goals

No additional execution, mutation, replay engine, automatic timing suppression, causal inference from wall-clock order, or equivalence claim for missing dimensions. Preserve complete observations and unknowns.

Alternatives considered

Raising the time bucket made terminal unchanged in one control, but did not explain the exit label. Inspecting entire captures manually recovers facts but leaves the comparison headline ambiguous.

Possible direction

Add precise outcome/content/timing facets or per-dimension explanations to the existing comparison contract; keep current evidence and provenance and evaluate compatibility before changing aggregate statuses.

References

Hands-on public CLI/MCP evaluation, findings F09. Baseline: REA 4.0.1, source revision c3bc38bf6c74d2175a4680530dcadd6cab0e7791; evaluation date 2026-10-06.

#61 previously considered repeatability and temporal variance; #558 addresses definite conclusions from partial coverage and has a different remaining scope. docs/process-capture.md

Related evaluation issues: #719.

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/724 ### Before submitting - [x] I searched the open and closed issues and relevant pull requests for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Current limitation Two captures of the exact same Node command had identical raw terminal text, final rendered lines, and exit objects `{code: 0, signal: 0, reason: "exited"}`. In a fresh connection the comparison nevertheless reported terminal=changed and exit=changed. first_divergence showed the same normalized command/PIDs at different sample times; process remained unknown. A new identical pair with time_bucket_ms=1000 reported terminal=unchanged but exit=changed with identical exit objects. Settlement elapsed time differed (182 ms versus 86 ms), and first_divergence pointed to timestamps of empty filesystem checkpoints although filesystem observations were unknown. Temporal capture differences are real. This request concerns explanation of comparison facets, not a claim that the entire traces are equivalent or that elapsed time may be discarded. ### Use cases A reconstruction analyst asks whether a command changed its printed result or exit outcome. The result should distinguish those facts from timing/settlement/sample variation and identify what remains unobserved. ### Desired outcome and success criteria - Explain each changed dimension using the compared facet/path and observed left/right values. - Make unchanged output content and exit outcome visible independently of temporal/lifecycle changes. - Retain raw/normalized time observations and normalization commitments. - Do not use an unobserved filesystem/process facet as proof of a semantic change or equivalence. - Exercise an identical-output/identical-exit control, a true output difference, a true exit-code difference, and incomplete capture coverage through CLI/MCP. - Caller-selected normalization remains explicit and cannot hide meaningful differences without being reflected in the result. ### Authority boundaries and non-goals No additional execution, mutation, replay engine, automatic timing suppression, causal inference from wall-clock order, or equivalence claim for missing dimensions. Preserve complete observations and unknowns. ### Alternatives considered Raising the time bucket made terminal unchanged in one control, but did not explain the exit label. Inspecting entire captures manually recovers facts but leaves the comparison headline ambiguous. ### Possible direction Add precise outcome/content/timing facets or per-dimension explanations to the existing comparison contract; keep current evidence and provenance and evaluate compatibility before changing aggregate statuses. ### References Hands-on public CLI/MCP evaluation, findings F09. Baseline: REA 4.0.1, source revision [c3bc38bf6c74d2175a4680530dcadd6cab0e7791](https://github.com/morluto/rea/commit/c3bc38bf6c74d2175a4680530dcadd6cab0e7791); evaluation date 2026-10-06. [#61](https://github.com/morluto/rea/issues/61) previously considered repeatability and temporal variance; [#558](https://github.com/morluto/rea/issues/558) addresses definite conclusions from partial coverage and has a different remaining scope. [docs/process-capture.md](https://github.com/morluto/rea/blob/c3bc38bf6c74d2175a4680530dcadd6cab0e7791/docs/process-capture.md) Related evaluation issues: [#719](https://github.com/morluto/rea/issues/719).
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@sunxiuguo commented on GitHub (Oct 6, 2026):

Source review at 9dd6cef (no runtime validation): terminal comparison includes the full timestamped frames, and exit comparison includes settlement, including elapsed_ms. This explains how the reported content and exit outcome can match while those dimensions change.

A bounded first step could add optional explanations, preserving the existing aggregate statuses and acceptance of historical captures/results.

  • Terminal: distinguish concatenated observed normalized stream, final rendered state (including size, cursor, buffer and serialized state), and frame timing/chunk boundaries. Matching the final screen would not erase earlier stream differences.
  • Exit: distinguish code/signal/reason, settlement state/cleanup outcome, and settlement duration.
  • Each explanation: include the facet/path, observed left/right values, normalization context and coverage. Preserve unknown/truncated conclusions; matching bounded observations must not imply complete equivalence. A timestamp difference from an unobserved filesystem/process facet should remain explicitly limited to observed timing.
  • Raw timing needs a separate producer change: terminal timestamps are bucketed before storage, and process samples are also normalized before persistence. Where only bucketed values exist, report raw times as unavailable rather than reconstructing them.

Regression cases should include identical content/exit with timing changes, identical stream with different chunking, different streams that render identically, true content/exit-code changes, and incomplete coverage, plus historical-payload parsing and CLI/MCP output checks.

Does this bounded additive approach fit the intended contract before any implementation changes?

<!-- gh-comment-id:6015350035 --> @sunxiuguo commented on GitHub (Oct 6, 2026): Source review at 9dd6cef (no runtime validation): [terminal comparison includes the full timestamped frames](https://github.com/morluto/rea/blob/9dd6cef90abf89f9191bce942e31900870231968/src/domain/processComparison.ts#L98-L110), and [exit comparison includes settlement](https://github.com/morluto/rea/blob/9dd6cef90abf89f9191bce942e31900870231968/src/domain/processComparison.ts#L336-L344), including elapsed_ms. This explains how the reported content and exit outcome can match while those dimensions change. A bounded first step could add optional explanations, preserving the existing aggregate statuses and acceptance of historical captures/results. - Terminal: distinguish concatenated observed normalized stream, final rendered state (including size, cursor, buffer and serialized state), and frame timing/chunk boundaries. Matching the final screen would not erase earlier stream differences. - Exit: distinguish code/signal/reason, settlement state/cleanup outcome, and settlement duration. - Each explanation: include the facet/path, observed left/right values, normalization context and coverage. Preserve unknown/truncated conclusions; matching bounded observations must not imply complete equivalence. A timestamp difference from an unobserved filesystem/process facet should remain explicitly limited to observed timing. - Raw timing needs a separate producer change: [terminal timestamps are bucketed before storage](https://github.com/morluto/rea/blob/9dd6cef90abf89f9191bce942e31900870231968/src/application/ProcessCaptureLifecycle.ts#L408-L424), and process samples are also normalized before persistence. Where only bucketed values exist, report raw times as unavailable rather than reconstructing them. Regression cases should include identical content/exit with timing changes, identical stream with different chunking, different streams that render identically, true content/exit-code changes, and incomplete coverage, plus historical-payload parsing and CLI/MCP output checks. Does this bounded additive approach fit the intended contract before any implementation changes?
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 9, 2026):

TL;DR: Still there on 6.2.0 on Linux. I captured the exact same one-line Node command twice. Most pairs read status: changed. The only value that differs is settlement.elapsed_ms. That number is not bucketed, so time_bucket_ms can't help. When the milliseconds happen to match, the verdict is unknown, never unchanged.

Runtime check of the source review above, on the published rea-agents@6.2.0. Main at f6cfbfcc5 still folds settlement into the exit dimension (src/domain/process/processComparison.ts lines 139-140 and 380-381). It also folds at_ms into each compared raw frame (processObservation.ts lines 10-14).

Scenario: node -e "console.log('hello'); process.exit(0)", defaults except timeout_ms 10000, idle_timeout_ms 5000, settle_ms 100. Captured 7 times at time_bucket_ms: 10 and 7 times at time_bucket_ms: 1000, then compared five pairs from each set with rea compare-process-captures:

time_bucket_ms identical pairs changed unknown unchanged
10 5 4 1 0
1000 5 2 3 0

So the facts the source review predicted hold at runtime on 6.2.0. The smallest split that would fix the headline here is this. Separate settlement.elapsed_ms (and probably settlement.state / cleanup_outcome) from code / signal / reason inside the exit dimension.

More detail

Every changed pair looks the same: terminal: unchanged, interaction: unchanged, exit: changed, filesystem: unknown, process: unknown, and first_divergence on exit with identical { code: 0, signal: 0, reason: "exited", settlement.state: "quiesced", cleanup_outcome: "not_required" } on both sides. The one differing value is settlement.elapsed_ms. It ranged from 99 to 105 across the 14 captures.

settlement.elapsed_ms is not bucketed: at time_bucket_ms: 1000 the frame and process-sample times all read 0, but elapsed_ms still read 99, 100 or 101. So it comes down to whether two runs settle on the same millisecond. That decides whether an identical pair compares as changed or unknown.

The unknown pairs are the ones where the milliseconds matched. They stop short of unchanged, as documented. That is because filesystem (no paths selected) and process (sampled trees) carry residual unknowns.

Terminal stayed unchanged here. The single output frame got the same bucketed time within each setting. That was 80 ms in all seven bucket-10 captures and 0 in all seven at 1000. So the terminal: changed from the report needs a frame landing near a bucket edge. A one-line program didn't hit that. Same mechanism (at_ms is part of the compared frame), just not triggered.

Two controls, same contract. Each was compared against one of the bucket-10 captures:

  • True output difference (hello world instead of hello): terminal: changed with a correct first_divergence (data: "hello\r\n" vs "hello world\r\n", both at 80 ms). But exit: changed as well, from 102 vs 103 ms.
  • True exit-code difference (process.exit(3)): exit: changed with first_divergence showing code: 0 vs code: 3. That is under the same label as the timing-only pairs. The headline can't tell the two apart without reading first_divergence.
Test setup
  • Box: Linux x86-64
  • Node: 22.22.2

🤖 Generated with Claude Code (Fable 5.1) 🦊

<!-- gh-comment-id:6087173354 --> @CassieFox23 commented on GitHub (Oct 9, 2026): **TL;DR:** Still there on 6.2.0 on Linux. I captured the exact same one-line Node command twice. Most pairs read `status: changed`. The only value that differs is `settlement.elapsed_ms`. That number is not bucketed, so `time_bucket_ms` can't help. When the milliseconds happen to match, the verdict is `unknown`, never `unchanged`. Runtime check of the source review above, on the published `rea-agents@6.2.0`. Main at f6cfbfcc5 still folds `settlement` into the exit dimension (`src/domain/process/processComparison.ts` lines 139-140 and 380-381). It also folds `at_ms` into each compared raw frame (`processObservation.ts` lines 10-14). Scenario: `node -e "console.log('hello'); process.exit(0)"`, defaults except `timeout_ms` 10000, `idle_timeout_ms` 5000, `settle_ms` 100. Captured 7 times at `time_bucket_ms: 10` and 7 times at `time_bucket_ms: 1000`, then compared five pairs from each set with `rea compare-process-captures`: | `time_bucket_ms` | identical pairs | `changed` | `unknown` | `unchanged` | | --- | --- | --- | --- | --- | | 10 | 5 | 4 | 1 | 0 | | 1000 | 5 | 2 | 3 | 0 | So the facts the source review predicted hold at runtime on 6.2.0. The smallest split that would fix the headline here is this. Separate `settlement.elapsed_ms` (and probably `settlement.state` / `cleanup_outcome`) from `code` / `signal` / `reason` inside the exit dimension. <details> <summary>More detail</summary> Every `changed` pair looks the same: `terminal: unchanged`, `interaction: unchanged`, `exit: changed`, `filesystem: unknown`, `process: unknown`, and `first_divergence` on `exit` with identical `{ code: 0, signal: 0, reason: "exited", settlement.state: "quiesced", cleanup_outcome: "not_required" }` on both sides. The one differing value is `settlement.elapsed_ms`. It ranged from 99 to 105 across the 14 captures. `settlement.elapsed_ms` is not bucketed: at `time_bucket_ms: 1000` the frame and process-sample times all read 0, but `elapsed_ms` still read 99, 100 or 101. So it comes down to whether two runs settle on the same millisecond. That decides whether an identical pair compares as `changed` or `unknown`. The `unknown` pairs are the ones where the milliseconds matched. They stop short of `unchanged`, as documented. That is because `filesystem` (no paths selected) and `process` (sampled trees) carry residual unknowns. Terminal stayed `unchanged` here. The single output frame got the same bucketed time within each setting. That was 80 ms in all seven bucket-10 captures and 0 in all seven at 1000. So the `terminal: changed` from the report needs a frame landing near a bucket edge. A one-line program didn't hit that. Same mechanism (`at_ms` is part of the compared frame), just not triggered. Two controls, same contract. Each was compared against one of the bucket-10 captures: - True output difference (`hello world` instead of `hello`): `terminal: changed` with a correct `first_divergence` (`data: "hello\r\n"` vs `"hello world\r\n"`, both at 80 ms). But `exit: changed` as well, from 102 vs 103 ms. - True exit-code difference (`process.exit(3)`): `exit: changed` with `first_divergence` showing `code: 0` vs `code: 3`. That is under the same label as the timing-only pairs. The headline can't tell the two apart without reading `first_divergence`. </details> <details> <summary>Test setup</summary> - Box: Linux x86-64 - Node: 22.22.2 </details> 🤖 Generated with Claude Code (Fable 5.1) 🦊
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@sunxiuguo commented on GitHub (Oct 10, 2026):

Thanks @CassieFox23. I independently reproduced this using the CLI built from current main 7aa4d768 (source version 6.3.0, Linux x64, Node 24.19.0), with the same one-line Node command and timeout/settlement settings.

Across six adjacent pairs at each bucket size:

  • time_bucket_ms: 10: five changed, one unknown.
  • time_bucket_ms: 1000: four changed, two unknown; terminal unchanged in all six. One pair differed only within the compared exit observation at settlement.elapsed_ms: 68 versus 65 ms.
  • I also hit the terminal timing case: identical hello\r\n at 30 versus 40 ms, with the exit dimension unchanged.
  • True output and exit-code-3 controls remained changed. Matching observed dimensions still yielded overall unknown because filesystem/process coverage was incomplete.

There is already an explicit opt-in for the settlement-only case: a lifecycle trace declaring the expected exit and settlement events, with ignore_fields: ["elapsed_ms"] on both declarations. For a 65/66 ms pair this returned exit: unchanged, trace equivalent, and overall unknown; the code-3 control still returned exit: changed, trace different. The compiled stdio MCP gave the same comparison results. This does not suppress terminal timing or establish whole-capture equivalence. The first attempt with the ignore field on only one event was correctly rejected because declarations from one source must use the same ignored fields.

Before changing default semantics, would you prefer the existing exit dimension to remain an exact lifecycle comparison with additive outcome/timing explanations, or split outcome and settlement/timing into separate dimensions? Simply dropping elapsed time from the current comparison would lose a reported difference. I have not changed the runtime contract.

AI-assisted investigation. The source build and six existing focused comparison/trace tests passed; no full-suite or non-Linux verification was performed.

Standalone synthetic reproduction and explicit lifecycle trace

Run from a checkout built at the commit above (npm ci && npm run build:cached), on a supported Linux/macOS PTY host. node must resolve to a supported Node version. Use a new sibling directory named rea-724-repro; all input and output files below are synthetic. The capture commands must run from the same working directory.

mkdir ../rea-724-repro
cat > ../rea-724-repro/scenario.json <<'JSON'
{
  "executable": "node",
  "arguments": ["-e", "console.log('hello'); process.exit(0)"],
  "timeout_ms": 10000,
  "idle_timeout_ms": 5000,
  "settle_ms": 100,
  "normalization": {
    "paths": true,
    "pids": true,
    "ports": true,
    "time_bucket_ms": 1000,
    "patterns": []
  }
}
JSON
cat > ../rea-724-repro/lifecycle.json <<'JSON'
{
  "events": [
    {
      "id": "exit",
      "source": "lifecycle",
      "exact": { "event": "exit", "code": 0, "signal": 0, "reason": "exited" },
      "ignore_fields": ["elapsed_ms"]
    },
    {
      "id": "settlement",
      "source": "lifecycle",
      "exact": {
        "event": "settlement",
        "state": "quiesced",
        "cleanup_outcome": "not_required"
      },
      "ignore_fields": ["elapsed_ms"]
    }
  ],
  "language": {
    "kind": "finite_traces",
    "variants": [{ "id": "normal", "trace": ["exit", "settlement"] }]
  }
}
JSON
node scripts/rea.mjs capture-process ../rea-724-repro/scenario.json --json > ../rea-724-repro/left.json
node scripts/rea.mjs capture-process ../rea-724-repro/scenario.json --json > ../rea-724-repro/right.json
node scripts/rea.mjs compare-process-captures ../rea-724-repro/left.json ../rea-724-repro/right.json --json > ../rea-724-repro/exact.json
node scripts/rea.mjs compare-process-captures ../rea-724-repro/left.json ../rea-724-repro/right.json ../rea-724-repro/lifecycle.json --json > ../rea-724-repro/declared.json

Compare normalized_result in exact.json and declared.json. The elapsed values are measured, so two fresh captures may happen to match; repeat the captures if you want to observe a timing-only difference. The expected lifecycle events here are deliberately specific to this synthetic command. A different exit, settlement state, cleanup outcome, or undeclared lifecycle event must not be silently accepted. Both lifecycle declarations need the same ignore_fields list. Terminal timing/content and residual filesystem/process unknowns remain outside this lifecycle-only equivalence claim.

<!-- gh-comment-id:6092233447 --> @sunxiuguo commented on GitHub (Oct 10, 2026): Thanks @CassieFox23. I independently reproduced this using the CLI built from current main [7aa4d768](https://github.com/morluto/rea/commit/7aa4d768eb15317a63a476431ed75bafec086033) (source version 6.3.0, Linux x64, Node 24.19.0), with the same one-line Node command and timeout/settlement settings. Across six adjacent pairs at each bucket size: - `time_bucket_ms: 10`: five `changed`, one `unknown`. - `time_bucket_ms: 1000`: four `changed`, two `unknown`; terminal unchanged in all six. One pair differed only within the compared exit observation at `settlement.elapsed_ms`: 68 versus 65 ms. - I also hit the terminal timing case: identical `hello\r\n` at 30 versus 40 ms, with the exit dimension unchanged. - True output and exit-code-3 controls remained changed. Matching observed dimensions still yielded overall `unknown` because filesystem/process coverage was incomplete. There is already an explicit opt-in for the settlement-only case: a lifecycle trace declaring the expected exit and settlement events, with `ignore_fields: ["elapsed_ms"]` on both declarations. For a 65/66 ms pair this returned `exit: unchanged`, trace `equivalent`, and overall `unknown`; the code-3 control still returned `exit: changed`, trace `different`. The compiled stdio MCP gave the same comparison results. This does not suppress terminal timing or establish whole-capture equivalence. The first attempt with the ignore field on only one event was correctly rejected because declarations from one source must use the same ignored fields. Before changing default semantics, would you prefer the existing `exit` dimension to remain an exact lifecycle comparison with additive outcome/timing explanations, or split outcome and settlement/timing into separate dimensions? Simply dropping elapsed time from the current comparison would lose a reported difference. I have not changed the runtime contract. AI-assisted investigation. The source build and six existing focused comparison/trace tests passed; no full-suite or non-Linux verification was performed. <details> <summary>Standalone synthetic reproduction and explicit lifecycle trace</summary> Run from a checkout built at the commit above (`npm ci && npm run build:cached`), on a supported Linux/macOS PTY host. `node` must resolve to a supported Node version. Use a new sibling directory named `rea-724-repro`; all input and output files below are synthetic. The capture commands must run from the same working directory. ```sh mkdir ../rea-724-repro cat > ../rea-724-repro/scenario.json <<'JSON' { "executable": "node", "arguments": ["-e", "console.log('hello'); process.exit(0)"], "timeout_ms": 10000, "idle_timeout_ms": 5000, "settle_ms": 100, "normalization": { "paths": true, "pids": true, "ports": true, "time_bucket_ms": 1000, "patterns": [] } } JSON cat > ../rea-724-repro/lifecycle.json <<'JSON' { "events": [ { "id": "exit", "source": "lifecycle", "exact": { "event": "exit", "code": 0, "signal": 0, "reason": "exited" }, "ignore_fields": ["elapsed_ms"] }, { "id": "settlement", "source": "lifecycle", "exact": { "event": "settlement", "state": "quiesced", "cleanup_outcome": "not_required" }, "ignore_fields": ["elapsed_ms"] } ], "language": { "kind": "finite_traces", "variants": [{ "id": "normal", "trace": ["exit", "settlement"] }] } } JSON node scripts/rea.mjs capture-process ../rea-724-repro/scenario.json --json > ../rea-724-repro/left.json node scripts/rea.mjs capture-process ../rea-724-repro/scenario.json --json > ../rea-724-repro/right.json node scripts/rea.mjs compare-process-captures ../rea-724-repro/left.json ../rea-724-repro/right.json --json > ../rea-724-repro/exact.json node scripts/rea.mjs compare-process-captures ../rea-724-repro/left.json ../rea-724-repro/right.json ../rea-724-repro/lifecycle.json --json > ../rea-724-repro/declared.json ``` Compare `normalized_result` in `exact.json` and `declared.json`. The elapsed values are measured, so two fresh captures may happen to match; repeat the captures if you want to observe a timing-only difference. The expected lifecycle events here are deliberately specific to this synthetic command. A different exit, settlement state, cleanup outcome, or undeclared lifecycle event must not be silently accepted. Both lifecycle declarations need the same `ignore_fields` list. Terminal timing/content and residual filesystem/process unknowns remain outside this lifecycle-only equivalence claim. </details>
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: The workaround works. I tried it on three Node versions. Each time, the exit check stopped flagging timing. A real exit-code change still showed up.

I ran your exact files on the released 6.3.0 CLI. I took four captures per Node and compared three pairs.

Node Plain With lifecycle Code 3
22.23.1 3 changed, elapsed_ms only 3/3 exit unchanged, trace equivalent changed, trace different
24.18.0 1 changed, 2 unknown 3/3 exit unchanged, trace equivalent changed, trace different
26.11.1 2 changed, 1 unknown 3/3 exit unchanged, trace equivalent changed, trace different

Overall stayed unknown with the opt-in, as you said. The plain unknown pairs were the ones whose milliseconds matched. Terminal stayed unchanged in all 21 compares. At a 1000 ms bucket every frame read 0. So this run did not test terminal timing. The opt-in does not claim to cover that.

Thanks, sunxiuguo.

🤖 Generated with Claude Code (Opus 5.5) 🦊

<!-- gh-comment-id:6092352098 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** The workaround works. I tried it on three Node versions. Each time, the exit check stopped flagging timing. A real exit-code change still showed up. I ran your exact files on the released 6.3.0 CLI. I took four captures per Node and compared three pairs. | Node | Plain | With lifecycle | Code 3 | | --- | --- | --- | --- | | 22.23.1 | 3 `changed`, `elapsed_ms` only | 3/3 exit `unchanged`, trace `equivalent` | `changed`, trace `different` | | 24.18.0 | 1 `changed`, 2 `unknown` | 3/3 exit `unchanged`, trace `equivalent` | `changed`, trace `different` | | 26.11.1 | 2 `changed`, 1 `unknown` | 3/3 exit `unchanged`, trace `equivalent` | `changed`, trace `different` | Overall stayed `unknown` with the opt-in, as you said. The plain `unknown` pairs were the ones whose milliseconds matched. Terminal stayed `unchanged` in all 21 compares. At a 1000 ms bucket every frame read 0. So this run did not test terminal timing. The opt-in does not claim to cover that. Thanks, sunxiuguo. 🤖 Generated with Claude Code (Opus 5.5) 🦊
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: Terminal timing flips too, at a 1 ms bucket. The lifecycle file does not touch it. A trace file that also declares the terminal events does fix it, with two catches.

Follow-up on released 6.3.0. Six captures per bucket, five pairs each. Output was always hello\r\n. The table shows terminal status.

Bucket Plain Lifecycle at_ms hullo
1 11/15 changed same 11/15 59 vs 62, 97 vs 105, 35 vs 36 changed
10 0/15 changed 0/15 all equal (60, 100, 30) changed

At 10 the frames sat mid-bucket here, so an edge case can still flip it. No flag or normalization setting covers frame timing. The trace file schema does accept terminal_raw and terminal_rendered events with ignore_fields: ["sequence", "at_ms"]. Declaring both kinds gave terminal unchanged on all 30 pairs. The hullo control still came back changed. Catch one: the screen event and exit race, so it needs two trace variants. Catch two: a failed trace marks every declared dimension changed. So hullo also flipped exit, with equal elapsed_ms.

Test setup
  • Node: 24.18.0, 22.23.1 and 26.11.1

🤖 Generated with Claude Code (Opus 5.5) 🦊

<!-- gh-comment-id:6092493334 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** Terminal timing flips too, at a 1 ms bucket. The lifecycle file does not touch it. A trace file that also declares the terminal events does fix it, with two catches. Follow-up on released 6.3.0. Six captures per bucket, five pairs each. Output was always `hello\r\n`. The table shows terminal status. | Bucket | Plain | Lifecycle | `at_ms` | `hullo` | | --- | --- | --- | --- | --- | | 1 | 11/15 `changed` | same 11/15 | 59 vs 62, 97 vs 105, 35 vs 36 | `changed` | | 10 | 0/15 `changed` | 0/15 | all equal (60, 100, 30) | `changed` | At 10 the frames sat mid-bucket here, so an edge case can still flip it. No flag or normalization setting covers frame timing. The trace file schema does accept `terminal_raw` and `terminal_rendered` events with `ignore_fields: ["sequence", "at_ms"]`. Declaring both kinds gave terminal `unchanged` on all 30 pairs. The `hullo` control still came back `changed`. Catch one: the screen event and exit race, so it needs two trace variants. Catch two: a failed trace marks every declared dimension `changed`. So `hullo` also flipped exit, with equal `elapsed_ms`. <details> <summary>Test setup</summary> - Node: 24.18.0, 22.23.1 and 26.11.1 </details> 🤖 Generated with Claude Code (Opus 5.5) 🦊
bsakaguchi commented 2026-10-15 18:23:53 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: One piece of history that may help here. REA once had a runner that sorted timing noise from real change. It was removed as unused on 2026-09-29.

This may be obvious, since morluto wrote that code. I'm adding it so the thread has the link.

  • #61 asked for a "paired experiment" runner. It ran each side more than once first. Then it sorted timing variance from real change.
  • Commit e1d755f7 removed it: ProcessPairedExperiment.ts, its domain file and its tests.
    github.com/morluto/rea@e1d755f7cd
  • #61 and #558 also explain the unknown results here. Partial evidence must stay unknown. It is never called unchanged. So that part looks like design, not a bug.

So this request may be a smaller slice of that old idea. It would explain timing, not hide it.

🤖 Generated with Claude Code (Opus 5.5) 🦊

<!-- gh-comment-id:6092537827 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** One piece of history that may help here. REA once had a runner that sorted timing noise from real change. It was removed as unused on 2026-09-29. This may be obvious, since morluto wrote that code. I'm adding it so the thread has the link. - #61 asked for a "paired experiment" runner. It ran each side more than once first. Then it sorted timing variance from real change. - Commit e1d755f7 removed it: `ProcessPairedExperiment.ts`, its domain file and its tests. https://github.com/morluto/rea/commit/e1d755f7cd574860f59965a6244dbd43f5032f2c - #61 and #558 also explain the `unknown` results here. Partial evidence must stay unknown. It is never called unchanged. So that part looks like design, not a bug. So this request may be a smaller slice of that old idea. It would explain timing, not hide it. 🤖 Generated with Claude Code (Opus 5.5) 🦊
bsakaguchi commented 2026-10-15 18:23:54 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: I'd like to build the first slice of sunxiuguo's shape. It adds explanations and changes no verdicts. Before I start, does that scope fit?

What I'd propose for one PR:

  • Keep every current status as it is. Old captures and results still parse.
  • Add an optional note when exit or terminal shows a change.
  • Exit: say whether the code, signal or reason changed. Or the settlement state. Or only the settlement time.
  • Terminal: say whether the stream changed, the final screen, or only frame timing.
  • Each explanation names the path and shows the left and right values.
  • Unknown stays unknown. A match on observed parts never claims full equivalence.

Not in this PR:

  • Raw timing in the producer. That is a separate change, as sunxiuguo noted.
  • Any new way to hide differences.

For tests, I'd cover the cases from the issue and that review. Same output with different timing. Same stream with different chunks. A real output change. A real exit-code change. Incomplete coverage. Old payloads. CLI and MCP both. I already have these runs set up on Linux, on Node 22, 24 and 26.

If a different first slice fits better, I'm happy to follow that instead.

🤖 Generated with Claude Code (Opus 5.5) 🦊

<!-- gh-comment-id:6092564707 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** I'd like to build the first slice of sunxiuguo's shape. It adds explanations and changes no verdicts. Before I start, does that scope fit? What I'd propose for one PR: - Keep every current status as it is. Old captures and results still parse. - Add an optional note when exit or terminal shows a change. - Exit: say whether the code, signal or reason changed. Or the settlement state. Or only the settlement time. - Terminal: say whether the stream changed, the final screen, or only frame timing. - Each explanation names the path and shows the left and right values. - Unknown stays unknown. A match on observed parts never claims full equivalence. Not in this PR: - Raw timing in the producer. That is a separate change, as sunxiuguo noted. - Any new way to hide differences. For tests, I'd cover the cases from the issue and that review. Same output with different timing. Same stream with different chunks. A real output change. A real exit-code change. Incomplete coverage. Old payloads. CLI and MCP both. I already have these runs set up on Linux, on Node 22, 24 and 26. If a different first slice fits better, I'm happy to follow that instead. 🤖 Generated with Claude Code (Opus 5.5) 🦊
bsakaguchi commented 2026-10-15 18:23:54 +00:00
Author
Owner
Copy link

@sunxiuguo commented on GitHub (Oct 10, 2026):

Thanks @CassieFox23 for checking the released package across Node versions and separating the lifecycle and terminal cases. Your proposed additive slice matches the shape I suggested. I’m another contributor, so the contract decision remains with @morluto.

A few details from that earlier scope seem worth keeping explicit:

  • Include settlement cleanup outcome, and distinguish the concatenated observed normalized stream, full final rendered state, and frame timing/chunk boundaries. Add the different-streams/same-final-screen control; a matching final screen must not erase earlier output differences.
  • Keep normalization and coverage context beside the paths and observed values. “Timing only” needs evidence that the other compared facets match, and unavailable raw timestamps should remain unavailable.
  • Explain trace-derived statuses separately from observed field differences. In current main, a different trace with differing outcomes can mark every declared dimension changed, while nonconforming retains the exact comparisons. An explanation should therefore not call equal exit fields an exit-outcome change just because a terminal mismatch changed the combined trace verdict.

Preserving the current statuses, historical payload parsing and unknown/truncated evidence, with CLI/MCP checks, keeps this bounded. The terminal trace results above are your additional runtime evidence; I have only checked the projection logic in source here.

AI-assisted review.

<!-- gh-comment-id:6092653463 --> @sunxiuguo commented on GitHub (Oct 10, 2026): Thanks @CassieFox23 for checking the released package across Node versions and separating the lifecycle and terminal cases. Your proposed additive slice matches the shape I suggested. I’m another contributor, so the contract decision remains with @morluto. A few details from that earlier scope seem worth keeping explicit: - Include settlement cleanup outcome, and distinguish the concatenated observed normalized stream, full final rendered state, and frame timing/chunk boundaries. Add the different-streams/same-final-screen control; a matching final screen must not erase earlier output differences. - Keep normalization and coverage context beside the paths and observed values. “Timing only” needs evidence that the other compared facets match, and unavailable raw timestamps should remain unavailable. - Explain trace-derived statuses separately from observed field differences. In [current main](https://github.com/morluto/rea/blob/7aa4d768eb15317a63a476431ed75bafec086033/src/domain/process/processComparison.ts#L297-L329), a `different` trace with differing outcomes can mark every declared dimension changed, while `nonconforming` retains the exact comparisons. An explanation should therefore not call equal exit fields an exit-outcome change just because a terminal mismatch changed the combined trace verdict. Preserving the current statuses, historical payload parsing and unknown/truncated evidence, with CLI/MCP checks, keeps this bounded. The terminal trace results above are your additional runtime evidence; I have only checked the projection logic in source here. AI-assisted review.
bsakaguchi commented 2026-10-15 18:23:54 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: Here is the full scope in one place. It adds explanations and changes no verdicts. It folds in every point from sunxiuguo's review. I'll wait for morluto's call before I start.

Edit: added a frame guard and test 10 from sunxiuguo's review.

Thanks, sunxiuguo. Your review filled in the gaps in my offer. Here is the whole list, so morluto can say yes or no to one thing.

What this PR would do:

  • Keep every current status as it is. Old captures and results still parse.
  • Add an optional note when exit or terminal shows a change.
  • Exit: say whether the code, signal or reason changed. Or the settlement state. Or only the settlement time.
  • Terminal: say whether the stream changed, the final screen, or only frame timing.
  • Each note names the path and shows the left and right values.
  • Unknown stays unknown. A match on observed parts never claims full equivalence.

Points sunxiuguo asked me to keep explicit, so they are part of the scope:

  • The exit split includes settlement.cleanup_outcome, not only the state.
  • The terminal split keeps stream, final screen and frame timing apart. Final screen means size, cursor, buffer and serialized state. Frame timing includes chunk edges.
  • A matching final screen never hides an earlier stream difference.
  • Each path and value shows its normalization and coverage context next to it.
  • "Timing only" is said only when every other compared facet matches.
  • Where only bucketed times exist, raw time reads as unavailable. The PR never rebuilds it.
  • A time difference on an unobserved facet stays a timing note. It is never read as a real change.
  • Notes come from the exact field comparison, not the trace verdict. A terminal mismatch can flip the trace to different. That marks every declared dimension changed. The note must still show the exit fields as equal. I checked applyTraceVerdict at commit 7aa4d768 and read the same thing. This is "catch two" from my terminal comment.

Not in this PR:

  • Raw timing in the producer. That is a separate change, as sunxiuguo noted.
  • Any new way to hide differences.
  • Any change to how a trace verdict sets a status.

Tests I'd add:

  • Same output and exit, different timing.
  • Same stream, split into different chunks.
  • Different streams that render the same final screen. The note must still show the stream change.
  • A real output change.
  • A real exit-code change.
  • A trace flipped by a terminal mismatch, with equal exit fields. The exit note must not call it an outcome change.
  • Coverage that is incomplete or truncated. The note stays unknown.
  • Old payloads still parse.
  • CLI and MCP both show the same notes.
  • A middle screen frame that differs, plus a time difference. The note must not say timing only. sunxiuguo caught this one.

Six questions before I start

Each one has a default. A short answer like "1 yes, 3 always" is enough.

  1. Field name and place. Default: top-level explanations with exit and terminal keys. Alternatives: exit_explanation and terminal_explanation, or a nested dimensions object.
  2. Facet and summary names. Default: outcome, settlement, settlement_timing; stream, final_screen, frame_timing. Summaries as in the plan below. The flipped-trace case is trace_only.
  3. Presence rule. Default: a note only when the dimension is changed, added or removed, and none when it is unknown. Or: always present, every facet unchanged. Or: an explicit all-unknown record when the dimension is unknown.
  4. Size cap. Default: differences reuses the one-frame bound of first_divergence; frame_timing keeps the first 16 entries plus a count. Is a cap wanted, and where should it live?
  5. Tool description. Default: add one sentence to compare_process_captures in sessionToolContracts.ts 100-104 in this PR.
  6. Per-facet coverage. Default: out of this PR. A later slice could keep stream observable even when rendered frames are cut short.

I have these runs set up on Linux, on Node 22, 24 and 26. Nothing is built yet. I'll wait for morluto's call on the contract/approval before I start.

Thanks, sunxiuguo, for the frame guard catch. It is in the plan below.

Draft plan for this PR (for review, nothing built yet)

Draft plan, read against main 6875d4a6. Every name in it is open to change. Nothing is built. Re-checked at de8fc671: the process files are unchanged. Changes from sunxiuguo's second review are marked (review 2).

Contract shape

The result schema is zod: processCaptureComparisonSchema in src/domain/process/processComparison.ts, lines 59-102. It is a non-strict z.object, so unknown keys are stripped, not rejected. There is no schema version id in the process domain. The optional trace field at line 86, spread at line 482, is the precedent to copy. The MCP output schema derives from the same object at src/contracts/toolOutputSchemaGroups.ts:499.

Proposed field:

explanations?: {
  exit?: DimensionExplanation;      // only when exit is changed
  terminal?: DimensionExplanation;  // only when terminal is changed, added or removed
}

DimensionExplanation = {
  observed_status: ComparisonStatus;  // the exact status before any trace verdict
  facets: {                           // fixed keys per dimension (z.strictObject)
    <facet>: {
      status: "unchanged" | "changed" | "unknown";
      paths: string[];                // e.g. ["exit.code", "exit.signal", "exit.reason"]
      timing_basis: "raw" | "bucketed" | "none";
    }
  };
  summary: "outcome" | "settlement" | "stream" | "timing_only" | "mixed"
         | "unexplained" | "trace_only" | "unknown";
  differences: Array<{ path: string; left: JsonValue | null; right: JsonValue | null }>;
  context: {
    normalization: Normalization;     // both sides; they are equal or the dimension is already unknown
    time_bucket_ms: number;
    raw_timestamps: "unavailable";    // captures store bucketed at_ms only
    coverage: { left: Retention; right: Retention } | null;  // terminal retention counters
  };
}

Exit facets:

  • outcome: exit.code, exit.signal, exit.reason.
  • settlement: settlement.state, settlement.cleanup_outcome.
  • settlement_timing: settlement.elapsed_ms. Basis raw. The producer writes Date.now() - started with no bucket (src/process/capture/ProcessCaptureLifecycle.ts 627, 633, 641).

These three cover { exit, settlement } exactly. That is the object lines 140 and 379-382 compare today.

Terminal facets:

  • stream: frames[].data joined in sequence order.
  • final_screen: the last rendered_frames[] entry without sequence and at_ms. If either side has no rendered frames, this facet is unknown, never unchanged.
  • rendered_payloads (review 2): every rendered_frames[] entry, index-aligned, without sequence and at_ms. A count mismatch or any differing earlier payload marks it changed. No rendered frames on either side marks it unknown.
  • frame_timing: the at_ms lists for raw and rendered frames, plus the chunk layout (frame count and per-frame lengths). Basis bucketed. ProcessCaptureLifecycle.ts 908-911 buckets at_ms through normalizeProcessElapsedTime (ProcessNormalization.ts 7-10) before the frame is stored. So no raw time exists in a capture. The note says raw_timestamps: "unavailable" and never rebuilds one.

With the fourth facet, the four cover the whole list from 104-119. Raw data is stream. Rendered fields other than at_ms and sequence are rendered_payloads, and the last one is also final_screen. Times, sequence numbers and counts are frame_timing.

Guards, as a second superRefine next to lines 89-99:

  • summary: "timing_only" needs every other facet unchanged. For terminal that includes rendered_payloads (review 2). So every compared rendered payload must be equal once at_ms and sequence are removed.
  • When frame_timing changed beside a changed or unknown facet, summary is mixed. The differences list then carries the timing entries and the first differing payload path. So the narrower timing difference is reported next to its remainder (review 2).
  • An observed change with no changed facet is unexplained. It is never all-unchanged.

Old payloads: a stored result with no explanations key parses as before. Three consumers parse this result. They read only status, the five dimension statuses and limitations. Those are changedBehavior.ts 205-215 and 257-284, reconstructionVerification.ts 218-227 and 262-270, and staticRuntimeCorrelation.ts 240-278. No version bump exists, and none is needed.

One visible side effect: evidence_id hashes normalized_result (src/domain/evidence.ts 219). A new comparison that carries a note gets a different evidence_id than before. Stored records still check out against their own content (parseEvidence, 233-240).

Examples, trimmed to the keys that matter. Captures follow the EMPTY_PROCESS_CAPTURE_EXAMPLE shape.

Test 1, same output and exit, different timing (exit side):

"exit": "changed",
"explanations": { "exit": {
  "observed_status": "changed",
  "facets": {
    "outcome": { "status": "unchanged", "timing_basis": "none" },
    "settlement": { "status": "unchanged", "timing_basis": "none" },
    "settlement_timing": { "status": "changed", "timing_basis": "raw" } },
  "summary": "timing_only",
  "differences": [ { "path": "settlement.elapsed_ms", "left": 99, "right": 105 } ],
  "context": { "time_bucket_ms": 10, "raw_timestamps": "unavailable", "coverage": null } } }

Test 1, terminal side (same hello\r\n, at_ms 30 vs 40):

"terminal": "changed",
"explanations": { "terminal": {
  "observed_status": "changed",
  "facets": {
    "stream": { "status": "unchanged" },
    "final_screen": { "status": "unchanged" },
    "rendered_payloads": { "status": "unchanged" },
    "frame_timing": { "status": "changed", "timing_basis": "bucketed" } },
  "summary": "timing_only",
  "differences": [ { "path": "frames[0].at_ms", "left": 30, "right": 40 } ],
  "context": { "time_bucket_ms": 10, "raw_timestamps": "unavailable",
               "coverage": { "left": { "observed_frames": 1, "retained_frames": 1 },
                             "right": { "observed_frames": 1, "retained_frames": 1 } } } } }

Test 2, same stream, different chunks ("hel" + "lo" vs "hello"). Facets: stream and final_screen unchanged, frame_timing changed. Summary timing_only. Differences: frames.length 2 vs 1, plus the first frame's two values.

Test 3, different streams, same final screen ("bar" vs "foo\rbar", identical rendered_frames):

"facets": {
  "stream": { "status": "changed" },
  "final_screen": { "status": "unchanged" },
  "rendered_payloads": { "status": "unchanged" },
  "frame_timing": { "status": "unchanged" } },
"summary": "stream",
"differences": [ { "path": "frames[0].data", "left": "bar", "right": "foo\rbar" } ]

Test 4, real output change: stream changed, final_screen changed when rendered frames differ. summary is never timing_only.

Test 5, real exit-code change (code 0 vs 3, elapsed_ms also differs): outcome changed, settlement_timing changed, summary: "mixed". Differences list both paths.

Test 6, trace flipped by a terminal mismatch, exit fields equal:

"exit": "changed",
"trace": { "verdict": "different" },
"explanations": { "exit": {
  "observed_status": "unchanged",
  "facets": {
    "outcome": { "status": "unchanged" },
    "settlement": { "status": "unchanged" },
    "settlement_timing": { "status": "unchanged" } },
  "summary": "trace_only",
  "differences": [] } }

The reported changed came from applyTraceVerdict, lines 315-316, not from a field.

Test 7, coverage incomplete or truncated: the dimension is already unknown (classify 356-362, or hasUnknown at 377). The note is omitted, or (Q3) present with every facet unknown and summary: "unknown". No facet reads unchanged either way.

Test 8, old payload: a result JSON with no explanations key parses through processCaptureComparisonSchema and parseEvidence unchanged.

Test 9, CLI and MCP: normalized_result from compareProcessEvidenceFiles equals structuredContent.normalized_result from the tool, notes included, same evidence_id.

Test 10 (review 2), an earlier rendered payload that differs together with its time. Both sides have the same raw data and two rendered frames each. Frame 0 differs in lines and in at_ms (10 vs 20). Frame 1, the final screen, is the same.

"terminal": "changed",
"explanations": { "terminal": {
  "observed_status": "changed",
  "facets": {
    "stream": { "status": "unchanged" },
    "final_screen": { "status": "unchanged" },
    "rendered_payloads": { "status": "changed" },
    "frame_timing": { "status": "changed", "timing_basis": "bucketed" } },
  "summary": "mixed",
  "differences": [
    { "path": "rendered_frames[0].lines", "left": ["hello"], "right": ["hullo"] },
    { "path": "rendered_frames[0].at_ms", "left": 10, "right": 20 } ] } }

Never timing_only.

Files to touch (lines at 6875d4a6)

  • New src/domain/process/processComparisonExplanations.ts: the field schema and a pure builder, roughly explainProcessDimensions(left, right, exactDimensions, reportedDimensions). One file, so a rename is a one-file edit.
  • src/domain/process/processComparison.ts. Field after line 86. Guard beside 89-99. Build after 431, where both the exact and post-trace dimensions are in scope. Spread beside 482. The functions applyTraceVerdict (294-338) and compareDimensions (345-398) stay as they are. The helper terminalObservations (104-119) may need a sibling for the stream and final-screen projections.
  • processTraceComparison.ts and processTraceDimensionProjection.ts: no edit expected. The builder reads exactDimensions, which sits upstream of both.
  • src/domain/process/processCapture.fixture.ts: two or three shared builders, so the ten tests do not repeat setup. .jscpd.json counts test files at a 2.5% threshold.
  • src/application/process/ProcessCli.ts 84-108 and src/server/registerProcessComparisonTool.ts 107-139: no change. Both pass the whole result through jsonValueSchema.parse, so no key may hold undefined.
  • src/contracts/sessionToolContracts.ts 100-104: one optional sentence in the tool description (Q5).
  • docs/process-capture.md "Compare two captures", 179-209: one paragraph after line 191. The capture recipe at the top stays untouched; tests/boundary/process/processCli.test.ts 26-75 parses it.
  • Generated outputs are not committed: .cache/mcp-tool-catalog.json, docs/public/product-catalog.json, the CATALOG_IDENTITY digests (src/catalogIdentity.ts 66-75) and skills/. CI regenerates them. Nothing in skill-src names this tool. CHANGELOG.md is release-please owned.

Logic

Inputs: left, right, exactDimensions from compareDimensions (line 420), and dimensions after applyTraceVerdict (425-431).

  1. Exit. Build the note when dimensions.exit is changed. Each facet runs sameJsonValue over its paths on { exit, settlement }, the object lines 379-382 compare. Each differing path becomes one differences entry. The observed_status value is exactDimensions.exit. When that is unchanged but the reported status is changed, the cause is applyTraceVerdict 315-316 (different trace with processTraceOutcomesDiffer, processTraceComparison.ts 54-59). The note then shows every facet unchanged and an empty list. The reported status is not touched.
  2. Terminal. Build when dimensions.terminal is changed, added or removed. The stream facet compares the joined frames[].data. The final_screen facet compares the last rendered frame on each side, minus sequence and at_ms. Missing rendered frames make it unknown. The rendered_payloads facet (review 2) compares every rendered frame, minus sequence and at_ms, index by index. A count mismatch is changed. This closes the hole from the review. An earlier rendered payload differed with its time. The stream and final screen still matched. The frame_timing facet compares the at_ms lists and the chunk layout. The four facets now cover the whole merged list that terminalObservations sorts (104-119). The unexplained guard stays as a safety net. A changed dimension with no changed facet gets summary: "unexplained" and the first divergence path. It is never all-unchanged for an observed change.
  3. "Timing only" needs the timing facet changed and every other facet unchanged. For terminal that means stream, final_screen and rendered_payloads all unchanged (review 2). An unknown facet blocks it. When timing changed beside another changed or unknown facet, summary is mixed. Then differences lists the timing entries with the first differing payload path. The superRefine enforces this on the way out.
  4. Unknown stays unknown. When a dimension is unknown, no note is built for it. The causes sit in classify (350-362) and line 377. Three things cause that. A residual unknown. A truncated source, per processSourceTruncated. Or normalization rules that differ. A note never changes status, a dimension, or first_divergence (432-477, not edited).
  5. Truncated. The context.coverage value carries the raw_terminal and rendered_terminal retention counters from truncation_details, both sides. A truncated terminal is already unknown (359-361). So on a changed dimension, the counters show that both sides were fully retained.
  6. Unavailable. The value raw_timestamps: "unavailable" is a constant in this slice. The producer stores only bucketed at_ms. The timing_basis field says which facet is raw (settlement.elapsed_ms) and which is bucketed. A later producer change can flip the constant without changing the shape.
  7. Size. The differences values reuse the bound first_divergence already accepts: one frame's data per side. For stream, the entry is the first differing frame index, its two values, and the two total lengths. The frame_timing list is capped (Q4).

Tests

Placement follows docs/testing.md 35-56: domain semantics in src/**/*.test.ts, CLI plus MCP parity in tests/boundary/mcp/**. New files need npm run verify:test-discovery.

# Case File Fixture reuse
1 Same output and exit, different timing new src/domain/process/processComparison.explanations.test.ts emptyProcessCapture, accountFullyObservedProcessCapture, compareUnverifiedProcessCaptures (processCapture.fixture.ts 11, 19, 105)
2 Same stream, different chunks same file frames ["hel", "lo"] vs ["hello"]
3 Different streams, same final screen same file "bar" vs "foo\rbar" (as in processCapture.comparison.test.ts 101-108), same rendered_frames on both sides
4 Real output change same file hello vs hello world
5 Real exit-code change same file exit.code 0 vs 3, elapsed_ms 99 vs 105
6 Trace flips exit, exit fields equal src/domain/process/processTraceAdapters.test.ts, describe at 71 hoist withTerminal (99-120); see the note below
7 Coverage incomplete or truncated same file as 1 capture(values(...), { omittedTerminalFrame: true }) from processTraceComparison.fixture.ts 20-81; residual_unknowns: [{ scope: "exit" }] for the exit side
8 Old payload still parses same file as 1 PROCESS_COMPARISON_EVIDENCE (investigationExamples.ts 59-80), a self-comparison with no note
9 CLI and MCP show the same note new tests/boundary/mcp/processComparisonExplanationsMcp.test.ts the shape of processTraceAdaptersParity.test.ts 104-159; settlement.elapsed_ms 50 vs 60
10 Earlier rendered payload differs, plus a time difference (review 2) same file as 1 withRenderedFrame builder. Two rendered frames per side. Frame 0 differs in lines and at_ms. Frame 1 and raw data are the same.

Test 6 in full: the spec declares terminal_raw and both lifecycle events in one finite variant. The left capture passes and the right fails on terminal. The verdict is different, so exit flips to changed while its fields stay equal.

Tests 2, 3, 7 and 10 need synthetic fixtures. Real captures cannot produce them on demand. There is an acceptance test at tests/acceptance/tools/localToolContracts.test.ts 266-280. It parses a real self-comparison through the output schema. It keeps passing with no edit.

Fixtures may vary settlement.elapsed_ms alone. The validator (processCaptureValidation.ts) orders at_ms within a collection (18-22). It orders the manifest timestamps (47-50). It needs both checkpoints (114). Nothing ties elapsed_ms to after_settlement.at_ms.

Commands:

npm run test:local -- src/domain/process/processComparison.explanations.test.ts src/domain/process/processTraceAdapters.test.ts src/domain/process/processCapture.comparison.test.ts
npm run test:focused -- tests/boundary/mcp/processComparisonExplanationsMcp.test.ts tests/boundary/mcp/processTraceAdaptersParity.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/contractPresentation.test.ts
npm run test:focused -- tests/boundary/process/processCli.test.ts tests/acceptance/tools/localToolContracts.test.ts
npm run verify:test-discovery
npm run check:fast
npm run check
npm run jscpd
npm run docs:check

The full list runs on Node 24.18.0, the toolchain the repo pins. The domain tests also run on Node 22 and 26 through test:local.

Known risks

The six open questions are at the top of this comment.

Two known risks. The first draft's terminal facets had a hole, which sunxiuguo caught (review 2). A differing earlier rendered payload could pass as timing only. The fourth facet closes it. The unexplained guard stays as a safety net. And jscpd counts test files at a 2.5% threshold, so fixtures are shared.

Size estimate

This is an estimate, not a measurement. About 550-700 lines added and about 10 removed, across 7 files. Give or take 30 percent. Where it comes from:

  • The closest analogue is #1284 (33bf58518): +739/-50 over 14 files, with producer changes this PR does not make.
  • The new module is sized against processCaptureCoverage.ts (105 lines) plus a schema.
  • Ten tests at 25-40 lines each, in the existing style; processCapture.comparison.test.ts is 194 lines for four tests.
  • The fourth terminal facet and the tenth test add about 40-60 lines, inside this range (review 2).
  • The MCP parity file copies a 160-line pattern.
  • The docs paragraph is 15-30 lines; processComparison.ts itself gains 25-40 lines.

🤖 Generated with Claude Code (Fable 5.1) 🦊

<!-- gh-comment-id:6092777944 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** Here is the full scope in one place. It adds explanations and changes no verdicts. It folds in every point from sunxiuguo's review. I'll wait for morluto's call before I start. Edit: added a frame guard and test 10 from sunxiuguo's review. Thanks, sunxiuguo. Your review filled in the gaps in my offer. Here is the whole list, so morluto can say yes or no to one thing. What this PR would do: - Keep every current status as it is. Old captures and results still parse. - Add an optional note when exit or terminal shows a change. - Exit: say whether the code, signal or reason changed. Or the settlement state. Or only the settlement time. - Terminal: say whether the stream changed, the final screen, or only frame timing. - Each note names the path and shows the left and right values. - Unknown stays unknown. A match on observed parts never claims full equivalence. Points sunxiuguo asked me to keep explicit, so they are part of the scope: - The exit split includes `settlement.cleanup_outcome`, not only the state. - The terminal split keeps stream, final screen and frame timing apart. Final screen means size, cursor, buffer and serialized state. Frame timing includes chunk edges. - A matching final screen never hides an earlier stream difference. - Each path and value shows its normalization and coverage context next to it. - "Timing only" is said only when every other compared facet matches. - Where only bucketed times exist, raw time reads as unavailable. The PR never rebuilds it. - A time difference on an unobserved facet stays a timing note. It is never read as a real change. - Notes come from the exact field comparison, not the trace verdict. A terminal mismatch can flip the trace to `different`. That marks every declared dimension `changed`. The note must still show the exit fields as equal. I checked `applyTraceVerdict` at commit `7aa4d768` and read the same thing. This is "catch two" from my terminal comment. Not in this PR: - Raw timing in the producer. That is a separate change, as sunxiuguo noted. - Any new way to hide differences. - Any change to how a trace verdict sets a status. Tests I'd add: - Same output and exit, different timing. - Same stream, split into different chunks. - Different streams that render the same final screen. The note must still show the stream change. - A real output change. - A real exit-code change. - A trace flipped by a terminal mismatch, with equal exit fields. The exit note must not call it an outcome change. - Coverage that is incomplete or truncated. The note stays unknown. - Old payloads still parse. - CLI and MCP both show the same notes. - A middle screen frame that differs, plus a time difference. The note must not say timing only. sunxiuguo caught this one. ### Six questions before I start Each one has a default. A short answer like "1 yes, 3 always" is enough. 1. Field name and place. Default: top-level `explanations` with `exit` and `terminal` keys. Alternatives: `exit_explanation` and `terminal_explanation`, or a nested `dimensions` object. 2. Facet and summary names. Default: `outcome`, `settlement`, `settlement_timing`; `stream`, `final_screen`, `frame_timing`. Summaries as in the plan below. The flipped-trace case is `trace_only`. 3. Presence rule. Default: a note only when the dimension is `changed`, `added` or `removed`, and none when it is `unknown`. Or: always present, every facet unchanged. Or: an explicit all-unknown record when the dimension is unknown. 4. Size cap. Default: `differences` reuses the one-frame bound of `first_divergence`; `frame_timing` keeps the first 16 entries plus a count. Is a cap wanted, and where should it live? 5. Tool description. Default: add one sentence to `compare_process_captures` in `sessionToolContracts.ts` 100-104 in this PR. 6. Per-facet coverage. Default: out of this PR. A later slice could keep `stream` observable even when rendered frames are cut short. I have these runs set up on Linux, on Node 22, 24 and 26. Nothing is built yet. I'll wait for morluto's call on the contract/approval before I start. Thanks, sunxiuguo, for the frame guard catch. It is in the plan below. <details> <summary>Draft plan for this PR (for review, nothing built yet)</summary> Draft plan, read against main `6875d4a6`. Every name in it is open to change. Nothing is built. Re-checked at `de8fc671`: the process files are unchanged. Changes from sunxiuguo's second review are marked (review 2). ### Contract shape The result schema is zod: `processCaptureComparisonSchema` in `src/domain/process/processComparison.ts`, lines 59-102. It is a non-strict `z.object`, so unknown keys are stripped, not rejected. There is no schema version id in the process domain. The optional `trace` field at line 86, spread at line 482, is the precedent to copy. The MCP output schema derives from the same object at `src/contracts/toolOutputSchemaGroups.ts:499`. Proposed field: ```ts explanations?: { exit?: DimensionExplanation; // only when exit is changed terminal?: DimensionExplanation; // only when terminal is changed, added or removed } DimensionExplanation = { observed_status: ComparisonStatus; // the exact status before any trace verdict facets: { // fixed keys per dimension (z.strictObject) <facet>: { status: "unchanged" | "changed" | "unknown"; paths: string[]; // e.g. ["exit.code", "exit.signal", "exit.reason"] timing_basis: "raw" | "bucketed" | "none"; } }; summary: "outcome" | "settlement" | "stream" | "timing_only" | "mixed" | "unexplained" | "trace_only" | "unknown"; differences: Array<{ path: string; left: JsonValue | null; right: JsonValue | null }>; context: { normalization: Normalization; // both sides; they are equal or the dimension is already unknown time_bucket_ms: number; raw_timestamps: "unavailable"; // captures store bucketed at_ms only coverage: { left: Retention; right: Retention } | null; // terminal retention counters }; } ``` Exit facets: - `outcome`: `exit.code`, `exit.signal`, `exit.reason`. - `settlement`: `settlement.state`, `settlement.cleanup_outcome`. - `settlement_timing`: `settlement.elapsed_ms`. Basis `raw`. The producer writes `Date.now() - started` with no bucket (`src/process/capture/ProcessCaptureLifecycle.ts` 627, 633, 641). These three cover `{ exit, settlement }` exactly. That is the object lines 140 and 379-382 compare today. Terminal facets: - `stream`: `frames[].data` joined in `sequence` order. - `final_screen`: the last `rendered_frames[]` entry without `sequence` and `at_ms`. If either side has no rendered frames, this facet is `unknown`, never `unchanged`. - `rendered_payloads` (review 2): every `rendered_frames[]` entry, index-aligned, without `sequence` and `at_ms`. A count mismatch or any differing earlier payload marks it `changed`. No rendered frames on either side marks it `unknown`. - `frame_timing`: the `at_ms` lists for raw and rendered frames, plus the chunk layout (frame count and per-frame lengths). Basis `bucketed`. `ProcessCaptureLifecycle.ts` 908-911 buckets `at_ms` through `normalizeProcessElapsedTime` (`ProcessNormalization.ts` 7-10) before the frame is stored. So no raw time exists in a capture. The note says `raw_timestamps: "unavailable"` and never rebuilds one. With the fourth facet, the four cover the whole list from 104-119. Raw `data` is `stream`. Rendered fields other than `at_ms` and `sequence` are `rendered_payloads`, and the last one is also `final_screen`. Times, sequence numbers and counts are `frame_timing`. Guards, as a second `superRefine` next to lines 89-99: - `summary: "timing_only"` needs every other facet `unchanged`. For terminal that includes `rendered_payloads` (review 2). So every compared rendered payload must be equal once `at_ms` and `sequence` are removed. - When `frame_timing` changed beside a changed or unknown facet, `summary` is `mixed`. The `differences` list then carries the timing entries and the first differing payload path. So the narrower timing difference is reported next to its remainder (review 2). - An observed change with no changed facet is `unexplained`. It is never all-unchanged. Old payloads: a stored result with no `explanations` key parses as before. Three consumers parse this result. They read only `status`, the five dimension statuses and `limitations`. Those are `changedBehavior.ts` 205-215 and 257-284, `reconstructionVerification.ts` 218-227 and 262-270, and `staticRuntimeCorrelation.ts` 240-278. No version bump exists, and none is needed. One visible side effect: `evidence_id` hashes `normalized_result` (`src/domain/evidence.ts` 219). A new comparison that carries a note gets a different `evidence_id` than before. Stored records still check out against their own content (`parseEvidence`, 233-240). Examples, trimmed to the keys that matter. Captures follow the `EMPTY_PROCESS_CAPTURE_EXAMPLE` shape. Test 1, same output and exit, different timing (exit side): ```json "exit": "changed", "explanations": { "exit": { "observed_status": "changed", "facets": { "outcome": { "status": "unchanged", "timing_basis": "none" }, "settlement": { "status": "unchanged", "timing_basis": "none" }, "settlement_timing": { "status": "changed", "timing_basis": "raw" } }, "summary": "timing_only", "differences": [ { "path": "settlement.elapsed_ms", "left": 99, "right": 105 } ], "context": { "time_bucket_ms": 10, "raw_timestamps": "unavailable", "coverage": null } } } ``` Test 1, terminal side (same `hello\r\n`, `at_ms` 30 vs 40): ```json "terminal": "changed", "explanations": { "terminal": { "observed_status": "changed", "facets": { "stream": { "status": "unchanged" }, "final_screen": { "status": "unchanged" }, "rendered_payloads": { "status": "unchanged" }, "frame_timing": { "status": "changed", "timing_basis": "bucketed" } }, "summary": "timing_only", "differences": [ { "path": "frames[0].at_ms", "left": 30, "right": 40 } ], "context": { "time_bucket_ms": 10, "raw_timestamps": "unavailable", "coverage": { "left": { "observed_frames": 1, "retained_frames": 1 }, "right": { "observed_frames": 1, "retained_frames": 1 } } } } } ``` Test 2, same stream, different chunks (`"hel"` + `"lo"` vs `"hello"`). Facets: `stream` and `final_screen` unchanged, `frame_timing` changed. Summary `timing_only`. Differences: `frames.length` 2 vs 1, plus the first frame's two values. Test 3, different streams, same final screen (`"bar"` vs `"foo\rbar"`, identical `rendered_frames`): ```json "facets": { "stream": { "status": "changed" }, "final_screen": { "status": "unchanged" }, "rendered_payloads": { "status": "unchanged" }, "frame_timing": { "status": "unchanged" } }, "summary": "stream", "differences": [ { "path": "frames[0].data", "left": "bar", "right": "foo\rbar" } ] ``` Test 4, real output change: `stream` changed, `final_screen` changed when rendered frames differ. `summary` is never `timing_only`. Test 5, real exit-code change (`code` 0 vs 3, `elapsed_ms` also differs): `outcome` changed, `settlement_timing` changed, `summary: "mixed"`. Differences list both paths. Test 6, trace flipped by a terminal mismatch, exit fields equal: ```json "exit": "changed", "trace": { "verdict": "different" }, "explanations": { "exit": { "observed_status": "unchanged", "facets": { "outcome": { "status": "unchanged" }, "settlement": { "status": "unchanged" }, "settlement_timing": { "status": "unchanged" } }, "summary": "trace_only", "differences": [] } } ``` The reported `changed` came from `applyTraceVerdict`, lines 315-316, not from a field. Test 7, coverage incomplete or truncated: the dimension is already `unknown` (`classify` 356-362, or `hasUnknown` at 377). The note is omitted, or (Q3) present with every facet `unknown` and `summary: "unknown"`. No facet reads `unchanged` either way. Test 8, old payload: a result JSON with no `explanations` key parses through `processCaptureComparisonSchema` and `parseEvidence` unchanged. Test 9, CLI and MCP: `normalized_result` from `compareProcessEvidenceFiles` equals `structuredContent.normalized_result` from the tool, notes included, same `evidence_id`. Test 10 (review 2), an earlier rendered payload that differs together with its time. Both sides have the same raw `data` and two rendered frames each. Frame 0 differs in `lines` and in `at_ms` (10 vs 20). Frame 1, the final screen, is the same. ```json "terminal": "changed", "explanations": { "terminal": { "observed_status": "changed", "facets": { "stream": { "status": "unchanged" }, "final_screen": { "status": "unchanged" }, "rendered_payloads": { "status": "changed" }, "frame_timing": { "status": "changed", "timing_basis": "bucketed" } }, "summary": "mixed", "differences": [ { "path": "rendered_frames[0].lines", "left": ["hello"], "right": ["hullo"] }, { "path": "rendered_frames[0].at_ms", "left": 10, "right": 20 } ] } } ``` Never `timing_only`. ### Files to touch (lines at 6875d4a6) - New `src/domain/process/processComparisonExplanations.ts`: the field schema and a pure builder, roughly `explainProcessDimensions(left, right, exactDimensions, reportedDimensions)`. One file, so a rename is a one-file edit. - `src/domain/process/processComparison.ts`. Field after line 86. Guard beside 89-99. Build after 431, where both the exact and post-trace dimensions are in scope. Spread beside 482. The functions `applyTraceVerdict` (294-338) and `compareDimensions` (345-398) stay as they are. The helper `terminalObservations` (104-119) may need a sibling for the stream and final-screen projections. - `processTraceComparison.ts` and `processTraceDimensionProjection.ts`: no edit expected. The builder reads `exactDimensions`, which sits upstream of both. - `src/domain/process/processCapture.fixture.ts`: two or three shared builders, so the ten tests do not repeat setup. `.jscpd.json` counts test files at a 2.5% threshold. - `src/application/process/ProcessCli.ts` 84-108 and `src/server/registerProcessComparisonTool.ts` 107-139: no change. Both pass the whole result through `jsonValueSchema.parse`, so no key may hold `undefined`. - `src/contracts/sessionToolContracts.ts` 100-104: one optional sentence in the tool description (Q5). - `docs/process-capture.md` "Compare two captures", 179-209: one paragraph after line 191. The capture recipe at the top stays untouched; `tests/boundary/process/processCli.test.ts` 26-75 parses it. - Generated outputs are not committed: `.cache/mcp-tool-catalog.json`, `docs/public/product-catalog.json`, the `CATALOG_IDENTITY` digests (`src/catalogIdentity.ts` 66-75) and `skills/`. CI regenerates them. Nothing in `skill-src` names this tool. `CHANGELOG.md` is release-please owned. ### Logic Inputs: `left`, `right`, `exactDimensions` from `compareDimensions` (line 420), and `dimensions` after `applyTraceVerdict` (425-431). 1. Exit. Build the note when `dimensions.exit` is `changed`. Each facet runs `sameJsonValue` over its paths on `{ exit, settlement }`, the object lines 379-382 compare. Each differing path becomes one `differences` entry. The `observed_status` value is `exactDimensions.exit`. When that is `unchanged` but the reported status is `changed`, the cause is `applyTraceVerdict` 315-316 (`different` trace with `processTraceOutcomesDiffer`, `processTraceComparison.ts` 54-59). The note then shows every facet `unchanged` and an empty list. The reported status is not touched. 2. Terminal. Build when `dimensions.terminal` is `changed`, `added` or `removed`. The `stream` facet compares the joined `frames[].data`. The `final_screen` facet compares the last rendered frame on each side, minus `sequence` and `at_ms`. Missing rendered frames make it `unknown`. The `rendered_payloads` facet (review 2) compares every rendered frame, minus `sequence` and `at_ms`, index by index. A count mismatch is `changed`. This closes the hole from the review. An earlier rendered payload differed with its time. The stream and final screen still matched. The `frame_timing` facet compares the `at_ms` lists and the chunk layout. The four facets now cover the whole merged list that `terminalObservations` sorts (104-119). The `unexplained` guard stays as a safety net. A `changed` dimension with no changed facet gets `summary: "unexplained"` and the first divergence path. It is never all-unchanged for an observed change. 3. "Timing only" needs the timing facet `changed` and every other facet `unchanged`. For terminal that means `stream`, `final_screen` and `rendered_payloads` all `unchanged` (review 2). An `unknown` facet blocks it. When timing changed beside another changed or unknown facet, `summary` is `mixed`. Then `differences` lists the timing entries with the first differing payload path. The `superRefine` enforces this on the way out. 4. Unknown stays unknown. When a dimension is `unknown`, no note is built for it. The causes sit in `classify` (350-362) and line 377. Three things cause that. A residual unknown. A truncated source, per `processSourceTruncated`. Or normalization rules that differ. A note never changes `status`, a dimension, or `first_divergence` (432-477, not edited). 5. Truncated. The `context.coverage` value carries the `raw_terminal` and `rendered_terminal` retention counters from `truncation_details`, both sides. A truncated terminal is already `unknown` (359-361). So on a `changed` dimension, the counters show that both sides were fully retained. 6. Unavailable. The value `raw_timestamps: "unavailable"` is a constant in this slice. The producer stores only bucketed `at_ms`. The `timing_basis` field says which facet is raw (`settlement.elapsed_ms`) and which is bucketed. A later producer change can flip the constant without changing the shape. 7. Size. The `differences` values reuse the bound `first_divergence` already accepts: one frame's `data` per side. For `stream`, the entry is the first differing frame index, its two values, and the two total lengths. The `frame_timing` list is capped (Q4). ### Tests Placement follows `docs/testing.md` 35-56: domain semantics in `src/**/*.test.ts`, CLI plus MCP parity in `tests/boundary/mcp/**`. New files need `npm run verify:test-discovery`. | # | Case | File | Fixture reuse | | --- | --- | --- | --- | | 1 | Same output and exit, different timing | new `src/domain/process/processComparison.explanations.test.ts` | `emptyProcessCapture`, `accountFullyObservedProcessCapture`, `compareUnverifiedProcessCaptures` (`processCapture.fixture.ts` 11, 19, 105) | | 2 | Same stream, different chunks | same file | frames `["hel", "lo"]` vs `["hello"]` | | 3 | Different streams, same final screen | same file | `"bar"` vs `"foo\rbar"` (as in `processCapture.comparison.test.ts` 101-108), same `rendered_frames` on both sides | | 4 | Real output change | same file | `hello` vs `hello world` | | 5 | Real exit-code change | same file | `exit.code` 0 vs 3, `elapsed_ms` 99 vs 105 | | 6 | Trace flips exit, exit fields equal | `src/domain/process/processTraceAdapters.test.ts`, `describe` at 71 | hoist `withTerminal` (99-120); see the note below | | 7 | Coverage incomplete or truncated | same file as 1 | `capture(values(...), { omittedTerminalFrame: true })` from `processTraceComparison.fixture.ts` 20-81; `residual_unknowns: [{ scope: "exit" }]` for the exit side | | 8 | Old payload still parses | same file as 1 | `PROCESS_COMPARISON_EVIDENCE` (`investigationExamples.ts` 59-80), a self-comparison with no note | | 9 | CLI and MCP show the same note | new `tests/boundary/mcp/processComparisonExplanationsMcp.test.ts` | the shape of `processTraceAdaptersParity.test.ts` 104-159; `settlement.elapsed_ms` 50 vs 60 | | 10 | Earlier rendered payload differs, plus a time difference (review 2) | same file as 1 | `withRenderedFrame` builder. Two rendered frames per side. Frame 0 differs in `lines` and `at_ms`. Frame 1 and raw `data` are the same. | Test 6 in full: the spec declares `terminal_raw` and both `lifecycle` events in one finite variant. The left capture passes and the right fails on terminal. The verdict is `different`, so `exit` flips to `changed` while its fields stay equal. Tests 2, 3, 7 and 10 need synthetic fixtures. Real captures cannot produce them on demand. There is an acceptance test at `tests/acceptance/tools/localToolContracts.test.ts` 266-280. It parses a real self-comparison through the output schema. It keeps passing with no edit. Fixtures may vary `settlement.elapsed_ms` alone. The validator (`processCaptureValidation.ts`) orders `at_ms` within a collection (18-22). It orders the manifest timestamps (47-50). It needs both checkpoints (114). Nothing ties `elapsed_ms` to `after_settlement.at_ms`. Commands: ```bash npm run test:local -- src/domain/process/processComparison.explanations.test.ts src/domain/process/processTraceAdapters.test.ts src/domain/process/processCapture.comparison.test.ts npm run test:focused -- tests/boundary/mcp/processComparisonExplanationsMcp.test.ts tests/boundary/mcp/processTraceAdaptersParity.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/contractPresentation.test.ts npm run test:focused -- tests/boundary/process/processCli.test.ts tests/acceptance/tools/localToolContracts.test.ts npm run verify:test-discovery npm run check:fast npm run check npm run jscpd npm run docs:check ``` The full list runs on Node 24.18.0, the toolchain the repo pins. The domain tests also run on Node 22 and 26 through `test:local`. ### Known risks The six open questions are at the top of this comment. Two known risks. The first draft's terminal facets had a hole, which sunxiuguo caught (review 2). A differing earlier rendered payload could pass as timing only. The fourth facet closes it. The `unexplained` guard stays as a safety net. And `jscpd` counts test files at a 2.5% threshold, so fixtures are shared. ### Size estimate This is an estimate, not a measurement. About 550-700 lines added and about 10 removed, across 7 files. Give or take 30 percent. Where it comes from: - The closest analogue is #1284 (`33bf58518`): +739/-50 over 14 files, with producer changes this PR does not make. - The new module is sized against `processCaptureCoverage.ts` (105 lines) plus a schema. - Ten tests at 25-40 lines each, in the existing style; `processCapture.comparison.test.ts` is 194 lines for four tests. - The fourth terminal facet and the tenth test add about 40-60 lines, inside this range (review 2). - The MCP parity file copies a 160-line pattern. - The docs paragraph is 15-30 lines; `processComparison.ts` itself gains 25-40 lines. </details> 🤖 Generated with Claude Code (Fable 5.1) 🦊
bsakaguchi commented 2026-10-15 18:23:54 +00:00
Author
Owner
Copy link

@sunxiuguo commented on GitHub (Oct 10, 2026):

One guard detail to tighten in the draft: the current terminal comparison includes every rendered frame. If the joined stream and final screen match, but an earlier rendered payload and its timestamp both differ, the listed facets would allow timing_only. The acknowledged unexplained fallback would not trigger because frame_timing already changed.

Before emitting timing_only, please require equality of all compared rendered payloads after removing the explicitly classified timing fields, or report the narrower observed timing difference with an unexplained remainder. A synthetic regression combining an intermediate-rendered-payload difference with a timestamp difference would cover this gap.

This is static review of the proposed logic, not runtime reproduction; the contract decision remains with morluto.

<!-- gh-comment-id:6093232692 --> @sunxiuguo commented on GitHub (Oct 10, 2026): One guard detail to tighten in the draft: [the current terminal comparison includes every rendered frame](https://github.com/morluto/rea/blob/8550913cda311e5340cf012b9c4afd622e36e35b/src/domain/process/processComparison.ts#L104-L119). If the joined stream and final screen match, but an earlier rendered payload and its timestamp both differ, the listed facets would allow `timing_only`. The acknowledged `unexplained` fallback would not trigger because `frame_timing` already changed. Before emitting `timing_only`, please require equality of all compared rendered payloads after removing the explicitly classified timing fields, or report the narrower observed timing difference with an unexplained remainder. A synthetic regression combining an intermediate-rendered-payload difference with a timestamp difference would cover this gap. This is static review of the proposed logic, not runtime reproduction; the contract decision remains with morluto.
bsakaguchi commented 2026-10-15 18:23:54 +00:00
Author
Owner
Copy link

@CassieFox23 commented on GitHub (Oct 10, 2026):

TL;DR: Thanks, sunxiuguo. Your frame guard catch is now in the plan. I updated the scope comment above.

You were right. I checked the source at de8fc671. Every rendered frame is compared, not just the last one. So a middle frame could differ along with its time, and the old guard would still say timing_only.

What changed in the plan:

  • A new terminal facet, rendered_payloads. It compares every rendered frame with at_ms and sequence removed.
  • timing_only now needs the stream, the final screen and every rendered payload to match.
  • If timing changed next to anything else, the note says mixed. It lists the timing change and the first payload that differs.
  • Test 10 covers it. A middle frame differs, plus its time. The note must never say timing_only.

Nothing is built yet. The call on the contract is still morluto's.

🤖 Generated with Claude Code (Opus 5.5) 🦊

<!-- gh-comment-id:6093368028 --> @CassieFox23 commented on GitHub (Oct 10, 2026): **TL;DR:** Thanks, sunxiuguo. Your frame guard catch is now in the plan. I updated [the scope comment above](https://github.com/morluto/rea/issues/724#issuecomment-6092777944). You were right. I checked the source at `de8fc671`. Every rendered frame is compared, not just the last one. So a middle frame could differ along with its time, and the old guard would still say `timing_only`. What changed in the plan: - A new terminal facet, `rendered_payloads`. It compares every rendered frame with `at_ms` and `sequence` removed. - `timing_only` now needs the stream, the final screen and every rendered payload to match. - If timing changed next to anything else, the note says `mixed`. It lists the timing change and the first payload that differs. - Test 10 covers it. A middle frame differs, plus its time. The note must never say `timing_only`. Nothing is built yet. The call on the contract is still morluto's. 🤖 Generated with Claude Code (Opus 5.5) 🦊
bsakaguchi commented 2026-10-31 00:29:01 +00:00
Author
Owner
Copy link

@Rinkia commented on GitHub (Oct 30, 2026):

This still reproduces on rea-agents@6.8.0 and on main at 45cc93596 (Linux x64 container, MCP). It now also reaches correlate_static_and_runtime.

I ran the same self-written binary twice (capture_process_scenario, arguments: ["1"], normalization.time_bucket_ms: 1000). The terminal text and the exit object {code: 0, signal: 0, reason: "exited"} were identical. compare_process_captures reported terminal: "unchanged" and exit: "changed", and first_divergence pointed to exit, where the only difference was settlement.elapsed_ms (76 vs 68). With the default 10 ms bucket, the same pair also reported terminal: "changed" when the single output frame landed at at_ms 10 vs 50.

In correlate_static_and_runtime, a mapping of a compare_functions pseudocode change to the exit dimension with expected_pattern: "static_only" therefore gives a different classification from run to run with the same binaries. It is contradiction when elapsed_ms differs and hypothesis when it happens to match. Two consecutive runs of the same 7-mapping input on main gave 3 and then 2 contradictions. The terminal mappings were stable with the 1000 ms bucket and correct in every case: an internal-only change gave static_only, a changed constant gave cochanged, and an unchanged function gave both_unchanged.

That matches the per-facet explanation proposed here. A caller using the exit dimension as a stand-in for "exit outcome" currently gets timing noise.

<!-- gh-comment-id:6517800520 --> @Rinkia commented on GitHub (Oct 30, 2026): This still reproduces on `rea-agents@6.8.0` and on `main` at `45cc93596` (Linux x64 container, MCP). It now also reaches `correlate_static_and_runtime`. I ran the same self-written binary twice (`capture_process_scenario`, `arguments: ["1"]`, `normalization.time_bucket_ms: 1000`). The terminal text and the `exit` object `{code: 0, signal: 0, reason: "exited"}` were identical. `compare_process_captures` reported `terminal: "unchanged"` and `exit: "changed"`, and `first_divergence` pointed to `exit`, where the only difference was `settlement.elapsed_ms` (76 vs 68). With the default 10 ms bucket, the same pair also reported `terminal: "changed"` when the single output frame landed at `at_ms` 10 vs 50. In `correlate_static_and_runtime`, a mapping of a `compare_functions` pseudocode change to the `exit` dimension with `expected_pattern: "static_only"` therefore gives a different classification from run to run with the same binaries. It is `contradiction` when `elapsed_ms` differs and `hypothesis` when it happens to match. Two consecutive runs of the same 7-mapping input on main gave 3 and then 2 contradictions. The terminal mappings were stable with the 1000 ms bucket and correct in every case: an internal-only change gave `static_only`, a changed constant gave `cochanged`, and an unchanged function gave `both_unchanged`. That matches the per-facet explanation proposed here. A caller using the `exit` dimension as a stand-in for "exit outcome" currently gets timing noise.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/procedure-reference-stacked
release-please--branches--main--components--rea-agents
release-please--branches--main--components--rea-agents--release-notes
codex/kuna-independent-retry
codex/byte-search-overlay-coverage
codex/open-failure-target-receipt
codex/kuna-xref-evidence
codex/ghidra-ambiguous-procedure-recovery
codex/function-source-bundle
codex/trace-function-integration-prototype
codex/snapshot-source-closure
codex/recorded-evidence-array-pages
codex/evidence-recovery-redesign
codex/trace-feature-address-scope
codex/call-discovery-scope
codex/ghidra-literal-candidate-evidence
codex/decompile-directory-independent-retry
codex/retry-provenance-base
codex/value-flow-capability-experiment
codex/ghidra-containing-instruction
cursor/readme-discord-button-ef9f
dependabot/github_actions/actions/setup-node-7.1.0
dependabot/github_actions/actions/upload-artifact-7.0.2
dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df
codex/kuna-1989
fix/8559-native-ui-fixture-ownership
fix/8555-go-verifier-mcp-ownership
codex/provider-capability-recovery
fix/8642-rizin-target-stdin
codex/ghidra-memory-effect-scope
codex/ghidra-annotation-entry-selector
codex/ghidra-readonly-recovery
codex/ghidra-regex-verifier-recovery
codex/evidence-discovery-view-guidance
fix/8545-bulk-append-overflow
fix/8428-navigation-refusal-alternatives
codex/address-context-recovery
codex/matching-member-limitations
codex/unknown-scope-contract
codex/evidence-view-guidance
feat/8373-native-trace-views
codex/ghidra-snapshot-verifier-binding
codex/elf-export-visibility
codex/evidence-view-recovery
codex/native-reference-cli-help
codex/provider-candidate-observation-scope
codex/kuna-session-verification
codex/ghidra-switch-lifetime-guidance
codex/pseudocode-paging-discovery
codex/clarify-evidence-view-support
codex/kuna-xrefs-selector-recovery
codex/context-capacity-recovery
codex/evidence-closure-large-links
codex/same-function-evidence-identity
codex/ghidra-verifier-inspected-jdk
codex/audit-guided-workflows
codex/crash-prompt-retained-dossiers
codex/analysis-view-error-recovery
codex/kuna-verifier-target-selection
codex/ghidra-selected-project-call-coverage
codex/docs-native-analysis-scope
codex/ghidra-java-socket-boundary
codex/ghidra-retained-snapshot-verifier
codex/ghidra-search-inventory-facts
fix/8284-zero-argument-parameter-bindings
fix/8198-ui-action-handler-offset
fix/8257-elf-entry-label-identity
fix/8304-deferred-output-schema-projection
fix/8277-call-path-computed-target-uncertainty
fix/pr-8089-conflicts
fix/ci-static-duplicate-export
dependabot/npm_and_yarn/lint-staged-17.6.0
dependabot/github_actions/actions/download-artifact-8.0.2
dependabot/npm_and_yarn/write-file-atomic-8.0.0
dependabot/npm_and_yarn/canonicalize-5.1.0
dependabot/github_actions/actions/cache/restore-6.1.0
dependabot/npm_and_yarn/toon-format/toon-4.4.0
dependabot/github_actions/actions/deploy-pages-5.0.1
codex/review-fixes-pr-2403
codex/review-fixes-pr-2400
codex/process-terminal-width
codex/test-brittleness-cleanup
verify/7167-browser-connect-macos
fix/7167-browser-connect
feat/native-matching-allowance-setting
feat/browser-scenario-dry-run
test/managed-caller-parameters-20261024
codex/parser-fortythird-boundaries
fix/6926-pinvoke-platform-decorations-20261023
fix/6889-har-body-encoding-20261023
fix/6888-sqlite-schema-budget-20261023
fix/dependency-security-gates-6708
cursor/evidence-overwrite-rebase-b4ac
feat/4057-verify-native-exports
fix/windows-acl-observer-overhead-20261022
cursor/erlang-beam-restack-4476
cursor/pe-certificate-review-bf32
fix/6116-javascript-application-fs-errors
fix/6061-unmapped-runtime-script-unknown
feat/browser-scenario-summary
fix/pe-certificates
codex/verify-3407
cursor/rebase-5699
cursor/rebase-5352
cursor/rebase-5600
cursor/fix-cli-schema-omission-10a8
codex/audit-hopper-shutdown-reasons-20261021
fix/demangle-swift-omitted-record
fix/nib-boolean-value-types
cursor/cancellation-partial-observation-a5ac
fix/ib-object-limit-reasons
fix/electron-fuse-limit-advice
fix/kuna-doctor-dangling-cache
fix/fixed-budget-refusal-advice
fix/js-ipc-listener-dedup
fix/js-return-member-provenance
fix/dmg-integrity-retry-advice
fix/ida-doctor-readiness
fix/kuna-version-probe-output-drain
fix/legacy-swift-runtime-name-matching
fix/dylib-lazy-load-commands
fix/browser-scenario-startup-cancel
fix/swift-nested-punycode-classification
fix/compare-unselected-scopes
fix/objc-method-budget-coverage
fix/dmg-mounted-root-observations
audit/ci-tests-20261019-round2
release/6.7.0
fix/wabt-cleanup-retry-resolution
fix/electron-fuse-elf-pe-architecture
fix/native-plist-descriptor-ownership
fix/kuna-caller-cleanup-ownership
experiment/atomic-codemode-20261016
review/first-contributor-3776
fix/call-resolution-full-read-cycles
fix/member-mutation-selection-repeat
experiment/atomic-re-tools-20261015
perf/member-mutation-lazy-origins
perf/static-network-mutation-index
fix/hopper-startup-lifecycle
audit/3050-windows-proof-20261015
feat/windows-window-capture-2183
audit/3049-windows-proof-20261015
audit/3735-macos-fixtures-20261015
fix/batch-decompile-null-pseudocode
fix/kuna-inventory-honors-jobs
experiment/browser-codemode-tool-design-20261015
fix/hopper-clean-runtime-argv
codex/review-2049-27f1b367
review-fixes/pr-2787-routing
fix/3360-scoped-this-emitters
fix/3367-replaced-global-effects
fix/3838-compact-any-value-items
test/hopper-download-probe-20261015
codex/windows-runtime-creation-red-green-20261015
codex/windows-readonly-root-red-green-20261015
codex/windows-root-share-red-green-20261015
codex/exp-1685-win-mcp-startup
codex/bug-sweep-20261015
investigate/2190-windows-package-phases
investigate/3543-macos-cli-deadline
perf/3281-reverse-tools-feedback-quality
fix/3406-hopper-doctor-fixture
fix/3341-wrapper-require-test-context
fix/3301-owned-uninstall-fixtures
fix/3164-compiler-input-ownership
fix/pcap-cleanup-stdout-newline-20261014
perf/interface-builder-summary
codex/review-fixes-pr-1977
fix/ghidra-effective-jvm-platform
fix/docs-turkish-rust-link-20261013
fix/main-maintenance-ci-20261010
fix/recover-binary-session-process-effects
codex/review-fixes-pr-2457
codex/ci-required-cancellation-gate
fix/main-format-baseline-20261013
fix/setup-preserve-entry-fields
cursor/restore-main-checks-1ffc
fix/ghidra-bridge-hashmap-import
codex/shared-ci-jadx-json-default-20261013
codex/shared-ci-ghidra-test-groups-20261013
codex/shared-ci-java-fixture-format-20261013
codex/fix-intrinsic-invocation-boundaries
test/release-preflight-reusable-jobs
test/js-parameter-provenance-expectation
fix/linux-hopper-verification-current
test/2419-ghidra-elf-probe-size
followup/pr-1544-native-ida
codex/fold-pr-2230-contribution
docs/readme-responsible-use
fix/issue-2064
fix/issue-2050
fix/issue-2053
fix/issue-1968
fix/issue-1945
experiment/codemode-bridge-20261011
fix/tool-effects-negative-wording-1904-20261012
fix/1842-electron-interop-provenance
feat/recover-javascript-multi-input
refactor/website-i18n-prep
fix/nativeaot-pe-virtual-extent
fix/browser-verifier-ci-hang
fix/pe-resources-input-selection
docs/beginner-friendly-guides
docs/beginner-setup-guide
fix/javascript-analysis-worker-1563
fix/1649-ghidra-duplicate-names
fix/javascript-export-keys-integration-1369
test/maintenance-boundary-contracts-20261011
review/javascript-export-keys-1369
review/process-live-progress-1483
test/javascript-1495-growth-ratio
fix/ci-escape-timing
docs/normalized-terminal-geometry-1450
release/6.3.0
archive/release-6.3.0-pr1381
release/6.2.0
fix/electron-preload-dynamic-override
fix/javascript-self-referential-bundler-edges
release/6.1.0
cursor/grok-client-cherry-pick-0b56
feat/signature-posture
feat/dyld-shared-cache
feat/artifact-integrity-policy
feat/pkg-inventory
release/6.0.0
chore/sync-release-5.0.0
release/5.0.0
cursor/cloud-agent-node-787e
fix/javascript-xhr-open-endpoints
review/pr-711-realpath-fallback-wording
cleanup/replay-confinement
codex/test-suite-performance
codex/issue-foundations
codex/address-open-issues-2026-07-20
agent/javascript-vite-rollup-esbuild-adapters
agent/process-cleanup
agent/triage-integration
agent/unknown-commands-registry
feat/mcp-identity-metadata
morlvps/issue_cleanup
codex/installer-redesign
morl/roadmap-investigation-core
morl/roadmap-investigation-docs
release-please--branches--main--components--rea
rea-agents-6.8.0
rea-agents-6.7.0
rea-agents-6.6.0
experiment/atomic-codemode-wait-runtime-20261016
rea-agents-6.5.0
rea-agents-6.4.0
rea-agents-6.3.0
rea-agents-6.2.0
rea-agents-6.1.0
rea-agents-6.0.0
rea-agents-5.0.0
rea-agents-4.1.0
rea-agents-4.0.1
rea-agents-4.0.0
rea-agents-3.2.1
rea-agents-3.2.0
rea-agents-3.1.0
rea-agents-3.0.0
rea-agents-2.7.0
rea-agents-2.6.0
rea-agents-2.5.1
rea-agents-2.5.0
rea-agents-2.4.0
rea-agents-2.3.0
rea-agents-2.2.0
rea-agents-2.1.0
rea-agents-2.0.0
rea-agents-1.7.0
rea-agents-1.6.0
rea-agents-1.5.0
rea-agents-1.4.0
rea-agents-1.3.0
rea-agents-1.2.0
rea-agents-1.1.0
rea-agents-1.0.0
rea-agents-0.5.0
rea-agents-0.4.0
rea-agents-0.3.0
rea-0.2.1
rea-0.2.0
Labels
Clear labels   
area: bridge

   
area: bridge

   
area: build

   
area: build

   
area: build

   
area: cli

   
area: domain

   
area: evidence

   
area: mcp

   
area: native

   
area: providers

   
area: runtime

   
area: server

   
area: setup

   
area: tests

   
bug

   
design

   
documentation

   
enhancement

   
enhancement

   
enhancement

   
experiment

   
help wanted

   
i18n

   
invalid

   
P0: critical

   
P1: high

   
P2: medium

   
P2: medium

   
P2: medium

   
P3: low

   
performance

   
provider-request

   
pull-request
Mirrored from GitHub Pull Request

  
question

   
roadmap

   
triage
No labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
question
roadmap
triage
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#322
No description provided.