[GH-ISSUE #61] Add normalized differential comparison for process captures

bsakaguchi commented

2026-10-15 18:20:27 +00:00

Owner

Originally created by @morluto on GitHub (Jul 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/61

Problem

Raw process captures contain unstable clocks, PIDs, ports, and paths, making faithful implementations appear different and missing evidence appear equal.

Outcome

Add compare_process_captures over two evidence-backed captures.

Scope

Compare normalized PTY frames, process lifecycles, filesystem effects, HTTP requests/responses, WebSocket frames, exits/signals, cleanup, truncation, and residual unknowns. Classify facts as unchanged, added, removed, changed, or unknown and cite both evidence IDs.

Normalization rules are explicit inputs and included in derived evidence identity. Raw evidence remains available for audit.

Acceptance criteria

Equivalent scenarios with different PIDs/ports/temp paths compare unchanged.
Real text, ordering, exit, filesystem, or protocol changes are classified precisely.
Missing, truncated, unsupported, or redacted fields become unknown, never unchanged.
Comparison is deterministic, bounded, cancellable, and provider-neutral.

Dependencies

Parent: #48. Requires Evidence v2 bundles and all process capture projections.

Non-goals

Automatic claims of implementation equivalence or prose-only summaries.

Root capability expansion: paired experiments and first-divergence localization

Comparison must grow from two supplied captures into a workflow that executes, repeats, compares, and explains two implementations under one committed scenario.

Paired-run contract

Accept authority/candidate identities, shared scenario with explicit side bindings, repeat count, freshness policy, approved normalization, required dimensions, and finite reconstruction claims. Validate scenario, replay, shim, normalization, schema, and comparison-contract compatibility before launch.

Repeatability

Compare same-side runs first. Classify stable facts, timing variance, nondeterministic output, flaky exit, environmental drift, and insufficient samples. Do not claim cross-side equivalence while either side is unstable in a required dimension. Preserve all run evidence and a canonical repeatability digest. Proposed normalization changes require explicit approval.

First-divergence report

Return dimension/path, authority/candidate values, raw and normalized PTY context, rendered before/after state, nearest interaction/process/file/protocol/shim events, scenario milestone, primary/downstream/unknown classification, evidence IDs, and a bounded next probe. Earliest wall-clock timestamp is not automatically causal.

Acceptance

Run a stable authority and deliberately faulty candidate three times through a reactive terminal/network/filesystem scenario. Identify the injected first divergence, relate downstream differences without mislabeling them primary, and fail when normalization hides a semantic field.

Originally created by @morluto on GitHub (Jul 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/61 ## Problem Raw process captures contain unstable clocks, PIDs, ports, and paths, making faithful implementations appear different and missing evidence appear equal. ## Outcome Add `compare_process_captures` over two evidence-backed captures. ## Scope Compare normalized PTY frames, process lifecycles, filesystem effects, HTTP requests/responses, WebSocket frames, exits/signals, cleanup, truncation, and residual unknowns. Classify facts as unchanged, added, removed, changed, or unknown and cite both evidence IDs. Normalization rules are explicit inputs and included in derived evidence identity. Raw evidence remains available for audit. ## Acceptance criteria - Equivalent scenarios with different PIDs/ports/temp paths compare unchanged. - Real text, ordering, exit, filesystem, or protocol changes are classified precisely. - Missing, truncated, unsupported, or redacted fields become unknown, never unchanged. - Comparison is deterministic, bounded, cancellable, and provider-neutral. ## Dependencies Parent: #48. Requires Evidence v2 bundles and all process capture projections. ## Non-goals Automatic claims of implementation equivalence or prose-only summaries. ## Root capability expansion: paired experiments and first-divergence localization Comparison must grow from two supplied captures into a workflow that executes, repeats, compares, and explains two implementations under one committed scenario. ### Paired-run contract Accept authority/candidate identities, shared scenario with explicit side bindings, repeat count, freshness policy, approved normalization, required dimensions, and finite reconstruction claims. Validate scenario, replay, shim, normalization, schema, and comparison-contract compatibility before launch. ### Repeatability Compare same-side runs first. Classify stable facts, timing variance, nondeterministic output, flaky exit, environmental drift, and insufficient samples. Do not claim cross-side equivalence while either side is unstable in a required dimension. Preserve all run evidence and a canonical repeatability digest. Proposed normalization changes require explicit approval. ### First-divergence report Return dimension/path, authority/candidate values, raw and normalized PTY context, rendered before/after state, nearest interaction/process/file/protocol/shim events, scenario milestone, primary/downstream/unknown classification, evidence IDs, and a bounded next probe. Earliest wall-clock timestamp is not automatically causal. ### Acceptance Run a stable authority and deliberately faulty candidate three times through a reactive terminal/network/filesystem scenario. Identify the injected first divergence, relate downstream differences without mislabeling them primary, and fail when normalization hides a semantic field.

bsakaguchi

2026-10-15 18:20:27 +00:00

closed this issue
added the
area: server

P1: high

area: domain

roadmap

enhancement
labels

bsakaguchi commented

2026-10-15 18:20:29 +00:00

Author

Owner

@morluto commented on GitHub (Jul 17, 2026):

Current main ships normalized supplied-capture comparison, unknown handling, and first-divergence reporting. The remaining scope is the paired experiment runner, same-side repeatability and instability classification, multi-run orchestration, causal primary/downstream context, and normalization approval. It depends on the expanded #57/#60 contracts.

@morluto commented on GitHub (Jul 17, 2026): Current main ships normalized supplied-capture comparison, unknown handling, and first-divergence reporting. The remaining scope is the paired experiment runner, same-side repeatability and instability classification, multi-run orchestration, causal primary/downstream context, and normalization approval. It depends on the expanded #57/#60 contracts.

bsakaguchi commented

2026-10-15 18:20:29 +00:00

Author

Owner

@morluto commented on GitHub (Jul 17, 2026):

Current-main audit: normalized seven-dimension Process Capture comparison, first-divergence reporting, CLI/MCP parity, and tests are delivered. Remaining scope is a paired experiment runner with repeated same-side trials and explicit repeatability/instability classification; the current tool compares only two supplied captures. Keep open for that orchestration layer.

@morluto commented on GitHub (Jul 17, 2026): Current-main audit: normalized seven-dimension Process Capture comparison, first-divergence reporting, CLI/MCP parity, and tests are delivered. Remaining scope is a paired experiment runner with repeated same-side trials and explicit repeatability/instability classification; the current tool compares only two supplied captures. Keep open for that orchestration layer.

bsakaguchi commented

2026-10-15 18:20:29 +00:00

Author

Owner

@morluto commented on GitHub (Jul 20, 2026):

Closing as completed. The normalized process-capture comparison and paired-experiment orchestration are now in the tree:

compare_process_captures is a first-class MCP/CLI tool comparing two Process Capture v4 observations across terminal, interaction, exit, filesystem, HTTP, WebSocket, and other dimensions, with first-divergence reporting.
src/application/ProcessPairedExperiment.ts / src/domain/processPairedExperiment.ts run repeatability-first authority/candidate experiments: same-side repeats, stable-side classification, timing-vs-semantic drift detection, freshness-gated cross-side comparison, and cancellation.
tests/processPairedExperiment.test.ts exercises the paired-run contract and repeatability acceptance scenarios.

If CLI/MCP wiring for the paired workflow itself is still desired, that can be opened as a focused follow-up.

@morluto commented on GitHub (Jul 20, 2026): Closing as completed. The normalized process-capture comparison and paired-experiment orchestration are now in the tree: - `compare_process_captures` is a first-class MCP/CLI tool comparing two Process Capture v4 observations across terminal, interaction, exit, filesystem, HTTP, WebSocket, and other dimensions, with first-divergence reporting. - `src/application/ProcessPairedExperiment.ts` / `src/domain/processPairedExperiment.ts` run repeatability-first authority/candidate experiments: same-side repeats, stable-side classification, timing-vs-semantic drift detection, freshness-gated cross-side comparison, and cancellation. - `tests/processPairedExperiment.test.ts` exercises the paired-run contract and repeatability acceptance scenarios. If CLI/MCP wiring for the paired workflow itself is still desired, that can be opened as a focused follow-up.

Rows
Columns

[GH-ISSUE #61] Add normalized differential comparison for process captures #48

Problem

Outcome

Scope

Acceptance criteria

Dependencies

Non-goals

Root capability expansion: paired experiments and first-divergence localization

Paired-run contract

Repeatability

First-divergence report

Acceptance