[GH-ISSUE #406] Add end-to-end agent reconstruction readiness conformance #252

Closed
opened 2026-10-15 18:23:00 +00:00 by bsakaguchi · 0 comments
Owner

Originally created by @morluto on GitHub (Jul 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/406

Problem

REA has individual capabilities for artifact classification, provider analysis, Evidence v2, process/browser/Electron capture, differential comparison, residual unknowns, and reconstruction verification. Individual feature tests do not prove that an agent can traverse the complete workflow without discovering hidden limits, incompatible providers, stale servers, missing authority, broken host runtimes, or false equivalence only after invocation.

Several previously separate failures illustrate the integration risk:

  • a tool declaration can omit an effective numeric limit even when runtime validation knows it;
  • an artifact can be accepted for inventory but routed to an analysis provider that cannot interpret its format;
  • a newly installed CLI can coexist with a stale connected MCP server/catalog;
  • a gated operation can require startup policy even when a narrow session grant would be sufficient;
  • host runtime/linker failure can be presented as an analysis failure;
  • a structured operation failure can be mistaken for process success;
  • one integrity contradiction can either abort all unaffected comparison or be ignored unsafely;
  • concurrent runtime schedules can be compared with unsafe total-order or broad-sorting semantics;
  • green component tests can coexist with unowned reconstruction surfaces.

These are not adequately covered by testing each tool in isolation. REA needs a product-agnostic, public-contract conformance journey that proves the catalog, schemas, permissions, providers, evidence, comparison, and closure layers compose.

Outcome

Add an end-to-end agent reconstruction readiness suite and a machine-readable report. The suite should run representative native, generated JavaScript CLI/TUI, and Electron fixtures through the supported public CLI and MCP contracts.

The result should answer:

  1. Could an agent discover the correct workflow before making an invalid call?
  2. Were exact limits, authority requirements, side effects, and unavailable capabilities visible?
  3. Did setup/preflight distinguish server drift, provider absence, target incompatibility, and broken host dependencies?
  4. Did every failure propagate consistently through structured results and process status?
  5. Did capture preserve complete, bounded, attributable Evidence?
  6. Did comparison handle concurrency, contradictions, truncation, and missing authority without false equivalence?
  7. Did reconstruction verification identify every remaining unowned or unverified obligation?

Conformance journey

1. Discover and classify

  • Start MCP target-free.
  • Inventory the artifact without launching an incompatible provider.
  • Classify format, runtime family, packaging, architecture, and candidate workflow.
  • Query capability-aware discovery and select a compatible provider family.
  • Confirm exact schema limits and effective host/provider/policy constraints are visible before invocation.

2. Diagnose environment and version state

  • Detect a deliberately broken runtime dependency separately from target-analysis failure.
  • Detect installed CLI, connected MCP server, catalog/skill, and provider version skew.
  • Return an actionable restart/setup path without claiming the requested analysis ran.
  • Re-run after correction and prove discovery state/revision changes.

3. Acquire bounded authority

  • Begin with process/evidence/workspace authority absent.
  • Report the exact missing scope.
  • Where elicitation is supported, grant only the required operation/session scope and resume.
  • Where it is unsupported, return exact narrow configuration and restart requirements.
  • Prove that scope expansion re-prompts and that denial/cancellation launches nothing.

4. Perform static analysis

  • Route native artifacts to a compatible native provider.
  • Route generated JavaScript to application-graph and semantic workflows rather than Hopper/Ghidra procedure analysis.
  • Preserve static observations, derived relations, ambiguities, unsupported syntax, truncation, and residual unknowns distinctly.

5. Run reactive dynamic scenarios

  • Wait for terminal/process/filesystem/HTTP/WebSocket predicates rather than sleeping.
  • Send bounded input, resize, signal, answer controlled peers, and checkpoint.
  • Capture root and descendant lifecycle through settlement and teardown.
  • For Electron, correlate main, preload, renderer, utility, IPC, shell, deep-link, and error evidence.
  • Cancel one run and prove owned cleanup plus retained diagnostic Evidence.

6. Compare authority and candidate

  • Run repeated authority and candidate scenarios from reset environments.
  • Validate exact event fields before schedule canonicalization.
  • Accept declared concurrent schedules through partial-order/finite-trace semantics.
  • Localize a deliberate primary divergence and distinguish downstream effects.
  • Treat truncation, unavailable authority, and instability as unknown rather than pass.

7. Preserve contradictions

  • Introduce one declared-versus-observed artifact integrity mismatch.
  • Require explicit record-and-continue policy.
  • Preserve both hashes, mark affected evidence untrusted, and continue unaffected comparisons.
  • Prevent any equivalence claim over the contradicted entry.

8. Verify reconstruction closure

  • Import observed boundaries into a reconstruction obligation workspace/ledger.
  • Bind owners, parsers/types, positive and negative fixtures, and runtime verifiers.
  • Keep the result partial while one packaged/process/persistence/Electron obligation is missing despite green component tests.
  • Close only the obligation proven by each new verifier.
  • Fail a candidate that delegates back to the authority.

9. Export and replay

  • Export deterministic Evidence bundles, comparison specification, scenario, obligation ledger, and conformance report.
  • Replay from a clean checkout/isolated environment using public contracts.
  • Detect tampering, stale digests, widened normalization, missing dimensions, and version incompatibility.

Fixtures

Use synthetic, redistributable fixtures with no product-specific identifiers or external production services:

  • a small native executable with stable symbols plus a deliberate version change;
  • a generated/minified JavaScript CLI with prompts, alternate-screen TUI, configuration precedence, child process, filesystem persistence, loopback HTTP/WebSocket behavior, and signal teardown;
  • an Electron application with main/preload/renderer/utility roles, bidirectional IPC, multiple windows, renderer restart, blocked external shell effect, and deterministic deep link;
  • an intentionally incomplete reconstruction whose component tests pass;
  • a broken-runtime fixture or controlled dependency probe that does not modify the user's global installation.

Every fixture should include deliberate faults so the suite proves diagnostic quality, not just happy-path execution.

Machine-readable report

Emit a versioned report containing:

  • REA CLI/server/catalog/provider versions and digests;
  • negotiated MCP client capabilities;
  • effective capability records and exact limits;
  • fixture/artifact/environment identities;
  • selected and rejected workflow candidates with reason codes;
  • grants/approvals by opaque ID and scope, excluding secrets;
  • Evidence bundle IDs and completeness/authority classifications;
  • comparison verdicts and minimal divergence references;
  • contradiction and residual-unknown records;
  • reconstruction obligations by status;
  • cleanup, cancellation, and replay results;
  • per-stage pass, fail, unknown, unsupported, truncated, or skipped status.

An aggregate pass is allowed only when every required stage passes. Unsupported, skipped, truncated, and unknown must never collapse into success.

CLI and MCP parity

  • Exercise both surfaces from the same fixture definitions.
  • Structured outcomes, evidence identities, limits, and error classifications must agree.
  • Expected operation failure returns nonzero CLI status while MCP returns a typed non-success result.
  • Long-running stages expose meaningful progress and cancellation where negotiated.
  • Large reports and artifacts are available as resources rather than oversized tool responses.

Acceptance criteria

  • A documented command runs the complete synthetic reconstruction journey from a clean checkout.
  • A corresponding MCP-driven evaluation reaches the same semantic results without private/internal APIs.
  • Invalid provider routing is prevented before expensive invocation.
  • Exact effective limits are present in discovery/schema and repeated in validation errors.
  • Broken host runtime and stale connected server/catalog cases have distinct actionable diagnoses.
  • Missing authority supports narrow elicitation or a precise non-elicitation fallback.
  • Reactive CLI/TUI and Electron scenarios require no external Expect/PTY orchestration framework.
  • Concurrent valid schedules compare equivalent only under explicit partial-order/finite-trace rules.
  • A deliberate divergence is localized; a deliberate contradiction remains visible and blocks affected equivalence.
  • The incomplete reconstruction remains partial until every named obligation has comparable Evidence and a passing verifier.
  • Cancellation cleans only owned resources and preserves diagnostic Evidence.
  • CLI/MCP failures cannot be interpreted as success by ordinary automation.
  • Exported conformance artifacts replay deterministically and detect tampering or stale inputs.
  • CI publishes a compact summary and retains full reports as artifacts/resources.
  • Documentation maps every failing stage to the responsible capability issue and next action.

Evaluation metrics

Track at least:

  • first valid workflow/tool selection rate;
  • malformed or over-limit call rate;
  • incompatible-provider invocation rate;
  • number of failed calls required to discover prerequisites;
  • recovery rate after typed setup/permission errors;
  • authority-to-candidate divergence localization precision;
  • false-equivalence rate under truncation/concurrency/contradiction;
  • unowned-obligation detection rate;
  • cleanup leaks and nondeterministic replay rate.

Relationships

  • Agent-first platform roadmap: #32
  • Evidence/process fidelity epic: #48
  • Investigation and reconstruction workflows: #18 and #37
  • Reactive scenarios: #330
  • Partial-order comparison: #403
  • JavaScript runtime semantics: #404
  • Reconstruction closure: #405
  • Electron lifecycle capture: #375
  • Scoped grants: #96
  • Capability-aware discovery: #101
  • Integrity continuation, exit status, stale-server detection, runtime diagnosis, and schema-limit work: #112, #113, #115, #262, and #261

Non-goals

  • Declaring global behavioral equivalence from a finite suite.
  • Adding product-specific fixtures, credentials, endpoints, or identifiers.
  • Replacing focused unit/provider conformance tests.
  • Requiring unavailable private server behavior to be reconstructed.
  • Automatically granting authority or weakening startup policy.
Originally created by @morluto on GitHub (Jul 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/406 ## Problem REA has individual capabilities for artifact classification, provider analysis, Evidence v2, process/browser/Electron capture, differential comparison, residual unknowns, and reconstruction verification. Individual feature tests do not prove that an agent can traverse the complete workflow without discovering hidden limits, incompatible providers, stale servers, missing authority, broken host runtimes, or false equivalence only after invocation. Several previously separate failures illustrate the integration risk: - a tool declaration can omit an effective numeric limit even when runtime validation knows it; - an artifact can be accepted for inventory but routed to an analysis provider that cannot interpret its format; - a newly installed CLI can coexist with a stale connected MCP server/catalog; - a gated operation can require startup policy even when a narrow session grant would be sufficient; - host runtime/linker failure can be presented as an analysis failure; - a structured operation failure can be mistaken for process success; - one integrity contradiction can either abort all unaffected comparison or be ignored unsafely; - concurrent runtime schedules can be compared with unsafe total-order or broad-sorting semantics; - green component tests can coexist with unowned reconstruction surfaces. These are not adequately covered by testing each tool in isolation. REA needs a product-agnostic, public-contract conformance journey that proves the catalog, schemas, permissions, providers, evidence, comparison, and closure layers compose. ## Outcome Add an end-to-end agent reconstruction readiness suite and a machine-readable report. The suite should run representative native, generated JavaScript CLI/TUI, and Electron fixtures through the supported public CLI and MCP contracts. The result should answer: 1. Could an agent discover the correct workflow before making an invalid call? 2. Were exact limits, authority requirements, side effects, and unavailable capabilities visible? 3. Did setup/preflight distinguish server drift, provider absence, target incompatibility, and broken host dependencies? 4. Did every failure propagate consistently through structured results and process status? 5. Did capture preserve complete, bounded, attributable Evidence? 6. Did comparison handle concurrency, contradictions, truncation, and missing authority without false equivalence? 7. Did reconstruction verification identify every remaining unowned or unverified obligation? ## Conformance journey ### 1. Discover and classify - Start MCP target-free. - Inventory the artifact without launching an incompatible provider. - Classify format, runtime family, packaging, architecture, and candidate workflow. - Query capability-aware discovery and select a compatible provider family. - Confirm exact schema limits and effective host/provider/policy constraints are visible before invocation. ### 2. Diagnose environment and version state - Detect a deliberately broken runtime dependency separately from target-analysis failure. - Detect installed CLI, connected MCP server, catalog/skill, and provider version skew. - Return an actionable restart/setup path without claiming the requested analysis ran. - Re-run after correction and prove discovery state/revision changes. ### 3. Acquire bounded authority - Begin with process/evidence/workspace authority absent. - Report the exact missing scope. - Where elicitation is supported, grant only the required operation/session scope and resume. - Where it is unsupported, return exact narrow configuration and restart requirements. - Prove that scope expansion re-prompts and that denial/cancellation launches nothing. ### 4. Perform static analysis - Route native artifacts to a compatible native provider. - Route generated JavaScript to application-graph and semantic workflows rather than Hopper/Ghidra procedure analysis. - Preserve static observations, derived relations, ambiguities, unsupported syntax, truncation, and residual unknowns distinctly. ### 5. Run reactive dynamic scenarios - Wait for terminal/process/filesystem/HTTP/WebSocket predicates rather than sleeping. - Send bounded input, resize, signal, answer controlled peers, and checkpoint. - Capture root and descendant lifecycle through settlement and teardown. - For Electron, correlate main, preload, renderer, utility, IPC, shell, deep-link, and error evidence. - Cancel one run and prove owned cleanup plus retained diagnostic Evidence. ### 6. Compare authority and candidate - Run repeated authority and candidate scenarios from reset environments. - Validate exact event fields before schedule canonicalization. - Accept declared concurrent schedules through partial-order/finite-trace semantics. - Localize a deliberate primary divergence and distinguish downstream effects. - Treat truncation, unavailable authority, and instability as unknown rather than pass. ### 7. Preserve contradictions - Introduce one declared-versus-observed artifact integrity mismatch. - Require explicit record-and-continue policy. - Preserve both hashes, mark affected evidence untrusted, and continue unaffected comparisons. - Prevent any equivalence claim over the contradicted entry. ### 8. Verify reconstruction closure - Import observed boundaries into a reconstruction obligation workspace/ledger. - Bind owners, parsers/types, positive and negative fixtures, and runtime verifiers. - Keep the result partial while one packaged/process/persistence/Electron obligation is missing despite green component tests. - Close only the obligation proven by each new verifier. - Fail a candidate that delegates back to the authority. ### 9. Export and replay - Export deterministic Evidence bundles, comparison specification, scenario, obligation ledger, and conformance report. - Replay from a clean checkout/isolated environment using public contracts. - Detect tampering, stale digests, widened normalization, missing dimensions, and version incompatibility. ## Fixtures Use synthetic, redistributable fixtures with no product-specific identifiers or external production services: - a small native executable with stable symbols plus a deliberate version change; - a generated/minified JavaScript CLI with prompts, alternate-screen TUI, configuration precedence, child process, filesystem persistence, loopback HTTP/WebSocket behavior, and signal teardown; - an Electron application with main/preload/renderer/utility roles, bidirectional IPC, multiple windows, renderer restart, blocked external shell effect, and deterministic deep link; - an intentionally incomplete reconstruction whose component tests pass; - a broken-runtime fixture or controlled dependency probe that does not modify the user's global installation. Every fixture should include deliberate faults so the suite proves diagnostic quality, not just happy-path execution. ## Machine-readable report Emit a versioned report containing: - REA CLI/server/catalog/provider versions and digests; - negotiated MCP client capabilities; - effective capability records and exact limits; - fixture/artifact/environment identities; - selected and rejected workflow candidates with reason codes; - grants/approvals by opaque ID and scope, excluding secrets; - Evidence bundle IDs and completeness/authority classifications; - comparison verdicts and minimal divergence references; - contradiction and residual-unknown records; - reconstruction obligations by status; - cleanup, cancellation, and replay results; - per-stage `pass`, `fail`, `unknown`, `unsupported`, `truncated`, or `skipped` status. An aggregate pass is allowed only when every required stage passes. Unsupported, skipped, truncated, and unknown must never collapse into success. ## CLI and MCP parity - Exercise both surfaces from the same fixture definitions. - Structured outcomes, evidence identities, limits, and error classifications must agree. - Expected operation failure returns nonzero CLI status while MCP returns a typed non-success result. - Long-running stages expose meaningful progress and cancellation where negotiated. - Large reports and artifacts are available as resources rather than oversized tool responses. ## Acceptance criteria - [ ] A documented command runs the complete synthetic reconstruction journey from a clean checkout. - [ ] A corresponding MCP-driven evaluation reaches the same semantic results without private/internal APIs. - [ ] Invalid provider routing is prevented before expensive invocation. - [ ] Exact effective limits are present in discovery/schema and repeated in validation errors. - [ ] Broken host runtime and stale connected server/catalog cases have distinct actionable diagnoses. - [ ] Missing authority supports narrow elicitation or a precise non-elicitation fallback. - [ ] Reactive CLI/TUI and Electron scenarios require no external Expect/PTY orchestration framework. - [ ] Concurrent valid schedules compare equivalent only under explicit partial-order/finite-trace rules. - [ ] A deliberate divergence is localized; a deliberate contradiction remains visible and blocks affected equivalence. - [ ] The incomplete reconstruction remains partial until every named obligation has comparable Evidence and a passing verifier. - [ ] Cancellation cleans only owned resources and preserves diagnostic Evidence. - [ ] CLI/MCP failures cannot be interpreted as success by ordinary automation. - [ ] Exported conformance artifacts replay deterministically and detect tampering or stale inputs. - [ ] CI publishes a compact summary and retains full reports as artifacts/resources. - [ ] Documentation maps every failing stage to the responsible capability issue and next action. ## Evaluation metrics Track at least: - first valid workflow/tool selection rate; - malformed or over-limit call rate; - incompatible-provider invocation rate; - number of failed calls required to discover prerequisites; - recovery rate after typed setup/permission errors; - authority-to-candidate divergence localization precision; - false-equivalence rate under truncation/concurrency/contradiction; - unowned-obligation detection rate; - cleanup leaks and nondeterministic replay rate. ## Relationships - Agent-first platform roadmap: #32 - Evidence/process fidelity epic: #48 - Investigation and reconstruction workflows: #18 and #37 - Reactive scenarios: #330 - Partial-order comparison: #403 - JavaScript runtime semantics: #404 - Reconstruction closure: #405 - Electron lifecycle capture: #375 - Scoped grants: #96 - Capability-aware discovery: #101 - Integrity continuation, exit status, stale-server detection, runtime diagnosis, and schema-limit work: #112, #113, #115, #262, and #261 ## Non-goals - Declaring global behavioral equivalence from a finite suite. - Adding product-specific fixtures, credentials, endpoints, or identifiers. - Replacing focused unit/provider conformance tests. - Requiring unavailable private server behavior to be reconstructed. - Automatically granting authority or weakening startup policy.
bsakaguchi 2026-10-15 18:23:00 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#252
No description provided.