[PR #103] [MERGED] feat: add persistent cross-version investigation workspaces #1834

New issue

Closed

opened 2026-10-16 21:11:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:11:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/103
Author: @N0zoM1z0
Created: 7/13/2026
Status: ✅ Merged
Merged: 7/13/2026
Merged by: @morluto

Base: main ← Head: feat/persistent-workspace-investigation-runs

📝 Commits (4)

41366ed feat: add persistent cross-version investigation workspaces
0dc7d9a chore: merge main into persistent workspace branch
ee785ca Merge remote-tracking branch 'origin/main' into feat/persistent-workspace-investigation-runs
a2076b4 fix(security): restrict investigation artifact inputs

📊 Changes

120 files changed (+4086 additions, -387 deletions)

View changed files

📝 README.md (+19 -3)
📝 docs/api/assets/hierarchy.js (+1 -1)
📝 docs/api/assets/navigation.js (+1 -1)
📝 docs/api/assets/search.js (+1 -1)
📝 docs/api/classes/application_EvidenceLedger.EvidenceLedger.html (+13 -13)
📝 docs/api/classes/application_ProcessHarness.ProcessCaptureError.html (+1 -1)
📝 docs/api/classes/domain_errors.AnalysisCancelledError.html (+5 -5)
📝 docs/api/classes/domain_errors.AnalysisCapabilityUnavailableError.html (+7 -7)
📝 docs/api/classes/domain_errors.AnalysisError.html (+5 -4)
📝 docs/api/classes/domain_errors.AnalysisInputError.html (+5 -5)
📝 docs/api/classes/domain_errors.AnalysisOutputError.html (+6 -6)
📝 docs/api/classes/domain_errors.AnalysisProtocolError.html (+3 -3)
📝 docs/api/classes/domain_errors.AnalysisTimeoutError.html (+6 -6)
📝 docs/api/classes/domain_errors.ArtifactOperationError.html (+7 -7)
📝 docs/api/classes/domain_errors.BinaryTargetError.html (+5 -5)
📝 docs/api/classes/domain_errors.ConfigurationError.html (+3 -3)
📝 docs/api/classes/domain_errors.EvidenceFileError.html (+6 -6)
📝 docs/api/classes/domain_errors.EvidenceIntegrityError.html (+3 -3)
📝 docs/api/classes/domain_errors.EvidenceLimitError.html (+6 -6)
📝 docs/api/classes/domain_errors.HopperCancelledError.html (+4 -4)

...and 80 more files

📄 Description

Summary

This PR adds persistent, resumable cross-version investigation workspaces to REA. It turns the existing artifact inventory, artifact comparison, changed-behavior, and Evidence primitives into one durable workflow shared by the CLI and MCP server.

The new workflow:

inventories two artifact versions under the same bounded policy;
checkpoints canonical Evidence after inventory, comparison, and final reporting;
resumes an interrupted run from its last valid checkpoint;
reuses a completed run when artifact commitments and budgets are unchanged;
appends a new run when content or investigation limits change, preserving prior Evidence;
keeps static artifact differences classified as behavior candidates rather than runtime observations.

Motivation

REA already exposed strong individual reverse-engineering and comparison tools, but cross-version investigations were process-local and required callers to manually coordinate Evidence pages and intermediate comparisons. A terminated CLI process or MCP session could not continue that orchestration from durable state.

Persistent workspaces provide the missing continuity layer without moving provider-specific behavior into the domain or application layers. They preserve REA's existing truth model: automatic artifact comparison can identify static candidates, but cannot claim observed runtime behavior or causality.

Implementation

Canonical workspace domain

Adds a versioned workspace schema containing a canonical Evidence v2 bundle and bounded run manifests.
Derives deterministic ws_, run_, and wrev_ commitments from canonical JSON and SHA-256.
Validates workspace identity, run identity, revision commitments, canonical ordering, unique run IDs, stage consistency, completion state, and every run-to-Evidence reference.
Keeps local paths and timestamps out of run identity; artifact graph commitments and normalized budgets determine reuse.

Safe persistent storage

Restricts workspace paths to REA_EVIDENCE_ROOTS_JSON using the existing Evidence filesystem policy.
Rejects disabled policies, paths outside approved roots, symlink destinations, non-files, malformed or oversized JSON, excessive depth/node counts, integrity failures, stale revisions, and name conflicts.
Writes canonical JSON atomically with fsync and mode 0600.
Uses exclusive PID locks and compare-and-swap revision checks so concurrent writers fail closed.
Conservatively recovers only locks whose well-formed PID is confirmed absent; live, malformed, inaccessible, and symlink locks remain locked.

Shared investigation workflow

Scans both versions once and materializes complete bounded artifact inventory Evidence page sets.
Checkpoints inventory Evidence first, then artifact comparison Evidence, then the final changed-behavior Evidence.
Resumes partial runs without changing their deterministic run identity.
Imports only the completed run's transitive Evidence records into an MCP session.
Preserves incomplete comparison pagination as an explicit unknown with a limitation instead of rejecting or implying completeness.

CLI and MCP

Adds rea investigate-versions <left> <right> <workspace> --yes with workspace name, CAS revision, traversal, pagination, and change limits.
Adds an investigation_run mode to the existing find_changed_behavior MCP tool.
Requires explicit write approval through --yes or approved: true.
Preserves the public MCP inventory at exactly 68 tools; no new tool name is introduced.
Keeps the original find_changed_behavior comparison-aggregation mode intact. Inputs must supply exactly one of existing comparisons or an automatic investigation run.

Example:

export REA_EVIDENCE_ROOTS_JSON='["/absolute/path/to/evidence"]'

rea investigate-versions \
  /absolute/path/to/version-1 \
  /absolute/path/to/version-2 \
  /absolute/path/to/evidence/releases.json \
  --workspace-name releases \
  --yes

Additional changes

Moves deterministic investigation provider identities into the application layer so the artifact provider and shared workflow use one provider-neutral definition.
Makes Evidence ledger duplicate detection canonical-key-order insensitive, which is required when persisted canonical JSON is imported into a live session.
Updates the REA skill, README, architecture diagram, dedicated workspace documentation, package verification, and generated TypeDoc API pages.

Validation

All authoritative commands were run under Node.js 22.19.0, matching the repository engine requirement.

npm run check
- strict TypeScript checks passed;
- oxlint passed with the repository's existing seven warnings only;
- Prettier validation passed;
- 57 test files and all 357 tests passed;
- coverage: 80.13% statements, 68.85% branches, 81.69% functions, and 82.30% lines.
Focused feature matrix: 8 test files and all 37 workspace, MCP, changed-behavior, Evidence, error, contract, and skill tests passed.
Independent built-CLI black-box matrix passed approval gating, three-stage checkpointing, inode-stable reuse, stale CAS, live locking, name conflicts, changed-content append, prior Evidence retention, bounded pagination, root enforcement, symlink rejection, tamper rejection, disabled policy, and concurrent idempotence.
npm run verify:package passed against an isolated packed installation, including the persistent investigation CLI first-run/reuse path and the exact 68-tool MCP surface.
npm run knip passed with existing configuration hints only.
npm run jscpd passed at 1.20% duplicated lines, within the configured threshold.
npm run scan:todos reported no TODO, FIXME, or HACK markers.
npm run docs:generate completed with zero errors.

Real Hopper verification was not run because the automatic workflow in this PR uses the provider-neutral artifact graph and does not launch Hopper or make new Hopper claims.

Review notes

The main review areas are:

Workspace integrity and compare-and-swap invariants in the domain and filesystem store.
Resume/reuse behavior across the three persisted checkpoints.
The additive find_changed_behavior input contract and preservation of the original aggregation mode.
The static-analysis truth boundary: final behavior status remains unknown without controlled runtime Evidence.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/103 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 7/13/2026 **Status:** ✅ Merged **Merged:** 7/13/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `feat/persistent-workspace-investigation-runs` --- ### 📝 Commits (4) - [`41366ed`](https://github.com/morluto/rea/commit/41366edd566ba17cce81cd0cc3b503065775febc) feat: add persistent cross-version investigation workspaces - [`0dc7d9a`](https://github.com/morluto/rea/commit/0dc7d9a3c0d7719507d3307d0d9af929899b29d4) chore: merge main into persistent workspace branch - [`ee785ca`](https://github.com/morluto/rea/commit/ee785ca9a9aab3f2a8abc878486ba6d2781e0049) Merge remote-tracking branch 'origin/main' into feat/persistent-workspace-investigation-runs - [`a2076b4`](https://github.com/morluto/rea/commit/a2076b47a6b43d9c80396ee4a5954af590d65088) fix(security): restrict investigation artifact inputs ### 📊 Changes **120 files changed** (+4086 additions, -387 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+19 -3) 📝 `docs/api/assets/hierarchy.js` (+1 -1) 📝 `docs/api/assets/navigation.js` (+1 -1) 📝 `docs/api/assets/search.js` (+1 -1) 📝 `docs/api/classes/application_EvidenceLedger.EvidenceLedger.html` (+13 -13) 📝 `docs/api/classes/application_ProcessHarness.ProcessCaptureError.html` (+1 -1) 📝 `docs/api/classes/domain_errors.AnalysisCancelledError.html` (+5 -5) 📝 `docs/api/classes/domain_errors.AnalysisCapabilityUnavailableError.html` (+7 -7) 📝 `docs/api/classes/domain_errors.AnalysisError.html` (+5 -4) 📝 `docs/api/classes/domain_errors.AnalysisInputError.html` (+5 -5) 📝 `docs/api/classes/domain_errors.AnalysisOutputError.html` (+6 -6) 📝 `docs/api/classes/domain_errors.AnalysisProtocolError.html` (+3 -3) 📝 `docs/api/classes/domain_errors.AnalysisTimeoutError.html` (+6 -6) 📝 `docs/api/classes/domain_errors.ArtifactOperationError.html` (+7 -7) 📝 `docs/api/classes/domain_errors.BinaryTargetError.html` (+5 -5) 📝 `docs/api/classes/domain_errors.ConfigurationError.html` (+3 -3) 📝 `docs/api/classes/domain_errors.EvidenceFileError.html` (+6 -6) 📝 `docs/api/classes/domain_errors.EvidenceIntegrityError.html` (+3 -3) 📝 `docs/api/classes/domain_errors.EvidenceLimitError.html` (+6 -6) 📝 `docs/api/classes/domain_errors.HopperCancelledError.html` (+4 -4) _...and 80 more files_ </details> ### 📄 Description ## Summary This PR adds persistent, resumable cross-version investigation workspaces to REA. It turns the existing artifact inventory, artifact comparison, changed-behavior, and Evidence primitives into one durable workflow shared by the CLI and MCP server. The new workflow: - inventories two artifact versions under the same bounded policy; - checkpoints canonical Evidence after inventory, comparison, and final reporting; - resumes an interrupted run from its last valid checkpoint; - reuses a completed run when artifact commitments and budgets are unchanged; - appends a new run when content or investigation limits change, preserving prior Evidence; - keeps static artifact differences classified as behavior candidates rather than runtime observations. ## Motivation REA already exposed strong individual reverse-engineering and comparison tools, but cross-version investigations were process-local and required callers to manually coordinate Evidence pages and intermediate comparisons. A terminated CLI process or MCP session could not continue that orchestration from durable state. Persistent workspaces provide the missing continuity layer without moving provider-specific behavior into the domain or application layers. They preserve REA's existing truth model: automatic artifact comparison can identify static candidates, but cannot claim observed runtime behavior or causality. ## Implementation ### Canonical workspace domain - Adds a versioned workspace schema containing a canonical Evidence v2 bundle and bounded run manifests. - Derives deterministic `ws_`, `run_`, and `wrev_` commitments from canonical JSON and SHA-256. - Validates workspace identity, run identity, revision commitments, canonical ordering, unique run IDs, stage consistency, completion state, and every run-to-Evidence reference. - Keeps local paths and timestamps out of run identity; artifact graph commitments and normalized budgets determine reuse. ### Safe persistent storage - Restricts workspace paths to `REA_EVIDENCE_ROOTS_JSON` using the existing Evidence filesystem policy. - Rejects disabled policies, paths outside approved roots, symlink destinations, non-files, malformed or oversized JSON, excessive depth/node counts, integrity failures, stale revisions, and name conflicts. - Writes canonical JSON atomically with fsync and mode `0600`. - Uses exclusive PID locks and compare-and-swap revision checks so concurrent writers fail closed. - Conservatively recovers only locks whose well-formed PID is confirmed absent; live, malformed, inaccessible, and symlink locks remain locked. ### Shared investigation workflow - Scans both versions once and materializes complete bounded artifact inventory Evidence page sets. - Checkpoints inventory Evidence first, then artifact comparison Evidence, then the final changed-behavior Evidence. - Resumes partial runs without changing their deterministic run identity. - Imports only the completed run's transitive Evidence records into an MCP session. - Preserves incomplete comparison pagination as an explicit unknown with a limitation instead of rejecting or implying completeness. ### CLI and MCP - Adds `rea investigate-versions <left> <right> <workspace> --yes` with workspace name, CAS revision, traversal, pagination, and change limits. - Adds an `investigation_run` mode to the existing `find_changed_behavior` MCP tool. - Requires explicit write approval through `--yes` or `approved: true`. - Preserves the public MCP inventory at exactly 68 tools; no new tool name is introduced. - Keeps the original `find_changed_behavior` comparison-aggregation mode intact. Inputs must supply exactly one of existing comparisons or an automatic investigation run. Example: ```bash export REA_EVIDENCE_ROOTS_JSON='["/absolute/path/to/evidence"]' rea investigate-versions \ /absolute/path/to/version-1 \ /absolute/path/to/version-2 \ /absolute/path/to/evidence/releases.json \ --workspace-name releases \ --yes ``` ## Additional changes - Moves deterministic investigation provider identities into the application layer so the artifact provider and shared workflow use one provider-neutral definition. - Makes Evidence ledger duplicate detection canonical-key-order insensitive, which is required when persisted canonical JSON is imported into a live session. - Updates the REA skill, README, architecture diagram, dedicated workspace documentation, package verification, and generated TypeDoc API pages. ## Validation All authoritative commands were run under Node.js 22.19.0, matching the repository engine requirement. - `npm run check` - strict TypeScript checks passed; - oxlint passed with the repository's existing seven warnings only; - Prettier validation passed; - 57 test files and all 357 tests passed; - coverage: 80.13% statements, 68.85% branches, 81.69% functions, and 82.30% lines. - Focused feature matrix: 8 test files and all 37 workspace, MCP, changed-behavior, Evidence, error, contract, and skill tests passed. - Independent built-CLI black-box matrix passed approval gating, three-stage checkpointing, inode-stable reuse, stale CAS, live locking, name conflicts, changed-content append, prior Evidence retention, bounded pagination, root enforcement, symlink rejection, tamper rejection, disabled policy, and concurrent idempotence. - `npm run verify:package` passed against an isolated packed installation, including the persistent investigation CLI first-run/reuse path and the exact 68-tool MCP surface. - `npm run knip` passed with existing configuration hints only. - `npm run jscpd` passed at 1.20% duplicated lines, within the configured threshold. - `npm run scan:todos` reported no TODO, FIXME, or HACK markers. - `npm run docs:generate` completed with zero errors. Real Hopper verification was not run because the automatic workflow in this PR uses the provider-neutral artifact graph and does not launch Hopper or make new Hopper claims. ## Review notes The main review areas are: 1. Workspace integrity and compare-and-swap invariants in the domain and filesystem store. 2. Resume/reuse behavior across the three persisted checkpoints. 3. The additive `find_changed_behavior` input contract and preservation of the original aggregation mode. 4. The static-analysis truth boundary: final behavior status remains unknown without controlled runtime Evidence. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-16 21:11:35 +00:00