[PR #3812] feat(experiments): prototype evidence-gated findings triage #5918

Open
opened 2026-10-31 01:10:26 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3812
Author: @Lopesnextgen
Created: 10/15/2026
Status: 🔄 Open

Base: main ← Head: feat/findings-triage


📝 Commits (3)

  • 364882b feat(experiments): prototype evidence-gated findings triage
  • 6d42322 fix(experiments): keep triage fingerprints stable across builds
  • fb8127e fix(experiments): keep triage output owner-only and test cleanup

📊 Changes

16 files changed (+1639 additions, -0 deletions)

View changed files

➕ experiments/findings-triage/README.md (+265 -0)
➕ experiments/findings-triage/cli.mjs (+106 -0)
➕ experiments/findings-triage/cli.test.mjs (+135 -0)
➕ experiments/findings-triage/fixture/app/index.html (+21 -0)
➕ experiments/findings-triage/fixture/app/main.js (+29 -0)
➕ experiments/findings-triage/fixture/app/package.json (+6 -0)
➕ experiments/findings-triage/fixture/app/preload.js (+8 -0)
➕ experiments/findings-triage/fixture/bundle.ndjson (+1 -0)
➕ experiments/findings-triage/fixture/expected/advisories/notes-save-unchecked-path.md (+33 -0)
➕ experiments/findings-triage/fixture/expected/issues.json (+8 -0)
➕ experiments/findings-triage/fixture/expected/issues/insecure-web-preferences.md (+34 -0)
➕ experiments/findings-triage/fixture/expected/report.md (+113 -0)
➕ experiments/findings-triage/fixture/generate.mjs (+114 -0)
➕ experiments/findings-triage/fixture/triage.json (+68 -0)
➕ experiments/findings-triage/triage.mjs (+411 -0)
➕ experiments/findings-triage/triage.test.mjs (+287 -0)

📄 Description

Summary

Adds an isolated prototype under experiments/findings-triage/ that turns a finished REA investigation into a triage report. Every finding is classified as defect, suspect, acceptable or unknown. The validator refuses any verdict the cited Evidence cannot carry. Nothing is registered in the CLI, the MCP catalog or the package.

Refs #1142. That issue proposes evidence-gated Markdown reports from bundles (TraceBrief), and this draft covers the step that proposal leaves open: verdicts, remediation, the next probe and issue export.

Problem and expected behavior

A bundle records what REA observed and what remains unknown. It does not record which findings are defects in the target, which are fine, and what to do about each one. Analysts write that by hand today, and nothing checks the verdicts against the Evidence.

Inputs are an Evidence bundle from export_evidence_bundle and an analyst-written triage.json. validate reports every problem at once. render writes report.md, one issue body per public defect or suspect with an issues.json index for filing with gh, and one advisory draft per private finding.

Change and scope

  • The bundle is parsed with REA's own parseEvidenceBundle, which also authenticates Evidence IDs and unknown revisions, rather than a second parser.
  • defect and acceptable need at least one cited record that is observed or derived and not analyst-inference. record_unknown and update_unknown records are derived but only record intent.
  • A runtime-behavior defect or acceptable verdict needs that same record to carry controlled-replay or external-service authority. suspect and unknown may rest on static Evidence; that gap is what next_probe names.
  • Every residual unknown whose current revision is not resolved must be linked by a finding. Linking a resolved unknown, or settling a verdict on an open one, is an error.
  • Evidence strength is computed from the cited records, never authored.
  • Exported bodies carry only triage text, Evidence IDs, operations and limitations. They never include raw_result, parameters or environment, which hold local paths.
  • render creates a new directory with a non-recursive mkdir and exclusive writes. If a write fails, it removes that directory.

Not included: creating GitHub issues (the tool makes no network calls), any src/ integration, and any skill or doc change outside the experiment.

Follow the contribution scope.
Unresolved scope, authority, side effects, or required verification block
integration. Link the required maintainer scope decision for new providers or
capabilities involving target execution, injection, mutation, or protection
recovery; ordinary fixes within existing boundaries may state "not applicable".
Draft PRs may leave the scope decision pending and the readiness and merge checklists
unchecked. Maintainers can help resolve open questions.

Contract and boundary impact

  • Semantic owner and earliest changed stage: the experiment owns triage validation and rendering. Bundle parsing stays with src/domain/evidenceBundle.ts.
  • Contribution scope fit and linked maintainer decision: evidence reporting, which CONTRIBUTING lists as a supported contribution. Maintainer scope decision: pending (#1142).
  • CLI and MCP/tool-catalog contract: unchanged.
  • Provider, bridge, target-format, or platform compatibility: unchanged.
  • Evidence, artifact, provenance, or reconstruction contract: unchanged. The experiment only reads exported bundles.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): fixture/generate.mjs starts the built REA MCP server, analyzes a copy of the self-owned sample app in an mkdtemp directory, and removes both the directory and the server on every path. The sample app is never executed.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

render writes only a new output directory. The fixture generator starts REA's own MCP server as a child process; it does not launch the target.

Evidence and regression coverage

  • Tests added or updated: triage.test.mjs and cli.test.mjs, 36 tests.
    • Each rule fails on a minimal mutation of the real fixture.
    • Analyst-inference-only citations are covered, and so is basis and runtime authority split across two records.
    • Revision-head logic.
    • Finding-ID path safety: separators, device names and non-ASCII case folding.
    • Ordering, and the absence of local paths and raw results in rendered files.
    • CLI exit codes, --json, refusal of an existing --out, malformed JSON, and tampered or unparseable bundles.
    • An owner-only output directory, and its removal after a write fails mid-render.
  • Base reproduction or other evidence: fixture/bundle.ndjson is real output of analyze_javascript_application, record_unknown and update_unknown over fixture/app/, kept as REA serialized it. One unknown is resolved, so revision history is exercised. Against the pre-fix validator, the new tests for analyst-inference basis, split runtime authority, static suspects and unknowns, and ID safety fail.
  • User-visible CLI/MCP output (if applicable): see fixture/expected/report.md, issues/insecure-web-preferences.md and advisories/notes-save-unchecked-path.md.
  • Remaining proof gaps:
    • Only the static JavaScript provider and the unknown registry produced fixture Evidence. The runtime-authority rule is covered by in-memory records, not a real capture.
    • The gh filing loop in the README has not been run.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

All run on Linux x64 with Node 24.21.0. The pinned 24.18.0 was not used.

  • node --test experiments/findings-triage/*.test.mjs: 36 passed, 0 failed.
  • node experiments/findings-triage/cli.mjs render on the fixture, then diff -r against fixture/expected/: identical.
  • node experiments/findings-triage/fixture/generate.mjs: exit 0, 5 records and 4 unknown revisions. No temp directory or server process was left behind.
  • npx oxlint experiments/findings-triage: 0 warnings, 0 errors.
  • npx oxfmt --check experiments/findings-triage: clean.
  • npx jscpd experiments/findings-triage: 0 clones.
  • Pre-commit lint-staged and npm run check:fast: passed.
  • experiments/ is outside tsconfig.json, so the typecheck does not cover these .mjs files. This folder is covered by oxlint and the node --test suites only.

Compatibility, safety, and release

  • Breaking changes or migration steps: none.
  • Real Hopper/Ghidra, browser, or OS coverage: none. Linux only; macOS and Windows were not run.
  • Package or release metadata impact: none. experiments/ is not packaged.
  • Security, privacy, process, or containment review:
    • No network access.
    • Output never overwrites existing paths and is created owner-only (0700).
    • Finding IDs are restricted before being used as file names.
    • Private findings are kept out of issues.json.
    • The fixture bundle holds only a temporary path for the sample app; no user or host identifiers.

Author readiness

Human owner: @Lopesnextgen

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3812 **Author:** [@Lopesnextgen](https://github.com/Lopesnextgen) **Created:** 10/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/findings-triage` --- ### 📝 Commits (3) - [`364882b`](https://github.com/morluto/rea/commit/364882b353e8d1520870d44ea059bac8cec7173b) feat(experiments): prototype evidence-gated findings triage - [`6d42322`](https://github.com/morluto/rea/commit/6d42322e8332cd2e029ad3497fdfe594fd6041cd) fix(experiments): keep triage fingerprints stable across builds - [`fb8127e`](https://github.com/morluto/rea/commit/fb8127ec1525cec3615608b4d3e297f5bc9e4dc1) fix(experiments): keep triage output owner-only and test cleanup ### 📊 Changes **16 files changed** (+1639 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `experiments/findings-triage/README.md` (+265 -0) ➕ `experiments/findings-triage/cli.mjs` (+106 -0) ➕ `experiments/findings-triage/cli.test.mjs` (+135 -0) ➕ `experiments/findings-triage/fixture/app/index.html` (+21 -0) ➕ `experiments/findings-triage/fixture/app/main.js` (+29 -0) ➕ `experiments/findings-triage/fixture/app/package.json` (+6 -0) ➕ `experiments/findings-triage/fixture/app/preload.js` (+8 -0) ➕ `experiments/findings-triage/fixture/bundle.ndjson` (+1 -0) ➕ `experiments/findings-triage/fixture/expected/advisories/notes-save-unchecked-path.md` (+33 -0) ➕ `experiments/findings-triage/fixture/expected/issues.json` (+8 -0) ➕ `experiments/findings-triage/fixture/expected/issues/insecure-web-preferences.md` (+34 -0) ➕ `experiments/findings-triage/fixture/expected/report.md` (+113 -0) ➕ `experiments/findings-triage/fixture/generate.mjs` (+114 -0) ➕ `experiments/findings-triage/fixture/triage.json` (+68 -0) ➕ `experiments/findings-triage/triage.mjs` (+411 -0) ➕ `experiments/findings-triage/triage.test.mjs` (+287 -0) </details> ### 📄 Description ## Summary Adds an isolated prototype under `experiments/findings-triage/` that turns a finished REA investigation into a triage report. Every finding is classified as `defect`, `suspect`, `acceptable` or `unknown`. The validator refuses any verdict the cited Evidence cannot carry. Nothing is registered in the CLI, the MCP catalog or the package. Refs #1142. That issue proposes evidence-gated Markdown reports from bundles (TraceBrief), and this draft covers the step that proposal leaves open: verdicts, remediation, the next probe and issue export. ## Problem and expected behavior A bundle records what REA observed and what remains unknown. It does not record which findings are defects in the target, which are fine, and what to do about each one. Analysts write that by hand today, and nothing checks the verdicts against the Evidence. Inputs are an Evidence bundle from `export_evidence_bundle` and an analyst-written `triage.json`. `validate` reports every problem at once. `render` writes `report.md`, one issue body per public `defect` or `suspect` with an `issues.json` index for filing with `gh`, and one advisory draft per private finding. ## Change and scope - The bundle is parsed with REA's own `parseEvidenceBundle`, which also authenticates Evidence IDs and unknown revisions, rather than a second parser. - `defect` and `acceptable` need at least one cited record that is `observed` or `derived` and not `analyst-inference`. `record_unknown` and `update_unknown` records are derived but only record intent. - A `runtime-behavior` defect or acceptable verdict needs that same record to carry `controlled-replay` or `external-service` authority. `suspect` and `unknown` may rest on static Evidence; that gap is what `next_probe` names. - Every residual unknown whose current revision is not resolved must be linked by a finding. Linking a resolved unknown, or settling a verdict on an open one, is an error. - Evidence strength is computed from the cited records, never authored. - Exported bodies carry only triage text, Evidence IDs, operations and limitations. They never include `raw_result`, `parameters` or `environment`, which hold local paths. - `render` creates a new directory with a non-recursive `mkdir` and exclusive writes. If a write fails, it removes that directory. Not included: creating GitHub issues (the tool makes no network calls), any `src/` integration, and any skill or doc change outside the experiment. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". Draft PRs may leave the scope decision pending and the readiness and merge checklists unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact - Semantic owner and earliest changed stage: the experiment owns triage validation and rendering. Bundle parsing stays with `src/domain/evidenceBundle.ts`. - Contribution scope fit and linked maintainer decision: evidence reporting, which CONTRIBUTING lists as a supported contribution. Maintainer scope decision: pending (#1142). - CLI and MCP/tool-catalog contract: unchanged. - Provider, bridge, target-format, or platform compatibility: unchanged. - Evidence, artifact, provenance, or reconstruction contract: unchanged. The experiment only reads exported bundles. - Target selection, authority, upstream behavior, and cleanup details (if applicable): `fixture/generate.mjs` starts the built REA MCP server, analyzes a copy of the self-owned sample app in an `mkdtemp` directory, and removes both the directory and the server on every path. The sample app is never executed. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs): none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. `render` writes only a new output directory. The fixture generator starts REA's own MCP server as a child process; it does not launch the target. ## Evidence and regression coverage - Tests added or updated: `triage.test.mjs` and `cli.test.mjs`, 36 tests. - Each rule fails on a minimal mutation of the real fixture. - Analyst-inference-only citations are covered, and so is basis and runtime authority split across two records. - Revision-head logic. - Finding-ID path safety: separators, device names and non-ASCII case folding. - Ordering, and the absence of local paths and raw results in rendered files. - CLI exit codes, `--json`, refusal of an existing `--out`, malformed JSON, and tampered or unparseable bundles. - An owner-only output directory, and its removal after a write fails mid-render. - Base reproduction or other evidence: `fixture/bundle.ndjson` is real output of `analyze_javascript_application`, `record_unknown` and `update_unknown` over `fixture/app/`, kept as REA serialized it. One unknown is resolved, so revision history is exercised. Against the pre-fix validator, the new tests for analyst-inference basis, split runtime authority, static suspects and unknowns, and ID safety fail. - User-visible CLI/MCP output (if applicable): see `fixture/expected/report.md`, `issues/insecure-web-preferences.md` and `advisories/notes-save-unchecked-path.md`. - Remaining proof gaps: - Only the static JavaScript provider and the unknown registry produced fixture Evidence. The runtime-authority rule is covered by in-memory records, not a real capture. - The `gh` filing loop in the README has not been run. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed All run on Linux x64 with Node 24.21.0. The pinned 24.18.0 was not used. - `node --test experiments/findings-triage/*.test.mjs`: 36 passed, 0 failed. - `node experiments/findings-triage/cli.mjs render` on the fixture, then `diff -r` against `fixture/expected/`: identical. - `node experiments/findings-triage/fixture/generate.mjs`: exit 0, 5 records and 4 unknown revisions. No temp directory or server process was left behind. - `npx oxlint experiments/findings-triage`: 0 warnings, 0 errors. - `npx oxfmt --check experiments/findings-triage`: clean. - `npx jscpd experiments/findings-triage`: 0 clones. - Pre-commit `lint-staged` and `npm run check:fast`: passed. - `experiments/` is outside `tsconfig.json`, so the typecheck does not cover these `.mjs` files. This folder is covered by oxlint and the `node --test` suites only. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real Hopper/Ghidra, browser, or OS coverage: none. Linux only; macOS and Windows were not run. - Package or release metadata impact: none. `experiments/` is not packaged. - Security, privacy, process, or containment review: - No network access. - Output never overwrites existing paths and is created owner-only (`0700`). - Finding IDs are restricted before being used as file names. - Private findings are kept out of `issues.json`. - The fixture bundle holds only a temporary path for the sample app; no user or host identifiers. ## Author readiness Human owner: @Lopesnextgen - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5918
No description provided.