morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1142] Proposal: TraceBrief — evidence-gated Markdown reports from REA investigations #403

New issue

Open

opened 2026-10-15 18:24:47 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:24:47 +00:00

Owner

Originally created by @dany331x on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1142

I'd like to propose adding TraceBrief — an evidence-gated Markdown report generator for finished REA investigations. This is a scope question, not a finished patch: I'm asking which integration shape (if any) is worth carrying before I write a port.

What TraceBrief is

A standalone, zero-dependency Node CLI that turns a finished REA investigation into a Markdown report another developer can read, reproduce and check.

REA investigation -> export_evidence_bundle -> bundle.json
                       |
        agent authors claims (report.json)
                       |
              validate -> fail -> actionable errors, no file
                       | pass
                reports/<name>.md

The design point is the split. An agent writes claims; a deterministic script verifies the citations and renders the document. A prompt alone cannot verify that an ev_<64 hex> id exists, that a runtime claim is not resting on static evidence, or that every residual unknown in the bundle is carried into the report as an open question.

TraceBrief analyses nothing and calls no REA tools. It consumes the JSON that export_evidence_bundle already produces.

Why this is not a drop-in patch

TraceBrief as written does not satisfy REA's contribution gates, and presenting it as a ready PR would waste review time:

REA expects	TraceBrief today
TypeScript across `src/`	plain `.mjs` modules
`vitest`	`node --test`
`oxlint` / `oxfmt`	not formatted for either
`knip` clean	would flag unused exports
`jscpd` duplication gate	not measured
layered domain / contracts / provider / application / server / adapter	a flat 4-module pipeline
`verify:test-discovery`, `metadata:generate`	n/a

Porting is straightforward but real: ~2,165 lines of tests become vitest specs, and the modules need a home inside REA's layering. That is ongoing maintenance surface, so it needs agreement before anyone writes it.

Three integration shapes, cheapest first

Documentation pointer. A page in docs/ describing the investigation-to-report workflow, linking to TraceBrief as an external tool. Smallest diff, no maintenance burden, and adjacent in spirit to the existing reconstruction-obligation-ledgers.md and reconstruction-readiness.md guides.
A managed skill in skill-src/. REA already ships managed skills, and /tracebrief is essentially a workflow guide plus a validator invocation. Reuses REA's existing distribution; needs a metadata:generate bump.
First-party tool. Port TraceBrief into src/ as a supported command with full toolchain conformance. Largest benefit, largest ongoing cost, and it puts a second opinion about report structure inside the same repository.

What has actually been verified

Against real REA 6.0.0 output on Windows / Node 24.21.0:

179 tests, 0 failures, zero dependencies, no node_modules.
25/25 mutations caught by the intended test. npm run test:mutations reverts each fix in turn and asserts a named test fails. The harness runs a baseline control first and exits 2 if that control does not pass — an earlier version of the harness reported a false 20/20 by running a path Node could not resolve, so every run failed and "all caught" was vacuously true.
Three sample reports built from live REA exports, not synthetic fixtures: an Electron feature trace, real residual unknowns created through REA's own record_unknown / update_unknown bookkeeping, and a genuinely managed System.Data.dll (34 metadata tables, 9 assembly references, three mixed file-offset locations).
/tracebrief exercised through OpenCode itself. opencode run "/tracebrief" in an empty directory resolved the installed command and the agent followed it, including refusing to invent ev_ ids when the session had zero records.
Real bug found by real data, worth flagging to anyone consuming bundles. REA exports retain unknown revision history, so a single unk_ id appears once as open and later as resolved in the same bundle. Any consumer treating "an unknown with this id exists" as "this unknown is unresolved" will mis-report. residualUnknownHeads() reduces the history to one head per id.

What has not been verified

Being explicit, per CONTRIBUTING's "be clear about anything you couldn't test":

Native decompilation was never exercised. Hopper is macOS/Linux only, and Ghidra/IDA need a JDK that is not installed on the verification host. TraceBrief consumes their output, but no native-decompiler record has passed through the validator.
No EVM bytecode, firmware, Android (no JDK), pwntools ELF / core-dump, or live process-capture targets.
Windows-only host: no macOS or Linux verification of any kind.

Question

Which of the three shapes, if any, is worth carrying? If the answer is "none — ship it standalone and link it from the docs", that is a fine outcome and costs this project nothing.

Happy to do the port if shape 2 or 3 is the answer. If it is shape 1, I will keep it out of tree and only submit the docs change.

Originally created by @dany331x on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1142 I'd like to propose adding **[TraceBrief](https://github.com/dany331x/tracebrief)** — an evidence-gated Markdown report generator for finished REA investigations. This is a scope question, not a finished patch: I'm asking which integration shape (if any) is worth carrying before I write a port. ## What TraceBrief is A standalone, zero-dependency Node CLI that turns a finished REA investigation into a Markdown report another developer can read, reproduce and check. ``` REA investigation -> export_evidence_bundle -> bundle.json | agent authors claims (report.json) | validate -> fail -> actionable errors, no file | pass reports/<name>.md ``` The design point is the split. An agent writes *claims*; a deterministic script verifies the citations and renders the document. A prompt alone cannot verify that an `ev_<64 hex>` id exists, that a runtime claim is not resting on static evidence, or that every residual unknown in the bundle is carried into the report as an open question. TraceBrief analyses nothing and calls no REA tools. It consumes the JSON that `export_evidence_bundle` already produces. ## Why this is not a drop-in patch TraceBrief as written does not satisfy REA's contribution gates, and presenting it as a ready PR would waste review time: | REA expects | TraceBrief today | | --- | --- | | TypeScript across `src/` | plain `.mjs` modules | | `vitest` | `node --test` | | `oxlint` / `oxfmt` | not formatted for either | | `knip` clean | would flag unused exports | | `jscpd` duplication gate | not measured | | layered domain / contracts / provider / application / server / adapter | a flat 4-module pipeline | | `verify:test-discovery`, `metadata:generate` | n/a | Porting is straightforward but real: ~2,165 lines of tests become vitest specs, and the modules need a home inside REA's layering. That is ongoing maintenance surface, so it needs agreement before anyone writes it. ## Three integration shapes, cheapest first 1. **Documentation pointer.** A page in `docs/` describing the investigation-to-report workflow, linking to TraceBrief as an external tool. Smallest diff, no maintenance burden, and adjacent in spirit to the existing `reconstruction-obligation-ledgers.md` and `reconstruction-readiness.md` guides. 2. **A managed skill in `skill-src/`.** REA already ships managed skills, and `/tracebrief` is essentially a workflow guide plus a validator invocation. Reuses REA's existing distribution; needs a `metadata:generate` bump. 3. **First-party tool.** Port TraceBrief into `src/` as a supported command with full toolchain conformance. Largest benefit, largest ongoing cost, and it puts a second opinion about report structure inside the same repository. ## What has actually been verified Against real REA 6.0.0 output on Windows / Node 24.21.0: - **179 tests, 0 failures**, zero dependencies, no `node_modules`. - **25/25 mutations caught by the intended test.** `npm run test:mutations` reverts each fix in turn and asserts a named test fails. The harness runs a baseline control first and exits 2 if that control does not pass — an earlier version of the harness reported a false 20/20 by running a path Node could not resolve, so every run failed and "all caught" was vacuously true. - **Three sample reports built from live REA exports**, not synthetic fixtures: an Electron feature trace, real residual unknowns created through REA's own `record_unknown` / `update_unknown` bookkeeping, and a genuinely managed `System.Data.dll` (34 metadata tables, 9 assembly references, three mixed `file-offset` locations). - **`/tracebrief` exercised through OpenCode itself.** `opencode run "/tracebrief"` in an empty directory resolved the installed command and the agent followed it, including refusing to invent `ev_` ids when the session had zero records. - **Real bug found by real data, worth flagging to anyone consuming bundles.** REA exports retain unknown *revision history*, so a single `unk_` id appears once as `open` and later as `resolved` in the same bundle. Any consumer treating "an unknown with this id exists" as "this unknown is unresolved" will mis-report. `residualUnknownHeads()` reduces the history to one head per id. ## What has **not** been verified Being explicit, per CONTRIBUTING's "be clear about anything you couldn't test": - **Native decompilation was never exercised.** Hopper is macOS/Linux only, and Ghidra/IDA need a JDK that is not installed on the verification host. TraceBrief consumes their output, but no native-decompiler record has passed through the validator. - No EVM bytecode, firmware, Android (no JDK), pwntools ELF / core-dump, or live process-capture targets. - Windows-only host: no macOS or Linux verification of any kind. ## Question Which of the three shapes, if any, is worth carrying? If the answer is "none — ship it standalone and link it from the docs", that is a fine outcome and costs this project nothing. Happy to do the port if shape 2 or 3 is the answer. If it is shape 1, I will keep it out of tree and only submit the docs change.

bsakaguchi added the

labels

2026-10-15 18:24:47 +00:00

bsakaguchi commented

2026-10-15 18:24:48 +00:00

Author

Owner

@dany331x commented on GitHub (Oct 9, 2026):

Following up on my own issue with something concrete, in case it is useful for judging scope.

I measured the port rather than guessing at it. Typechecking the original sources under this repo's exact compiler options (exactOptionalPropertyTypes, noUncheckedIndexedAccess, verbatimModuleSyntax) with allowJs/checkJs reports 300 errors — 67% implicit-any parameters, ~12% unknown narrowing in catch clauses, and the rest needing genuine option types. So roughly two thirds is mechanical.

Partial port: https://github.com/dany331x/tracebrief-rea-port

All five product modules are ported and tsc --noEmit is clean under your exact options. What is not done: the CLI, 13 of 14 test files (9 of 179 tests ported), and all of the repo-specific integration work — verify:test-discovery project globs, verify:module-boundaries placement, knip export checking, and the 2.5% jscpd threshold that does not exempt test files. PORTING.md lists each of those with the specific obstacle. This should not be read as mergeable.

The part I would point at is the verification approach. Typechecking a port proves nothing about behaviour, so parity.mjs runs the original .mjs and the compiled TypeScript over the same inputs and compares byte for byte — issue lists, index sizes and full rendered Markdown, across the three real REA fixtures plus malformed and empty inputs. It caught three defects that no type error would have surfaced:

readStringPath walked the final key as if it were an intermediate object, so isRecord() was tested against a digest string. It returned undefined for every call, silently disabling checkTarget and the evidence-subject mismatch rule. Two integrity rules were dead and the port still compiled clean.
The renderer dropped confidence/authority counts; section headers must read derived 1, inferred 2.
loadBundleFile had been narrowed to string while callers legitimately pass a file: URL from import.meta.url.

I also proved the parity harness can fail. Changing MIN_QUOTE_WORDS from 4 to 3 initially slipped through, because no fixture sits at that threshold — a gap I only found by injecting the mutation. Explicit 3-word and 4-word cases were added, and it now fails as it should.

Two questions this raises rather than answers:

zod. This repo validates protocol input with safeParse. TraceBrief's contract is "return every problem at once, never throw", which my hand-written guards express directly and zod expresses only awkwardly. Whether that difference is acceptable is a design call I did not want to make unilaterally.
Placement. A flat 5-module pipeline does not fit the domain/contracts/provider/application/server/adapter layering. If shape 3 is the answer, I would rather let verify:module-boundaries and verify:test-discovery dictate where these files live than port the remaining files in a flat layout and relocate everything afterwards.

If the answer is shape 1 (docs pointer) or "no", none of this is needed and I will leave it where it is.

@dany331x commented on GitHub (Oct 9, 2026): Following up on my own issue with something concrete, in case it is useful for judging scope. I measured the port rather than guessing at it. Typechecking the original sources under this repo's exact compiler options (`exactOptionalPropertyTypes`, `noUncheckedIndexedAccess`, `verbatimModuleSyntax`) with `allowJs`/`checkJs` reports **300 errors** — 67% implicit-any parameters, ~12% `unknown` narrowing in `catch` clauses, and the rest needing genuine option types. So roughly two thirds is mechanical. **Partial port:** https://github.com/dany331x/tracebrief-rea-port All five product modules are ported and `tsc --noEmit` is clean under your exact options. What is **not** done: the CLI, 13 of 14 test files (9 of 179 tests ported), and all of the repo-specific integration work — `verify:test-discovery` project globs, `verify:module-boundaries` placement, `knip` export checking, and the 2.5% `jscpd` threshold that does not exempt test files. `PORTING.md` lists each of those with the specific obstacle. This should not be read as mergeable. The part I would point at is the verification approach. Typechecking a port proves nothing about behaviour, so `parity.mjs` runs the original `.mjs` and the compiled TypeScript over the same inputs and compares byte for byte — issue lists, index sizes and full rendered Markdown, across the three real REA fixtures plus malformed and empty inputs. It caught three defects that no type error would have surfaced: - `readStringPath` walked the final key as if it were an intermediate object, so `isRecord()` was tested against a digest string. It returned `undefined` for **every** call, silently disabling `checkTarget` and the evidence-subject mismatch rule. Two integrity rules were dead and the port still compiled clean. - The renderer dropped confidence/authority counts; section headers must read `derived 1, inferred 2`. - `loadBundleFile` had been narrowed to `string` while callers legitimately pass a `file:` URL from `import.meta.url`. I also proved the parity harness can fail. Changing `MIN_QUOTE_WORDS` from 4 to 3 initially **slipped through**, because no fixture sits at that threshold — a gap I only found by injecting the mutation. Explicit 3-word and 4-word cases were added, and it now fails as it should. Two questions this raises rather than answers: 1. **zod.** This repo validates protocol input with `safeParse`. TraceBrief's contract is "return every problem at once, never throw", which my hand-written guards express directly and zod expresses only awkwardly. Whether that difference is acceptable is a design call I did not want to make unilaterally. 2. **Placement.** A flat 5-module pipeline does not fit the domain/contracts/provider/application/server/adapter layering. If shape 3 is the answer, I would rather let `verify:module-boundaries` and `verify:test-discovery` dictate where these files live than port the remaining files in a flat layout and relocate everything afterwards. If the answer is shape 1 (docs pointer) or "no", none of this is needed and I will leave it where it is.

bsakaguchi commented

2026-10-15 18:24:48 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 12, 2026):

Integration Shape & Ecosystem Review

TraceBrief offers a valuable reporting concept for evidence-backed markdown synthesis:

Ecosystem Fit: Per REA's core architecture (AGENTS.md), core REA focuses on local evidence generation, session supervision, and typed contracts. Downstream synthesis and report generation are best kept either:
1. As an external companion tool/plugin that reads REA's JSON Evidence outputs.
2. Or packaged as an agent skill guidance workflow that prompts the orchestrating LLM to summarize retained Evidence records into formatted Markdown.

@matbrgz commented on GitHub (Oct 12, 2026): ### Integration Shape & Ecosystem Review TraceBrief offers a valuable reporting concept for evidence-backed markdown synthesis: - **Ecosystem Fit:** Per REA's core architecture (`AGENTS.md`), core REA focuses on local evidence generation, session supervision, and typed contracts. Downstream synthesis and report generation are best kept either: 1. As an external companion tool/plugin that reads REA's JSON Evidence outputs. 2. Or packaged as an agent skill guidance workflow that prompts the orchestrating LLM to summarize retained Evidence records into formatted Markdown.

bsakaguchi commented

2026-10-15 18:24:49 +00:00

Author

Owner

@Lopesnextgen commented on GitHub (Oct 15, 2026):

This proposal leaves one step open: deciding what the findings mean for the target. I prototyped that step as a separate layer on the same input.

Draft PR: #3812 (experiments/findings-triage/). Nothing is registered in the CLI, the MCP catalog or the package.

It reads export_evidence_bundle output plus an analyst-written triage.json. Each finding gets a verdict: defect, suspect, acceptable or unknown. Depending on the verdict, it also needs severity, impact, remediation and the next REA operation that would close the gap.

The validator refuses verdicts the cited records cannot carry:

defect and acceptable need at least one observed or derived record that is not analyst-inference. record_unknown records are derived, but they only record intent.
A runtime-behavior defect or acceptable needs that same record to carry runtime authority. suspect and unknown may rest on static Evidence; next_probe names what would settle them.
Every residual unknown whose current revision is not resolved must be triaged.

Evidence strength is computed from the cited records, never written by the analyst. The bundle goes through REA's own parseEvidenceBundle instead of a second parser.

Output:

report.md.
One body per actionable finding. Public findings go to issues/, with an issues.json index for filing with gh. Private findings go to advisories/ for private vulnerability reporting.
Bodies carry only triage text, Evidence ids and limitations. They never include raw_result, parameters or environment.

The tool itself never contacts GitHub.

@dany331x This builds on your claim model rather than competing with it. I kept the vocabulary on purpose: claim_type, issue objects and exit codes. If shape 2 or 3 is chosen, I'm glad to help finish the port and fold triage into it.

Questions for maintainers:

Is verdict and remediation triage in scope for REA, or should it stay out of tree next to TraceBrief?
If it is in scope, should reporting and triage be one tool or two?

@Lopesnextgen commented on GitHub (Oct 15, 2026): This proposal leaves one step open: deciding what the findings mean for the target. I prototyped that step as a separate layer on the same input. Draft PR: #3812 (`experiments/findings-triage/`). Nothing is registered in the CLI, the MCP catalog or the package. It reads `export_evidence_bundle` output plus an analyst-written `triage.json`. Each finding gets a verdict: `defect`, `suspect`, `acceptable` or `unknown`. Depending on the verdict, it also needs severity, impact, remediation and the next REA operation that would close the gap. The validator refuses verdicts the cited records cannot carry: - `defect` and `acceptable` need at least one `observed` or `derived` record that is not `analyst-inference`. `record_unknown` records are derived, but they only record intent. - A runtime-behavior `defect` or `acceptable` needs that same record to carry runtime authority. `suspect` and `unknown` may rest on static Evidence; `next_probe` names what would settle them. - Every residual unknown whose current revision is not resolved must be triaged. Evidence strength is computed from the cited records, never written by the analyst. The bundle goes through REA's own `parseEvidenceBundle` instead of a second parser. Output: - `report.md`. - One body per actionable finding. Public findings go to `issues/`, with an `issues.json` index for filing with `gh`. Private findings go to `advisories/` for private vulnerability reporting. - Bodies carry only triage text, Evidence ids and limitations. They never include `raw_result`, `parameters` or `environment`. The tool itself never contacts GitHub. @dany331x This builds on your claim model rather than competing with it. I kept the vocabulary on purpose: `claim_type`, issue objects and exit codes. If shape 2 or 3 is chosen, I'm glad to help finish the port and fold triage into it. Questions for maintainers: 1. Is verdict and remediation triage in scope for REA, or should it stay out of tree next to TraceBrief? 2. If it is in scope, should reporting and triage be one tool or two?

bsakaguchi referenced this issue

2026-10-16 21:12:56 +00:00

[PR #419] [MERGED] feat(process): compare declared concurrent traces #1968

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:09 +00:00