morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4809] [Design] Opt-in journal of a session's Evidence ledger, written as records are added #2500

New issue

Open

opened 2026-10-17 21:05:49 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-17 21:05:49 +00:00

Owner

Originally created by @pablodiazjorge on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4809

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Should REA offer an opt-in, append-only journal of a session's Evidence and residual unknowns, written by REA as records are added, so the record outlives close_binary and the process without the agent calling export_evidence_bundle? If so, where should it attach and how should it be switched on?

Context and evidence

Observed. In my benchmark (rea-lab, five preregistered studies on Windows with rea 6.4.0 over MCP in Claude Code 2.1.296), no run kept its Evidence:

Runs that used REA	Exported a bundle or saved a snapshot	Distinct Evidence IDs produced
339	0	3,364

No task asked the agent to keep Evidence, and the prompt said "Close any REA analysis session you open before finishing", so 322 runs ended with close_binary and 17 ended with the process. Either way, none of the 3,364 records outlived its run. That matches the ledger contract in docs/mcp-contracts.md: "Export a bundle before closing if the investigation needs it later."

The reader who needed the record afterwards was not the agent but the evaluator. My grader re-authenticates every cited evidence_id with REA's own code, and with the ledger gone it rebuilds each record from the client's transcript. That went wrong once: Claude Code writes results too large to return inline to a file, the grader first read only inline results, and 2 of 144 citations in study 3 were reported as unauthenticated until a published correction. A record written by REA would have made that step unnecessary and independent of the client.

rea-casebook, a Claude Code mod I published, does the same reconstruction live for one client: it rebuilds the session's Evidence and unknowns from the tool results passing through the client, re-reads results the client saved to files, and rewrites a per-session JSONL journal after each change. It works, but only for that client, and only as faithfully as the client hands results over. The server is the one place that sees every record whole.

Source-derived. BinarySessionRecords.onAnalysisSnapshotChanged is declared on the session port and fires when Evidence is added, a bundle or snapshot is imported, a snapshot or workflow record is stored, an unknown is recorded or updated, and when the session is invalidated or cleared. Nothing in src/ subscribes to it on main; only tests do.

Expected interaction with targets

No target interaction (documentation or usability only)

Options and tradeoffs

Attachment point. A listener on onAnalysisSnapshotChanged. The event carries no payload, so a listener diffs against what it last wrote (cheap, since Evidence IDs are content-addressed), or the event could pass the change. Alternatively the journal lives in InvestigationRecords, next to the ledger.
Format. JSONL, one line per Evidence record or unknown change, append-only and in order, like the process-capture event journal and the bounded journal in docs/native-investigation.md. Or a complete bundle rewritten atomically after each change: importable as is through import_evidence_bundle, at the cost of one rewrite per record.
Switch. Off by default. An environment variable or CLI flag naming a directory, or an open_binary option. Files named by session and target digest, never by target path.
Safety. The rules export_evidence_bundle already follows: canonical paths, no overwrite, a size bound with an explicit truncation line, and nothing about the host beyond what the Evidence holds.
close_binary. It still clears the ledger; with the journal on, it first writes a closing line, and the file stays.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

With the journal on, a session that closes, or whose process exits, leaves a file from which import_evidence_bundle (directly or after a documented conversion) restores every Evidence record with the same IDs.
With it off, the default, outputs and behaviour are unchanged.
A killed process leaves a journal whose complete lines are valid; a torn last line is ignored with a stated limitation.
Open: core feature, or a documented convention plus a validator? #4394 asks the same for curated notes. This would be the raw half, written by REA; #4394 the curated half, written by the agent. Its own alternatives name manual export as the gap.
Open: Evidence only, or unknowns and workflow records too?

The results above are observed; there is no REA prototype. I'm happy to prepare a PR if this is wanted.

References

#4394, #1142, #3812, #51, #133.
docs/mcp-contracts.md (ledger lifetime), docs/native-investigation.md (bounded journal).
rea-lab: benchmark and study 3 correction; rea-casebook.
Checked against main f4322b15e.

Originally created by @pablodiazjorge on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4809 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Should REA offer an opt-in, append-only journal of a session's Evidence and residual unknowns, written by REA as records are added, so the record outlives `close_binary` and the process without the agent calling `export_evidence_bundle`? If so, where should it attach and how should it be switched on? ### Context and evidence **Observed.** In my benchmark ([rea-lab](https://github.com/pablodiazjorge/rea-lab), five preregistered studies on Windows with rea 6.4.0 over MCP in Claude Code 2.1.296), no run kept its Evidence: | Runs that used REA | Exported a bundle or saved a snapshot | Distinct Evidence IDs produced | | ---: | ---: | ---: | | 339 | 0 | 3,364 | No task asked the agent to keep Evidence, and the prompt said "Close any REA analysis session you open before finishing", so 322 runs ended with `close_binary` and 17 ended with the process. Either way, none of the 3,364 records outlived its run. That matches the ledger contract in [`docs/mcp-contracts.md`](https://github.com/morluto/rea/blob/f4322b15ed1423d736bd2572cf7a8eadf2b466e1/docs/mcp-contracts.md?plain=1#L313-L322): "Export a bundle before closing if the investigation needs it later." The reader who needed the record afterwards was not the agent but the evaluator. My grader re-authenticates every cited `evidence_id` with REA's own code, and with the ledger gone it rebuilds each record from the client's transcript. That went wrong once: Claude Code writes results too large to return inline to a file, the grader first read only inline results, and 2 of 144 citations in study 3 were reported as unauthenticated until a [published correction](https://github.com/pablodiazjorge/rea-lab/blob/main/writeups/2026-10-study-3/README.md#correction-16-october-2026). A record written by REA would have made that step unnecessary and independent of the client. [rea-casebook](https://github.com/pablodiazjorge/rea-casebook), a Claude Code mod I published, does the same reconstruction live for one client: it rebuilds the session's Evidence and unknowns from the tool results passing through the client, re-reads results the client saved to files, and rewrites a per-session JSONL journal after each change. It works, but only for that client, and only as faithfully as the client hands results over. The server is the one place that sees every record whole. **Source-derived.** [`BinarySessionRecords.onAnalysisSnapshotChanged`](https://github.com/morluto/rea/blob/f4322b15ed1423d736bd2572cf7a8eadf2b466e1/src/application/binary/BinarySessionRecords.ts#L70-L74) is declared on the session port and fires when Evidence is added, a bundle or snapshot is imported, a snapshot or workflow record is stored, an unknown is recorded or updated, and when the session is invalidated or cleared. Nothing in `src/` subscribes to it on main; only tests do. ### Expected interaction with targets No target interaction (documentation or usability only) ### Options and tradeoffs - **Attachment point.** A listener on `onAnalysisSnapshotChanged`. The event carries no payload, so a listener diffs against what it last wrote (cheap, since Evidence IDs are content-addressed), or the event could pass the change. Alternatively the journal lives in `InvestigationRecords`, next to the ledger. - **Format.** JSONL, one line per Evidence record or unknown change, append-only and in order, like the process-capture event journal and the bounded journal in [`docs/native-investigation.md`](https://github.com/morluto/rea/blob/f4322b15ed1423d736bd2572cf7a8eadf2b466e1/docs/native-investigation.md?plain=1#L501-L507). Or a complete bundle rewritten atomically after each change: importable as is through `import_evidence_bundle`, at the cost of one rewrite per record. - **Switch.** Off by default. An environment variable or CLI flag naming a directory, or an `open_binary` option. Files named by session and target digest, never by target path. - **Safety.** The rules `export_evidence_bundle` already follows: canonical paths, no overwrite, a size bound with an explicit truncation line, and nothing about the host beyond what the Evidence holds. - **`close_binary`.** It still clears the ledger; with the journal on, it first writes a closing line, and the file stays. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions - With the journal on, a session that closes, or whose process exits, leaves a file from which `import_evidence_bundle` (directly or after a documented conversion) restores every Evidence record with the same IDs. - With it off, the default, outputs and behaviour are unchanged. - A killed process leaves a journal whose complete lines are valid; a torn last line is ignored with a stated limitation. - Open: core feature, or a documented convention plus a validator? #4394 asks the same for curated notes. This would be the raw half, written by REA; #4394 the curated half, written by the agent. Its own alternatives name manual export as the gap. - Open: Evidence only, or unknowns and workflow records too? The results above are observed; there is no REA prototype. I'm happy to prepare a PR if this is wanted. ### References - #4394, #1142, #3812, #51, #133. - [`docs/mcp-contracts.md`](https://github.com/morluto/rea/blob/f4322b15ed1423d736bd2572cf7a8eadf2b466e1/docs/mcp-contracts.md?plain=1#L313-L322) (ledger lifetime), [`docs/native-investigation.md`](https://github.com/morluto/rea/blob/f4322b15ed1423d736bd2572cf7a8eadf2b466e1/docs/native-investigation.md?plain=1#L501-L507) (bounded journal). - rea-lab: [benchmark](https://github.com/pablodiazjorge/rea-lab/blob/main/bench/README.md) and [study 3 correction](https://github.com/pablodiazjorge/rea-lab/blob/main/writeups/2026-10-study-3/README.md#correction-16-october-2026); [rea-casebook](https://github.com/pablodiazjorge/rea-casebook). - Checked against main `f4322b15e`.

bsakaguchi commented

2026-10-31 00:34:33 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

The benchmark data is compelling — 3,364 Evidence IDs across 339 runs, none surviving the session, and the study 3 correction shows the real cost of client-side reconstruction. The server-side journal is clearly the right attachment point since it is the only place that sees every record whole. I'd lean toward JSONL over atomic bundle rewrites: content-addressed IDs make diffing cheap, append-only survives crashes better, and a torn last line is a recoverable, stated limitation. On the open question, I'd argue for the core feature with a validator rather than convention-only — the whole point is that the evaluator shouldn't have to trust the client transcript, and a convention without enforcement recreates exactly the failure mode this issue documents.

@XinwangTao commented on GitHub (Oct 18, 2026): The benchmark data is compelling — 3,364 Evidence IDs across 339 runs, none surviving the session, and the study 3 correction shows the real cost of client-side reconstruction. The server-side journal is clearly the right attachment point since it is the only place that sees every record whole. I'd lean toward JSONL over atomic bundle rewrites: content-addressed IDs make diffing cheap, append-only survives crashes better, and a torn last line is a recoverable, stated limitation. On the open question, I'd argue for the core feature with a validator rather than convention-only — the whole point is that the evaluator shouldn't have to trust the client transcript, and a convention without enforcement recreates exactly the failure mode this issue documents.

bsakaguchi referenced this issue

2026-10-31 01:05:23 +00:00

[PR #2500] [CLOSED] fix(electron): name the failing path in capture_electron_scenario input errors #5278

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#2500

No description provided.

Rows
Columns