morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8427] [Design] Recover inline unretained Evidence into a new investigation #6599

New issue

Open

opened 2026-11-01 21:12:34 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8427

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

How should an agent save and later reuse a complete inline unretained Evidence record from an error without inventing a bundle format or rerunning the provider?

Context and evidence

On draft #8426 at 458674fba8bc5adc7873adb99b5256ed0e16fc30, a real Kuna 1.986 address_name call with an injected zero-byte investigation allowance returned _main as partial_observation.kind: unretained-evidence; the original session contained no record. In a fresh target-free MCP session, writing the inline record to JSON and passing that path to import_evidence_bundle failed because it is a record, not a canonical bundle. Manually assembling the canonical manifests around this particular single record allowed import, preserved its Evidence ID, provider, profile and result, and did not rerun Kuna. Altering the normalized result while keeping its ID was rejected, and the already imported record remained unchanged.

This example had no Evidence links, environment, capture or scenario manifests. It does not establish a general manual procedure for linked records or other Evidence types. The limit was test injection, not production-default exhaustion. Real REA MCP server and TypeScript SDK in-memory transport, macOS arm64, Node 24.22.1, owned static Mach-O; the target was not executed and its digest did not change.

Expected interaction with targets

Read existing files or captured data; write or change files/configuration if the caller chooses a file-based handoff. No target execution or provider restart should be required merely to recover the already returned record.

Boundary details (optional)

Preserve Evidence identity, provenance, and the distinction between inline, committed, and omitted observations. Authenticate input and apply existing memory/size limits; do not assert linked-source closure when links are unavailable. Retained records already present in the old session and an unretained error payload are separate. See #8425 for a different source-admission case where the completed primary result is not returned at all.

Options and tradeoffs

Consider an authenticated record-import or bundle-construction workflow, improving the existing error recovery and documentation, or another bounded operation that serves the agent's handoff. This is a design question, not a request to expose every ledger function or accept unvalidated records.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Exercise the advertised CLI/MCP handoff from the actual error into a fresh session, including duplicate and tampered input, size limits, linked/missing sources, and failed admission. Keep canonical import atomic and report what remains unavailable. Establish whether the same mechanism helps other completed partial observations without promising that every partial is a standalone Evidence record.

References

#8425, #8364, #8426. import_evidence_bundle currently accepts a path to a canonical bundle; this report concerns making the returned record usable without requiring the agent to reconstruct that persisted contract.

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Real Kuna capacity-recovery handoff control

A controlled in-memory MCP test on draft #8444 head e619141b9 paused a real Kuna 1.986 batch, imported synthetic Evidence into an intentionally small ledger, and resumed the provider. The pseudocode completed, but retention failed and the error returned an unretained batch Evidence record. A fresh target-free session rejected that individual JSON record through import_evidence_bundle; a canonical one-record bundle constructed with REA's internal test helper imported it exactly, without rerunning Kuna. This is a workaround, not a public bundle-construction capability.

An unmerged raw-import experiment based on an earlier #8444 head accepted the individual recovered record in fresh sessions under both full and compact advertised MCP schema profiles. It rejected a modified record without retaining it, and repeated raw and canonical-bundle imports left exactly one record. The real target was unexecuted and unchanged. Timing and capacity were test-controlled; this does not establish autonomous agent success, natural contention frequency, linked-source closure, other providers, or production readiness.

Mixed retained-source and unretained-result recovery

A separate real-Kuna control with a synthetic source association and test-controlled ledger pressure retained the source record but could not retain the completed batch. The error returned the batch as partial_observation.kind: unretained-evidence, with its source ID, while the original session's exported bundle included the source. On exact #8444 head e619141b9, both full and compact MCP profiles advised: “Export the retained investigation and start a new investigation, or increase the parent Node process heap before starting it.” That instruction does not explain that the returned batch is absent from the export or how to preserve it separately.

An unmerged raw-import prototype imported the returned batch into a fresh target-free session. At that point its linked source was absent; importing the separately extracted source from the old bundle restored both exact records. The source association, contention, and small capacity were injected; the Kuna pseudocode was real and the unexecuted target was unchanged. This does not establish autonomous agent behavior. Recovery guidance should reflect the final delivered fields: when MCP response limits omit an inline observation, it must not instruct the caller to save an observation that was not delivered. Preserve the distinction between a source present in the old session, a returned but unretained result, and source closure that has not been established.

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8427 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed How should an agent save and later reuse a complete inline **unretained** Evidence record from an error without inventing a bundle format or rerunning the provider? ### Context and evidence On draft #8426 at `458674fba8bc5adc7873adb99b5256ed0e16fc30`, a real Kuna 1.986 `address_name` call with an injected zero-byte investigation allowance returned `_main` as `partial_observation.kind: unretained-evidence`; the original session contained no record. In a fresh target-free MCP session, writing the inline record to JSON and passing that path to `import_evidence_bundle` failed because it is a record, not a canonical bundle. Manually assembling the canonical manifests around this particular single record allowed import, preserved its Evidence ID, provider, profile and result, and did not rerun Kuna. Altering the normalized result while keeping its ID was rejected, and the already imported record remained unchanged. This example had no Evidence links, environment, capture or scenario manifests. It does not establish a general manual procedure for linked records or other Evidence types. The limit was test injection, not production-default exhaustion. Real REA MCP server and TypeScript SDK in-memory transport, macOS arm64, Node 24.22.1, owned static Mach-O; the target was not executed and its digest did not change. ### Expected interaction with targets Read existing files or captured data; write or change files/configuration if the caller chooses a file-based handoff. No target execution or provider restart should be required merely to recover the already returned record. ### Boundary details (optional) Preserve Evidence identity, provenance, and the distinction between inline, committed, and omitted observations. Authenticate input and apply existing memory/size limits; do not assert linked-source closure when links are unavailable. Retained records already present in the old session and an unretained error payload are separate. See #8425 for a different source-admission case where the completed primary result is not returned at all. ### Options and tradeoffs Consider an authenticated record-import or bundle-construction workflow, improving the existing error recovery and documentation, or another bounded operation that serves the agent's handoff. This is a design question, not a request to expose every ledger function or accept unvalidated records. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Exercise the advertised CLI/MCP handoff from the actual error into a fresh session, including duplicate and tampered input, size limits, linked/missing sources, and failed admission. Keep canonical import atomic and report what remains unavailable. Establish whether the same mechanism helps other completed partial observations without promising that every partial is a standalone Evidence record. ### References #8425, #8364, #8426. `import_evidence_bundle` currently accepts a path to a canonical bundle; this report concerns making the returned record usable without requiring the agent to reconstruct that persisted contract. cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Real Kuna capacity-recovery handoff control A controlled in-memory MCP test on draft #8444 head `e619141b9` paused a real Kuna 1.986 batch, imported synthetic Evidence into an intentionally small ledger, and resumed the provider. The pseudocode completed, but retention failed and the error returned an unretained batch Evidence record. A fresh target-free session rejected that individual JSON record through `import_evidence_bundle`; a canonical one-record bundle constructed with REA's internal test helper imported it exactly, without rerunning Kuna. This is a workaround, not a public bundle-construction capability. An unmerged raw-import experiment based on an earlier #8444 head accepted the individual recovered record in fresh sessions under both full and compact advertised MCP schema profiles. It rejected a modified record without retaining it, and repeated raw and canonical-bundle imports left exactly one record. The real target was unexecuted and unchanged. Timing and capacity were test-controlled; this does not establish autonomous agent success, natural contention frequency, linked-source closure, other providers, or production readiness. ### Mixed retained-source and unretained-result recovery A separate real-Kuna control with a synthetic source association and test-controlled ledger pressure retained the source record but could not retain the completed batch. The error returned the batch as `partial_observation.kind: unretained-evidence`, with its source ID, while the original session's exported bundle included the source. On exact #8444 head `e619141b9`, both full and compact MCP profiles advised: “Export the retained investigation and start a new investigation, or increase the parent Node process heap before starting it.” That instruction does not explain that the returned batch is absent from the export or how to preserve it separately. An unmerged raw-import prototype imported the returned batch into a fresh target-free session. At that point its linked source was absent; importing the separately extracted source from the old bundle restored both exact records. The source association, contention, and small capacity were injected; the Kuna pseudocode was real and the unexecuted target was unchanged. This does not establish autonomous agent behavior. Recovery guidance should reflect the final delivered fields: when MCP response limits omit an inline observation, it must not instruct the caller to save an observation that was not delivered. Preserve the distinction between a source present in the old session, a returned but unretained result, and source closure that has not been established.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:34 +00:00

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Real linked-record control on #8426 head 458674fba: with a Ghidra 12.1.4 selected-project copy containing synthetic historical report metadata and an injected 20,000-byte allowance, REA retained the historical health source, then refused the completed address_name record and returned that primary record inline with the source ID in evidence_links. The source was available in a public export_evidence_bundle.

For this one pair, I manually combined that exported source bundle with the inline primary record. Import into a fresh target-free MCP session restored both exact records and resolved the link without rerunning Ghidra. A primary-only canonical bundle was also accepted while its source was absent; import success by itself therefore did not establish link closure in this test. This is an observed contract boundary, not a claim that all bundles must be closed or that all downstream consumers fail to check closure. Source-derived: workflows that follow links use resolveInvestigationEvidenceClosure and explicitly reject missing references.

The project and static target digests were unchanged during REA work; the target was not executed. The report metadata was deliberately created in a disposable project copy and does not prove actual analyzer settings. This is a controlled capacity test, not evidence of production-default exhaustion. A general recovery design should account for partial bundles and sources committed before a failure, and avoid presenting successful record import as proof of complete provenance.

@morluto commented on GitHub (Nov 1, 2026): Real linked-record control on #8426 head `458674fba`: with a Ghidra 12.1.4 selected-project copy containing **synthetic** historical report metadata and an injected 20,000-byte allowance, REA retained the historical `health` source, then refused the completed `address_name` record and returned that primary record inline with the source ID in `evidence_links`. The source was available in a public `export_evidence_bundle`. For this one pair, I manually combined that exported source bundle with the inline primary record. Import into a fresh target-free MCP session restored both exact records and resolved the link without rerunning Ghidra. A primary-only canonical bundle was also accepted while its source was absent; import success by itself therefore did not establish link closure in this test. This is an observed contract boundary, not a claim that all bundles must be closed or that all downstream consumers fail to check closure. Source-derived: workflows that follow links use `resolveInvestigationEvidenceClosure` and explicitly reject missing references. The project and static target digests were unchanged during REA work; the target was not executed. The report metadata was deliberately created in a disposable project copy and does not prove actual analyzer settings. This is a controlled capacity test, not evidence of production-default exhaustion. A general recovery design should account for partial bundles and sources committed before a failure, and avoid presenting successful record import as proof of complete provenance.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Follow-up public-workflow check with real Ghidra 12.1.4: I obtained an analyze_function dossier from the selected disposable project with synthetic history metadata, constructed a primary-only bundle for the test, and imported it into a fresh target-free MCP session. build_call_path using that retained dossier rejected the missing linked source with evidence_integrity_mismatch and named its Evidence ID. After importing the original complete bundle, the same call succeeded as a zero-hop path and linked both the dossier and history source; no provider was started in the offline session.

This verifies explicit closure checking for that particular public workflow, not all consumers or a nontrivial call graph. The primary-only bundle was constructed using an internal test helper, not a public record-import API. REA left the project copy and static target unchanged; the target was not executed.

The observed error supplied details.reason: missing, reference_scope: retained-closure, and the missing evidence_id; its remediation specifically suggested importing the originating bundle with complete linked sources and retrying. That advertised recovery was the action exercised successfully above.

@morluto commented on GitHub (Nov 1, 2026): Follow-up public-workflow check with real Ghidra 12.1.4: I obtained an `analyze_function` dossier from the selected disposable project with synthetic history metadata, constructed a primary-only bundle for the test, and imported it into a fresh target-free MCP session. `build_call_path` using that retained dossier rejected the missing linked source with `evidence_integrity_mismatch` and named its Evidence ID. After importing the original complete bundle, the same call succeeded as a zero-hop path and linked both the dossier and history source; no provider was started in the offline session. This verifies explicit closure checking for that particular public workflow, not all consumers or a nontrivial call graph. The primary-only bundle was constructed using an internal test helper, not a public record-import API. REA left the project copy and static target unchanged; the target was not executed. The observed error supplied `details.reason: missing`, `reference_scope: retained-closure`, and the missing `evidence_id`; its remediation specifically suggested importing the originating bundle with complete linked sources and retrying. That advertised recovery was the action exercised successfully above.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Related REA ownership issue: #8433 reproduces an inline procedure_address observation accepted by inspect_analysis_view on main, but only the resulting view is retained. A later lookup of its parent fails as missing; importing the original authenticated source restores it. This is distinct from how to package an unretained source for import.

@morluto commented on GitHub (Nov 1, 2026): Related REA ownership issue: #8433 reproduces an inline `procedure_address` observation accepted by `inspect_analysis_view` on main, but only the resulting view is retained. A later lookup of its parent fails as missing; importing the original authenticated source restores it. This is distinct from how to package an unretained source for import.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Existing partial workaround verified on main: a caller can save a single recovered Evidence record as JSON and use inspect_analysis_view with source: {kind: "evidence-file", path: <absolute path>}; repeated MCP views and the public CLI returned the same result without a target/provider, and a tampered record was rejected. See #8433 for the test and its limit: this does not import the record into the session, and only tools advertising Evidence-file selectors can use it.

@morluto commented on GitHub (Nov 1, 2026): Existing partial workaround verified on main: a caller can save a single recovered Evidence record as JSON and use `inspect_analysis_view` with `source: {kind: "evidence-file", path: <absolute path>}`; repeated MCP views and the public CLI returned the same result without a target/provider, and a tampered record was rejected. See #8433 for the test and its limit: this does not import the record into the session, and only tools advertising Evidence-file selectors can use it.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Inline composition after an atomic refusal

A controlled public MCP handoff on #8426 head 050362143 shows another existing route for one linked record. With two synthetic function dossiers and an injected ledger limit, compare_functions completed but the atomic comparison-and-Unknown write was refused. The response returned the authenticated comparison inline as unretained-evidence; both dossier sources were already retained and the Unknown revision count was zero.

I exported that source bundle, closed the session, and imported it into a fresh target-free session. find_changed_behavior rejected a retained reference to the uncommitted comparison, but accepted the complete returned comparison inline and produced Evidence linking it. A preexisting-comparison control could use the retained reference after import; before import the new session rejected that reference. Both controls passed with the default input-element allowance for these small synthetic records, and also with an explicit 65,536-element allowance. No provider was started.

This is a composition route when the consuming tool accepts inline Evidence and the linked sources are available. It does not import a standalone record directly, guarantee that larger records fit the input limit, repair missing or invalid linked sources, or address the source-first loss in #8425. It does not establish real-provider or production-default exhaustion behavior.

@morluto commented on GitHub (Nov 1, 2026): ### Inline composition after an atomic refusal A controlled public MCP handoff on #8426 head `050362143` shows another existing route for one linked record. With two synthetic function dossiers and an injected ledger limit, `compare_functions` completed but the atomic comparison-and-Unknown write was refused. The response returned the authenticated comparison inline as `unretained-evidence`; both dossier sources were already retained and the Unknown revision count was zero. I exported that source bundle, closed the session, and imported it into a fresh target-free session. `find_changed_behavior` rejected a retained reference to the uncommitted comparison, but accepted the complete returned comparison inline and produced Evidence linking it. A preexisting-comparison control could use the retained reference after import; before import the new session rejected that reference. Both controls passed with the default input-element allowance for these small synthetic records, and also with an explicit 65,536-element allowance. No provider was started. This is a composition route when the consuming tool accepts inline Evidence **and** the linked sources are available. It does not import a standalone record directly, guarantee that larger records fit the input limit, repair missing or invalid linked sources, or address the source-first loss in #8425. It does not establish real-provider or production-default exhaustion behavior.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Real-provider handoff and an import prototype

On #8444 at f2de739f6, an injected zero-byte Evidence ledger caused a real Ghidra 12.1.4 analyze_function request to return both its source and completed dossier inline. The selected disposable project contained synthetic historical metadata. REA did not execute the target or change the target/project hashes.

In a fresh target-free MCP session, a file-backed view of the 33,622-byte dossier succeeded but did not retain its parent. Passing the dossier inline to build_call_path exceeded the default 1,024 input-element allowance; passing its ID failed because it was not retained. import_evidence_bundle rejected the raw record and recommended exporting a session bundle, but neither record was retained in the failed session. With a larger input allowance and the linked source imported in a canonical bundle made by an internal test helper, inline composition succeeded and made the parent reference usable.

A real Kuna 1.986 control with the same injected allowance returned a 6,589-byte dossier with no links. In a fresh default-limit stdio session, its inline zero-hop call path succeeded and made the parent reference usable. This is a working route for that observation, not a general solution for larger or linked records.

I prototyped extending the existing path-based import/export boundary to accept an authenticated standalone record or a canonical bundle. At the default input limit, importing the actual Ghidra dossier alone correctly exposed the missing source; importing the raw source then enabled the retained-reference path. The Kuna record also worked. Full/compact stdio controls rejected tampering, treated duplicate import as zero additions, and round-tripped an exported bundle into a fresh session. The existing CLI import/export paths were prototyped as well: CLI-exported single-record bundles imported into the unmodified MCP server, which identified the missing source until its bundle was imported. The CLI remains process-local; it does not retain an investigation.

Focused prototype controls also rejected an authenticated but malformed process-capture payload and a zero-capacity admission without retaining records. They preserved the cancellation reason after authentication, allowed retry after SDK cancellation, and retained a failed file-close owner until cleanup succeeded. A separate controlled final-close test preserved the exact cancellation reason with both successful and refused descriptor cleanup; neither case admitted the record, and the refused owner closed on retry. After updating the changed input expectation and the test's read-boundary spy, 42 focused CLI/MCP and file-backed-view tests passed. Tampered raw input reports a record authentication failure and CLI export does not publish an output. A record missing its ID and a bundle missing its records now identify the appropriate missing field in the prototype; an earlier discriminator based only on presence of the ID misclassified the incomplete record as a bundle.

This is an unmerged prototype, not a complete proposal: one file per call is not an atomic multi-record import; successful import does not prove linked-source closure; large-record headroom, additional provider types, full CLI cancellation/finalization, and an agent's ability to discover the workflow still need verification. A production design should share one file-kind/authentication owner with existing file-backed views, preserve payload validation and canonical-bundle atomicity, and avoid duplicating parsers. The revised prototype now uses one authenticated document reader for import and file-backed views, with record- and bundle-specific selection preserved. It rejects a synthetic malformed process-capture record through both operations in raw and bundle form; the earlier view accepted the raw record. A valid controlled-replay process capture imported from either format produced the same canonical records, environment, scenario, and capture manifests. A 69-test affected CLI/MCP/view run passed. An additional 98-test focused run passed after preserving the file path and field-level diagnostics on recognized malformed records; the public MCP error also passed its declared JSON Schema in full and compact profiles. Typecheck and lint passed. Previously captured Ghidra and Kuna Evidence again completed the offline stdio handoff in full and compact profiles. No provider was started for that replay. #8449 also shows why enforcing predicate identity only in the file path would miss inline views. The tests cover a zero-hop path, injected capacity failures and one host; they do not establish production-default exhaustion or runtime reachability.

Discoverability and delivery follow-up: The experimental MCP catalog now describes single-record recovery and warns that linked sources may be missing. Its error hint is only added when the response still fits: a measured boundary control showed the full record would fit without the added guidance but not with it, and the candidate delivered the record without the optional hint. A controlled public MCP test followed a synthetic zero-capacity provider error, wrote the returned record, discovered the import tool, and imported it into a fresh session in both profiles. The CLI help now states that its validation does not retain an MCP session; a real CLI/fresh-MCP control confirmed the count is process-local. No autonomous-agent selection trial was run. Mixed record/bundle fields now produce a specific wrong-kind error rather than being guessed as a bundle. These changes remain in the isolated unmerged prototype.

Diagnostic-budget control: In the prototype, a synthetic record with 2,000 malformed fields caused both full and compact MCP inspect_analysis_view to return a transport constraint instead of its file-integrity error when the server's response budget was deliberately set to 32 KiB. This is an injected budget below the normal configuration floor, not a claim about default-size failures. The revised prototype defers field-issue projection until the actual response budget is known; it preserves the integrity error and selected path, and explicitly marks the field issues omitted. The same control passed for import, views, and paired artifact comparison, and an unrelated JSON document with 2,000 keys preserved its wrong-kind error. Small malformed records still report field details. Explicitly sensitive failure redaction omits deferred details and preserves the omission marker. After the changes, 117 focused projection/file/redaction/MCP tests and 28 CLI/import/cancellation tests passed, as did typecheck and lint.

This is still exploratory: the check bounds the projection of already-produced schema issues, not the validator's production of those issues. Multiple nested failures, large attacker-controlled first-issue messages, default response/heap limits, and live agent selection are not proven. The shared reader and diagnostics changes remain unmerged and need a simpler final ownership/resource review before integration.

A subsequent default-budget control reproduced diagnostic overflow on current main with a 376,625-byte synthetic file containing 125,000 invalid values; the full error was retained and a later summary call worked. This separate existing file-reader issue is tracked in #8453. On the same full-profile input, the unmerged prototype returned the file-integrity error and path with explicit field-detail omission. Zod's issue-allocation cost remains open.

A separate current-main control showed this discoverability question is not specific to raw-record import: an internally constructed valid one-record bundle imported and round-tripped while its linked Ghidra source was absent. Summary correctly reported complete-record but did not expose link membership; the full record and the downstream typed missing-reference error did. Importing the source restored the zero-hop path. The general bounded discovery question is now tracked in #8454; partial bundles remain valid and this does not establish a producer defect.

@morluto commented on GitHub (Nov 1, 2026): ### Real-provider handoff and an import prototype On #8444 at `f2de739f6`, an injected zero-byte Evidence ledger caused a real Ghidra 12.1.4 `analyze_function` request to return both its source and completed dossier inline. The selected disposable project contained **synthetic** historical metadata. REA did not execute the target or change the target/project hashes. In a fresh target-free MCP session, a file-backed view of the 33,622-byte dossier succeeded but did not retain its parent. Passing the dossier inline to `build_call_path` exceeded the default 1,024 input-element allowance; passing its ID failed because it was not retained. `import_evidence_bundle` rejected the raw record and recommended exporting a session bundle, but neither record was retained in the failed session. With a larger input allowance and the linked source imported in a canonical bundle made by an internal test helper, inline composition succeeded and made the parent reference usable. A real Kuna 1.986 control with the same injected allowance returned a 6,589-byte dossier with no links. In a fresh default-limit stdio session, its inline zero-hop call path succeeded and made the parent reference usable. This is a working route for that observation, not a general solution for larger or linked records. I prototyped extending the **existing** path-based import/export boundary to accept an authenticated standalone record or a canonical bundle. At the default input limit, importing the actual Ghidra dossier alone correctly exposed the missing source; importing the raw source then enabled the retained-reference path. The Kuna record also worked. Full/compact stdio controls rejected tampering, treated duplicate import as zero additions, and round-tripped an exported bundle into a fresh session. The existing CLI import/export paths were prototyped as well: CLI-exported single-record bundles imported into the unmodified MCP server, which identified the missing source until its bundle was imported. The CLI remains process-local; it does not retain an investigation. Focused prototype controls also rejected an authenticated but malformed process-capture payload and a zero-capacity admission without retaining records. They preserved the cancellation reason after authentication, allowed retry after SDK cancellation, and retained a failed file-close owner until cleanup succeeded. A separate controlled final-close test preserved the exact cancellation reason with both successful and refused descriptor cleanup; neither case admitted the record, and the refused owner closed on retry. After updating the changed input expectation and the test's read-boundary spy, 42 focused CLI/MCP and file-backed-view tests passed. Tampered raw input reports a record authentication failure and CLI export does not publish an output. A record missing its ID and a bundle missing its records now identify the appropriate missing field in the prototype; an earlier discriminator based only on presence of the ID misclassified the incomplete record as a bundle. This is an **unmerged prototype**, not a complete proposal: one file per call is not an atomic multi-record import; successful import does not prove linked-source closure; large-record headroom, additional provider types, full CLI cancellation/finalization, and an agent's ability to discover the workflow still need verification. A production design should share one file-kind/authentication owner with existing file-backed views, preserve payload validation and canonical-bundle atomicity, and avoid duplicating parsers. The revised prototype now uses one authenticated document reader for import and file-backed views, with record- and bundle-specific selection preserved. It rejects a synthetic malformed process-capture record through both operations in raw and bundle form; the earlier view accepted the raw record. A valid controlled-replay process capture imported from either format produced the same canonical records, environment, scenario, and capture manifests. A 69-test affected CLI/MCP/view run passed. An additional 98-test focused run passed after preserving the file path and field-level diagnostics on recognized malformed records; the public MCP error also passed its declared JSON Schema in full and compact profiles. Typecheck and lint passed. Previously captured Ghidra and Kuna Evidence again completed the offline stdio handoff in full and compact profiles. No provider was started for that replay. #8449 also shows why enforcing predicate identity only in the file path would miss inline views. The tests cover a zero-hop path, injected capacity failures and one host; they do not establish production-default exhaustion or runtime reachability. **Discoverability and delivery follow-up:** The experimental MCP catalog now describes single-record recovery and warns that linked sources may be missing. Its error hint is only added when the response still fits: a measured boundary control showed the full record would fit without the added guidance but not with it, and the candidate delivered the record without the optional hint. A controlled public MCP test followed a synthetic zero-capacity provider error, wrote the returned record, discovered the import tool, and imported it into a fresh session in both profiles. The CLI help now states that its validation does not retain an MCP session; a real CLI/fresh-MCP control confirmed the count is process-local. No autonomous-agent selection trial was run. Mixed record/bundle fields now produce a specific wrong-kind error rather than being guessed as a bundle. These changes remain in the isolated unmerged prototype. **Diagnostic-budget control:** In the prototype, a synthetic record with 2,000 malformed fields caused both full and compact MCP `inspect_analysis_view` to return a transport constraint instead of its file-integrity error when the server's response budget was deliberately set to 32 KiB. This is an injected budget below the normal configuration floor, not a claim about default-size failures. The revised prototype defers field-issue projection until the actual response budget is known; it preserves the integrity error and selected path, and explicitly marks the field issues omitted. The same control passed for import, views, and paired artifact comparison, and an unrelated JSON document with 2,000 keys preserved its wrong-kind error. Small malformed records still report field details. Explicitly sensitive failure redaction omits deferred details and preserves the omission marker. After the changes, 117 focused projection/file/redaction/MCP tests and 28 CLI/import/cancellation tests passed, as did typecheck and lint. This is still exploratory: the check bounds the **projection** of already-produced schema issues, not the validator's production of those issues. Multiple nested failures, large attacker-controlled first-issue messages, default response/heap limits, and live agent selection are not proven. The shared reader and diagnostics changes remain unmerged and need a simpler final ownership/resource review before integration. A subsequent default-budget control reproduced diagnostic overflow on current main with a 376,625-byte synthetic file containing 125,000 invalid values; the full error was retained and a later summary call worked. This separate existing file-reader issue is tracked in #8453. On the same full-profile input, the unmerged prototype returned the file-integrity error and path with explicit field-detail omission. Zod's issue-allocation cost remains open. A separate current-main control showed this discoverability question is not specific to raw-record import: an internally constructed valid one-record bundle imported and round-tripped while its linked Ghidra source was absent. Summary correctly reported `complete-record` but did not expose link membership; the full record and the downstream typed missing-reference error did. Importing the source restored the zero-hop path. The general bounded discovery question is now tracked in #8454; partial bundles remain valid and this does not establish a producer defect.

bsakaguchi commented

2026-11-01 21:12:34 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

Live completed-trace and source recovery

I exercised the two-record case end to end using the isolated #8425 shared-recovery experiment together with this raw-import prototype. A test-only stdio subprocess assembled the production session/server with an injected zero-byte retention allowance and real Kuna 1.986. trace_native_values returned an unretained function source and a separately completed unretained trace that linked it; the source-owned target was not executed and its SHA-256 was unchanged.

For each MCP schema profile, an SDK client wrote those two returned evidence objects directly as ordinary JSON files—no internal canonical-bundle helper. A fresh target-free prototype stdio server then:

Imported the trace file. Looking up the linked source correctly returned missing.
Rejected a deliberately tampered source file with evidence_integrity_mismatch; the previously imported trace remained intact and the source remained absent.
Imported the unmodified source file and resolved its summary view.
Exported the session as a canonical bundle. A second fresh target-free server imported that bundle and returned the exact original trace record and source link.

Both profiles passed; no provider ran during either recovery session. The two file imports are separate commits, not an atomic multi-record operation, and the first successful import does not establish source closure. The control used a test-only zero allowance and two unmerged experiments; it does not establish natural exhaustion, a released rea mcp zero-limit option, autonomous-agent tool selection, or safety of the still-unfinished preconstruction headroom and large-file validation paths. Related #8425 and #8454.

@morluto commented on GitHub (Nov 1, 2026): ### Live completed-trace and source recovery I exercised the two-record case end to end using the isolated #8425 shared-recovery experiment together with this raw-import prototype. A test-only stdio subprocess assembled the production session/server with an injected zero-byte retention allowance and real Kuna 1.986. `trace_native_values` returned an unretained function source and a separately completed unretained trace that linked it; the source-owned target was not executed and its SHA-256 was unchanged. For each MCP schema profile, an SDK client wrote those two returned `evidence` objects directly as ordinary JSON files—no internal canonical-bundle helper. A fresh target-free **prototype** stdio server then: 1. Imported the trace file. Looking up the linked source correctly returned missing. 2. Rejected a deliberately tampered source file with `evidence_integrity_mismatch`; the previously imported trace remained intact and the source remained absent. 3. Imported the unmodified source file and resolved its summary view. 4. Exported the session as a canonical bundle. A second fresh target-free server imported that bundle and returned the exact original trace record and source link. Both profiles passed; no provider ran during either recovery session. The two file imports are separate commits, not an atomic multi-record operation, and the first successful import does not establish source closure. The control used a test-only zero allowance and two **unmerged** experiments; it does not establish natural exhaustion, a released `rea mcp` zero-limit option, autonomous-agent tool selection, or safety of the still-unfinished preconstruction headroom and large-file validation paths. Related #8425 and #8454.

bsakaguchi commented

2026-11-01 21:12:35 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

CLI conversion and existing-server handoff

Using the previously captured real-Kuna failure records, I also ran the prototype's actual CLI processes: evidence-import validated the raw trace, and evidence-export separately converted the raw trace and raw source to canonical single-record bundles. A fresh target-free stdio MCP server on unmodified main e18594b started with zero retained records, imported those two CLI-produced bundles, resolved the source view after its import, and retained the trace's source link. No provider was started during this offline control.

The CLI import count is local validation and did not populate the fresh MCP session; export is the explicit handoff. This is a useful compatibility route for canonical bundles, not a claim of an atomic multi-file import or a released raw-record importer. The CLI and MCP raw-import changes remain unmerged, with the validation/headroom limitations recorded above.

@morluto commented on GitHub (Nov 1, 2026): ### CLI conversion and existing-server handoff Using the previously captured real-Kuna failure records, I also ran the prototype's **actual CLI processes**: `evidence-import` validated the raw trace, and `evidence-export` separately converted the raw trace and raw source to canonical single-record bundles. A fresh target-free stdio MCP server on unmodified main `e18594b` started with zero retained records, imported those two CLI-produced bundles, resolved the source view after its import, and retained the trace's source link. No provider was started during this offline control. The CLI import count is local validation and did not populate the fresh MCP session; export is the explicit handoff. This is a useful compatibility route for canonical bundles, not a claim of an atomic multi-file import or a released raw-record importer. The CLI and MCP raw-import changes remain unmerged, with the validation/headroom limitations recorded above.

bsakaguchi commented

2026-11-01 21:12:35 +00:00

Author

Owner

@morluto commented on GitHub (Nov 1, 2026):

The two-record live handoff above works, but the prototype is not ready to merge: a separate 128 MiB old-space control with a 381,310-byte malformed standalone record caused both the tested main revision and the prototype stdio process to exhaust its heap during import. Instrumentation localized the prototype failure to Evidence schema/authentication, after JSON read and the existing headroom check but before error projection. A valid raw-record control completed. Full details, stage controls and scope: https://github.com/morluto/rea/issues/8453#issuecomment-6555620875.

A boolean Zod precheck rejected that particular malformed array without exhausting the constrained server, but it does not authenticate Evidence IDs and does not prove bounded behavior for all schema shapes. The raw-record workflow should not be integrated on the strength of the happy-path handoff alone.

@morluto commented on GitHub (Nov 1, 2026): The two-record live handoff above works, but the prototype is not ready to merge: a separate 128 MiB old-space control with a 381,310-byte malformed standalone record caused both the tested main revision and the prototype stdio process to exhaust its heap during import. Instrumentation localized the prototype failure to Evidence schema/authentication, after JSON read and the existing headroom check but before error projection. A valid raw-record control completed. Full details, stage controls and scope: https://github.com/morluto/rea/issues/8453#issuecomment-6555620875. A boolean Zod precheck rejected that particular malformed array without exhausting the constrained server, but it does not authenticate Evidence IDs and does not prove bounded behavior for all schema shapes. The raw-record workflow should not be integrated on the strength of the happy-path handoff alone.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6599

No description provided.

Rows
Columns