morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4063] [Experiment] Code-friendly browser/Electron data contracts and agent code composition #1611

New issue

Closed

opened 2026-10-16 21:07:57 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:07:57 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4063

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Deferred by the workstream decision in #4091. Production code mode is not being pursued now; preserve the scripts and results for possible future evaluation. Prioritize shared data contracts and existing domain tools under #4027.

Original experimental question; success is not assumed. Can browser/Electron/JavaScript investigation tools expose reusable typed records and snapshot-bound references that make agent-written JavaScript composition more effective than direct tool calls?

Evaluate the data interface separately from code execution: current direct tools → redesigned direct tools → code composition over exactly the same redesigned queries. Keep null and negative outcomes.

Context and evidence

Related umbrella: #4027. The local pilot branch experiment/browser-electron-codemode-20261015 contains a real Obsidian 1.14.4 ASAR and Excalidraw network-capture comparison using Codex CLI, Terra high. Its controlled action-loop pilot scored direct 4/4 and code 3/4; code mostly wrapped calls instead of composing them. A dependency follow-up succeeded directly and failed in code mode. These are small, inconclusive experiments, not evidence that code mode is inferior or faster. Native-host preflights were excluded when direct calls could not be verified.

Source-derived problem: JavaScript module pages expose IDs/kinds/paths, while reusable identities and retained range counts require full individual item envelopes. Generic JSON result types and repeated metadata make thin wrappers awkward. A recorded real-Obsidian projection probe preserved the selected task's facts and partial coverage in 17,989 bytes versus 325,215 bytes for the current page plus 23 item envelopes. This measures representation only; bytes do not establish token or completed-task improvements.

Expected interaction with targets

Read existing files or captured data. Reuse existing offline analysis and authenticated retained Evidence reads. Agent code composes these reads in its code host; this proposal adds no target injection, execution, attachment, mutation, or protection recovery. Acquiring a public browser capture for evaluation is a separately explicit owned browser/network action.

Boundary details

Extend the closest existing retained-view contract with a typed module-reference projection; keep interpretation in the existing producer/domain owner.
Preserve canonical identity, parent/query Evidence, artifact digests, per-row support, source selection, unknowns and partial coverage. Exhausting a page is not complete source/runtime coverage.
Provide narrow typed code methods generated from canonical contracts, ordinary JavaScript composition, and references passed as values. Do not create a separate parser or task-specific dependency oracle.
Start as an exploratory prototype. Resolve material authority and compatibility questions before any production integration; this issue is not integration approval.

Options and tradeoffs

Compare current direct reads, redesigned direct reads (including equivalent batching), and redesigned code composition with identical information. A later factorial comparison can include dossiers/retained answer views. Extra schema, discovery, provenance and code-debugging overhead may erase any benefit.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Use real public applications and genuine analyst questions, never CTF/demo tasks. Freeze artifact/capture versions, tasks, scoring and model settings; verify actual composition.
Save evaluation scripts, task/oracle definitions, executed-source fingerprints, actual model trajectories, failures and measured results. Keep acquisition/regeneration instructions and checksums; do not archive downloadable applications or regenerable bulky data. The user explicitly requested that these inputs remain regenerable rather than saved in a data bundle.
Count whole-task input/cached/output tokens, context peaks, discovery, retries and failed attempts. Report elapsed and CPU time and process-family peak memory; separate cold/warm, preparation and answering. Failed tasks earn no efficiency win.
Preserve CLI/MCP/Evidence parity, explicit errors, partial facts, resource admission and cleanup. Bound the complete process family to 8 GiB, stop early at 6 GiB, one CPU, one heavy job at a time.
Decide whether a narrow typed SDK is sufficient, where server-side filtering pays off, and whether ordinary code composition actually helps. No production rollout or performance conclusion is promised.

References

Recorded exploratory progress

Prototype and preserved results: closed, unmerged #4082. Exploratory work stays on the experiment branch; no exploratory PR is planned. Detailed findings and replay scripts.

The first typed-interface cohort passed direct 2/2 and code 1/2; code did not actually compose the passing network answer. An adaptive pair then used identical authenticated snapshot/query Evidence. Code used the typed SDK to aggregate the ten external modules correctly; direct returned the same ten-entry dictionary but claimed eleven. This is a feasibility signal, not a completed-task efficiency win or a population-level comparison.

All model outcomes, failures, usage and resource receipts are retained, including an operator-aborted attempt with incomplete usage. Downloadable applications, HARs and full regenerable Evidence/bundles are not archived. Executed: 135 focused tests, build, SDK declaration/type checking, full static and documentation checks; no OOM, largest local check peak about 5.87 GiB. Native-host latency, repeated samples and a controlled legacy-direct third arm remain unverified.

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4063 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed **Deferred by the workstream decision in #4091.** Production code mode is not being pursued now; preserve the scripts and results for possible future evaluation. Prioritize shared data contracts and existing domain tools under #4027. **Original experimental question; success is not assumed.** Can browser/Electron/JavaScript investigation tools expose reusable typed records and snapshot-bound references that make agent-written JavaScript composition more effective than direct tool calls? Evaluate the data interface separately from code execution: current direct tools → redesigned direct tools → code composition over exactly the same redesigned queries. Keep null and negative outcomes. ### Context and evidence Related umbrella: #4027. The local pilot branch `experiment/browser-electron-codemode-20261015` contains a real Obsidian 1.14.4 ASAR and Excalidraw network-capture comparison using Codex CLI, Terra high. Its controlled action-loop pilot scored direct 4/4 and code 3/4; code mostly wrapped calls instead of composing them. A dependency follow-up succeeded directly and failed in code mode. These are small, inconclusive experiments, not evidence that code mode is inferior or faster. Native-host preflights were excluded when direct calls could not be verified. Source-derived problem: JavaScript module pages expose IDs/kinds/paths, while reusable identities and retained range counts require full individual item envelopes. Generic JSON result types and repeated metadata make thin wrappers awkward. A recorded real-Obsidian projection probe preserved the selected task's facts and partial coverage in 17,989 bytes versus 325,215 bytes for the current page plus 23 item envelopes. This measures representation only; bytes do not establish token or completed-task improvements. ### Expected interaction with targets Read existing files or captured data. Reuse existing offline analysis and authenticated retained Evidence reads. Agent code composes these reads in its code host; this proposal adds no target injection, execution, attachment, mutation, or protection recovery. Acquiring a public browser capture for evaluation is a separately explicit owned browser/network action. ### Boundary details - Extend the closest existing retained-view contract with a typed module-reference projection; keep interpretation in the existing producer/domain owner. - Preserve canonical identity, parent/query Evidence, artifact digests, per-row support, source selection, unknowns and partial coverage. Exhausting a page is not complete source/runtime coverage. - Provide narrow typed code methods generated from canonical contracts, ordinary JavaScript composition, and references passed as values. Do not create a separate parser or task-specific dependency oracle. - Start as an exploratory prototype. Resolve material authority and compatibility questions before any production integration; this issue is not integration approval. ### Options and tradeoffs Compare current direct reads, redesigned direct reads (including equivalent batching), and redesigned code composition with identical information. A later factorial comparison can include dossiers/retained answer views. Extra schema, discovery, provenance and code-debugging overhead may erase any benefit. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions - Use real public applications and genuine analyst questions, never CTF/demo tasks. Freeze artifact/capture versions, tasks, scoring and model settings; verify actual composition. - Save evaluation scripts, task/oracle definitions, executed-source fingerprints, actual model trajectories, failures and measured results. Keep acquisition/regeneration instructions and checksums; do not archive downloadable applications or regenerable bulky data. The user explicitly requested that these inputs remain regenerable rather than saved in a data bundle. - Count whole-task input/cached/output tokens, context peaks, discovery, retries and failed attempts. Report elapsed and CPU time and process-family peak memory; separate cold/warm, preparation and answering. Failed tasks earn no efficiency win. - Preserve CLI/MCP/Evidence parity, explicit errors, partial facts, resource admission and cleanup. Bound the complete process family to 8 GiB, stop early at 6 GiB, one CPU, one heavy job at a time. - Decide whether a narrow typed SDK is sufficient, where server-side filtering pays off, and whether ordinary code composition actually helps. No production rollout or performance conclusion is promised. ### References - [Cloudflare Code Mode](https://blog.cloudflare.com/code-mode/) - [Code execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp) - [REA tool-design guide](https://github.com/morluto/rea/blob/main/docs/tool-design.md) - [Experiment branch with scripts and saved results](https://github.com/morluto/rea/tree/experiment/browser-codemode-tool-design-20261015/experiments/codemode-data-contract). ### Recorded exploratory progress Prototype and preserved results: closed, unmerged #4082. Exploratory work stays on the experiment branch; no exploratory PR is planned. [Detailed findings and replay scripts](https://github.com/morluto/rea/blob/experiment/browser-codemode-tool-design-20261015/experiments/codemode-data-contract/results-2026-10-15.md). The first typed-interface cohort passed direct 2/2 and code 1/2; code did not actually compose the passing network answer. An adaptive pair then used identical authenticated snapshot/query Evidence. Code used the typed SDK to aggregate the ten external modules correctly; direct returned the same ten-entry dictionary but claimed eleven. This is a feasibility signal, not a completed-task efficiency win or a population-level comparison. All model outcomes, failures, usage and resource receipts are retained, including an operator-aborted attempt with incomplete usage. Downloadable applications, HARs and full regenerable Evidence/bundles are not archived. Executed: 135 focused tests, build, SDK declaration/type checking, full static and documentation checks; no OOM, largest local check peak about 5.87 GiB. Native-host latency, repeated samples and a controlled legacy-direct third arm remain unverified.

bsakaguchi

2026-10-16 21:07:57 +00:00

closed this issue
added the
performance

experiment

design

area: evidence

triage

enhancement
labels

bsakaguchi commented

2026-10-16 21:07:58 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

The evaluation design here is the right kind of rigorous — separating representation size from token/completed-task gains, and refusing efficiency credit for failed tasks, keeps the pilot honest. The "code mostly wrapped calls instead of composing them" observation is the most diagnostic result so far; one idea is to add a probe that explicitly requires at least one composition step (e.g. filter by identity derived from another read) so the pilot distinguishes "SDK ergonomics" from "agent chooses not to compose". A narrow typed module-reference projection with envelopes preserved seems like the right minimal contract to test that.

@XinwangTao commented on GitHub (Oct 15, 2026): The evaluation design here is the right kind of rigorous — separating representation size from token/completed-task gains, and refusing efficiency credit for failed tasks, keeps the pilot honest. The "code mostly wrapped calls instead of composing them" observation is the most diagnostic result so far; one idea is to add a probe that explicitly requires at least one composition step (e.g. filter by identity derived from another read) so the pilot distinguishes "SDK ergonomics" from "agent chooses not to compose". A narrow typed module-reference projection with envelopes preserved seems like the right minimal contract to test that.

bsakaguchi commented

2026-10-16 21:07:58 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 15, 2026):

Deferred by the workstream decision in #4091. Preserve the experiment branch, scripts, measurements and negative outcomes for possible future reevaluation. Research PR #4082 remains closed and unmerged; current implementation work follows #4027, starting with shared data-contract improvement #4092 / #4097.

@N0zoM1z0 commented on GitHub (Oct 15, 2026): Deferred by the workstream decision in #4091. Preserve the experiment branch, scripts, measurements and negative outcomes for possible future reevaluation. Research PR #4082 remains closed and unmerged; current implementation work follows #4027, starting with shared data-contract improvement #4092 / #4097.

bsakaguchi referenced this issue

2026-10-17 21:12:47 +00:00

[PR #1612] [CLOSED] fix(browser): redact serialized accessibility secret names #3003

bsakaguchi referenced this issue

2026-10-17 21:12:47 +00:00

[PR #1611] [CLOSED] fix(browser): redact escaped scenario artifact secrets #3004

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:31:44 +00:00