JavaScript evidence #1617

New issue

Open

opened 2026-10-16 21:08:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:08:02 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4091

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Decision for this workstream: defer production code mode. Improve browser/Electron/JavaScript tools and their shared data contracts first. This is a prioritization decision, not a finding that code mode cannot work. Reconsider only when repeated real workflows demonstrate a useful additional benefit over optimized direct tools using the same backend and information.

Context and evidence

Related tracker: #4027. Exploration: #4063. Research PR #4082 is closed without merging; retain the experiment branch, scripts and small result records. Exploratory work should remain in issues/branches, without an exploratory PR that could be merged accidentally.

Observed with real Obsidian 1.14.4 and an Excalidraw browser capture:

The original four-question pilot passed direct 4/4 and code 3/4. A dependency follow-up passed directly and failed in code mode. Most code runs printed tool replies instead of programmatically composing an answer.
The first typed-interface cohort passed direct 2/2 and code 1/2. Mixed Evidence/Result return conventions and inappropriate discovery caused concrete errors and extra work.
A final adaptive pair used identical retained snapshot/query Evidence. Code correctly aggregated the dependency facts; direct made a count error. One adaptive pair with a failed baseline does not establish an efficiency advantage.
A deterministic real-MCP comparison reduced selected module-reference replies from 338,208 to 23,402 structured bytes by replacing a page plus 23 item reads with one typed query. Both direct and code can use this improvement. This establishes representation/call-count changes, not token or speed savings.

The fallback agent harness restarted Codex CLI for every action and replayed history. Native-host latency, repeated outcomes, richer source/IPC joins, and isolated interface/runtime effects remain NOT_ENOUGH_INFO. Preserve these limitations and failed attempts. Full report and saved records.

Source-derived comparison: Hex-Rays code mode builds on an existing IDAPython/IDA Domain SDK, persistent lease-scoped execution, and on-demand API reference. REA would additionally maintain the SDK, execution/state lifecycle, cancellation, bounds and derived-Evidence semantics. Its incremental benefit here is not yet established. The vendor's internal token-saving claim is not a REA measurement.

Expected interaction with targets

No target interaction (this decision record). Follow-up improvements initially read existing files/captured data and authenticated retained Evidence within existing authority. This decision authorizes no new target execution, injection, mutation or protection recovery capability.

Boundary details (optional)

Prioritize these invariants in independently reviewable implementation issues:

Composable facts and relationships: canonical identity, exact typed fields, and batching where measured repeated reads justify it; avoid requiring full reports to retrieve one fact.
On-demand data: useful filtering, field selection, paging and demonstrated aggregation needs, while keeping complete Evidence accessible.
Consistent Evidence semantics: compact inline facts retain necessary source support, selection/coverage, partial/unknown states and parent references. Page exhaustion does not establish complete source or runtime coverage.
One implementation: domain/application owners interpret facts once; CLI/MCP and any future SDK consume the same canonical contracts and workflows.

Start with the measured JavaScript module-identity/reference read amplification. Revalidate against latest main; reuse existing owners and avoid overlapping fixes. The prototype is evidence for a candidate, not a wholesale integration plan.

Options and tradeoffs

Chosen now: improve shared data queries and existing domain tools. These benefits apply to direct use, ordinary scripts and any later code mode.
Deferred: a production generic code host, persistent REPL, new SDK/runtime product surface, and selecting code mode as the default interface.
Avoid adding one task-specific tool per question or building a general query language without observed demand.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Each implementation starts from current main on its own branch and links #4027 plus its concrete issue. Complete correctness/performance verification before integration.
Measure equivalent completed real CLI/MCP workflows: whole-task token/context costs (label estimates and unmeasured model costs), elapsed time, CPU and peak process-family memory; separate first-process/warm-session runs. Failures earn no efficiency credit. Keep total local work within the agreed 8 GiB budget, with bounded concurrency.
Save reusable verification scripts, acquisition instructions, checksums and compact measurements. Do not archive downloadable applications or regenerable large inputs/Evidence.
Revisit code mode only after optimized direct tools still require substantial, variable multi-step data integration and repeated controlled comparisons show a useful incremental benefit without reduced reliability or lost provenance. Freeze tasks/scoring/model settings and use identical information; include discovery, retries, errors and output costs.

References

First implementation slice

#4092 is implemented in production PR #4097: typed retained module-reference facts through the existing shared CLI/MCP view. No code-mode runtime or SDK is included. Real Obsidian verification preserves the selected 23 rows and partial coverage; full measurements and limits are in the PR.

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4091 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed **Decision for this workstream: defer production code mode. Improve browser/Electron/JavaScript tools and their shared data contracts first.** This is a prioritization decision, not a finding that code mode cannot work. Reconsider only when repeated real workflows demonstrate a useful additional benefit over optimized direct tools using the same backend and information. ### Context and evidence Related tracker: #4027. Exploration: #4063. Research PR #4082 is closed without merging; retain the experiment branch, scripts and small result records. Exploratory work should remain in issues/branches, without an exploratory PR that could be merged accidentally. Observed with real Obsidian 1.14.4 and an Excalidraw browser capture: - The original four-question pilot passed direct 4/4 and code 3/4. A dependency follow-up passed directly and failed in code mode. Most code runs printed tool replies instead of programmatically composing an answer. - The first typed-interface cohort passed direct 2/2 and code 1/2. Mixed Evidence/Result return conventions and inappropriate discovery caused concrete errors and extra work. - A final adaptive pair used identical retained snapshot/query Evidence. Code correctly aggregated the dependency facts; direct made a count error. One adaptive pair with a failed baseline does not establish an efficiency advantage. - A deterministic real-MCP comparison reduced selected module-reference replies from 338,208 to 23,402 structured bytes by replacing a page plus 23 item reads with one typed query. Both direct and code can use this improvement. This establishes representation/call-count changes, not token or speed savings. The fallback agent harness restarted Codex CLI for every action and replayed history. Native-host latency, repeated outcomes, richer source/IPC joins, and isolated interface/runtime effects remain **NOT_ENOUGH_INFO**. Preserve these limitations and failed attempts. [Full report and saved records](https://github.com/morluto/rea/blob/experiment/browser-codemode-tool-design-20261015/experiments/codemode-data-contract/results-2026-10-15.md). Source-derived comparison: Hex-Rays code mode builds on an existing IDAPython/IDA Domain SDK, persistent lease-scoped execution, and on-demand API reference. REA would additionally maintain the SDK, execution/state lifecycle, cancellation, bounds and derived-Evidence semantics. Its incremental benefit here is not yet established. The vendor's internal token-saving claim is not a REA measurement. ### Expected interaction with targets No target interaction (this decision record). Follow-up improvements initially read existing files/captured data and authenticated retained Evidence within existing authority. This decision authorizes no new target execution, injection, mutation or protection recovery capability. ### Boundary details (optional) Prioritize these invariants in independently reviewable implementation issues: 1. **Composable facts and relationships:** canonical identity, exact typed fields, and batching where measured repeated reads justify it; avoid requiring full reports to retrieve one fact. 2. **On-demand data:** useful filtering, field selection, paging and demonstrated aggregation needs, while keeping complete Evidence accessible. 3. **Consistent Evidence semantics:** compact inline facts retain necessary source support, selection/coverage, partial/unknown states and parent references. Page exhaustion does not establish complete source or runtime coverage. 4. **One implementation:** domain/application owners interpret facts once; CLI/MCP and any future SDK consume the same canonical contracts and workflows. Start with the measured JavaScript module-identity/reference read amplification. Revalidate against latest main; reuse existing owners and avoid overlapping fixes. The prototype is evidence for a candidate, not a wholesale integration plan. ### Options and tradeoffs - Chosen now: improve shared data queries and existing domain tools. These benefits apply to direct use, ordinary scripts and any later code mode. - Deferred: a production generic code host, persistent REPL, new SDK/runtime product surface, and selecting code mode as the default interface. - Avoid adding one task-specific tool per question or building a general query language without observed demand. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions - Each implementation starts from current main on its own branch and links #4027 plus its concrete issue. Complete correctness/performance verification before integration. - Measure equivalent completed real CLI/MCP workflows: whole-task token/context costs (label estimates and unmeasured model costs), elapsed time, CPU and peak process-family memory; separate first-process/warm-session runs. Failures earn no efficiency credit. Keep total local work within the agreed 8 GiB budget, with bounded concurrency. - Save reusable verification scripts, acquisition instructions, checksums and compact measurements. Do not archive downloadable applications or regenerable large inputs/Evidence. - Revisit code mode only after optimized direct tools still require substantial, variable multi-step data integration and repeated controlled comparisons show a useful incremental benefit without reduced reliability or lost provenance. Freeze tasks/scoring/model settings and use identical information; include discovery, retries, errors and output costs. ### References - #4027, #4063, closed #4082 - [Hex-Rays architecture](https://github.com/HexRaysSA/ida-mcp/blob/df6de7985b534a964c125d5a86e41b26b17b14f6/docs/ARCHITECTURE.md) - [REA tool-design guide](https://github.com/morluto/rea/blob/main/docs/tool-design.md) ### First implementation slice #4092 is implemented in production PR #4097: typed retained module-reference facts through the existing shared CLI/MCP view. No code-mode runtime or SDK is included. Real Obsidian verification preserves the selected 23 rows and partial coverage; full measurements and limits are in the PR.

bsakaguchi added the

labels

2026-10-16 21:08:02 +00:00

bsakaguchi referenced this issue

2026-10-17 21:12:43 +00:00

[PR #1601] [MERGED] perf(javascript): reduce retained semantic analysis allocations #2996

bsakaguchi added

enhancement