[GH-ISSUE #3136] research(perf): establish whether Ghidra database reuse adds value beyond snapshots #1098

New issue

Open

opened 2026-10-15 18:31:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:31:02 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3136

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. #1048 owns pre-imported Ghidra project support; #740/#1065 remain the architecture/performance roadmaps. This does not duplicate their feature implementations.

Analyst question and outcome

On a recurring native analyst question, is startup/import/analysis work still repeated after using existing authenticated snapshots, and is the remaining cost large enough to justify cross-session database reuse?

Existing evidence and decision boundary

Status: demand-and-benefit research; persistent cache implementation is deferred.

#2349 verifies existing same-profile snapshot reuse and pre-launch rejection of profile mismatches on real GNU true/jq. Fresh jq calls took about 295–299 seconds and existing MCP snapshot reads about 0.17–0.22 seconds, but these verification calls are not a repeated whole-question performance cohort. Recorded scope. Existing replay may already solve the requested workload.

Bounded work

Identify one recurring question with observable repeated expensive analysis on one larger public native artifact. Pin the current Ghidra/JDK/runtime/profile/target identity and public invocations.
Compare fresh sessions, supported live-session warm queries, authenticated JSON snapshot continuation, and any available pre-imported-project workflow owned by #1048. Preserve query entries, coverage and identity; do not call equal records a cache hit. Repeat equivalent completed tasks with startup/analysis/query and whole-task costs distinguished.
If existing mechanisms suffice or demand is absent, document a no-cache decision. Only if a material gap remains, write a separate opt-in reuse proposal with target/content/provider/version/profile/analysis-revision identity, one resource owner, capacity, invalidation and cleanup/retry rules. Concurrent callers or different Program instances must not be merged merely because they share an adapter.
No database persistence, pool/shared-query concurrency or ambient configuration changes are implemented here. Any future prototype is private/owned and requires demonstrated demand; preserving failed-cleanup ownership and explicit unknowns is mandatory.

Acceptance

One recurring workload and equivalent repeated controls identify whether analysis is genuinely duplicated after existing reuse.
Compare snapshots and applicable existing #1048 behavior, with honest unavailable stages and no unsupported cache-hit or whole-task claim.
Document a no-cache conclusion or a measured gap and a bounded opt-in design; any implementation follows in a separate issue.
Persistent engine caches and shared-query concurrency stay deferred unless the measurements justify their lifecycle/compatibility costs.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3136 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. #1048 owns pre-imported Ghidra project support; #740/#1065 remain the architecture/performance roadmaps. This does not duplicate their feature implementations. ## Analyst question and outcome On a recurring native analyst question, is startup/import/analysis work still repeated after using existing authenticated snapshots, and is the remaining cost large enough to justify cross-session database reuse? ## Existing evidence and decision boundary Status: demand-and-benefit research; persistent cache implementation is deferred. #2349 verifies existing same-profile snapshot reuse and pre-launch rejection of profile mismatches on real GNU true/jq. Fresh jq calls took about 295–299 seconds and existing MCP snapshot reads about 0.17–0.22 seconds, but these verification calls are not a repeated whole-question performance cohort. [Recorded scope](https://github.com/morluto/rea/blob/1dd20ce1ff4ab66c74350929577e753aa17a9006/experiments/analysis-efficiency/preset-lifecycle.md). Existing replay may already solve the requested workload. ## Bounded work 1. Identify one recurring question with observable repeated expensive analysis on one larger public native artifact. Pin the current Ghidra/JDK/runtime/profile/target identity and public invocations. 2. Compare fresh sessions, supported live-session warm queries, authenticated JSON snapshot continuation, and any available pre-imported-project workflow owned by #1048. Preserve query entries, coverage and identity; do not call equal records a cache hit. Repeat equivalent completed tasks with startup/analysis/query and whole-task costs distinguished. 3. If existing mechanisms suffice or demand is absent, document a no-cache decision. Only if a material gap remains, write a separate opt-in reuse proposal with target/content/provider/version/profile/analysis-revision identity, one resource owner, capacity, invalidation and cleanup/retry rules. Concurrent callers or different Program instances must not be merged merely because they share an adapter. 4. No database persistence, pool/shared-query concurrency or ambient configuration changes are implemented here. Any future prototype is private/owned and requires demonstrated demand; preserving failed-cleanup ownership and explicit unknowns is mandatory. ## Acceptance - [ ] One recurring workload and equivalent repeated controls identify whether analysis is genuinely duplicated after existing reuse. - [ ] Compare snapshots and applicable existing #1048 behavior, with honest unavailable stages and no unsupported cache-hit or whole-task claim. - [ ] Document a no-cache conclusion or a measured gap and a bounded opt-in design; any implementation follows in a separate issue. - [ ] Persistent engine caches and shared-query concurrency stay deferred unless the measurements justify their lifecycle/compatibility costs. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi added the

labels

2026-10-15 18:31:02 +00:00

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794