morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3135] design(perf): define bounded pilots for remaining real-artifact and provider coverage #1094

New issue

Open

opened 2026-10-15 18:31:01 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:31:01 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3135

Originally assigned to: @N0zoM1z0 on GitHub.

Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. #778 owns reproducible demo presentation, #2383 broad language conformance and #1065 the performance roadmap; this issue owns only bounded efficiency pilot selection.

Analyst question and outcome

Which remaining real-artifact/provider gaps are worth the next bounded efficiency experiment, and what exact question, oracle, authority and migration checks make each result reviewable?

Existing evidence and decision boundary

Completed #2016 evidence is deliberately scoped. Linux Ghidra verified native Windows PE and original compressed DOS MZ; native Windows runtime behavior, unpacked editor logic and DOS COM remain unverified. Selected JavaScript inventory/static analysis is not whole-app AST/runtime coverage; firmware region observations are not device execution or complete firmware analysis. Flutter, SQLite and compact-PCAP findings also cover specific questions rather than every app, provider or host.

Bounded work

This issue produces a coverage-and-experiment plan, not an all-domain benchmark or new provider implementation.

Reconcile current public contracts, existing real verification reports and the #2016 curated evidence. List concrete missing proof for larger/unpacked native questions and DOS COM, bounded JavaScript AST tasks, deeper firmware region questions, browser/runtime workflows and unavailable OS/providers.
Reuse shipped public inputs and available providers. Distinguish native PE static facts from Windows execution, DOS MZ from COM/unpacked logic, inventory from AST, and firmware extraction from runtime behavior. Coordinate existing scenarios with #778; language conformance remains owned by #2383.
Rank at most three next pilots by observed cost, recurring analyst question and feasible independent oracle. For each, define one question, artifact acquisition/digest, host/provider versions, exact old/current invocations, migration checks, resource bounds and a stopping condition. More missing rows do not mandate new product code or every host in one round.
Map each selected pilot to a narrow execution issue with its own completion condition. Mark unavailable hosts/providers and runtime-authority requirements explicitly; fixtures cannot substitute for actual providers. Leave excluded/unavailable lanes visible without blocking closure of the completed #2016 scope.

Acceptance

Current evidence is mapped to concrete observed/partial/unavailable scope without blanket domain or historical-client claims.
At most three ranked next experiments have exact questions, public artifact plans, independent oracles and compatibility/resource contracts.
Existing #778/#2383 ownership is respected; new execution tasks have non-overlapping scope and a success/no-benefit/unavailable stopping condition.
A concise plan records intentionally excluded/unavailable lanes; no all-provider run or new runtime authority is introduced by this planning task.

Compatibility, measurement and host constraints

Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions.

Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction.

Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing /tmp team handoff. No release is part of this task.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3135 Originally assigned to: @N0zoM1z0 on GitHub. Follow-up to #2016, whose original three-stage implementation and scoped verification are complete. This issue owns only the bounded continuation below. #778 owns reproducible demo presentation, #2383 broad language conformance and #1065 the performance roadmap; this issue owns only bounded efficiency pilot selection. ## Analyst question and outcome Which remaining real-artifact/provider gaps are worth the next bounded efficiency experiment, and what exact question, oracle, authority and migration checks make each result reviewable? ## Existing evidence and decision boundary Completed #2016 evidence is deliberately scoped. Linux Ghidra verified native Windows PE and original compressed DOS MZ; native Windows runtime behavior, unpacked editor logic and DOS COM remain unverified. Selected JavaScript inventory/static analysis is not whole-app AST/runtime coverage; firmware region observations are not device execution or complete firmware analysis. Flutter, SQLite and compact-PCAP findings also cover specific questions rather than every app, provider or host. ## Bounded work This issue produces a coverage-and-experiment plan, not an all-domain benchmark or new provider implementation. 1. Reconcile current public contracts, existing real verification reports and the #2016 curated evidence. List concrete missing proof for larger/unpacked native questions and DOS COM, bounded JavaScript AST tasks, deeper firmware region questions, browser/runtime workflows and unavailable OS/providers. 2. Reuse shipped public inputs and available providers. Distinguish native PE static facts from Windows execution, DOS MZ from COM/unpacked logic, inventory from AST, and firmware extraction from runtime behavior. Coordinate existing scenarios with #778; language conformance remains owned by #2383. 3. Rank at most three next pilots by observed cost, recurring analyst question and feasible independent oracle. For each, define one question, artifact acquisition/digest, host/provider versions, exact old/current invocations, migration checks, resource bounds and a stopping condition. More missing rows do not mandate new product code or every host in one round. 4. Map each selected pilot to a narrow execution issue with its own completion condition. Mark unavailable hosts/providers and runtime-authority requirements explicitly; fixtures cannot substitute for actual providers. Leave excluded/unavailable lanes visible without blocking closure of the completed #2016 scope. ## Acceptance - [ ] Current evidence is mapped to concrete observed/partial/unavailable scope without blanket domain or historical-client claims. - [ ] At most three ranked next experiments have exact questions, public artifact plans, independent oracles and compatibility/resource contracts. - [ ] Existing #778/#2383 ownership is respected; new execution tasks have non-overlapping scope and a success/no-benefit/unavailable stopping condition. - [ ] A concise plan records intentionally excluded/unavailable lanes; no all-provider run or new runtime authority is introduced by this planning task. ## Compatibility, measurement and host constraints Token consumption and time to a correct answer are the primary outcomes. Record total/cached/uncached input, output, discovery, RPCs, retries and failed attempts separately from JSON bytes. Preserve negative results; a smaller response alone is not a speed or token win. Separate observed producer/transport/client work and unknown stages, and compare equivalent completed questions. Keep current complete defaults, caller-selected inputs/providers, source bytes, Evidence identity/links, omission/null/type distinctions, partial coverage, and lifecycle/export/import behavior. Inspect the current advertised contracts and reuse existing selectors/views before adding code. Any implementation needs measured benefit and relevant compatibility verification; a documented no-change decision is a successful research outcome. Run checks for affected boundaries rather than repeating unrelated suites after each small correction. Run heavy work serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap, owned scopes and host-availability checks. Distinguish verified payload-phase observations from unavailable terminal whole-scope counters; missing counters stay unknown. For performance experiments, use real public CLI/MCP workflows and actual Codex CLI tasks when assessing agent token/time outcomes, with a private temporary profile pointing at the tested REA MCP; pin the requested model/reasoning and runtime within each cohort. Planning or demand-only work does not require an unrelated model/provider matrix. Simulated providers do not establish real-provider behavior. Keep raw captures/binaries/replies/transcripts/configuration private, commit source and curated reproducible findings, and record unrelated product bugs in the existing `/tmp` team handoff. No release is part of this task.

bsakaguchi added the

labels

2026-10-15 18:31:01 +00:00

bsakaguchi referenced this issue

2026-10-17 21:08:32 +00:00

[PR #1094] [MERGED] fix(analysis): bind overview document identity to the active target #2642

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:55 +00:00

bsakaguchi commented

2026-10-31 00:29:56 +00:00

Author

Owner

@pablodiazjorge commented on GitHub (Oct 15, 2026):

For the larger native questions, one ready-made oracle: rea-lab's study 3 builds four real open-source C programs from pinned source archives with llvm-mingw (x64, -O2, stripped, reproducible) and takes ground truth from the full build's linker map, symbol table, .pdata and inlining info (ADR 0024). Each task asks where a described function or table lives and is graded by address range, with no LLM judge; the 24 tasks and their ground truth are published. It runs on a native Windows host with real Ghidra 12.1.4, so if a bounded native pilot needs real-artifact questions with an exact oracle, these can be reused, or the builder pointed at another source archive.

@pablodiazjorge commented on GitHub (Oct 15, 2026): For the larger native questions, one ready-made oracle: [rea-lab's study 3](https://github.com/pablodiazjorge/rea-lab/tree/main/writeups/2026-10-study-3) builds four real open-source C programs from pinned source archives with llvm-mingw (x64, `-O2`, stripped, reproducible) and takes ground truth from the full build's linker map, symbol table, `.pdata` and inlining info ([ADR 0024](https://github.com/pablodiazjorge/rea-lab/blob/main/docs/adr/0024-real-software-tasks.md)). Each task asks where a described function or table lives and is graded by address range, with no LLM judge; the 24 tasks and their ground truth are published. It runs on a native Windows host with real Ghidra 12.1.4, so if a bounded native pilot needs real-artifact questions with an exact oracle, these can be reused, or the builder pointed at another source archive.

bsakaguchi commented

2026-10-31 00:29:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 20, 2026):

Current-main reconciliation and bounded follow-through (2026-10-20). Source inventory at 3fca2e0ca6e54ba700c286d9a85a9600dee6a19d has 45 immediate src/ directories; directory presence is not a completed performance qualification. This checkpoint does not close this planning issue or claim an all-provider benchmark.

Shared Evidence/investigations: #3134 is closed with real mixed native/JavaScript SDK/model measurements; #4810 subsequently addressed repeated immutable native/imported-JavaScript validation. Existing selection/paging remains the common CLI/MCP path. #5349 then addressed the separate retained-capture validation bottleneck, with real qlog data, unchanged selected facts and a documented first-read tradeoff.
Other decompilers already have scoped work: IDA Windows includes real whoami.exe before/after task measurements and batching/search improvements; Hopper #5017 reuses decoded string records on a real SQLite task; JEB Linux CE has six completed real-provider sessions and did not justify another metadata cache. None establishes every operation/platform or universal whole-task speedups.
Apktool is no longer pending shared decoding: #4777 is closed by #4804. Further real APK measurements also cover the optional table-only projection. Reimplementing the original repeated-decode fix would duplicate existing work.
Retained artifact inventories: #5338 remains open, but current main already has immutable-source validation reuse and mutation/identity regressions from #3634; source reconciliation explicitly makes no fresh ZIP timing claim.
This round's second merged fix: #5355 removes duplicate Ghidra raw representations through the owning parsers, preserving distinct original observations. Real true/jq CLI/MCP tasks, unchanged selected facts, a capacity regression and independent address-context verification are recorded. The jq replies shrink 46.8%, but whole-task latency/CPU improvement is not established; slower cold preparation and a failed unchanged-main discovery control remain visible. Actual model tokens are unmeasured.

Remaining proof is workflow-specific: #3133 still lacks clean accepted model pairs; #4415 remains an exploration of real Joplin preparation. Go/Rust, command-driven providers, device/runtime observations, and shared startup/resource chains still need concrete completed-task measurements before proposing further caches or interfaces. Prior managed/Flutter/SQLite/WASM/EVM/packet/signal checkpoints remain scoped evidence, not blanket module completion. The next experiment should select an available real target and an independent oracle, count discovery/retries and token/time/family-resource costs, and stop with a no-change result when existing controls suffice. No new runtime or mutation authority is implied.

#5355 merged normally as 58d6043303e722115f1da18b9505c24abeeab423 after its complete CI run and latest-head Greptile review passed, with no unresolved review threads. #5349 is merged as 4e716b06078d725a53234503cbefbb340731e809. The two implementations used separate branches/worktrees and refreshed main for each issue.

@gensokyo-dev commented on GitHub (Oct 20, 2026): Current-main reconciliation and bounded follow-through (2026-10-20). Source inventory at `3fca2e0ca6e54ba700c286d9a85a9600dee6a19d` has 45 immediate `src/` directories; directory presence is not a completed performance qualification. This checkpoint does not close this planning issue or claim an all-provider benchmark. - **Shared Evidence/investigations:** #3134 is closed with [real mixed native/JavaScript SDK/model measurements](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/retained-investigation-workflows.md); #4810 subsequently addressed repeated [immutable native/imported-JavaScript validation](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/immutable-parent-validation.md). Existing selection/paging remains the common CLI/MCP path. #5349 then addressed the separate retained-capture validation bottleneck, with real qlog data, unchanged selected facts and a documented first-read tradeoff. - **Other decompilers already have scoped work:** [IDA Windows](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/ida-native-windows.md) includes real `whoami.exe` before/after task measurements and batching/search improvements; [Hopper #5017](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/hopper-string-records.md) reuses decoded string records on a real SQLite task; [JEB Linux CE](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/jeb-linux-native.md) has six completed real-provider sessions and did not justify another metadata cache. None establishes every operation/platform or universal whole-task speedups. - **Apktool is no longer pending shared decoding:** #4777 is closed by #4804. [Further real APK measurements](https://github.com/morluto/rea/blob/3fca2e0ca6e54ba700c286d9a85a9600dee6a19d/experiments/analysis-efficiency/apktool-string-tables.md) also cover the optional table-only projection. Reimplementing the original repeated-decode fix would duplicate existing work. - **Retained artifact inventories:** #5338 remains open, but current main already has immutable-source validation reuse and mutation/identity regressions from #3634; [source reconciliation](https://github.com/morluto/rea/issues/5338#issuecomment-6283870298) explicitly makes no fresh ZIP timing claim. - **This round's second merged fix:** #5355 removes duplicate Ghidra raw representations through the owning parsers, preserving distinct original observations. Real `true`/jq CLI/MCP tasks, unchanged selected facts, a capacity regression and independent address-context verification are [recorded](https://github.com/morluto/rea/blob/ce97c2d4682ffcb97c7b25084485133c91746134/experiments/analysis-efficiency/ghidra-evidence-projection.md). The jq replies shrink 46.8%, but whole-task latency/CPU improvement is **not** established; slower cold preparation and a failed unchanged-main discovery control remain visible. Actual model tokens are unmeasured. Remaining proof is workflow-specific: #3133 still lacks clean accepted model pairs; #4415 remains an exploration of real Joplin preparation. Go/Rust, command-driven providers, device/runtime observations, and shared startup/resource chains still need concrete completed-task measurements before proposing further caches or interfaces. Prior managed/Flutter/SQLite/WASM/EVM/packet/signal checkpoints remain scoped evidence, not blanket module completion. The next experiment should select an available real target and an independent oracle, count discovery/retries and token/time/family-resource costs, and stop with a no-change result when existing controls suffice. No new runtime or mutation authority is implied. #5355 merged normally as `58d6043303e722115f1da18b9505c24abeeab423` after its complete CI run and latest-head Greptile review passed, with no unresolved review threads. #5349 is merged as `4e716b06078d725a53234503cbefbb340731e809`. The two implementations used separate branches/worktrees and refreshed main for each issue.

bsakaguchi commented

2026-10-31 00:29:56 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 21, 2026):

Coverage reconciliation and one measured producer follow-up are merged in #5781. The 2026-10-21 workflow table maps all46 source directories /212 tools at pinned main264862bf2 to located evidence and narrower gaps. It incorporates original feature-PR measurements: model #5603, IL2CPP #3807/#5327, firmware extraction #4602, selected IDA/Go/GDB/objdump/Frida tasks, and historical Apple NIB byte reductions. These must not be treated as wholly unaudited modules.

Three bounded further questions are Rust dependency identity (tiny real macOS correctness already exists, full task costs remain missing in the searched records), real Apple UI graph lookup, and reference/source-to-bundle comparison. Rizin/Cutter and device/live lanes remain explicit availability/coverage gaps. This is not a blanket real-provider qualification or closure of this tracker; #3133's actual model/task requirements remain separate from the completed qlog producer audit.

#5774 is closed by the GGUF omitted-string validation fix. On real stories15M: first/repeat parsing154→121 /54→37ms; full task6.667→6.567s with similar control variation, so no stable whole-task speedup, token or peak-memory saving is claimed. Main-task peak771→816MiB. Complete source/report/answer identities and independent reader checks agree. Curated records pin all before/after samples and original driver hashes. CI/review/remaining-job state is recorded on #5781.

@gensokyo-dev commented on GitHub (Oct 21, 2026): Coverage reconciliation and one measured producer follow-up are merged in #5781. The [2026-10-21 workflow table](https://github.com/morluto/rea/blob/7d8e4e43380b9dbea4d4061dfa9f12f8e9dc5e08/experiments/analysis-efficiency/coverage-review-20261021.md) maps all46 source directories /212 tools at pinned main264862bf2 to located evidence and narrower gaps. It incorporates original feature-PR measurements: model #5603, IL2CPP #3807/#5327, firmware extraction #4602, selected IDA/Go/GDB/objdump/Frida tasks, and historical Apple NIB byte reductions. These must not be treated as wholly unaudited modules. Three bounded further questions are Rust dependency identity (tiny real macOS correctness already exists, full task costs remain missing in the searched records), real Apple UI graph lookup, and reference/source-to-bundle comparison. Rizin/Cutter and device/live lanes remain explicit availability/coverage gaps. This is not a blanket real-provider qualification or closure of this tracker; #3133's actual model/task requirements remain separate from the completed qlog producer audit. #5774 is closed by the GGUF omitted-string validation fix. On real stories15M: first/repeat parsing154→121 /54→37ms; full task6.667→6.567s with similar control variation, so no stable whole-task speedup, token or peak-memory saving is claimed. Main-task peak771→816MiB. Complete source/report/answer identities and independent reader checks agree. Curated records pin all before/after samples and original driver hashes. CI/review/remaining-job state is recorded on #5781.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1094

No description provided.

Rows
Columns