morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #7146] [MERGED] feat(native): page retained observations and source context #7674

New issue

Closed

opened 2026-11-03 13:39:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 13:39:33 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/7146
Author: @gensokyo-dev
Created: 10/25/2026
Status: ✅ Merged
Merged: 10/25/2026
Merged by: @gensokyo-dev

Base: main ← Head: feat/6426-native-observation-views

📝 Commits (5)

1dc7326 feat(native): page retained observations and source context
e7ab53d docs(native): record complete observation workflow measurements
248245c fix(native): preserve context coverage and overflow recovery
4ac1971 fix(benchmarks): finalize partially initialized observation probes
2eb7131 docs(native): publish reviewed workflow verification

📊 Changes

29 files changed (+2266 additions, -78 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+15 -0)
📝 docs/mcp-contracts.md (+30 -0)
➕ experiments/analysis-efficiency/native-observation-views.md (+113 -0)
➕ experiments/analysis-efficiency/native-observation-views.mjs (+417 -0)
📝 experiments/analysis-efficiency/retained-query-finalization.mjs (+18 -8)
📝 experiments/analysis-efficiency/test_retained_queries.py (+20 -0)
📝 src/application/EnhancedTools.ts (+4 -3)
📝 src/application/analysisView/AnalysisViewService.ts (+10 -2)
➕ src/application/analysisView/NativeObservationViews.test.ts (+543 -0)
📝 src/contracts/enhancedToolContracts.ts (+5 -2)
📝 src/contracts/officialToolContracts.ts (+8 -8)
📝 src/contracts/toolOutputSchemaGroups.ts (+25 -14)
📝 src/contracts/toolOutputSchemaPrimitives.ts (+0 -17)
📝 src/domain/analysisView/analysisView.ts (+98 -1)
📝 src/domain/analysisView/commandObservationView.ts (+1 -1)
➕ src/domain/analysisView/nativeGraphView.ts (+98 -0)
📝 src/domain/analysisView/nativeInventoryView.ts (+1 -1)
📝 src/domain/analysisView/nativeLoadImageView.ts (+1 -1)
➕ src/domain/analysisView/nativeObservationSchemas.ts (+199 -0)
➕ src/domain/analysisView/nativeObservationView.ts (+326 -0)

...and 9 more files

📄 Description

Summary

Native fact queries repeated complete analysis profiles and caveats, while call graphs repeated diagnostic text for every node. Add opt-in detail: "summary" to procedure_address, procedure_callees, procedure_callers, resolve_containing_procedure and get_call_graph: return the first 64 observations and retain the complete source. native-observation pages preserve source indices, graph levels, failed nodes, duplicate messages and partial/unknown coverage; graph message dictionaries deduplicate exact text within each page. native-context facets page the complete metadata, profile and caveats through the same CLI/MCP projector. Default complete delivery and original Evidence identity remain unchanged.

Context identity reuses Evidence's semantic subject identity, preserving caller-selected path aliases. Coverage stays in the derived Evidence envelope instead of being repeated in its normalized payload. No provider execution, cache, target mutation or authority changes; maintainer scope decision is not applicable. Completes the observation/context portion of #6426 after #7134 and #7140.

Validation

Source regressions: earlier 61/57-case runs passed; after updating to main 5d428390996, all 47 observation/schema cases passed, including partial/unknown context coverage. Path-alias and context-coverage regressions failed before their owner fixes.
Built CLI/MCP boundaries: earlier 32 cases and two graph coverage cases passed; both final SDK/schema and five-operation overflow-recovery cases passed. Thirteen receipt/finalization cases and actual missing-target/Git-failure invocations pass without leaks. check:fast, formatting, docs:check (nine consistency cases), and build passed. Overflow regression uses a small injected budget; no real >10 MiB graph claim is made.
Final real Ghidra 12.1.4 / Node 24.22.1 / JDK 21.0.12.1, unmodified stripped /usr/bin/ls: both completed investigations preserve all 65 source Evidence IDs and every reconstructed fact, complete profile and caveat. Cold/warm fact digests match. The client reuses exact context/profile digests during one investigation; the server adds no cache.

Completed task	Complete control	Summary + complete context
RPCs	137	179
Catalog token estimate	919,115	932,545
Catalog + arguments + text token estimate	1,243,161	1,234,768
Catalog + arguments + raw replies token estimate	1,587,863	1,554,038
All reply text bytes	1,331,358	1,006,503
Whole-family elapsed seconds	39.38	41.06
Whole-family CPU seconds	69.26	70.4
Sampled family peak RSS GiB	1.45	1.69

The completed-task text estimate decreases 0.68%; raw-reply accounting decreases
2.13%, and text bytes decrease 24.4%. The large forward graph's first reply is
63,888 → 5,150 estimated text tokens (283,990 → 15,995 bytes); remaining facts
and all caveats are included in the completed-task table. A one-address callee
reply is only 754 → 744 tokens despite 3,104 → 2,326 bytes: bytes alone would
overstate its token improvement. There is no demonstrated time or memory
improvement. The complete-read workflow adds 42 RPCs; its warm phase is
0.762 → 1.005 seconds. Cold phases are 33.151 → 34.804 seconds. This one fresh
session per final lane cannot separate provider-startup variance from overhead.
The final candidate implementation and experiment driver are 4ac1971864;
the final control is 5d428390996. CPU and RSS remain within the stated guard
but are higher. An earlier pre-review pair measured 34.55 → 50.29 seconds,
so elapsed-time variance must not be mistaken for a speed improvement.

Providerless CLI replay from the control's exported real graph also passed: offset 64 / limit 3 retained exact source indices, addresses, calls, caveat references and the 327-observation total.

The reproducible driver and measurement limits are in experiments/analysis-efficiency/native-observation-views.{mjs,md}. Token counts are tokenizer estimates over catalog discovery, arguments, every reply and every continuation, not actual model usage. Real-provider preservation does not establish Ghidra discovery completeness. No cross-provider runtime claim is made.

Author readiness

Human owner: @gensokyo-dev

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

The repository owner authorized proceeding with these declarations unchecked and bypassing the readiness gate; no human-review attestation is made.

Large inventory follow-up

The existing native-inventory summary/page path is also verified on unmodified stripped LLVM 14 (436887791de0478d72c8323be99df69d6d0cf82745e5abec79d5e0374f4df560): 43,643 Ghidra procedures and 177,830 defined strings. Complete string delivery was refused at the 10 MiB MCP boundary; a retained summary and first/last pages succeeded without another provider read. Six UTF-8 observations, including .shstrtab addresses, match independently read original-file bytes. The 39,856,151-byte complete Evidence bundle was exported before closing, then authenticated offline; the function-only bundle preserves its unchanged real source record.

This is a deliberately narrower, explicitly committed 29-disabled-analyzer Ghidra 12.1.4 profile with ASCII Strings enabled, not default analysis or complete function-body coverage. Whole cold investigation: 280.14 s elapsed, 366.63 s CPU, 2.88 GiB sampled family RSS; two CPU affinity slots, 8 GiB family guard, Node/Ghidra 3 GiB heaps. No OOM or cleanup failure. Earlier attempts include a 600 s SDK timeout during constant propagation, a rejected unregistered analyzer name, and a driver exporting after session close; all are retained as failed verification attempts and excluded from the completed-run number. Both export ordering and retained-summary reuse were corrected in the verification driver, with no production change.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/7146 **Author:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Created:** 10/25/2026 **Status:** ✅ Merged **Merged:** 10/25/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `feat/6426-native-observation-views` --- ### 📝 Commits (5) - [`1dc7326`](https://github.com/morluto/rea/commit/1dc73267caae464b49fa878c426a2703d87f75f0) feat(native): page retained observations and source context - [`e7ab53d`](https://github.com/morluto/rea/commit/e7ab53d110dbf8bd11f8f4cdeb11550742cbee67) docs(native): record complete observation workflow measurements - [`248245c`](https://github.com/morluto/rea/commit/248245c9cde08fb98e7c57aefa737b1bcc21a14e) fix(native): preserve context coverage and overflow recovery - [`4ac1971`](https://github.com/morluto/rea/commit/4ac1971864b76796bc424ae5df8a1f41b0b3dfff) fix(benchmarks): finalize partially initialized observation probes - [`2eb7131`](https://github.com/morluto/rea/commit/2eb713150c752a8074362c60121d1e0dfa03cf02) docs(native): publish reviewed workflow verification ### 📊 Changes **29 files changed** (+2266 additions, -78 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+15 -0) 📝 `docs/mcp-contracts.md` (+30 -0) ➕ `experiments/analysis-efficiency/native-observation-views.md` (+113 -0) ➕ `experiments/analysis-efficiency/native-observation-views.mjs` (+417 -0) 📝 `experiments/analysis-efficiency/retained-query-finalization.mjs` (+18 -8) 📝 `experiments/analysis-efficiency/test_retained_queries.py` (+20 -0) 📝 `src/application/EnhancedTools.ts` (+4 -3) 📝 `src/application/analysisView/AnalysisViewService.ts` (+10 -2) ➕ `src/application/analysisView/NativeObservationViews.test.ts` (+543 -0) 📝 `src/contracts/enhancedToolContracts.ts` (+5 -2) 📝 `src/contracts/officialToolContracts.ts` (+8 -8) 📝 `src/contracts/toolOutputSchemaGroups.ts` (+25 -14) 📝 `src/contracts/toolOutputSchemaPrimitives.ts` (+0 -17) 📝 `src/domain/analysisView/analysisView.ts` (+98 -1) 📝 `src/domain/analysisView/commandObservationView.ts` (+1 -1) ➕ `src/domain/analysisView/nativeGraphView.ts` (+98 -0) 📝 `src/domain/analysisView/nativeInventoryView.ts` (+1 -1) 📝 `src/domain/analysisView/nativeLoadImageView.ts` (+1 -1) ➕ `src/domain/analysisView/nativeObservationSchemas.ts` (+199 -0) ➕ `src/domain/analysisView/nativeObservationView.ts` (+326 -0) _...and 9 more files_ </details> ### 📄 Description ## Summary Native fact queries repeated complete analysis profiles and caveats, while call graphs repeated diagnostic text for every node. Add opt-in `detail: "summary"` to `procedure_address`, `procedure_callees`, `procedure_callers`, `resolve_containing_procedure` and `get_call_graph`: return the first 64 observations and retain the complete source. `native-observation` pages preserve source indices, graph levels, failed nodes, duplicate messages and partial/unknown coverage; graph message dictionaries deduplicate exact text within each page. `native-context` facets page the complete metadata, profile and caveats through the same CLI/MCP projector. Default complete delivery and original Evidence identity remain unchanged. Context identity reuses Evidence's semantic subject identity, preserving caller-selected path aliases. Coverage stays in the derived Evidence envelope instead of being repeated in its normalized payload. No provider execution, cache, target mutation or authority changes; maintainer scope decision is not applicable. Completes the observation/context portion of #6426 after #7134 and #7140. ## Validation - Source regressions: earlier 61/57-case runs passed; after updating to main `5d428390996`, all 47 observation/schema cases passed, including partial/unknown context coverage. Path-alias and context-coverage regressions failed before their owner fixes. - Built CLI/MCP boundaries: earlier 32 cases and two graph coverage cases passed; both final SDK/schema and five-operation overflow-recovery cases passed. Thirteen receipt/finalization cases and actual missing-target/Git-failure invocations pass without leaks. `check:fast`, formatting, `docs:check` (nine consistency cases), and build passed. Overflow regression uses a small injected budget; no real >10 MiB graph claim is made. - Final real Ghidra 12.1.4 / Node 24.22.1 / JDK 21.0.12.1, unmodified stripped `/usr/bin/ls`: both completed investigations preserve all 65 source Evidence IDs and every reconstructed fact, complete profile and caveat. Cold/warm fact digests match. The client reuses exact context/profile digests during one investigation; the server adds no cache. | Completed task | Complete control | Summary + complete context | | --- | ---: | ---: | | RPCs | 137 | 179 | | Catalog token estimate | 919,115 | 932,545 | | Catalog + arguments + text token estimate | 1,243,161 | 1,234,768 | | Catalog + arguments + raw replies token estimate | 1,587,863 | 1,554,038 | | All reply text bytes | 1,331,358 | 1,006,503 | | Whole-family elapsed seconds | 39.38 | 41.06 | | Whole-family CPU seconds | 69.26 | 70.4 | | Sampled family peak RSS GiB | 1.45 | 1.69 | The completed-task text estimate decreases 0.68%; raw-reply accounting decreases 2.13%, and text bytes decrease 24.4%. The large forward graph's first reply is 63,888 → 5,150 estimated text tokens (283,990 → 15,995 bytes); remaining facts and all caveats are included in the completed-task table. A one-address callee reply is only 754 → 744 tokens despite 3,104 → 2,326 bytes: bytes alone would overstate its token improvement. There is no demonstrated time or memory improvement. The complete-read workflow adds 42 RPCs; its warm phase is 0.762 → 1.005 seconds. Cold phases are 33.151 → 34.804 seconds. This one fresh session per final lane cannot separate provider-startup variance from overhead. The final candidate implementation and experiment driver are `4ac1971864`; the final control is `5d428390996`. CPU and RSS remain within the stated guard but are higher. An earlier pre-review pair measured 34.55 → 50.29 seconds, so elapsed-time variance must not be mistaken for a speed improvement. Providerless CLI replay from the control's exported real graph also passed: offset 64 / limit 3 retained exact source indices, addresses, calls, caveat references and the 327-observation total. The reproducible driver and measurement limits are in `experiments/analysis-efficiency/native-observation-views.{mjs,md}`. Token counts are tokenizer estimates over catalog discovery, arguments, every reply and every continuation, not actual model usage. Real-provider preservation does not establish Ghidra discovery completeness. No cross-provider runtime claim is made. ## Author readiness Human owner: @gensokyo-dev - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. The repository owner authorized proceeding with these declarations unchecked and bypassing the readiness gate; no human-review attestation is made. ## Large inventory follow-up The existing native-inventory summary/page path is also verified on unmodified stripped LLVM 14 (`436887791de0478d72c8323be99df69d6d0cf82745e5abec79d5e0374f4df560`): 43,643 Ghidra procedures and 177,830 defined strings. Complete string delivery was refused at the 10 MiB MCP boundary; a retained summary and first/last pages succeeded without another provider read. Six UTF-8 observations, including `.shstrtab` addresses, match independently read original-file bytes. The 39,856,151-byte complete Evidence bundle was exported before closing, then authenticated offline; the function-only bundle preserves its unchanged real source record. This is a deliberately narrower, explicitly committed 29-disabled-analyzer Ghidra 12.1.4 profile with ASCII Strings enabled, not default analysis or complete function-body coverage. Whole cold investigation: 280.14 s elapsed, 366.63 s CPU, 2.88 GiB sampled family RSS; two CPU affinity slots, 8 GiB family guard, Node/Ghidra 3 GiB heaps. No OOM or cleanup failure. Earlier attempts include a 600 s SDK timeout during constant propagation, a rejected unregistered analyzer name, and a driver exporting after session close; all are retained as failed verification attempts and excluded from the completed-run number. Both export ordering and retained-summary reuse were corrected in the verification driver, with no production change. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-03 13:39:33 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7674

No description provided.

Rows
Columns