morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #5010] [Bug] Hopper literal string search reprojects every cached record for sparse matches #3089

New issue

Closed

opened 2026-10-31 00:36:27 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:36:27 +00:00

Owner

Originally created by @gensokyo-dev on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5010

Before submitting

Searched open and closed reports and removed credentials/private paths.

Area

Native analysis / Hopper Python bridge.

Observed behavior

Source finding at c12d9eda87: every positive literal string search re-enters _string_inventory(document), sorts all object addresses and projects a complete record map after matches have already been selected. The search inventory first discarded those same records into a value-only view.

Reproduction and measurements

Executed: official publisher-authenticated Hopper 6.6.0 Linux demo, REA 6.7.0, Node 24.18.0, private Ubuntu Base 24.04 runtime on Debian 12. Three paired fresh-engine public stdio MCP workflows against Debian SQLite 3.40.1 (17 calls each), plus real source-fixture semantic and CLI/MCP parity checks. Targets were not executed.

Warm-query median: 14.849 -> 12.628 ms; six warm reads: 100.202 -> 75.552 ms. Whole guarded elapsed: 89.384 -> 93.630 s, CPU: 88.333 -> 92.590 s. No overall acceleration, memory reduction or context saving is established. Offline raw context estimates are about 1.066M tokens per validation task, including full catalog and inventory oracle. One resource-stopped attempt is retained and excluded from completed comparisons.

Expected behavior / proposed correction

Keep the existing owner's complete decoded records in the sorted search inventory and directly project matching records, removing the lossy _strings view and redundant full projection. No new cache or public contract. Preserve address order, typed metadata, decoding unknowns and Evidence. Regex retains the existing supervised worker.

Correctness evidence and limits

All 42 applicable Evidence payloads across the three pairs agree on facts, IDs, provenance, authority and coverage/limitations (live environment observations excluded). Both real conformance workflows passed 70 calls each, including six expected errors; CLI matches MCP facts and Evidence ID. Existing focused tests: 35 passed in five files. Native recognition misses some known source literals; their bytes are verified but decoded-object coverage stays unverified. No real UTF-16 provider or complex agent-investigation claim.

Separate reordered-call controls locate most preparation time in an unchanged procedure lookup before string search; first string queries then take 329.254 / 337.533 ms. This does not establish the cause of whole-task variance. Actual rea setup, other operating systems and broad provider parity remain unverified. The minimal-runtime Python bootstrap issue is separately tracked in #5015.

Originally created by @gensokyo-dev on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5010 ### Before submitting - [x] Searched open and closed reports and removed credentials/private paths. ### Area Native analysis / Hopper Python bridge. ### Observed behavior Source finding at c12d9eda87aa5a022ee7e18ea96d8d03bfd12e96: every positive literal string search re-enters `_string_inventory(document)`, sorts all object addresses and projects a complete record map after matches have already been selected. The search inventory first discarded those same records into a value-only view. ### Reproduction and measurements Executed: official publisher-authenticated Hopper 6.6.0 Linux demo, REA 6.7.0, Node 24.18.0, private Ubuntu Base 24.04 runtime on Debian 12. Three paired fresh-engine public stdio MCP workflows against Debian SQLite 3.40.1 (17 calls each), plus real source-fixture semantic and CLI/MCP parity checks. Targets were not executed. Warm-query median: 14.849 -> 12.628 ms; six warm reads: 100.202 -> 75.552 ms. Whole guarded elapsed: 89.384 -> 93.630 s, CPU: 88.333 -> 92.590 s. No overall acceleration, memory reduction or context saving is established. Offline raw context estimates are about 1.066M tokens per validation task, including full catalog and inventory oracle. One resource-stopped attempt is retained and excluded from completed comparisons. ### Expected behavior / proposed correction Keep the existing owner's complete decoded records in the sorted search inventory and directly project matching records, removing the lossy `_strings` view and redundant full projection. No new cache or public contract. Preserve address order, typed metadata, decoding unknowns and Evidence. Regex retains the existing supervised worker. ### Correctness evidence and limits All 42 applicable Evidence payloads across the three pairs agree on facts, IDs, provenance, authority and coverage/limitations (live environment observations excluded). Both real conformance workflows passed 70 calls each, including six expected errors; CLI matches MCP facts and Evidence ID. Existing focused tests: 35 passed in five files. Native recognition misses some known source literals; their bytes are verified but decoded-object coverage stays unverified. No real UTF-16 provider or complex agent-investigation claim. Separate reordered-call controls locate most preparation time in an unchanged procedure lookup before string search; first string queries then take 329.254 / 337.533 ms. This does not establish the cause of whole-task variance. Actual `rea setup`, other operating systems and broad provider parity remain unverified. The minimal-runtime Python bootstrap issue is separately tracked in #5015.

bsakaguchi

2026-10-31 00:36:27 +00:00

closed this issue
added the
triage

bug
labels

bsakaguchi commented

2026-10-31 00:36:28 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

Confirmed from the code path: for sparse matches the full record-map projection in _string_inventory runs on every positive hit, so cost scales with inventory size rather than match count — exactly the wrong trade for cached-string workloads. Reusing the owner's selected-address projection and keeping the metadata/order/decoding behavior intact is the right fix; the PR #5013 direction looks correct. Suggest adding a warm repeated-query benchmark in the verification plan, since repeated hits on the same addresses are the common Hopper workflow.

@XinwangTao commented on GitHub (Oct 18, 2026): Confirmed from the code path: for sparse matches the full record-map projection in _string_inventory runs on every positive hit, so cost scales with inventory size rather than match count — exactly the wrong trade for cached-string workloads. Reusing the owner's selected-address projection and keeping the metadata/order/decoding behavior intact is the right fix; the PR #5013 direction looks correct. Suggest adding a warm repeated-query benchmark in the verification plan, since repeated hits on the same addresses are the common Hopper workflow.

bsakaguchi commented

2026-10-31 00:36:29 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

Confirmed from the code path: for sparse matches the full record-map projection in _string_inventory runs on every positive hit, so cost scales with inventory size rather than match count — exactly the wrong trade for cached-string workloads. Reusing the owner selected-address projection and keeping the metadata/order/decoding behavior intact is the right fix; the PR #5013 direction looks correct. Suggest adding a warm repeated-query benchmark in the verification plan, since repeated hits on the same addresses are the common Hopper workflow.

@XinwangTao commented on GitHub (Oct 18, 2026): Confirmed from the code path: for sparse matches the full record-map projection in _string_inventory runs on every positive hit, so cost scales with inventory size rather than match count — exactly the wrong trade for cached-string workloads. Reusing the owner selected-address projection and keeping the metadata/order/decoding behavior intact is the right fix; the PR #5013 direction looks correct. Suggest adding a warm repeated-query benchmark in the verification plan, since repeated hits on the same addresses are the common Hopper workflow.

bsakaguchi referenced this issue

2026-10-31 01:07:56 +00:00

[PR #3089] [MERGED] fix(native): classify every Swift mangling prefix as invalid when undecodable #5606

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3089

No description provided.

Rows
Columns