morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #4695] [MERGED] perf(kuna): reuse sorted string inventory for exact lookups #6362

New issue

Closed

opened 2026-10-31 01:13:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:13:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4695
Author: @N0zoM1z0
Created: 10/17/2026
Status: ✅ Merged
Merged: 10/17/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/kuna-string-address-lookup-20261017

📝 Commits (3)

5f56496 perf(kuna): reuse sorted string inventory for exact lookups
c3df8b5 Merge branch 'main' into perf/kuna-string-address-lookup-20261017
3adfc7d Merge branch 'main' into perf/kuna-string-address-lookup-20261017

📊 Changes

5 files changed (+158 additions, -8 deletions)

View changed files

📝 docs/kuna-provider.md (+1 -1)
📝 src/kuna/KunaStringInventory.ts (+20 -0)
📝 src/kuna/KunaStringQueries.ts (+2 -1)
📝 tests/boundary/providers/kuna/kunaStringInventory.test.ts (+80 -1)
📝 tests/boundary/providers/kuna/kunaStrings.test.ts (+55 -5)

📄 Description

Summary

Kuna already caches its validated string inventory in numeric address order, but every list_strings(address) request scanned all rows again. Reuse that order for binary search, returning every encoding at the selected address in O(log n + matches) time without another lookup index.

Change and scope

The inventory owner supplies the lookup; the shared provider workflow consumes it. Existing streaming validation (including early duplicate rejection), session cache lifetime, complete listings, search semantics, source digests, coverage/limitations and retained Evidence remain unchanged. No new public contracts, native Kuna commands, capabilities, dependencies or generated outputs. This is an ordinary static provider optimization; additional maintainer scope approval is not applicable.

Evidence and regression coverage

Executed: all 22 Kuna boundary files, 241 tests passed, including cancellation, deadlines, capacity, identity, concurrent work and cleanup failures. New coverage checks empty inventories, numeric address boundaries through u64 max, same-address encodings, normalized selectors, misses and malformed-inventory retries.
Executed: real production stdio MCP against unmodified Kuna 1.833 on Linux x86-64, LLVM 16 (602,544 string facts) and SQLite (7,985 facts). Independently mapped selected native string addresses through ELF load headers and verified file bytes/terminators. All selected results, target digests, source linkage, retained Evidence discovery and process cleanup passed.
Executed: changed-file lint/format, module boundaries and git diff --check; production build passed. Latest main was refreshed and rebased to 7086768a2; its intervening change is ADB-only. npm run check stopped at the 3.5 GiB early memory guard during root typechecking (observed OOM counters zero): not a pass. Separately, npm exec -- turbo run lint:built format:check knip metadata:check --concurrency=1 and npm run verify:architecture-guards passed, including a fresh latest-main build. CI Static checks (including typecheck), Source tests, Build package, Docs Generation and Native package baseline (linux x64) passed. The unrelated CI baseline failure and cancelled coverage are detailed below.

Performance

Three alternating fresh-session observations per LLVM implementation; same target, engine and task (catalog discovery, cold selected read, 28 warm exact-address reads, retained Evidence summary and close). One CPU, 4 GiB hard family limit, 3 GiB early-stop threshold, no swap. Cold means a fresh session, not a cold OS file cache.

Median	Before	After
28 warm MCP reads	495.2 ms	90.8 ms
Cold selected MCP read	11.30 s	11.01 s
Whole verification harness wall	28.24 s	27.43 s
Whole family CPU	27.71 s	26.89 s
Kernel family peak, including file cache	2237.6 MiB	2240.6 MiB

Warm totals were 491.7–544.3 ms before and 84.8–99.5 ms after (~5.5x median). Cold/harness times and peak memory overlap; no stable cold, whole-task or memory improvement is claimed. Harness resources include the independent native oracle, target hashing, MCP catalog and correctness checks. Every analysis scope confirmed payload cleanup and zero OOM events. SQLite's one pair: 20 warm reads 88.6 → 80.5 ms; whole harness 13.68 → 13.94 s, CPU 13.22 → 13.57 s, peak 924.7 → 929.8 MiB. Small-target overhead dominates.

Token/context: the catalog hash, number of calls and normalized facts are identical. A cl100k_base single-pass estimate over raw catalog JSON + tool arguments + complete replies is 846,301–846,579 tokens before and 846,083–846,545 after (catalog alone 802,562 each). These transport estimates are not rendered model context or actual model usage; client discovery/rendering, prompts, replay and protocol framing are unmeasured. No token reduction is claimed. No model trial was needed to measure this provider-internal lookup.

Pinned LLVM SHA-256: f62d254b7f2bf42df8c8b07d46ee3bb4c2cafeca436b2e6bc6ccbe4581f58f40; Kuna SHA-256: d858918b0766435cd96b999a77f2d5a31a0cc4f07256421c883b5b9dc4dd58e4. All six LLVM normalized-result sequence digests: 4d2f90269da8568984090b20274163c4a7c2a0538aab63530bd2fe44840a0bba. Native discovery completeness remains unknown; empty exact reads do not establish absence of undiscovered strings.

Additional executed verification

node scripts/verify-real-kuna.mjs --report … — passed with Kuna 1.833: symbol-bearing and stripped source-built ELF fixtures, full string results, exact reads, literal/regex searches, references/calls, export reuse, file evidence and complete cleanup.
rea search /usr/lib/x86_64-linux-gnu/libLLVM-16.so.1 'Invalid bitcode signature' --provider kuna --format json — passed; the single result and encoded metadata matched the independent native/file-byte oracle and selected target digest. Its source reports 602,544 inventory facts and one selected fact.
Greptile review — 5/5, no actionable comments. Codex review could not run because the repository review quota is exhausted; this is unavailable coverage, not a code finding.

Baseline follow-up (2026-10-17)

Head 3adfc7d16bd9e71705d973ffe7456f8bdc7dd3a9 incorporates main 5b2439938, including #4691 and #4703. Exact merge-tree verification and equal stable patch IDs confirm the original Kuna optimization is unchanged. Those two test-only main repairs resolve the earlier JavaScript comparison and Kuna producer/caller assertion failures.

Complete current-head CI passed: all five built-dependent shards, Source/static checks (including typecheck), build, docs, Linux native package baseline and required gates. The latest title check passes; an older cancelled title run is superseded. Current-head Greptile review passed, with no unresolved review threads. Codex review remains unavailable due explicitly reported repository usage limits.

Additional local validation on this exact head: 292 passed / 1 failed in a combined 29-file Kuna/JavaScript selection. The single failure is the existing 300 ms export-deadline test expecting captured output; the exact case passes unchanged in isolation, and its entire provider file subsequently passes 31/31. This test/export implementation has no feature diff against main. A pre-execution timing path can omit output capture, but the failing phase was not independently recorded: root cause remains unconfirmed. The initial failure is retained in the local audit; no all-pass is claimed for that combined run, and no code/timeout/assertion change was applied for the retries. The complete hosted CI above supplies independent coverage. The original real Kuna/performance observations remain pinned to their stated feature revisions; no fresh native performance trial on this merge commit is claimed.

All local heavy tasks used two CPU cores and a 3 GiB Node heap. The combined run took 155.28 s wall / 279.16 s reaped CPU / 3.30 GiB sampled family RSS; complete provider recheck took 19.34 s / 18.62 s / 0.85 GiB. No OOM/guard stop. Sampled RSS excludes assistant/shared services; cache/host conditions and controlled cold/warm costs are unmeasured. No new speedup or model-token claim.

Author readiness

Human owner: @N0zoM1z0

Declarations are entered by the agent at the human owner's explicit direction. The diff and evidence inspections described here were performed by the agent; this does not assert a separately observed human diff review.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4695 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/17/2026 **Status:** ✅ Merged **Merged:** 10/17/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/kuna-string-address-lookup-20261017` --- ### 📝 Commits (3) - [`5f56496`](https://github.com/morluto/rea/commit/5f564962da0e1b17f00a68a74a312539f0209a01) perf(kuna): reuse sorted string inventory for exact lookups - [`c3df8b5`](https://github.com/morluto/rea/commit/c3df8b518cb6aefdb2431c79c105a513df35f84c) Merge branch 'main' into perf/kuna-string-address-lookup-20261017 - [`3adfc7d`](https://github.com/morluto/rea/commit/3adfc7d16bd9e71705d973ffe7456f8bdc7dd3a9) Merge branch 'main' into perf/kuna-string-address-lookup-20261017 ### 📊 Changes **5 files changed** (+158 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `docs/kuna-provider.md` (+1 -1) 📝 `src/kuna/KunaStringInventory.ts` (+20 -0) 📝 `src/kuna/KunaStringQueries.ts` (+2 -1) 📝 `tests/boundary/providers/kuna/kunaStringInventory.test.ts` (+80 -1) 📝 `tests/boundary/providers/kuna/kunaStrings.test.ts` (+55 -5) </details> ### 📄 Description ## Summary Kuna already caches its validated string inventory in numeric address order, but every `list_strings(address)` request scanned all rows again. Reuse that order for binary search, returning every encoding at the selected address in O(log n + matches) time without another lookup index. ## Change and scope The inventory owner supplies the lookup; the shared provider workflow consumes it. Existing streaming validation (including early duplicate rejection), session cache lifetime, complete listings, search semantics, source digests, coverage/limitations and retained Evidence remain unchanged. No new public contracts, native Kuna commands, capabilities, dependencies or generated outputs. This is an ordinary static provider optimization; additional maintainer scope approval is not applicable. ## Evidence and regression coverage - **Executed:** all 22 Kuna boundary files, **241 tests passed**, including cancellation, deadlines, capacity, identity, concurrent work and cleanup failures. New coverage checks empty inventories, numeric address boundaries through u64 max, same-address encodings, normalized selectors, misses and malformed-inventory retries. - **Executed:** real production stdio MCP against unmodified Kuna 1.833 on Linux x86-64, LLVM 16 (602,544 string facts) and SQLite (7,985 facts). Independently mapped selected native string addresses through ELF load headers and verified file bytes/terminators. All selected results, target digests, source linkage, retained Evidence discovery and process cleanup passed. - **Executed:** changed-file lint/format, module boundaries and `git diff --check`; production build passed. Latest main was refreshed and rebased to `7086768a2`; its intervening change is ADB-only. `npm run check` stopped at the 3.5 GiB early memory guard during root typechecking (observed OOM counters zero): **not a pass**. Separately, `npm exec -- turbo run lint:built format:check knip metadata:check --concurrency=1` and `npm run verify:architecture-guards` passed, including a fresh latest-main build. CI Static checks (including typecheck), Source tests, Build package, Docs Generation and Native package baseline (linux x64) passed. The unrelated CI baseline failure and cancelled coverage are detailed below. ### Performance Three alternating fresh-session observations per LLVM implementation; same target, engine and task (catalog discovery, cold selected read, 28 warm exact-address reads, retained Evidence summary and close). One CPU, 4 GiB hard family limit, 3 GiB early-stop threshold, no swap. Cold means a fresh session, **not** a cold OS file cache. | Median | Before | After | | --- | ---: | ---: | | 28 warm MCP reads | 495.2 ms | 90.8 ms | | Cold selected MCP read | 11.30 s | 11.01 s | | Whole verification harness wall | 28.24 s | 27.43 s | | Whole family CPU | 27.71 s | 26.89 s | | Kernel family peak, including file cache | 2237.6 MiB | 2240.6 MiB | Warm totals were 491.7–544.3 ms before and 84.8–99.5 ms after (~5.5x median). Cold/harness times and peak memory overlap; **no stable cold, whole-task or memory improvement is claimed**. Harness resources include the independent native oracle, target hashing, MCP catalog and correctness checks. Every analysis scope confirmed payload cleanup and zero OOM events. SQLite's one pair: 20 warm reads 88.6 → 80.5 ms; whole harness 13.68 → 13.94 s, CPU 13.22 → 13.57 s, peak 924.7 → 929.8 MiB. Small-target overhead dominates. Token/context: the catalog hash, number of calls and normalized facts are identical. A `cl100k_base` single-pass estimate over raw catalog JSON + tool arguments + complete replies is 846,301–846,579 tokens before and 846,083–846,545 after (catalog alone 802,562 each). These transport estimates are **not rendered model context or actual model usage**; client discovery/rendering, prompts, replay and protocol framing are unmeasured. No token reduction is claimed. No model trial was needed to measure this provider-internal lookup. Pinned LLVM SHA-256: `f62d254b7f2bf42df8c8b07d46ee3bb4c2cafeca436b2e6bc6ccbe4581f58f40`; Kuna SHA-256: `d858918b0766435cd96b999a77f2d5a31a0cc4f07256421c883b5b9dc4dd58e4`. All six LLVM normalized-result sequence digests: `4d2f90269da8568984090b20274163c4a7c2a0538aab63530bd2fe44840a0bba`. Native discovery completeness remains unknown; empty exact reads do not establish absence of undiscovered strings. ### Additional executed verification - `node scripts/verify-real-kuna.mjs --report …` — passed with Kuna 1.833: symbol-bearing and stripped source-built ELF fixtures, full string results, exact reads, literal/regex searches, references/calls, export reuse, file evidence and complete cleanup. - `rea search /usr/lib/x86_64-linux-gnu/libLLVM-16.so.1 'Invalid bitcode signature' --provider kuna --format json` — passed; the single result and encoded metadata matched the independent native/file-byte oracle and selected target digest. Its source reports 602,544 inventory facts and one selected fact. - [Greptile review](https://github.com/morluto/rea/pull/4695#issuecomment-6230444547) — 5/5, no actionable comments. [Codex review](https://github.com/morluto/rea/pull/4695#issuecomment-6230423469) could not run because the repository review quota is exhausted; this is unavailable coverage, not a code finding. ## Baseline follow-up (2026-10-17) Head `3adfc7d16bd9e71705d973ffe7456f8bdc7dd3a9` incorporates main `5b2439938`, including #4691 and #4703. Exact merge-tree verification and equal stable patch IDs confirm the original Kuna optimization is unchanged. Those two test-only main repairs resolve the earlier JavaScript comparison and Kuna producer/caller assertion failures. **[Complete current-head CI passed](https://github.com/morluto/rea/actions/runs/39011208118)**: all five built-dependent shards, Source/static checks (including typecheck), build, docs, Linux native package baseline and required gates. The latest title check passes; an older cancelled title run is superseded. Current-head Greptile review passed, with no unresolved review threads. Codex review remains unavailable due explicitly reported repository usage limits. Additional local validation on this exact head: **292 passed / 1 failed** in a combined 29-file Kuna/JavaScript selection. The single failure is the existing 300 ms export-deadline test expecting captured output; the exact case passes unchanged in isolation, and its entire provider file subsequently passes **31/31**. This test/export implementation has no feature diff against main. A pre-execution timing path can omit output capture, but the failing phase was not independently recorded: root cause remains unconfirmed. The initial failure is retained in the local audit; no all-pass is claimed for that combined run, and no code/timeout/assertion change was applied for the retries. The complete hosted CI above supplies independent coverage. The original real Kuna/performance observations remain pinned to their stated feature revisions; no fresh native performance trial on this merge commit is claimed. All local heavy tasks used two CPU cores and a 3 GiB Node heap. The combined run took 155.28 s wall / 279.16 s reaped CPU / 3.30 GiB sampled family RSS; complete provider recheck took 19.34 s / 18.62 s / 0.85 GiB. No OOM/guard stop. Sampled RSS excludes assistant/shared services; cache/host conditions and controlled cold/warm costs are unmeasured. No new speedup or model-token claim. ## Author readiness Human owner: @N0zoM1z0 Declarations are entered by the agent at the human owner's explicit direction. The diff and evidence inspections described here were performed by the agent; this does not assert a separately observed human diff review. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:13:52 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:46:55 +00:00

[PR #6362] [MERGED] fix(scripts): keep the first proxy error in the package-update verifier #7308

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6362

No description provided.

Rows
Columns