morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #4095] [MERGED] feat(kuna): add cached string inventory and search #6053

New issue

Closed

opened 2026-10-31 01:11:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:11:29 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4095
Author: @N0zoM1z0
Created: 10/15/2026
Status: ✅ Merged
Merged: 10/15/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/kuna-strings-20261015

📝 Commits (4)

0072855 feat(kuna): add cached string inventory and search
7c4735a docs(kuna): clarify static string scan wording
2e33e5b fix(kuna): validate string facts and defer regex allocations
4291a21 fix(kuna): preserve shared scan failure context and budgets

📊 Changes

23 files changed (+1632 additions, -212 deletions)

View changed files

📝 docs/kuna-provider.md (+14 -3)
➕ scripts/lib/kuna-string-verifier.mjs (+166 -0)
📝 scripts/verify-real-kuna.mjs (+33 -2)
📝 src/contracts/officialToolContracts.ts (+2 -2)
📝 src/kuna/KunaAddress.ts (+4 -0)
➕ src/kuna/KunaJson.ts (+31 -0)
📝 src/kuna/KunaLimits.ts (+4 -0)
📝 src/kuna/KunaProvider.ts (+8 -0)
📝 src/kuna/KunaProviderCapabilities.ts (+13 -0)
➕ src/kuna/KunaQueryErrors.ts (+94 -0)
➕ src/kuna/KunaQueryWork.ts (+138 -0)
📝 src/kuna/KunaReferenceFiles.ts (+8 -34)
📝 src/kuna/KunaReferenceQueries.ts (+20 -113)
📝 src/kuna/KunaSessionClient.ts (+90 -35)
➕ src/kuna/KunaStringInventory.ts (+140 -0)
➕ src/kuna/KunaStringQueries.ts (+195 -0)
📝 tests/boundary/providers/kuna/fakeKuna.ts (+21 -0)
📝 tests/boundary/providers/kuna/kunaProvider.test.ts (+1 -3)
➕ tests/boundary/providers/kuna/kunaQueryWork.test.ts (+69 -0)
📝 tests/boundary/providers/kuna/kunaRegexSearch.test.ts (+27 -10)

...and 3 more files

📄 Description

Summary

Kuna could decompile and follow calls, but REA rejected list_strings and search_strings. Add those existing operations so analysts can find a literal, keep its exact address and encoding facts, and request xrefs separately. Refs #1957.

Change and scope

One lazy kuna strings --encoding all --min-length 5 --termination any --no-xrefs scan serves complete, exact-address, literal and ECMAScript regex queries. Validate the selected input, scan policy, counts, identities and every row before caching, including agreement between text, encoding, character/byte lengths and terminators. Preserve numeric u64 tokens, source digest/size, inventory/selected counts and extraction limitations; no unrequested inventory text accompanies a search. Complete inventory Evidence remains available through list_strings and normal retention/export. Existing complete-result contracts are preserved; paging and function-referenced dossier strings are not added.

Extract shared native-query admission/cancellation from the reference owner and reuse it for strings. Procedure/string regex matching shares one session worker slot and drains on close; queued requests defer input arrays until admission, and procedure results are projected after selection. Failed native cleanup retains its admission leases. The 16 MiB scan cache expires with the target/executable-bound session; failures are not cached. CLI and MCP use the same provider implementation.

Contribution scope: static reads within the existing Kuna provider; no target execution, injection, mutation or protection recovery. Separate maintainer scope decision is not applicable. This launches the supplied Kuna and writes/removes private target copies. No dependencies, installation changes or new MCP tool names/input fields.

Validation performed

All 20 Kuna boundary files plus the effects audit: 194 tests passed, including cancellation, cleanup ownership, corrupt/oversized output, regex deadlines, u64 identities and verifier corruption oracles.
Pinned-toolchain typecheck (--checkers 1), lint/module boundaries, formatting, knip, metadata, architecture guards and docs:check: passed.
Real Kuna 1.833 on Linux: SQLite (7,985 rows), zlib (449), /usr/bin/printf (277), and an ELF with ASCII/UTF-8 (including supplementary characters)/UTF-16LE strings above 2^53. An empty-inventory ELF was also verified. Complete and selected MCP answers matched an independent native oracle. CLI/MCP SQLite searches matched exactly.
Updated verify-real-kuna.mjs --targets ... --functions _start: passed both source fixtures and both empty-inventory caller ELFs (original/stripped), required Unicode/wide fixture literals, existing decompilation/reference workflows and verified cleanup; 33.90 s wall, 48.13 s CPU, 697.2 MiB sampled peak process-family RSS.

Performance evidence

Same questions: literal lookup, repeat lookup, ECMAScript lookbehind, negative lookup and one exact address. Compare focused calls with a complete inventory read plus client-side selection, on the same implementation; the old provider did not support strings. Includes MCP startup/close. Final-code samples below are one completed task per workflow, limited to 2 CPUs / 3 GiB, with process-family RSS sampled every 50 ms.

Target/workflow	Wall	CPU	Peak family RSS	Requests + replies, estimated tokens
SQLite focused	4.76 s	7.01 s	671.1 MiB	9,320
SQLite complete	4.97 s	7.48 s	732.7 MiB	644,971
zlib focused	4.52 s	6.59 s	659.1 MiB	8,175
zlib complete	4.42 s	6.61 s	647.3 MiB	37,026

Earlier three-sample medians were 4.74 s focused / 5.24 s complete for SQLite and 4.58 s / 4.49 s for zlib. A later diagnostic interval was slower: SQLite focused 8.04–9.24 s and complete 11.21 s; the cause is unknown. A startup-only comparison measured old main at 4.40 s and this provider at 4.19 s. Treat these as observed samples, not a general speedup claim. Small inventories can be quicker to fetch completely.

Six real concurrent/repeated queries launched exactly one native string scan. Within-session selected reads were milliseconds; every fresh server rebuilds the scan, and OS caches were not flushed. Token estimates use o200k_base over captured requests/replies, including text/structured duplication. The final 207-tool output-schema-off catalog adds 206,638 estimated tokens: SQLite totals are 215,958 focused versus 851,609 complete; selected declarations instead yield 11,160 versus 646,371. Actual model billing, reasoning and selection costs were not measured. No macOS/Windows or other Kuna-release runtime claims.

CI initially caught the noun “runs” in the new descriptions as execution wording. Replaced it with “sequences”; the existing effects audit passed (6 tests), with no authority change or test exception.

All five actionable review findings are fixed at the producing boundaries: inconsistent native facts are rejected, queued regex allocations are deferred for both query families, only source-owned fixtures require positive string inventories, shared failures keep each waiter's operation and original typed producer facts, and both provider/oracle scans use the whole-program deadline. Regression tests and real empty caller verification passed; the corresponding threads are resolved. Latest main's shared export inventory, terminal export failures and deterministic output-bound handling were preserved during rebase; string output-bound failures are also retained until close.

Author readiness

Human owner: @N0zoM1z0

The owner explicitly authorized publication ready for review, continued fixes and merge, and waived further confirmation. Human-review declarations remain unasserted; the checks and review described here were performed by the agent.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4095 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/15/2026 **Status:** ✅ Merged **Merged:** 10/15/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/kuna-strings-20261015` --- ### 📝 Commits (4) - [`0072855`](https://github.com/morluto/rea/commit/00728553c5031e3a0599930a91459bda0bd884ac) feat(kuna): add cached string inventory and search - [`7c4735a`](https://github.com/morluto/rea/commit/7c4735a77b04b56a647d554618510cf67959512e) docs(kuna): clarify static string scan wording - [`2e33e5b`](https://github.com/morluto/rea/commit/2e33e5b2192effb700cce9320bc779378853838d) fix(kuna): validate string facts and defer regex allocations - [`4291a21`](https://github.com/morluto/rea/commit/4291a21a91fbd5f745c443dc6df6d40a9a8c1b00) fix(kuna): preserve shared scan failure context and budgets ### 📊 Changes **23 files changed** (+1632 additions, -212 deletions) <details> <summary>View changed files</summary> 📝 `docs/kuna-provider.md` (+14 -3) ➕ `scripts/lib/kuna-string-verifier.mjs` (+166 -0) 📝 `scripts/verify-real-kuna.mjs` (+33 -2) 📝 `src/contracts/officialToolContracts.ts` (+2 -2) 📝 `src/kuna/KunaAddress.ts` (+4 -0) ➕ `src/kuna/KunaJson.ts` (+31 -0) 📝 `src/kuna/KunaLimits.ts` (+4 -0) 📝 `src/kuna/KunaProvider.ts` (+8 -0) 📝 `src/kuna/KunaProviderCapabilities.ts` (+13 -0) ➕ `src/kuna/KunaQueryErrors.ts` (+94 -0) ➕ `src/kuna/KunaQueryWork.ts` (+138 -0) 📝 `src/kuna/KunaReferenceFiles.ts` (+8 -34) 📝 `src/kuna/KunaReferenceQueries.ts` (+20 -113) 📝 `src/kuna/KunaSessionClient.ts` (+90 -35) ➕ `src/kuna/KunaStringInventory.ts` (+140 -0) ➕ `src/kuna/KunaStringQueries.ts` (+195 -0) 📝 `tests/boundary/providers/kuna/fakeKuna.ts` (+21 -0) 📝 `tests/boundary/providers/kuna/kunaProvider.test.ts` (+1 -3) ➕ `tests/boundary/providers/kuna/kunaQueryWork.test.ts` (+69 -0) 📝 `tests/boundary/providers/kuna/kunaRegexSearch.test.ts` (+27 -10) _...and 3 more files_ </details> ### 📄 Description ## Summary Kuna could decompile and follow calls, but REA rejected `list_strings` and `search_strings`. Add those existing operations so analysts can find a literal, keep its exact address and encoding facts, and request xrefs separately. Refs #1957. ## Change and scope One lazy `kuna strings --encoding all --min-length 5 --termination any --no-xrefs` scan serves complete, exact-address, literal and ECMAScript regex queries. Validate the selected input, scan policy, counts, identities and every row before caching, including agreement between text, encoding, character/byte lengths and terminators. Preserve numeric u64 tokens, source digest/size, inventory/selected counts and extraction limitations; no unrequested inventory text accompanies a search. Complete inventory Evidence remains available through `list_strings` and normal retention/export. Existing complete-result contracts are preserved; paging and function-referenced dossier strings are not added. Extract shared native-query admission/cancellation from the reference owner and reuse it for strings. Procedure/string regex matching shares one session worker slot and drains on close; queued requests defer input arrays until admission, and procedure results are projected after selection. Failed native cleanup retains its admission leases. The 16 MiB scan cache expires with the target/executable-bound session; failures are not cached. CLI and MCP use the same provider implementation. Contribution scope: static reads within the existing Kuna provider; no target execution, injection, mutation or protection recovery. Separate maintainer scope decision is not applicable. This launches the supplied Kuna and writes/removes private target copies. No dependencies, installation changes or new MCP tool names/input fields. ## Validation performed - All 20 Kuna boundary files plus the effects audit: 194 tests passed, including cancellation, cleanup ownership, corrupt/oversized output, regex deadlines, u64 identities and verifier corruption oracles. - Pinned-toolchain typecheck (`--checkers 1`), lint/module boundaries, formatting, knip, metadata, architecture guards and `docs:check`: passed. - Real Kuna 1.833 on Linux: SQLite (7,985 rows), zlib (449), `/usr/bin/printf` (277), and an ELF with ASCII/UTF-8 (including supplementary characters)/UTF-16LE strings above `2^53`. An empty-inventory ELF was also verified. Complete and selected MCP answers matched an independent native oracle. CLI/MCP SQLite searches matched exactly. - Updated `verify-real-kuna.mjs --targets ... --functions _start`: passed both source fixtures and both empty-inventory caller ELFs (original/stripped), required Unicode/wide fixture literals, existing decompilation/reference workflows and verified cleanup; 33.90 s wall, 48.13 s CPU, 697.2 MiB sampled peak process-family RSS. ## Performance evidence Same questions: literal lookup, repeat lookup, ECMAScript lookbehind, negative lookup and one exact address. Compare focused calls with a complete inventory read plus client-side selection, on the same implementation; the old provider did not support strings. Includes MCP startup/close. Final-code samples below are one completed task per workflow, limited to 2 CPUs / 3 GiB, with process-family RSS sampled every 50 ms. | Target/workflow | Wall | CPU | Peak family RSS | Requests + replies, estimated tokens | | --- | ---: | ---: | ---: | ---: | | SQLite focused | 4.76 s | 7.01 s | 671.1 MiB | 9,320 | | SQLite complete | 4.97 s | 7.48 s | 732.7 MiB | 644,971 | | zlib focused | 4.52 s | 6.59 s | 659.1 MiB | 8,175 | | zlib complete | 4.42 s | 6.61 s | 647.3 MiB | 37,026 | Earlier three-sample medians were 4.74 s focused / 5.24 s complete for SQLite and 4.58 s / 4.49 s for zlib. A later diagnostic interval was slower: SQLite focused 8.04–9.24 s and complete 11.21 s; the cause is unknown. A startup-only comparison measured old main at 4.40 s and this provider at 4.19 s. Treat these as observed samples, not a general speedup claim. Small inventories can be quicker to fetch completely. Six real concurrent/repeated queries launched exactly one native string scan. Within-session selected reads were milliseconds; every fresh server rebuilds the scan, and OS caches were not flushed. Token estimates use `o200k_base` over captured requests/replies, including text/structured duplication. The final 207-tool output-schema-off catalog adds 206,638 estimated tokens: SQLite totals are 215,958 focused versus 851,609 complete; selected declarations instead yield 11,160 versus 646,371. Actual model billing, reasoning and selection costs were not measured. No macOS/Windows or other Kuna-release runtime claims. CI initially caught the noun “runs” in the new descriptions as execution wording. Replaced it with “sequences”; the existing effects audit passed (6 tests), with no authority change or test exception. All five actionable review findings are fixed at the producing boundaries: inconsistent native facts are rejected, queued regex allocations are deferred for both query families, only source-owned fixtures require positive string inventories, shared failures keep each waiter's operation and original typed producer facts, and both provider/oracle scans use the whole-program deadline. Regression tests and real empty caller verification passed; the corresponding threads are resolved. Latest main's shared export inventory, terminal export failures and deterministic output-bound handling were preserved during rebase; string output-bound failures are also retained until close. ## Author readiness Human owner: @N0zoM1z0 The owner explicitly authorized publication ready for review, continued fixes and merge, and waived further confirmation. Human-review declarations remain unasserted; the checks and review described here were performed by the agent. - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:11:29 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:47:12 +00:00

[PR #6414] [MERGED] fix(evm): report the carrier input allowance as a file-size constraint #7343

bsakaguchi changed title from ~~[PR #4095] [MERGED] feat(kuna): add cached string inventory and search~~ to [PR #4095] feat(kuna): add cached string inventory and search

2026-11-03 16:37:44 +00:00

bsakaguchi changed title from ~~[PR #4095] feat(kuna): add cached string inventory and search~~ to [PR #4095] [MERGED] feat(kuna): add cached string inventory and search

2026-11-03 17:39:14 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6053

No description provided.

Rows
Columns