morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #4329] [MERGED] fix(kuna): stream the call graph instead of capping it at 32 MiB #6175

New issue

Closed

opened 2026-10-31 01:12:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:12:26 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4329
Author: @kaoru0822-kitauji
Created: 10/16/2026
Status: ✅ Merged
Merged: 10/16/2026
Merged by: @kaoru0822-kitauji

Base: fix/kuna-stream-string-scan ← Head: fix/kuna-stream-call-graph

📝 Commits (3)

b0dfabf fix(kuna): stream the call graph instead of capping it at 32 MiB
4ef937c fix(kuna): index call-graph edges as they stream
0537c49 docs(kuna): state the graph's retained size after incremental indexing

📊 Changes

11 files changed (+429 additions, -184 deletions)

View changed files

📝 docs/kuna-provider.md (+1 -1)
📝 src/kuna/KunaJsonStream.ts (+36 -7)
📝 src/kuna/KunaQueryWork.ts (+43 -1)
📝 src/kuna/KunaReferenceFiles.ts (+151 -74)
📝 src/kuna/KunaReferenceQueries.ts (+20 -29)
📝 src/kuna/KunaStringInventory.ts (+5 -10)
📝 src/kuna/KunaStringQueries.ts (+26 -43)
📝 tests/boundary/providers/kuna/fakeKuna.ts (+6 -4)
📝 tests/boundary/providers/kuna/kunaCommandFailures.test.ts (+32 -3)
📝 tests/boundary/providers/kuna/kunaReferenceFiles.test.ts (+105 -11)
📝 tests/boundary/providers/kuna/kunaReferences.test.ts (+4 -1)

📄 Description

Stacked on #4318 (base fix/kuna-stream-string-scan). It reuses #4318's stdout sink and streaming document reader. Review #4318 first; this PR's diff is only the call-graph change.

Summary

Kuna call relationships failed on large apps. REA buffered kuna decompile-graph and refused output above 32 MiB, so procedure_callers, procedure_callees, analyze_function call facets, get_call_graph and trace_call_path all failed. The failure came only after Kuna's whole-program walk had finished. REA now parses the graph as Kuna writes it. Refs #1957.

Problem and expected behavior

Trigger: MCP procedure_callees { procedure: "main" } on Raycast's thin arm64 binary (53 MB). On origin/main with real Kuna 1.833 on macOS arm64, after 62 s:

{"code":"resource_constraint","message":"kuna decompile-graph wrote more than 33554432 bytes of output.","retryable":false}

The graph is 157 MB: 180,203 functions and 772,835 edges. The docs listed the 32 MiB cap as a limitation, and noted that a 14 MB Sublime Text binary already produces 31.8 MB. Like the string scan fixed in #4318, the cap followed no format or resource constraint. Expected: the call relationships, with memory bounded by what REA retains.

Change and scope

KunaCallGraphReader (src/kuna/KunaReferenceFiles.ts) replaces parseKunaCallGraph. It runs on #4318's KunaJsonDocumentReader.
- It keeps only the node and edge facts queries use.
- It skips fields it does not interpret, such as parameters, signature and size.
- It indexes each edge as it arrives, with no retained edge array, so the whole working set grows under the per-row heap admission.
- Endpoints and callee-order completeness are checked once every node is known; the validation-only maps are then released.
- The validation and index rules are unchanged.
The document reader gains must-be-null row fields, so an unrequested codeC/assembly/error body is refused before any of its text is assembled. Address fields may now be null, as forwardsTo is.
KunaQueryWork.scan now owns the run-and-validate sequence both producers repeated: diagnostics budget, early rejection, and retention of a deterministic overflow. This removes a 22-line clone between KunaStringQueries and KunaReferenceQueries.
Behavior change: a malformed graph is now AnalysisOutputError (unreadable_output), like the string scan. It was a generic ProviderAdapterError (execution_failure). Both are non-retryable for the same output.
Not changed: xrefs keep their 16 MiB buffer (a per-address query). Graph caching and session semantics are unchanged.

Contribution scope: a fix inside the existing Kuna provider. No new authority, target execution, injection, mutation or protection recovery. A maintainer scope decision is not applicable. There are no new dependencies and no new tool names or input fields.

Contract and boundary impact

Semantic owner and earliest changed stage: KunaCallGraphReader owns graph interpretation; KunaQueryWork.scan owns the streaming producer sequence.
Contribution scope fit and linked maintainer decision: fix within existing boundaries; not applicable.
CLI and MCP/tool-catalog contract: unchanged. Large targets now answer. A malformed graph is projected as unreadable_output instead of execution_failure.
Provider, bridge, target-format, or platform compatibility: same Kuna command and arguments. Unknown graph fields are still tolerated.
Evidence, artifact, provenance, or reconstruction contract: raw_result.references keeps its digest, size, counts and coverage.
Target selection, authority, upstream behavior, and cleanup details: a rejected graph stops Kuna early. Ownership and cleanup are unchanged.
Generated metadata, package, or installation impact: none.

Expected effects:

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated:
- kunaCommandFailures.test.ts: a graph padded past 40 MiB answers procedure_callees. It fails on base with the 32 MiB resource_constraint.
- kunaCommandFailures.test.ts: the retention case now floods diagnostics past 8 MiB and still reports once per session.
- kunaReferenceFiles.test.ts: 300 nodes and 3,000 edges index fully. When headroom runs out after the node rows, the reader is refused while edges stream, with retained_rows between the node count and the row total.
- kunaReferenceFiles.test.ts drives the reader in 7-byte chunks. Every existing contradiction case still rejects, now asserting AnalysisOutputError. Added cases: unrequested bodies refused as functions row codeC must be null, and ignored nested parameters and extra fields.
- kunaReferences.test.ts: unusable graphs are still not cached, now with the new error type.
Base reproduction: Raycast, shown above.
User-visible MCP output after the change, for procedure_callees main on Raycast: ok, 4 callees, with raw_result.references reporting bytes: 156648585, functions: 180203, edges: 772835.
Remaining proof gaps:
- No Linux host run.
- The headroom refusal is covered with a mocked heap (in #4318).
- Firefox XUL's graph was not run: Kuna's own walk of that image is expected to take very long.

Validation performed

npm run test:focused -- tests/boundary/providers/kuna/*.test.ts tests/boundary/process/ownedCommand.test.ts: 22 files, 235 tests passed.
npm run check:fast (typecheck and lint), npm run knip, npm run docs:check: passed.
npm run jscpd: no new Kuna source clones. The producer clone is removed; the remaining Kuna clones are already on main.
REA_KUNA_COMMAND=<kuna 1.833 macos-arm64> npm run verify:kuna on macOS 26.7.1 arm64: passed. Callers, callees and xrefs match the native oracle, with cleanup: "complete".

Performance

MCP procedure_callees main, then a warm procedure_callers in the same session. One cold server per row, real Kuna 1.833, peak RSS for the whole process family (REA plus Kuna) sampled every 50 ms:

Target	Base (`origin/main`)	This PR
zip (208 KB)	ok, first query 1.25 s, warm 5 ms; 2.02 s wall, 590 MiB	ok, first query 0.90–1.25 s, warm 2–4 ms; 1.58 s wall, 592 MiB
Raycast (53 MB)	`resource_constraint` after 62.0 s; 62.7 s wall, 73.3 s CPU, 2276 MiB	ok after 66.3 s, warm 2 ms; 67.1 s wall, 78.3 s CPU, 2210 MiB

Kuna's walk dominates the Raycast run. Run alone, kuna decompile-graph takes 38 s at 3.5 GiB.
Graph reader alone, on the saved 157 MB Raycast JSON: 1.65 s streaming plus 0.07 s finishing, 346 MiB RSS, 93 MB heap retained after GC.
On Ghostty's 15 MB graph, which base can buffer: base's parseKunaCallGraph takes 214 ms at 258 MiB RSS and retains about 20 MB; this reader takes 187 ms at 143 MiB and retains 18 MB.
Agent output is unchanged in shape: about 5 KB per call answer for both targets.
Not measured: token billing, Linux, and Kuna releases other than 1.833.

Compatibility, safety, and release

Breaking changes or migration steps: none to inputs. A malformed graph is classified unreadable_output instead of execution_failure; both are non-retryable.
Real Hopper/Ghidra, browser, or OS coverage: real Kuna 1.833 on macOS arm64. No Linux run.
Package or release metadata impact: none.
Security, privacy, process, or containment review:
- Kuna is still launched only through runOwnedCommand.
- Unrequested bodies are refused before assembly.
- Unknown fields are skipped, not assigned.

Author readiness

Human owner: @Kaoru0822

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4329 **Author:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Created:** 10/16/2026 **Status:** ✅ Merged **Merged:** 10/16/2026 **Merged by:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Base:** `fix/kuna-stream-string-scan` ← **Head:** `fix/kuna-stream-call-graph` --- ### 📝 Commits (3) - [`b0dfabf`](https://github.com/morluto/rea/commit/b0dfabf638a4c63eb597d9e6d207160ef7b8204e) fix(kuna): stream the call graph instead of capping it at 32 MiB - [`4ef937c`](https://github.com/morluto/rea/commit/4ef937c18e29a16404ee5d5ff673b8a9b94e94c2) fix(kuna): index call-graph edges as they stream - [`0537c49`](https://github.com/morluto/rea/commit/0537c49cef6316c7350df0a412fa50ec9e372d4b) docs(kuna): state the graph's retained size after incremental indexing ### 📊 Changes **11 files changed** (+429 additions, -184 deletions) <details> <summary>View changed files</summary> 📝 `docs/kuna-provider.md` (+1 -1) 📝 `src/kuna/KunaJsonStream.ts` (+36 -7) 📝 `src/kuna/KunaQueryWork.ts` (+43 -1) 📝 `src/kuna/KunaReferenceFiles.ts` (+151 -74) 📝 `src/kuna/KunaReferenceQueries.ts` (+20 -29) 📝 `src/kuna/KunaStringInventory.ts` (+5 -10) 📝 `src/kuna/KunaStringQueries.ts` (+26 -43) 📝 `tests/boundary/providers/kuna/fakeKuna.ts` (+6 -4) 📝 `tests/boundary/providers/kuna/kunaCommandFailures.test.ts` (+32 -3) 📝 `tests/boundary/providers/kuna/kunaReferenceFiles.test.ts` (+105 -11) 📝 `tests/boundary/providers/kuna/kunaReferences.test.ts` (+4 -1) </details> ### 📄 Description > Stacked on #4318 (base `fix/kuna-stream-string-scan`). It reuses #4318's stdout sink and streaming document reader. Review #4318 first; this PR's diff is only the call-graph change. ## Summary Kuna call relationships failed on large apps. REA buffered `kuna decompile-graph` and refused output above 32 MiB, so `procedure_callers`, `procedure_callees`, `analyze_function` call facets, `get_call_graph` and `trace_call_path` all failed. The failure came only after Kuna's whole-program walk had finished. REA now parses the graph as Kuna writes it. Refs #1957. ## Problem and expected behavior Trigger: MCP `procedure_callees { procedure: "main" }` on Raycast's thin arm64 binary (53 MB). On `origin/main` with real Kuna 1.833 on macOS arm64, after 62 s: ```json {"code":"resource_constraint","message":"kuna decompile-graph wrote more than 33554432 bytes of output.","retryable":false} ``` The graph is 157 MB: 180,203 functions and 772,835 edges. The docs listed the 32 MiB cap as a limitation, and noted that a 14 MB Sublime Text binary already produces 31.8 MB. Like the string scan fixed in #4318, the cap followed no format or resource constraint. Expected: the call relationships, with memory bounded by what REA retains. ## Change and scope - `KunaCallGraphReader` (`src/kuna/KunaReferenceFiles.ts`) replaces `parseKunaCallGraph`. It runs on #4318's `KunaJsonDocumentReader`. - It keeps only the node and edge facts queries use. - It skips fields it does not interpret, such as `parameters`, `signature` and `size`. - It indexes each edge as it arrives, with no retained edge array, so the whole working set grows under the per-row heap admission. - Endpoints and callee-order completeness are checked once every node is known; the validation-only maps are then released. - The validation and index rules are unchanged. - The document reader gains must-be-null row fields, so an unrequested `codeC`/`assembly`/`error` body is refused before any of its text is assembled. Address fields may now be null, as `forwardsTo` is. - `KunaQueryWork.scan` now owns the run-and-validate sequence both producers repeated: diagnostics budget, early rejection, and retention of a deterministic overflow. This removes a 22-line clone between `KunaStringQueries` and `KunaReferenceQueries`. - Behavior change: a malformed graph is now `AnalysisOutputError` (`unreadable_output`), like the string scan. It was a generic `ProviderAdapterError` (`execution_failure`). Both are non-retryable for the same output. - Not changed: xrefs keep their 16 MiB buffer (a per-address query). Graph caching and session semantics are unchanged. Contribution scope: a fix inside the existing Kuna provider. No new authority, target execution, injection, mutation or protection recovery. A maintainer scope decision is not applicable. There are no new dependencies and no new tool names or input fields. ## Contract and boundary impact - Semantic owner and earliest changed stage: `KunaCallGraphReader` owns graph interpretation; `KunaQueryWork.scan` owns the streaming producer sequence. - Contribution scope fit and linked maintainer decision: fix within existing boundaries; not applicable. - CLI and MCP/tool-catalog contract: unchanged. Large targets now answer. A malformed graph is projected as `unreadable_output` instead of `execution_failure`. - Provider, bridge, target-format, or platform compatibility: same Kuna command and arguments. Unknown graph fields are still tolerated. - Evidence, artifact, provenance, or reconstruction contract: `raw_result.references` keeps its digest, size, counts and coverage. - Target selection, authority, upstream behavior, and cleanup details: a rejected graph stops Kuna early. Ownership and cleanup are unchanged. - Generated metadata, package, or installation impact: none. Expected effects: - [x] Read existing files or captured data. - [x] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `kunaCommandFailures.test.ts`: a graph padded past 40 MiB answers `procedure_callees`. It fails on base with the 32 MiB `resource_constraint`. - `kunaCommandFailures.test.ts`: the retention case now floods diagnostics past 8 MiB and still reports once per session. - `kunaReferenceFiles.test.ts`: 300 nodes and 3,000 edges index fully. When headroom runs out after the node rows, the reader is refused while edges stream, with `retained_rows` between the node count and the row total. - `kunaReferenceFiles.test.ts` drives the reader in 7-byte chunks. Every existing contradiction case still rejects, now asserting `AnalysisOutputError`. Added cases: unrequested bodies refused as `functions row codeC must be null`, and ignored nested `parameters` and extra fields. - `kunaReferences.test.ts`: unusable graphs are still not cached, now with the new error type. - Base reproduction: Raycast, shown above. - User-visible MCP output after the change, for `procedure_callees main` on Raycast: `ok`, 4 callees, with `raw_result.references` reporting `bytes: 156648585, functions: 180203, edges: 772835`. - Remaining proof gaps: - No Linux host run. - The headroom refusal is covered with a mocked heap (in #4318). - Firefox `XUL`'s graph was not run: Kuna's own walk of that image is expected to take very long. ## Validation performed - `npm run test:focused -- tests/boundary/providers/kuna/*.test.ts tests/boundary/process/ownedCommand.test.ts`: 22 files, 235 tests passed. - `npm run check:fast` (typecheck and lint), `npm run knip`, `npm run docs:check`: passed. - `npm run jscpd`: no new Kuna source clones. The producer clone is removed; the remaining Kuna clones are already on main. - `REA_KUNA_COMMAND=<kuna 1.833 macos-arm64> npm run verify:kuna` on macOS 26.7.1 arm64: passed. Callers, callees and xrefs match the native oracle, with `cleanup: "complete"`. ### Performance MCP `procedure_callees main`, then a warm `procedure_callers` in the same session. One cold server per row, real Kuna 1.833, peak RSS for the whole process family (REA plus Kuna) sampled every 50 ms: | Target | Base (`origin/main`) | This PR | | --- | --- | --- | | zip (208 KB) | ok, first query 1.25 s, warm 5 ms; 2.02 s wall, 590 MiB | ok, first query 0.90–1.25 s, warm 2–4 ms; 1.58 s wall, 592 MiB | | Raycast (53 MB) | `resource_constraint` after 62.0 s; 62.7 s wall, 73.3 s CPU, 2276 MiB | ok after 66.3 s, warm 2 ms; 67.1 s wall, 78.3 s CPU, 2210 MiB | - Kuna's walk dominates the Raycast run. Run alone, `kuna decompile-graph` takes 38 s at 3.5 GiB. - Graph reader alone, on the saved 157 MB Raycast JSON: 1.65 s streaming plus 0.07 s finishing, 346 MiB RSS, 93 MB heap retained after GC. - On Ghostty's 15 MB graph, which base can buffer: base's `parseKunaCallGraph` takes 214 ms at 258 MiB RSS and retains about 20 MB; this reader takes 187 ms at 143 MiB and retains 18 MB. - Agent output is unchanged in shape: about 5 KB per call answer for both targets. - Not measured: token billing, Linux, and Kuna releases other than 1.833. ## Compatibility, safety, and release - Breaking changes or migration steps: none to inputs. A malformed graph is classified `unreadable_output` instead of `execution_failure`; both are non-retryable. - Real Hopper/Ghidra, browser, or OS coverage: real Kuna 1.833 on macOS arm64. No Linux run. - Package or release metadata impact: none. - Security, privacy, process, or containment review: - Kuna is still launched only through `runOwnedCommand`. - Unrequested bodies are refused before assembly. - Unknown fields are skipped, not assigned. ## Author readiness Human owner: @Kaoru0822 - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:12:26 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:45:58 +00:00

[PR #6175] [MERGED] fix(model): keep host I/O failures distinct from unavailable model targets #7184

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6175

No description provided.

Rows
Columns