[PR #1601] [MERGED] perf(javascript): reduce retained semantic analysis allocations #2996

New issue

Closed

opened 2026-10-17 21:12:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:12:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1601
Author: @N0zoM1z0
Created: 10/10/2026
Status: ✅ Merged
Merged: 10/10/2026
Merged by: @N0zoM1z0

Base: main ← Head: perf/javascript-semantic-ids-1563

📝 Commits (10+)

e2e098d perf(javascript): reduce retained semantic analysis allocations
bf07883 Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
16106be Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
e00c2bf Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
fc8bfda Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
dd48ba3 Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
fa85af8 perf(javascript): cache lexical scope ancestry on demand
c6ad27b Merge remote-tracking branch 'origin/perf/javascript-semantic-ids-1563' into perf/javascript-semantic-ids-1563
0399482 Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563
473e206 Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563

📊 Changes

7 files changed (+125 additions, -44 deletions)

View changed files

➕ src/domain/javascript/javascriptSemanticAnalysis.scope_depth.test.ts (+43 -0)
📝 src/domain/javascript/javascriptSemanticAnalysis.ts (+5 -1)
📝 src/domain/javascript/javascriptSemanticCalls.ts (+7 -1)
📝 src/domain/javascript/javascriptSemanticMemberMutations.ts (+30 -38)
📝 src/domain/javascript/javascriptSemanticObjects.ts (+7 -1)
📝 src/domain/javascript/javascriptSemanticReturns.ts (+8 -1)
📝 src/domain/javascript/javascriptSemanticState.ts (+25 -2)

📄 Description

Large JavaScript bundles retain concatenated semantic IDs, empty collections for every mutation observer, and an eager scope index for every AST node. Construct high-volume IDs with one join, materialize observer collections only when they acquire facts, and resolve lexical scopes through explicit boundaries and the existing AST parent index. Cache traversed ancestry on demand so repeated scope lookup remains linear in the controlled depth case.

Refs #1065 and #1563. This is an allocation reduction: no new source cutoff, skipped facts, result cap, schema, tool or authority change. The full-directory OOM remains; #1563 must stay open. The PR diff remains seven files and preserves the original implementation and independent cache follow-up history.

Validation:

Current head a2255f49 integrates main 16d9c5ad, including the existing #1618 cleanup/CI repair. Fresh runtime preparation, 85 affected browser/process-output/Qwen/EVM/firmware cases across seven files (2 host-only skips), and npm run check:fast pass. The preceding integration 30a33f92 passed 495 semantic cases across 20 files plus build/static checks; the seven changed files are identical to measured head 03994825.
Independent follow-up at fa85af8e: the depth regression fails before the ancestry cache and passes after it. At depths 80/160/320, AST-parent reads change from 7,161/27,081/105,321 to 332/652/1,292. Its verification reports 494 semantic tests across 21 files, all static/documentation/architecture/discovery checks, four unchanged complete-IR controls, and a real 674-byte public CLI/pinned-SDK MCP control preserving Evidence IDs, text/structured parity, source bytes and subsequent ping. These structural counts do not establish general throughput or billed-token improvements.
Real official Claude Code 2.1.296 member at 03994825: 4,762,637 bytes, SHA256 64516393b8f06dce5b74140781d43b1df4e743de3e283526a533508ca1c58147. Production domain analysis completes with 768 MiB old-space, retaining the complete 256,300,420-byte IR, SHA256 c109f190e1fb1719b73705096730bf4876338d191a9a07d88e3f6eaed019df04. Peak family RSS 1,006,104,576 bytes. Thirty additional complete-IR corpus digests matched the original source owners before the later integrations. Domain equality does not establish full public CLI success at that heap size.
Real pinned-SDK MCP at 03994825, parent 512/server 1024 MiB: complete 170-tool catalog; the completed member returns a typed oversized-transport constraint with retained Evidence. Summary, module page, exact node selection, one-node incoming trace (one match), ping, close/stale-reference and subsequent small analysis pass. Source SHA unchanged; Evidence ev_c11a23f39bc7f3f371eeea1c5754dbc2d3923b124b2ac83562c73112f27c87a9; peak family RSS 1,656,877,056 bytes. The application graph explicitly retains its existing ceiling/partial semantic coverage. These public proofs remain tied to the stated frozen revision.
Full original 2351-relevant-file directory at 30a33f92, including main dce81ca4 and all seven allocation/cache changes: still fatal heap exhaustion at chunk-kasbfbhj.js (1432/2351), exit 134 at 1024 MiB, zero CLI stdout, 177.39 seconds, peak family RSS 1,362,915,328 bytes. The external guard did not stop it. All 2589 extracted file digests were independently rechecked: zero mismatches. No causal speedup follows from single runs at different revisions.

Linux x64, Node 24.18.0/npm 11.16.0, CPUs 13/15, nice 10, one heavy lane, 2304 MiB process-family RSS guard and 2 GiB host reserve. Real Hopper/Ghidra and Windows/macOS verification were not performed for this JavaScript-only change. Final-head CI is pending; merge requires all checks on the exact final head to pass.

Separate follow-up: #1610's fixed source cutoff was reviewed through real full-directory CLI/MCP and a below-cutoff synthetic module and still permits host termination: https://github.com/morluto/rea/pull/1610#pullrequestreview-5480296727 . No cutoff or worker protocol is included here. My overlapping test-only #1617 was closed as superseded by #1618; it adds no files to this PR.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1601 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/10/2026 **Status:** ✅ Merged **Merged:** 10/10/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `perf/javascript-semantic-ids-1563` --- ### 📝 Commits (10+) - [`e2e098d`](https://github.com/morluto/rea/commit/e2e098d654d4813663e6b791c840e12131e035a0) perf(javascript): reduce retained semantic analysis allocations - [`bf07883`](https://github.com/morluto/rea/commit/bf07883ffd741830e7505e1fff1bd312a62d1bf7) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`16106be`](https://github.com/morluto/rea/commit/16106bec9e09132a6abb3a7ca1ae348afeb6deb7) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`e00c2bf`](https://github.com/morluto/rea/commit/e00c2bf123b6e006c753895fb084cd6f287a8b11) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`fc8bfda`](https://github.com/morluto/rea/commit/fc8bfda6035fbf26151fa4b81d595ec9ce4f5608) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`dd48ba3`](https://github.com/morluto/rea/commit/dd48ba3ef7214cf7a73f54367aeed1a9f1fd7490) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`fa85af8`](https://github.com/morluto/rea/commit/fa85af8e9fee288f26671f3b46e6e42e28037dd2) perf(javascript): cache lexical scope ancestry on demand - [`c6ad27b`](https://github.com/morluto/rea/commit/c6ad27b2641fd13fb7e3999ab0f619ec3b33f1af) Merge remote-tracking branch 'origin/perf/javascript-semantic-ids-1563' into perf/javascript-semantic-ids-1563 - [`0399482`](https://github.com/morluto/rea/commit/03994825c118ced5eef90d62d5757c49d76147bd) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 - [`473e206`](https://github.com/morluto/rea/commit/473e20696be726f35325c6e9602cbb98ac5b9762) Merge remote-tracking branch 'origin/main' into perf/javascript-semantic-ids-1563 ### 📊 Changes **7 files changed** (+125 additions, -44 deletions) <details> <summary>View changed files</summary> ➕ `src/domain/javascript/javascriptSemanticAnalysis.scope_depth.test.ts` (+43 -0) 📝 `src/domain/javascript/javascriptSemanticAnalysis.ts` (+5 -1) 📝 `src/domain/javascript/javascriptSemanticCalls.ts` (+7 -1) 📝 `src/domain/javascript/javascriptSemanticMemberMutations.ts` (+30 -38) 📝 `src/domain/javascript/javascriptSemanticObjects.ts` (+7 -1) 📝 `src/domain/javascript/javascriptSemanticReturns.ts` (+8 -1) 📝 `src/domain/javascript/javascriptSemanticState.ts` (+25 -2) </details> ### 📄 Description Large JavaScript bundles retain concatenated semantic IDs, empty collections for every mutation observer, and an eager scope index for every AST node. Construct high-volume IDs with one join, materialize observer collections only when they acquire facts, and resolve lexical scopes through explicit boundaries and the existing AST parent index. Cache traversed ancestry on demand so repeated scope lookup remains linear in the controlled depth case. Refs #1065 and #1563. This is an allocation reduction: no new source cutoff, skipped facts, result cap, schema, tool or authority change. **The full-directory OOM remains; #1563 must stay open.** The PR diff remains seven files and preserves the original implementation and independent cache follow-up history. Validation: - Current head `a2255f49` integrates main `16d9c5ad`, including the existing #1618 cleanup/CI repair. Fresh runtime preparation, 85 affected browser/process-output/Qwen/EVM/firmware cases across seven files (2 host-only skips), and `npm run check:fast` pass. The preceding integration `30a33f92` passed 495 semantic cases across 20 files plus build/static checks; the seven changed files are identical to measured head `03994825`. - Independent follow-up at `fa85af8e`: the depth regression fails before the ancestry cache and passes after it. At depths 80/160/320, AST-parent reads change from 7,161/27,081/105,321 to 332/652/1,292. Its verification reports 494 semantic tests across 21 files, all static/documentation/architecture/discovery checks, four unchanged complete-IR controls, and a real 674-byte public CLI/pinned-SDK MCP control preserving Evidence IDs, text/structured parity, source bytes and subsequent ping. These structural counts do not establish general throughput or billed-token improvements. - Real official Claude Code 2.1.296 member at `03994825`: 4,762,637 bytes, SHA256 `64516393b8f06dce5b74140781d43b1df4e743de3e283526a533508ca1c58147`. Production domain analysis completes with **768 MiB old-space**, retaining the complete **256,300,420-byte IR**, SHA256 `c109f190e1fb1719b73705096730bf4876338d191a9a07d88e3f6eaed019df04`. Peak family RSS 1,006,104,576 bytes. Thirty additional complete-IR corpus digests matched the original source owners before the later integrations. Domain equality does not establish full public CLI success at that heap size. - Real pinned-SDK MCP at `03994825`, parent 512/server 1024 MiB: complete 170-tool catalog; the completed member returns a typed oversized-transport constraint with retained Evidence. Summary, module page, exact node selection, one-node incoming trace (one match), ping, close/stale-reference and subsequent small analysis pass. Source SHA unchanged; Evidence `ev_c11a23f39bc7f3f371eeea1c5754dbc2d3923b124b2ac83562c73112f27c87a9`; peak family RSS 1,656,877,056 bytes. The application graph explicitly retains its existing ceiling/partial semantic coverage. These public proofs remain tied to the stated frozen revision. - **Full original 2351-relevant-file directory at `30a33f92`**, including main `dce81ca4` and all seven allocation/cache changes: still fatal heap exhaustion at `chunk-kasbfbhj.js` (1432/2351), **exit 134 at 1024 MiB**, zero CLI stdout, 177.39 seconds, peak family RSS 1,362,915,328 bytes. The external guard did not stop it. All 2589 extracted file digests were independently rechecked: zero mismatches. No causal speedup follows from single runs at different revisions. Linux x64, Node 24.18.0/npm 11.16.0, CPUs 13/15, nice 10, one heavy lane, 2304 MiB process-family RSS guard and 2 GiB host reserve. Real Hopper/Ghidra and Windows/macOS verification were not performed for this JavaScript-only change. Final-head CI is pending; merge requires all checks on the exact final head to pass. Separate follow-up: #1610's fixed source cutoff was reviewed through real full-directory CLI/MCP and a below-cutoff synthetic module and still permits host termination: https://github.com/morluto/rea/pull/1610#pullrequestreview-5480296727 . No cutoff or worker protocol is included here. My overlapping test-only #1617 was closed as superseded by #1618; it adds no files to this PR. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:12:43 +00:00