morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3779] [MERGED] perf(javascript): share one definitions pass and index wrapper IPC dedupe #5891

New issue

Closed

opened 2026-10-31 01:10:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:10:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3779
Author: @kaoru0822-kitauji
Created: 10/15/2026
Status: ✅ Merged
Merged: 10/15/2026
Merged by: @kaoru0822-kitauji

Base: main ← Head: fix/3670-large-javascript-parse

📝 Commits (1)

45e40cd perf(javascript): share one definitions pass and index wrapper IPC dedupe

📊 Changes

8 files changed (+217 additions, -184 deletions)

View changed files

📝 src/application/javascript/JavaScriptArtifactAnalysis.ts (+1 -7)
📝 src/composition/javascriptAnalysisWorkerProcess.ts (+2 -12)
📝 src/domain/javascript/electronWrappedCallPaths.test.ts (+16 -0)
📝 src/domain/javascript/electronWrappedCallPaths.ts (+34 -13)
📝 src/domain/javascript/javascriptSemanticAnalysis.ts (+146 -119)
📝 src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts (+5 -5)
📝 src/domain/javascript/javascriptStaticAnalysis.ts (+8 -18)
📝 src/domain/webBundleAnalyzerInspection.ts (+5 -10)

📄 Description

Refs #3670

Problem

analyze_javascript_application runs out of the 1 GiB worker heap on single-file bundles of about 5 MB (#3670). On the issue's 5 MB generator, current main fails during the static phase. Profiling that phase turned up two problems, both in the static pass itself:

Quadratic Electron wrapper dedupe. Each IPC registration found through wrapper substitution was checked against every earlier accumulator.ipc finding, calling JSON.stringify on locations in each comparison. A bundle with tens of thousands of registrations spent about 5.7 s of 13 s CPU in that one check (electronWrappedCallPaths.js some callback plus ipc).
The definitions index was built three times per file. collectDefinitionsSteps (the full scope/binding/parent index, about 390 MB retained for the 5 MB file) ran separately for the open-receiver classification, the RPC client-root classification and binding lookup. Two of those copies were alive at the same time.

Change

electronWrappedCallPaths: index IPC findings by [side, operation, channel, location] once, built lazily from the facts the syntax pass already found, and add each new wrapper finding to the index as it is pushed. The duplicate check now looks only at same-endpoint candidates, and the predicate is unchanged.
javascriptSemanticAnalysis: one definitions-only pass now feeds a single traversal that classifies both open receivers and tRPC client roots (classifyDefinitionReceiversSteps). The same state is then extended with ambient writes for binding lookup (indexJavaScriptBindingsWithReceiversSteps). The receivers are classified before ambient-write indexing, so they read exactly the definitions-only state they read before. The receiver/RPC classification only reads that state and does not change it.
analyzeParsedJavaScriptStaticSourceSteps now owns receiver classification, so the separate openReceiverFacts parameter and the duplicate pre-passes in the worker and in-process callers are gone. classifyParsedJavaScriptOpenReceivers* and classifyParsedJavaScriptRpcClientRootsSteps are replaced by classifyParsedJavaScriptReceivers*. The web bundle inspector now uses that one entry point too, which also saves it a definitions pass.

Validation

Equivalence. Per-file JSON.stringify of the static facts from an origin/main build vs this branch, using real application code:

corpus	JS files	identical	static ms main → branch
Discord `app.asar`	2	2	2,284 → 1,711
Notion `app.asar`	97	97	7,219 → 5,376
Signal `app.asar`	573	573	12,032 → 7,696
Claude `app.asar`	424	424	32,316 → 21,681
Tabby `app.asar`	2,185	2,185	7,277 → 5,123
VS Code `resources/app/out`	28 parsed	28	10,115 → 6,740

That is 3,309/3,309 files byte-identical, with static analysis 25–36% faster. The two VS Code workbench bundles fail to parse on both builds; that is a separate parser issue I'll file.

End-to-end CLI (analyze-javascript-application --json, macOS arm64, Node 26.5, default limits, wall time from /usr/bin/time -l):

input	main	branch	output
Discord `app.asar`	14.5 s, 908 MB RSS	14.1 s, 896 MB RSS	108,108,885 B both; normalized JSON identical
Notion `app.asar` (`--integrity-policy record-and-continue`)	37.0 s	35.1 s	125,791,536 B both; equivalent once host-budget numbers and hash ordering are normalized
Signal `app.asar` (same policy)	68.1 s	62.6 s	262.2 MB; differs only in which files land past the host-dependent parent result budget (1,992,383,260 vs 1,991,331,380 B for these two runs)
issue generator, 4 MB	run 1: `provider_timeout` after 320.8 s; run 2: 25.6 s	16.5 s / 17.7 s	8,292,248 B both
issue generator, 5 MB	worker heap exhausted during static	static completes; the file then stops at the static checkpoint byte budget (10.27 MB of facts)	44,970 → 36,235 B (shorter limitation text)

Static phase only, 5 MB generator (--max-old-space-size=6000, so peaks are visible):

	main	branch
receivers + static	1,101 + 10,915 ms	3,948 ms
peak heap	1,520 MB	1,052 MB
facts	10,268,128 B	10,268,128 B (identical)

Output size for agents is unchanged: the facts are identical, so no extra tokens.

Tests. All 85 JavaScript domain/application/composition/web-bundle test files pass (2,062 tests), along with npm run check:fast. I added keeps one wrapper registration when one invocation reaches it twice. It covers a wrapper finding added during this pass suppressing a later duplicate from the same invocation, and fails if the index isn't updated after a push.

Remaining (not in this PR)

#3670 is not fully resolved. Bundles of 5 MB and up now get past the heap limit in static analysis but stop at the static checkpoint byte budget. Larger files still report parse_status: null and contribute 0 to parse_failures, so the summary still reads as zero.
The VS Code / VSCodium workbench bundles (~20 MB) fail because .js sources are parsed with Babel's TypeScript plugin, which rejects valid JS such as c?H=>({…}):H=>({…}). I'll send a separate PR for that.
The semantic phase still builds its own definitions index. It uses a different static-import resolution mode and adds member mutations, so sharing it needs its own equivalence work.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3779 **Author:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Created:** 10/15/2026 **Status:** ✅ Merged **Merged:** 10/15/2026 **Merged by:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Base:** `main` ← **Head:** `fix/3670-large-javascript-parse` --- ### 📝 Commits (1) - [`45e40cd`](https://github.com/morluto/rea/commit/45e40cd6b4238be4e3ca567235c224189cca17b6) perf(javascript): share one definitions pass and index wrapper IPC dedupe ### 📊 Changes **8 files changed** (+217 additions, -184 deletions) <details> <summary>View changed files</summary> 📝 `src/application/javascript/JavaScriptArtifactAnalysis.ts` (+1 -7) 📝 `src/composition/javascriptAnalysisWorkerProcess.ts` (+2 -12) 📝 `src/domain/javascript/electronWrappedCallPaths.test.ts` (+16 -0) 📝 `src/domain/javascript/electronWrappedCallPaths.ts` (+34 -13) 📝 `src/domain/javascript/javascriptSemanticAnalysis.ts` (+146 -119) 📝 `src/domain/javascript/javascriptSemanticAnalysisSteps.test.ts` (+5 -5) 📝 `src/domain/javascript/javascriptStaticAnalysis.ts` (+8 -18) 📝 `src/domain/webBundleAnalyzerInspection.ts` (+5 -10) </details> ### 📄 Description Refs #3670 ## Problem `analyze_javascript_application` runs out of the 1 GiB worker heap on single-file bundles of about 5 MB (#3670). On the issue's 5 MB generator, current `main` fails during the **static** phase. Profiling that phase turned up two problems, both in the static pass itself: 1. **Quadratic Electron wrapper dedupe.** Each IPC registration found through wrapper substitution was checked against *every* earlier `accumulator.ipc` finding, calling `JSON.stringify` on locations in each comparison. A bundle with tens of thousands of registrations spent about 5.7 s of 13 s CPU in that one check (`electronWrappedCallPaths.js` `some` callback plus `ipc`). 2. **The definitions index was built three times per file.** `collectDefinitionsSteps` (the full scope/binding/parent index, about 390 MB retained for the 5 MB file) ran separately for the open-receiver classification, the RPC client-root classification and binding lookup. Two of those copies were alive at the same time. ## Change - `electronWrappedCallPaths`: index IPC findings by `[side, operation, channel, location]` once, built lazily from the facts the syntax pass already found, and add each new wrapper finding to the index as it is pushed. The duplicate check now looks only at same-endpoint candidates, and the predicate is unchanged. - `javascriptSemanticAnalysis`: one definitions-only pass now feeds a **single** traversal that classifies both open receivers and tRPC client roots (`classifyDefinitionReceiversSteps`). The same state is then extended with ambient writes for binding lookup (`indexJavaScriptBindingsWithReceiversSteps`). The receivers are classified *before* ambient-write indexing, so they read exactly the definitions-only state they read before. The receiver/RPC classification only reads that state and does not change it. - `analyzeParsedJavaScriptStaticSourceSteps` now owns receiver classification, so the separate `openReceiverFacts` parameter and the duplicate pre-passes in the worker and in-process callers are gone. `classifyParsedJavaScriptOpenReceivers*` and `classifyParsedJavaScriptRpcClientRootsSteps` are replaced by `classifyParsedJavaScriptReceivers*`. The web bundle inspector now uses that one entry point too, which also saves it a definitions pass. ## Validation **Equivalence.** Per-file `JSON.stringify` of the static facts from an `origin/main` build vs this branch, using real application code: | corpus | JS files | identical | static ms main → branch | |---|---:|---:|---:| | Discord `app.asar` | 2 | 2 | 2,284 → 1,711 | | Notion `app.asar` | 97 | 97 | 7,219 → 5,376 | | Signal `app.asar` | 573 | 573 | 12,032 → 7,696 | | Claude `app.asar` | 424 | 424 | 32,316 → 21,681 | | Tabby `app.asar` | 2,185 | 2,185 | 7,277 → 5,123 | | VS Code `resources/app/out` | 28 parsed | 28 | 10,115 → 6,740 | That is 3,309/3,309 files byte-identical, with static analysis 25–36% faster. The two VS Code workbench bundles fail to parse on both builds; that is a separate parser issue I'll file. **End-to-end CLI** (`analyze-javascript-application --json`, macOS arm64, Node 26.5, default limits, wall time from `/usr/bin/time -l`): | input | main | branch | output | |---|---|---|---| | Discord `app.asar` | 14.5 s, 908 MB RSS | 14.1 s, 896 MB RSS | 108,108,885 B both; normalized JSON identical | | Notion `app.asar` (`--integrity-policy record-and-continue`) | 37.0 s | 35.1 s | 125,791,536 B both; equivalent once host-budget numbers and hash ordering are normalized | | Signal `app.asar` (same policy) | 68.1 s | 62.6 s | 262.2 MB; differs only in which files land past the host-dependent parent result budget (1,992,383,260 vs 1,991,331,380 B for these two runs) | | issue generator, 4 MB | run 1: **`provider_timeout` after 320.8 s**; run 2: 25.6 s | 16.5 s / 17.7 s | 8,292,248 B both | | issue generator, 5 MB | worker heap exhausted during static | static completes; the file then stops at the static checkpoint byte budget (10.27 MB of facts) | 44,970 → 36,235 B (shorter limitation text) | **Static phase only, 5 MB generator** (`--max-old-space-size=6000`, so peaks are visible): | | main | branch | |---|---|---| | receivers + static | 1,101 + 10,915 ms | 3,948 ms | | peak heap | 1,520 MB | 1,052 MB | | facts | 10,268,128 B | 10,268,128 B (identical) | Output size for agents is unchanged: the facts are identical, so no extra tokens. **Tests.** All 85 JavaScript domain/application/composition/web-bundle test files pass (2,062 tests), along with `npm run check:fast`. I added `keeps one wrapper registration when one invocation reaches it twice`. It covers a wrapper finding added during this pass suppressing a later duplicate from the same invocation, and fails if the index isn't updated after a push. ## Remaining (not in this PR) - #3670 is not fully resolved. Bundles of 5 MB and up now get past the heap limit in static analysis but stop at the static checkpoint byte budget. Larger files still report `parse_status: null` and contribute 0 to `parse_failures`, so the summary still reads as zero. - The VS Code / VSCodium workbench bundles (~20 MB) fail because `.js` sources are parsed with Babel's TypeScript plugin, which rejects valid JS such as `c?H=>({…}):H=>({…})`. I'll send a separate PR for that. - The semantic phase still builds its own definitions index. It uses a different static-import resolution mode and adds member mutations, so sharing it needs its own equivalence work. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:10:14 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:45:26 +00:00

[PR #5927] [CLOSED] fix(reference)!: honor nested .gitignore files #7105

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5891

No description provided.

Rows
Columns