morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #1055] [MERGED] fix(javascript): complete large-ASAR analysis with file-local projection #2611

New issue

Closed

opened 2026-10-17 21:08:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:08:13 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1055
Author: @N0zoM1z0
Created: 10/8/2026
Status: ✅ Merged
Merged: 10/8/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/javascript-obsidian-asar-746

📝 Commits (1)

74a55db fix(javascript): complete large-ASAR analysis with file-local projection

📊 Changes

27 files changed (+1231 additions, -631 deletions)

View changed files

📝 docs/javascript-artifact-reconstruction.md (+15 -0)
📝 src/application/javascript/ElectronBoundaryAnalysis.ts (+3 -3)
📝 src/application/javascript/JavaScriptApplicationEvidence.ts (+1 -2)
📝 src/application/javascript/JavaScriptArtifactAnalysis.ts (+92 -16)
📝 src/application/javascript/JavaScriptArtifactAnalysisTypes.ts (+23 -4)
📝 src/application/javascript/JavaScriptArtifactGraphBuilder.ts (+2 -2)
📝 src/application/javascript/JavaScriptArtifactGraphContext.ts (+3 -3)
📝 src/application/javascript/JavaScriptArtifactGraphStructure.ts (+2 -2)
📝 src/application/javascript/JavaScriptArtifactReconstruction.ts (+23 -9)
📝 src/application/javascript/JavaScriptModuleRelationships.ts (+5 -4)
📝 src/application/javascript/JavaScriptReturnShapeProjection.ts (+2 -2)
📝 src/application/javascript/JavaScriptSemanticGraphAsyncProjection.ts (+42 -64)
📝 src/application/javascript/JavaScriptSemanticGraphBuilder.ts (+198 -208)
📝 src/application/javascript/JavaScriptSemanticGraphChildProcessProjection.ts (+53 -83)
📝 src/application/javascript/JavaScriptSemanticGraphConstruction.ts (+50 -0)
📝 src/application/javascript/JavaScriptSemanticGraphDataProjection.ts (+48 -70)
📝 src/application/javascript/JavaScriptSemanticGraphFlowProjection.ts (+22 -33)
📝 src/application/javascript/JavaScriptSemanticGraphObjectProjection.ts (+19 -27)
➕ src/application/javascript/JavaScriptSemanticGraphProjection.test.ts (+259 -0)
📝 src/application/javascript/JavaScriptSemanticGraphProjection.ts (+157 -24)

...and 7 more files

📄 Description

Large Electron ASAR analysis retained every file's semantic IR, repeatedly scanned source ranges, and allocated rejected graph nodes and duplicate validated payloads. Real shipped archives exhausted the bounded heap or failed to finish before any result was emitted. Project semantics file by file, retain only the module/export facts needed later, and index callable ownership, contained references and exact call locations.

The patch also checks existing node budgets before constructing a node, validates graph content without a second full graph clone, selects the known Evidence envelope before parsing, and avoids a redundant Evidence parse/hash. Evidence remains an independent snapshot and public parsers still validate arbitrary records and commitments. Small CLI string leaves use one encoding step while oversized leaves retain bounded streaming. Per-file progress yields to the event loop and honors cancellation between files.

Public schemas, return contracts, canonical IDs, existing budgets and truthful coverage remain unchanged. Complete large MCP delivery still fails at serialization; that separate return/transport work is tracked in #1053 and #1050.

Fixes #746.

Real artifact measurements

Linux: Node 24.18.0, two CPUs, below-normal priority, 1.5 GiB V8 heap, 2 GiB process-family RSS guard. Before is pristine 8d5c88f1, with identical locked dependencies. Failed baselines are not completed-run speedup measurements.

Target	Before	After	After peak family RSS
Official Obsidian 1.12.7, 24,060,763-byte ASAR	External time stop at 240.2 s; no result	Complete JSONL at 162.3 s	1.58 GiB
Shipped Notion 7.6.1, 8,903,750-byte ASAR	Native heap exhaustion at 182.7 s; no result	Complete JSONL at 97.8 s	1.37 GiB

Complete outputs were independently streamed through a JSON parser: approximately 409 MiB and 321 MiB respectively. No filtered view was used. Obsidian retains 100,000 semantic nodes, 77,137 relations, 144,053 unknowns and 23,440 fingerprints. Existing coverage remains partial. Its final complete Evidence bytes match the earlier successful candidate byte for byte.

The actual Obsidian app.js contains 21,050 callables and 197,401 reference ranges. On 2,048 evenly spaced actual reference ranges, three runs each of the previous scan/sort algorithm and the production index produced identical result digests. Median lookup time fell from 1,129.2 ms to 10.1 ms; index setup took 12.2 ms. This is a lookup measurement, not an end-to-end speedup claim.

Windows packaged verification reused an existing Node 22.19.0 installation, two CPUs and the same heap/Node-family guard. Full Obsidian passed in 185.9 s (1.39 GiB peak); full Notion passed in 128.1 s (1.11 GiB peak). Both complete outputs have the same statistics, counts, application graph IDs and semantic graph IDs as Linux. Cold first progress took 17.1 s and 14.0 s respectively. D-drive free space stayed above 32 GiB, owned processes exited, and the isolated package/cache/artifact/output directory was removed.

Artifact SHA-256:

Obsidian: 2b2483b2e1246772e0d25367ec055cbc5047ea2f0091b667c35656678f86d712, from the official release. Version and size match #746; the reporter supplied no digest, so byte identity with that host's file is unverified.
Notion: 97c41bc14fc0ea8c91cd34f502660963dfdf294d5eb35165ccac9d30963069d4.

Verification

25 focused files / 340 tests passed across JavaScript services, graph validation/query/traversal, Evidence, EvidenceBundle, reconstruction, compiled CLI, MCP and retained-Evidence workflows.
Pristine-main versus candidate complete Evidence compares equal for a source-owned 13-file Electron/Webpack/Rspack directory and packed ASAR, including imports, re-exports and nested exported return shapes.
npm run check:fast -- --concurrency=1, npm run format:check, npm run knip and npm run docs:check passed. Normal pre-commit and pre-push hooks passed.
npm run verify:package passed in 280.3 s at 1.36 GiB peak family RSS.
Full shipped-artifact CLI runs used rea analyze-javascript-application <selected.asar> --format jsonl on Linux and Windows. Inputs were parsed as inert text; target code was never executed.
Real Hopper/Ghidra verification was not performed; this patch changes the static JavaScript artifact workflow. The original macOS 4.0.1 terminal error was not reproduced on that host/runtime.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1055 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/8/2026 **Status:** ✅ Merged **Merged:** 10/8/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/javascript-obsidian-asar-746` --- ### 📝 Commits (1) - [`74a55db`](https://github.com/morluto/rea/commit/74a55dbda4552dbbe48027385ac3b0031879ae08) fix(javascript): complete large-ASAR analysis with file-local projection ### 📊 Changes **27 files changed** (+1231 additions, -631 deletions) <details> <summary>View changed files</summary> 📝 `docs/javascript-artifact-reconstruction.md` (+15 -0) 📝 `src/application/javascript/ElectronBoundaryAnalysis.ts` (+3 -3) 📝 `src/application/javascript/JavaScriptApplicationEvidence.ts` (+1 -2) 📝 `src/application/javascript/JavaScriptArtifactAnalysis.ts` (+92 -16) 📝 `src/application/javascript/JavaScriptArtifactAnalysisTypes.ts` (+23 -4) 📝 `src/application/javascript/JavaScriptArtifactGraphBuilder.ts` (+2 -2) 📝 `src/application/javascript/JavaScriptArtifactGraphContext.ts` (+3 -3) 📝 `src/application/javascript/JavaScriptArtifactGraphStructure.ts` (+2 -2) 📝 `src/application/javascript/JavaScriptArtifactReconstruction.ts` (+23 -9) 📝 `src/application/javascript/JavaScriptModuleRelationships.ts` (+5 -4) 📝 `src/application/javascript/JavaScriptReturnShapeProjection.ts` (+2 -2) 📝 `src/application/javascript/JavaScriptSemanticGraphAsyncProjection.ts` (+42 -64) 📝 `src/application/javascript/JavaScriptSemanticGraphBuilder.ts` (+198 -208) 📝 `src/application/javascript/JavaScriptSemanticGraphChildProcessProjection.ts` (+53 -83) 📝 `src/application/javascript/JavaScriptSemanticGraphConstruction.ts` (+50 -0) 📝 `src/application/javascript/JavaScriptSemanticGraphDataProjection.ts` (+48 -70) 📝 `src/application/javascript/JavaScriptSemanticGraphFlowProjection.ts` (+22 -33) 📝 `src/application/javascript/JavaScriptSemanticGraphObjectProjection.ts` (+19 -27) ➕ `src/application/javascript/JavaScriptSemanticGraphProjection.test.ts` (+259 -0) 📝 `src/application/javascript/JavaScriptSemanticGraphProjection.ts` (+157 -24) _...and 7 more files_ </details> ### 📄 Description Large Electron ASAR analysis retained every file's semantic IR, repeatedly scanned source ranges, and allocated rejected graph nodes and duplicate validated payloads. Real shipped archives exhausted the bounded heap or failed to finish before any result was emitted. Project semantics file by file, retain only the module/export facts needed later, and index callable ownership, contained references and exact call locations. The patch also checks existing node budgets before constructing a node, validates graph content without a second full graph clone, selects the known Evidence envelope before parsing, and avoids a redundant Evidence parse/hash. Evidence remains an independent snapshot and public parsers still validate arbitrary records and commitments. Small CLI string leaves use one encoding step while oversized leaves retain bounded streaming. Per-file progress yields to the event loop and honors cancellation between files. Public schemas, return contracts, canonical IDs, existing budgets and truthful coverage remain unchanged. Complete large MCP delivery still fails at serialization; that separate return/transport work is tracked in #1053 and #1050. Fixes #746. ### Real artifact measurements Linux: Node 24.18.0, two CPUs, below-normal priority, 1.5 GiB V8 heap, 2 GiB process-family RSS guard. Before is pristine `8d5c88f1`, with identical locked dependencies. Failed baselines are not completed-run speedup measurements. | Target | Before | After | After peak family RSS | | --- | --- | --- | --- | | Official Obsidian 1.12.7, 24,060,763-byte ASAR | External time stop at 240.2 s; no result | Complete JSONL at 162.3 s | 1.58 GiB | | Shipped Notion 7.6.1, 8,903,750-byte ASAR | Native heap exhaustion at 182.7 s; no result | Complete JSONL at 97.8 s | 1.37 GiB | Complete outputs were independently streamed through a JSON parser: approximately 409 MiB and 321 MiB respectively. No filtered view was used. Obsidian retains 100,000 semantic nodes, 77,137 relations, 144,053 unknowns and 23,440 fingerprints. Existing coverage remains partial. Its final complete Evidence bytes match the earlier successful candidate byte for byte. The actual Obsidian `app.js` contains 21,050 callables and 197,401 reference ranges. On 2,048 evenly spaced actual reference ranges, three runs each of the previous scan/sort algorithm and the production index produced identical result digests. Median lookup time fell from 1,129.2 ms to 10.1 ms; index setup took 12.2 ms. This is a lookup measurement, not an end-to-end speedup claim. Windows packaged verification reused an existing Node 22.19.0 installation, two CPUs and the same heap/Node-family guard. Full Obsidian passed in 185.9 s (1.39 GiB peak); full Notion passed in 128.1 s (1.11 GiB peak). Both complete outputs have the same statistics, counts, application graph IDs and semantic graph IDs as Linux. Cold first progress took 17.1 s and 14.0 s respectively. D-drive free space stayed above 32 GiB, owned processes exited, and the isolated package/cache/artifact/output directory was removed. Artifact SHA-256: - Obsidian: `2b2483b2e1246772e0d25367ec055cbc5047ea2f0091b667c35656678f86d712`, from the [official release](https://github.com/obsidianmd/obsidian-releases/releases/download/v1.12.7/obsidian-1.12.7.asar.gz). Version and size match #746; the reporter supplied no digest, so byte identity with that host's file is unverified. - Notion: `97c41bc14fc0ea8c91cd34f502660963dfdf294d5eb35165ccac9d30963069d4`. ### Verification - 25 focused files / 340 tests passed across JavaScript services, graph validation/query/traversal, Evidence, EvidenceBundle, reconstruction, compiled CLI, MCP and retained-Evidence workflows. - Pristine-main versus candidate complete Evidence compares equal for a source-owned 13-file Electron/Webpack/Rspack directory and packed ASAR, including imports, re-exports and nested exported return shapes. - `npm run check:fast -- --concurrency=1`, `npm run format:check`, `npm run knip` and `npm run docs:check` passed. Normal pre-commit and pre-push hooks passed. - `npm run verify:package` passed in 280.3 s at 1.36 GiB peak family RSS. - Full shipped-artifact CLI runs used `rea analyze-javascript-application <selected.asar> --format jsonl` on Linux and Windows. Inputs were parsed as inert text; target code was never executed. - Real Hopper/Ghidra verification was not performed; this patch changes the static JavaScript artifact workflow. The original macOS 4.0.1 terminal error was not reproduced on that host/runtime. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:08:13 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-10-31 01:10:33 +00:00

[PR #3833] [MERGED] fix(native): retain valid Objective-C properties beside a malformed record #5934

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#2611

No description provided.

Rows
Columns