mirror of
https://github.com/morluto/rea.git
synced 2026-11-05 06:45:22 +00:00
[GH-ISSUE #4415] [Perf][Exploration] Investigate scoped JavaScript preparation and sealing costs on real Joplin #1755
Open
opened 2026-10-16 21:09:51 +00:00 by bsakaguchi
·
3 comments
No Branch/Tag specified
main
codex/procedure-reference-stacked
release-please--branches--main--components--rea-agents
release-please--branches--main--components--rea-agents--release-notes
codex/kuna-independent-retry
codex/byte-search-overlay-coverage
codex/open-failure-target-receipt
codex/kuna-xref-evidence
codex/ghidra-ambiguous-procedure-recovery
codex/function-source-bundle
codex/trace-function-integration-prototype
codex/snapshot-source-closure
codex/recorded-evidence-array-pages
codex/evidence-recovery-redesign
codex/trace-feature-address-scope
codex/call-discovery-scope
codex/ghidra-literal-candidate-evidence
codex/decompile-directory-independent-retry
codex/retry-provenance-base
codex/value-flow-capability-experiment
codex/ghidra-containing-instruction
cursor/readme-discord-button-ef9f
dependabot/github_actions/actions/setup-node-7.1.0
dependabot/github_actions/actions/upload-artifact-7.0.2
dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df
codex/kuna-1989
fix/8559-native-ui-fixture-ownership
fix/8555-go-verifier-mcp-ownership
codex/provider-capability-recovery
fix/8642-rizin-target-stdin
codex/ghidra-memory-effect-scope
codex/ghidra-annotation-entry-selector
codex/ghidra-readonly-recovery
codex/ghidra-regex-verifier-recovery
codex/evidence-discovery-view-guidance
fix/8545-bulk-append-overflow
fix/8428-navigation-refusal-alternatives
codex/address-context-recovery
codex/matching-member-limitations
codex/unknown-scope-contract
codex/evidence-view-guidance
feat/8373-native-trace-views
codex/ghidra-snapshot-verifier-binding
codex/elf-export-visibility
codex/evidence-view-recovery
codex/native-reference-cli-help
codex/provider-candidate-observation-scope
codex/kuna-session-verification
codex/ghidra-switch-lifetime-guidance
codex/pseudocode-paging-discovery
codex/clarify-evidence-view-support
codex/kuna-xrefs-selector-recovery
codex/context-capacity-recovery
codex/evidence-closure-large-links
codex/same-function-evidence-identity
codex/ghidra-verifier-inspected-jdk
codex/audit-guided-workflows
codex/crash-prompt-retained-dossiers
codex/analysis-view-error-recovery
codex/kuna-verifier-target-selection
codex/ghidra-selected-project-call-coverage
codex/docs-native-analysis-scope
codex/ghidra-java-socket-boundary
codex/ghidra-retained-snapshot-verifier
codex/ghidra-search-inventory-facts
fix/8284-zero-argument-parameter-bindings
fix/8198-ui-action-handler-offset
fix/8257-elf-entry-label-identity
fix/8304-deferred-output-schema-projection
fix/8277-call-path-computed-target-uncertainty
fix/pr-8089-conflicts
fix/ci-static-duplicate-export
dependabot/npm_and_yarn/lint-staged-17.6.0
dependabot/github_actions/actions/download-artifact-8.0.2
dependabot/npm_and_yarn/write-file-atomic-8.0.0
dependabot/npm_and_yarn/canonicalize-5.1.0
dependabot/github_actions/actions/cache/restore-6.1.0
dependabot/npm_and_yarn/toon-format/toon-4.4.0
dependabot/github_actions/actions/deploy-pages-5.0.1
codex/review-fixes-pr-2403
codex/review-fixes-pr-2400
codex/process-terminal-width
codex/test-brittleness-cleanup
verify/7167-browser-connect-macos
fix/7167-browser-connect
feat/native-matching-allowance-setting
feat/browser-scenario-dry-run
test/managed-caller-parameters-20261024
codex/parser-fortythird-boundaries
fix/6926-pinvoke-platform-decorations-20261023
fix/6889-har-body-encoding-20261023
fix/6888-sqlite-schema-budget-20261023
fix/dependency-security-gates-6708
cursor/evidence-overwrite-rebase-b4ac
feat/4057-verify-native-exports
fix/windows-acl-observer-overhead-20261022
cursor/erlang-beam-restack-4476
cursor/pe-certificate-review-bf32
fix/6116-javascript-application-fs-errors
fix/6061-unmapped-runtime-script-unknown
feat/browser-scenario-summary
fix/pe-certificates
codex/verify-3407
cursor/rebase-5699
cursor/rebase-5352
cursor/rebase-5600
cursor/fix-cli-schema-omission-10a8
codex/audit-hopper-shutdown-reasons-20261021
fix/demangle-swift-omitted-record
fix/nib-boolean-value-types
cursor/cancellation-partial-observation-a5ac
fix/ib-object-limit-reasons
fix/electron-fuse-limit-advice
fix/kuna-doctor-dangling-cache
fix/fixed-budget-refusal-advice
fix/js-ipc-listener-dedup
fix/js-return-member-provenance
fix/dmg-integrity-retry-advice
fix/ida-doctor-readiness
fix/kuna-version-probe-output-drain
fix/legacy-swift-runtime-name-matching
fix/dylib-lazy-load-commands
fix/browser-scenario-startup-cancel
fix/swift-nested-punycode-classification
fix/compare-unselected-scopes
fix/objc-method-budget-coverage
fix/dmg-mounted-root-observations
audit/ci-tests-20261019-round2
release/6.7.0
fix/wabt-cleanup-retry-resolution
fix/electron-fuse-elf-pe-architecture
fix/native-plist-descriptor-ownership
fix/kuna-caller-cleanup-ownership
experiment/atomic-codemode-20261016
review/first-contributor-3776
fix/call-resolution-full-read-cycles
fix/member-mutation-selection-repeat
experiment/atomic-re-tools-20261015
perf/member-mutation-lazy-origins
perf/static-network-mutation-index
fix/hopper-startup-lifecycle
audit/3050-windows-proof-20261015
feat/windows-window-capture-2183
audit/3049-windows-proof-20261015
audit/3735-macos-fixtures-20261015
fix/batch-decompile-null-pseudocode
fix/kuna-inventory-honors-jobs
experiment/browser-codemode-tool-design-20261015
fix/hopper-clean-runtime-argv
codex/review-2049-27f1b367
review-fixes/pr-2787-routing
fix/3360-scoped-this-emitters
fix/3367-replaced-global-effects
fix/3838-compact-any-value-items
test/hopper-download-probe-20261015
codex/windows-runtime-creation-red-green-20261015
codex/windows-readonly-root-red-green-20261015
codex/windows-root-share-red-green-20261015
codex/exp-1685-win-mcp-startup
codex/bug-sweep-20261015
investigate/2190-windows-package-phases
investigate/3543-macos-cli-deadline
perf/3281-reverse-tools-feedback-quality
fix/3406-hopper-doctor-fixture
fix/3341-wrapper-require-test-context
fix/3301-owned-uninstall-fixtures
fix/3164-compiler-input-ownership
fix/pcap-cleanup-stdout-newline-20261014
perf/interface-builder-summary
codex/review-fixes-pr-1977
fix/ghidra-effective-jvm-platform
fix/docs-turkish-rust-link-20261013
fix/main-maintenance-ci-20261010
fix/recover-binary-session-process-effects
codex/review-fixes-pr-2457
codex/ci-required-cancellation-gate
fix/main-format-baseline-20261013
fix/setup-preserve-entry-fields
cursor/restore-main-checks-1ffc
fix/ghidra-bridge-hashmap-import
codex/shared-ci-jadx-json-default-20261013
codex/shared-ci-ghidra-test-groups-20261013
codex/shared-ci-java-fixture-format-20261013
codex/fix-intrinsic-invocation-boundaries
test/release-preflight-reusable-jobs
test/js-parameter-provenance-expectation
fix/linux-hopper-verification-current
test/2419-ghidra-elf-probe-size
followup/pr-1544-native-ida
codex/fold-pr-2230-contribution
docs/readme-responsible-use
fix/issue-2064
fix/issue-2050
fix/issue-2053
fix/issue-1968
fix/issue-1945
experiment/codemode-bridge-20261011
fix/tool-effects-negative-wording-1904-20261012
fix/1842-electron-interop-provenance
feat/recover-javascript-multi-input
refactor/website-i18n-prep
fix/nativeaot-pe-virtual-extent
fix/browser-verifier-ci-hang
fix/pe-resources-input-selection
docs/beginner-friendly-guides
docs/beginner-setup-guide
fix/javascript-analysis-worker-1563
fix/1649-ghidra-duplicate-names
fix/javascript-export-keys-integration-1369
test/maintenance-boundary-contracts-20261011
review/javascript-export-keys-1369
review/process-live-progress-1483
test/javascript-1495-growth-ratio
fix/ci-escape-timing
docs/normalized-terminal-geometry-1450
release/6.3.0
archive/release-6.3.0-pr1381
release/6.2.0
fix/electron-preload-dynamic-override
fix/javascript-self-referential-bundler-edges
release/6.1.0
cursor/grok-client-cherry-pick-0b56
feat/signature-posture
feat/dyld-shared-cache
feat/artifact-integrity-policy
feat/pkg-inventory
release/6.0.0
chore/sync-release-5.0.0
release/5.0.0
cursor/cloud-agent-node-787e
fix/javascript-xhr-open-endpoints
review/pr-711-realpath-fallback-wording
cleanup/replay-confinement
codex/test-suite-performance
codex/issue-foundations
codex/address-open-issues-2026-07-20
agent/javascript-vite-rollup-esbuild-adapters
agent/process-cleanup
agent/triage-integration
agent/unknown-commands-registry
feat/mcp-identity-metadata
morlvps/issue_cleanup
codex/installer-redesign
morl/roadmap-investigation-core
morl/roadmap-investigation-docs
release-please--branches--main--components--rea
rea-agents-6.8.0
rea-agents-6.7.0
rea-agents-6.6.0
experiment/atomic-codemode-wait-runtime-20261016
rea-agents-6.5.0
rea-agents-6.4.0
rea-agents-6.3.0
rea-agents-6.2.0
rea-agents-6.1.0
rea-agents-6.0.0
rea-agents-5.0.0
rea-agents-4.1.0
rea-agents-4.0.1
rea-agents-4.0.0
rea-agents-3.2.1
rea-agents-3.2.0
rea-agents-3.1.0
rea-agents-3.0.0
rea-agents-2.7.0
rea-agents-2.6.0
rea-agents-2.5.1
rea-agents-2.5.0
rea-agents-2.4.0
rea-agents-2.3.0
rea-agents-2.2.0
rea-agents-2.1.0
rea-agents-2.0.0
rea-agents-1.7.0
rea-agents-1.6.0
rea-agents-1.5.0
rea-agents-1.4.0
rea-agents-1.3.0
rea-agents-1.2.0
rea-agents-1.1.0
rea-agents-1.0.0
rea-agents-0.5.0
rea-agents-0.4.0
rea-agents-0.3.0
rea-0.2.1
rea-0.2.0
Labels
Clear labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
Mirrored from GitHub Pull Request
question
roadmap
triage
No labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
question
roadmap
triage
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
morluto/rea#1755
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @N0zoM1z0 on GitHub (Oct 16, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4415
Before submitting
Area
Evidence, artifacts, or reconstruction
Current limitation
At main
d72d856f58d4747c5eb4307426e80d20934e017c, a real Joplin 3.7.21 ASAR workflow selects onlyElectronAppWrapper.js(39,972 bytes) andpackage.json, then retrieves one BrowserWindow fact. It still takes 45.102 s to the first answer (preparation 40.531 s), 44.669897 process-family CPU s and 1,552,072,704 bytes peak cgroup memory. The existing fact view itself takes 120.8 ms initially and 21–24 ms on two same-session repeats. Those repeats are not cold analysis trials.This is an observed cost, not an established regression. Source selection limits source analysis; the enclosing artifact inventory and complete Evidence remain part of the current contract. A smaller requested answer alone does not authorize omitting that evidence.
Use cases
Repeated static analysis of real Electron apps with a narrow analyst question, while retaining full provenance/coverage and CLI/MCP parity.
Desired outcome and success criteria
Locate and reduce duplicated work at its owning boundary when a measured equivalent complete workflow justifies the change. Preserve identical facts, identity, full Evidence and unknown/partial semantics; preserve cancellation, JSON depth validation and ownership. Compare both run orders, cold-process and within-session behavior, projected whole-task tokens, elapsed time, CPU and process-family memory. Do not introduce a cache or new selector without an observed need and explicit validity/lifetime rules. No production change is proposed yet.
Expected interaction with targets
Read existing files or captured data. No Joplin execution, attachment or instrumentation.
Verification available so far
I have a prototype or observed results.
Verification details
Official Joplin 3.7.21 ASAR: 233,575,014 bytes, SHA256
767df409b18effe60421fbefcca873de237c7e813eac744ecdeebbc5e825a33a. The complete CLI/MCP fact answer, raw semantic fingerprint and Evidence parity match the #4400 verification record after merge. Structured whole-task BPE projection is 13,432 using the pinned verifier/tiktoken 0.12.0; this is not billed usage or live-agent context.A separate parent-only Node CPU-profile CLI diagnostic completed in 38.147 s / 37.736337 family CPU s / 1,010,319,360 peak cgroup bytes. Its 37.682 s sampling weight includes idle/native frames: graph construction/sealing, canonical hashing, input reads and GC all contribute. The analyzer worker is not profiled; these weights are neither exact phase CPU accounting nor an equivalent MCP timing comparison. Do not add overlapping inclusive rows. Both workflows used Node24.18.0, one CPU, 8GiB hard/6GiB early limits, no observed OOM and verified owned-payload cleanup. Build/OS caches are uncontrolled and input hashing warms file cache.
Reproduce the public CLI diagnostic after building the pinned main, under the existing resource observer:
A compact sampling summarizer and receipts are being retained on an exploration branch; no PR will be opened for this investigation alone. Regenerable app binaries, large Evidence exports and raw CPU profiles are not archived.
Alternatives considered
The existing retained-fact view already bounds the answer; it does not remove preparation. Reusing authenticated immutable roots is promising, but an immutable snapshot alone does not prove the full JSON depth/proxy contract: validation cannot simply be removed. The current profile does not justify speculative persistent caching.
References
Part of #4027; follows #4366 / #4400. Popup #3536 remains separate: its real Joplin helper passes webContents recursively, so a method name alone must not invent a proven opener.
Exploration checkpoint
The compact receipts and a sampling summarizer are preserved locally on
perf/javascript-analysis-cost-20261016in/home/yann/yann/rea-javascript-analysis-cost-20261016; no exploration PR is opened. Summarizer replay and independent sample-count comparison pass on the actual Node profile. It preserves the one observed -2us delta rather than clamping it; rows rank by sample counts, with signed delta weights explicitly labelled. There is no production code change or claimed optimization gain. The immutable owner still requires independent JSON depth/proxy validation, so that check has not been bypassed. Raw profiles and regenerable large outputs are removed after compact verification.@Antruly commented on GitHub (Oct 16, 2026):
I read the preparation path behind this report rather than reproducing it, so everything below is source-derived from
main@d72d856f— the same revision this issue measured — and I have not run the Joplin workflow myself. Nothing here is a measurement or a proposed production change; it is what the code says about where the cost is reachable from, plus two things I would check next.1. The inventory pass reads and hashes every entry before the caller's selection is consulted
reconstructJavaScriptArtifactbuilds the full snapshot first and only then passessource_pathsin:src/application/javascript/JavaScriptArtifactReconstruction.ts:92-96—scanCanonicalArtifactInventoryInScope(path, { resourceScope: scope, signal, integrity: … }).source_pathsis not part of this call.src/application/javascript/JavaScriptArtifactReconstruction.ts:120-126—readJavaScriptArtifactFiles(reader, snapshot, scope, signal, input.source_paths). Selection is applied to an already-complete snapshot.src/artifacts/inventory/scanReader.ts:288-293—digestArtifactEntryopens andhashReadables everyfile/sliceentry unconditionally;:296-316shows the integrity policy only decides what happens after a mismatch (failthrows,record-and-continuerecords a contradiction). The hash itself is not conditional.So
--source-pathbounds source-text acquisition and analysis, not the inventory pass:src/artifacts/javascript/JavaScriptArtifactFiles.ts:163-173reports unselected files asoutside-analysis-scopewhile thefiles/containerslisting stays complete.That is consistent with this issue's headline numbers: a request scoped to
ElectronAppWrapper.jsandpackage.jsonstill pays for the whole 233 MB archive before the first answer, and the same pass is why the documented scoped workflow still takes 143 s on the #3538 target.The only size-based refusals on this path are hard caps thrown during that pass —
src/artifacts/AsarHeader.ts:7(16 MiB header decode budget) andsrc/artifacts/AsarListing.ts:8(64 MiB expanded listing budget) — not an estimate of what the run will cost.2. The generic memory remediation names a mechanism this path does not implement
src/domain/analysisErrorPresentation.ts:71still ends with "…Retry with sufficient memory or a smaller artifact; REA retains tighter inherited limits."For JavaScript/Electron analysis there is no inherited limit to retain. The worker heap is selected explicitly and ambient heap flags are removed:
src/javascript/analysis/JavaScriptAnalysisWorker.ts:466—--max-old-space-size=${String(this.limits.heapMb)}src/javascript/analysis/JavaScriptAnalysisWorker.ts:462—delete environment.NODE_OPTIONS128–16384at:119-129; defaultsheapMb: 1024,timeoutMs: 300_000atsrc/domain/javascript/javascriptAnalysisResourceControls.ts:6-11; CLI wiringsrc/cli/electronCommands.ts:281-289The only code that reads or clamps inherited kernel limits is
src/process/LinuxResourceLimits.ts(:42-65clamp,:68-105read), and its consumers are the capture/analysis producers —src/captures/TsharkCaptureProvider.ts:13-15,src/pcap/PcapProvider.ts:50,src/signals/SigrokRecordingProvider.ts:12-14,src/process/ArtifactAnalysisResources.ts:22-24. Nothing undersrc/javascript/reads it. The remaining memory reads in this path are the process's own V8 heap (JavaScriptAnalysisWorker.ts:146-147,:575-576,:677-678;JavaScriptAnalysisWorkerServer.ts:67,84-85).So for this failure the actionable lever is the heap the caller or the default chose, and the numbers are usually already in
reported_limits(worker_old_space_mb,parent_result_budget_bytes). The parent side is a heap-derived heuristic, not a measured need —JavaScriptAnalysisWorker.ts:143-151:re-evaluated at
:575-576and:677-678. That shape — budget derived from the current heap rather than from what the work needs — is also a plausible source of the contradictoryapplication_projection_required_bytesvsparent_result_budget_bytesfigures in #3671; I have not verified that link.3. What I would check next
Not a change proposal, just the measurement the report itself says is missing ("The analyzer worker is not profiled"):
Disclosure
#3550 is mine and touched the sibling branch of this same remediation — it made the
parent_result_budget_exhaustedcase stop advising more worker memory and stated that the limit belongs to the parent result budget. Point 2 is the adjacent generic branch, which it did not change.@N0zoM1z0 commented on GitHub (Oct 16, 2026):
Additional single-call preflight observation, not a speedup or completed analyst-task benchmark:
Official VSCodium 1.140.06880 Linux x64 archive (212,538,680 B), SHA-256
61223b74f9e40834117311ec4177cb386d09b84c1aae4f9e674c96580569cef3, verified against the official GitHub release asset digest. Analyzed the full extractedresources/appwith exact--source-path out/main.js; that selected entry is 100,087 B, SHA-2560762b59a63f4ed640905bfa3c5bebf78d19508b06a2d3253d374ab3d111a6cad.Observed source selection reports the exact requested path and 4,539 unselected files. The full CLI output, including progress/Evidence and the resource receipt, is 87,316,011 B; inventory remains in the graph. Wall 33.320 s, payload-exit CPU 31.338796 s, cgroup peak 1,101,549,568 B, with verified complete-family observation at payload exit (observer still alive). This includes the call's startup/inventory/analysis/serialization, not later manual caller discovery or fuse inspection. No agent/billed token measurement or before/after comparison was performed.
Invocation:
node scripts/rea.mjs analyze-javascript-application APP --source-path out/main.js --max-heap-mb 1536 --analysis-timeout-ms 120000 --format json. Existing compiled static runtime built at709b5cbdwas reused from the #4605 worktree; not a rebuilt current-main acceptance run. One CPU, 8 GiB hard/6 GiB early, no swap, 192 tasks, parent V8 heap 2048 MiB. Warm filesystem/toolchain, single run. Peak includes file cache, not RSS.This strengthens the case for measuring preparation/inventory, sealing and serialization separately from the selected source analysis, before deciding what to cache or project. The entry itself has no
trashItem; later source-only inspection locates the real callers inout/mainImpl.js, so this one call does not complete that lookup task. No finding that current compact MCP replies have the same cost is claimed. Regenerable application and large raw output stay temporary; compact metrics/source identity are recorded locally for the follow-up.@SII-YDD commented on GitHub (Oct 18, 2026):
Independent current-main selected-source CLI preflight, not an optimization or completed multi-call analyst-task benchmark.
Pinned main
52b73ee857029a3ab8f5c6eeef233b295951a46c, Node24.18.0/npm11.16.0, Linux arm64; official VSCodium1.140.06880 asset SHA25661223b74f9e40834117311ec4177cb386d09b84c1aae4f9e674c96580569cef3(212,538,680B), checked against the release API digest. Onlyresources/appwas extracted; the downloaded app was not executed. The command was:Two fresh-process/container repeats with uncontrolled/warm OS caches, one CPU, 6GiB hard/no swap, 4,500MiB early memory guard, 192 tasks, UID1000, read-only runtime/input, no network or credentials:
Both payloads exited0 without guard stops. Exact selection was retained: one parsed file/100,087 text bytes, 4,539 unselected files, 10,705 graph nodes/11,926 edges.
out/main.jssource SHA256 is0762b59a63f4ed640905bfa3c5bebf78d19508b06a2d3253d374ab3d111a6cad. All three runs including the separate parent-only profile have identical sorted-key compact normalized-result SHA256aef44c7824c26c609fb081b691d38d130f8738bf5d5a5172a20d9d5672b21ea6. Coverage remains partial, untruncated, no limits, with selection omissions retained; this does not prove whole-app completeness.Elapsed progress-boundary intervals, rather than exact phase CPU accounting, were approximately:
read_javascript_artifacts(includes admission/inventory): 4.01–4.02s.Source-derived explanation for the second acquisition cost:
JavaScriptArtifactFiles.ts:readTextcallsverifyEntryByteseven foroutside-analysis-scopeentries. This preserves the inventory/acquisition byte identity contract; it is not evidence that the check can be removed. The progress intervals include adjacent owner work and scheduling, and should not be renamed pure hashing or parsing time.The separate parent CPU diagnostic contains7,030 samples (signed delta weight16.786s, including idle/native/GC). Canonical hashing, JSON validation/freezing, graph sealing and output appear among self frames; worker CPU is not profiled. No token count, improvement ratio, complete analyst-task comparison, or cache/scope proposal is claimed. A preliminary container-copy run contaminated its peak with dependency preparation and is excluded from the runtime figures above. Compact receipts/progress/sample summaries are retained locally. AI-assisted investigation; no production change or PR.