[GH-ISSUE #117] Add semantic analysis for JavaScript and packaged application bundles #74

Closed
opened 2026-10-15 18:20:50 +00:00 by bsakaguchi · 1 comment
Owner

Originally created by @morluto on GitHub (Jul 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/117

Problem Statement

REA can classify, open, hash, and inventory JavaScript targets, source maps, archives, and Electron application bundles. It can also import JavaScript and TypeScript as historical reference source. It cannot yet recover semantic structure from a shipped bundle.

For a published client distributed as one minified or bundled JavaScript artifact, agents cannot use REA to answer:

  • which logical modules and package versions are present;
  • which code is application-owned, generated, vendored, or runtime scaffolding;
  • how imports, exports, dynamic loads, and cross-chunk references connect;
  • which functions or behaviors changed between releases;
  • whether a source map matches the shipped bytes and what authority it provides;
  • where JavaScript crosses into Electron, Node.js native addons, WebAssembly, or other embedded runtimes.

Native Hopper procedure tools correctly do not claim support for JavaScript. Today agents must combine REA artifact inventory with external source indexing and runtime tests. The missing capability is semantic bundle analysis with REA evidence and comparison contracts.

This is a focused implementation issue under #42, not a replacement for the broader managed-runtime and package-provider roadmap.

Proposed Solution

Add a provider-neutral bundle-analysis capability family backed initially by a JavaScript/package provider. Keep parser, bundler, and runtime-specific code outside the domain and application layers.

Initial targets

  • plain JavaScript, ESM, and CommonJS;
  • Webpack, Rollup, esbuild, Parcel, Browserify, and similar bundles;
  • split chunks and lazy-loaded assets;
  • npm package trees and lockfile-backed distributions;
  • Electron ASAR applications and preload/renderer/main-process bundles;
  • Node.js snapshots or bytecode when a provider can make truthful claims;
  • embedded WebAssembly and native addon boundaries.

Unsupported or ambiguous formats must remain explicit unknowns.

Capability model

Candidate provider operations:

  • identify_bundle
  • inventory_modules
  • inspect_bundle_function
  • resolve_bundle_graph
  • map_source_artifacts
  • compare_bundles
  • trace_runtime_bridge

Public tool names and grouping may differ after contract review. CLI and MCP must expose equivalent behavior.

Required results

Bundle analysis should produce deterministic, bounded records for:

  • artifact digest and exact byte ranges;
  • detected bundler/runtime and confidence;
  • logical module, chunk, and package identities;
  • imports, exports, re-exports, dynamic imports, and loader edges;
  • recovered function identities with byte/line locations;
  • generated, vendored, dependency, and application-owned classifications;
  • package and lockfile provenance where observed;
  • source-map identity, validation status, and mapping authority;
  • JavaScript-to-native, JavaScript-to-WebAssembly, Electron IPC, and preload boundaries;
  • limitations, residual unknowns, and parser/provider versions.

Derived names or reconstructed source must be labeled as inference. Original shipped bytes, validated source maps, imported historical source, runtime observations, and analyst inference must remain distinct authorities.

Evidence and comparison

  • Reuse Evidence v2 identities and artifact graph occurrences rather than creating a separate cache.
  • Anchor every semantic record to the shipped artifact digest and exact location.
  • Make pagination deterministic for large bundles.
  • Compare logical modules and functions without treating missing mappings as equality.
  • Preserve minification, tree-shaking, wrapper, and source-map limitations.
  • Allow large records to become MCP resources through #99.

Discovery and routing

  • Provider capability discovery must state which bundle formats and operations are available.
  • Opening a JavaScript target must not imply native procedure analysis is supported.
  • Tool discovery should expose actionable availability through #101 rather than failing unexpectedly after invocation.
  • Hopper-specific code must not enter the bundle-analysis provider.

Alternatives Considered

  • Artifact inventory only: establishes provenance but cannot recover module or behavioral structure.
  • Treating CodeDB or another source index as the REA provider: useful for readable source, but does not by itself establish shipped-artifact identity, bundle mappings, package provenance, or Evidence v2 authority.
  • Sending JavaScript to Hopper: wrong provider boundary and cannot make truthful JavaScript semantic claims.
  • One bundler-specific parser: too narrow; bundler detection and adapters should sit behind shared provider contracts.

Acceptance Criteria

  • A provider-neutral bundle-analysis contract exists independently of any specific parser or bundler.
  • REA detects supported JavaScript/package bundle families and reports confidence and limitations.
  • Representative plain ESM/CommonJS and at least two bundled formats produce deterministic module/chunk inventories.
  • Recovered modules, functions, and graph edges cite shipped-artifact digests and exact locations.
  • Source maps are integrity-checked before contributing mapped names or source locations.
  • Application-owned, dependency, generated, and vendored classifications remain evidence-backed or explicitly inferred.
  • Dynamic imports, split chunks, and unresolved loader behavior remain distinguishable from absent edges.
  • Electron main, renderer, preload, IPC, native addon, and WebAssembly boundaries are represented when observed.
  • Bundle comparison never reports unchanged when either side lacks required semantic coverage.
  • Results preserve Evidence v2 authority, confidence, provider, limitations, and residual unknowns.
  • CLI and MCP expose equivalent analysis and typed failures.
  • Capability discovery distinguishes artifact inventory, bundle semantic analysis, and native procedure analysis.
  • Parsing, traversal, decompression, source-map, module, graph, output-byte, and time limits are enforced.
  • Tests cover minification, malformed bundles, forged source maps, path traversal, oversized inputs, split chunks, duplicated packages, unsupported wrappers, cancellation, and deterministic pagination.
  • Packaged-artifact tests analyze real generated bundles rather than only hand-authored fixtures.

Non-goals

  • Claiming recovered source is original source.
  • Executing arbitrary bundle code during static analysis.
  • Replacing controlled runtime capture for dynamic behavior.
  • Making every bundler heuristic part of the domain model.
  • Treating source-map presence as proof that it matches shipped bytes.
  • Folding native binary decompilation into the JavaScript provider.

Relationships

  • #42 tracks the broader managed-runtime, bytecode, and package-provider family.
  • #101 governs capability-aware discovery and unavailable-operation explanations.
  • #99 can expose large bundle records as addressable MCP resources.
  • #35 and the reference-source work govern historical-source provenance and mapping.
  • #98 should supply shared progress and cancellation for large bundle analysis.

This should become a focused child of #42 if GitHub issue hierarchy is enabled.

Root capability expansion: source-semantic JavaScript provider

A real-world bundled CLI investigation demonstrates the complete missing surface. REA accepted and hashed a shipped JavaScript artifact, but native providers correctly rejected semantic operations. The agent then had no REA-native route for definition/reference navigation, internal function ownership, minified state-machine recovery, or source-bound comparison.

This issue must deliver a coherent JavaScript semantic provider, not only bundle identification or text search.

Required semantic model

  • Parse JavaScript into an AST without executing it.
  • Preserve exact artifact byte offsets and generated line/column spans.
  • Recover chunks, wrappers, logical modules, functions, closures, classes, properties, constants, and package identities where supportable.
  • Model ESM, CommonJS, bundler registries, re-exports, dynamic imports, require indirection, split chunks, and unresolved loader edges.
  • Build bounded definition, reference, caller, callee, string-owner, constant-owner, property-read/write, protocol-consumer, and state-transition relationships.
  • Distinguish syntax, derived structural edges, source-map claims, runtime observations, and analyst hypotheses.
  • Preserve minifier-generated names as artifact-local identities without claiming they are original.
  • Represent parse holes, computed properties, eval, opaque wrappers, and unsupported syntax as scoped unknowns.

Agent-facing operations

Provide provider-neutral operations for bundle overview, module/function inventory, one-function inspection, literal/property/protocol queries, definition/references/callers/callees, state reads/writes, cross-version comparison, source-map mapping, and bridge tracing into Electron IPC, native addons, child processes, WebAssembly, HTTP/WebSocket, filesystem, and environment boundaries.

Source maps, scale, and evidence

  • Verify maps against exact shipped bytes before granting mapped authority.
  • Preserve generated and original coordinates together.
  • Treat sourcesContent, historical source, and inferred mapping as different authorities.
  • Cache by artifact digest, provider/parser version, analysis profile, and schema.
  • Bound AST nodes, source bytes, modules, functions, edges, query work, output, time, and memory.
  • Propagate cancellation/progress and expose large indexes as MCP resources.
  • Never report unchanged/removed when semantic coverage is incomplete.

Root completion gate

An agent must be able to open a real minified bundle, navigate from a literal to its owning function, inspect callers/callees and state effects, compare it with another version, and receive evidence-linked unknowns for unsupported edges. Tests must use real generated single-file and split-chunk bundles, duplicated packages, ASAR main/preload/renderer graphs, malformed partial parses, dynamic ambiguity, stale maps, native/Wasm bridges, and cross-version minifier renaming.

Originally created by @morluto on GitHub (Jul 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/117 ## Problem Statement REA can classify, open, hash, and inventory JavaScript targets, source maps, archives, and Electron application bundles. It can also import JavaScript and TypeScript as historical reference source. It cannot yet recover semantic structure from a shipped bundle. For a published client distributed as one minified or bundled JavaScript artifact, agents cannot use REA to answer: - which logical modules and package versions are present; - which code is application-owned, generated, vendored, or runtime scaffolding; - how imports, exports, dynamic loads, and cross-chunk references connect; - which functions or behaviors changed between releases; - whether a source map matches the shipped bytes and what authority it provides; - where JavaScript crosses into Electron, Node.js native addons, WebAssembly, or other embedded runtimes. Native Hopper procedure tools correctly do not claim support for JavaScript. Today agents must combine REA artifact inventory with external source indexing and runtime tests. The missing capability is semantic bundle analysis with REA evidence and comparison contracts. This is a focused implementation issue under #42, not a replacement for the broader managed-runtime and package-provider roadmap. ## Proposed Solution Add a provider-neutral bundle-analysis capability family backed initially by a JavaScript/package provider. Keep parser, bundler, and runtime-specific code outside the domain and application layers. ### Initial targets - plain JavaScript, ESM, and CommonJS; - Webpack, Rollup, esbuild, Parcel, Browserify, and similar bundles; - split chunks and lazy-loaded assets; - npm package trees and lockfile-backed distributions; - Electron ASAR applications and preload/renderer/main-process bundles; - Node.js snapshots or bytecode when a provider can make truthful claims; - embedded WebAssembly and native addon boundaries. Unsupported or ambiguous formats must remain explicit unknowns. ### Capability model Candidate provider operations: - `identify_bundle` - `inventory_modules` - `inspect_bundle_function` - `resolve_bundle_graph` - `map_source_artifacts` - `compare_bundles` - `trace_runtime_bridge` Public tool names and grouping may differ after contract review. CLI and MCP must expose equivalent behavior. ### Required results Bundle analysis should produce deterministic, bounded records for: - artifact digest and exact byte ranges; - detected bundler/runtime and confidence; - logical module, chunk, and package identities; - imports, exports, re-exports, dynamic imports, and loader edges; - recovered function identities with byte/line locations; - generated, vendored, dependency, and application-owned classifications; - package and lockfile provenance where observed; - source-map identity, validation status, and mapping authority; - JavaScript-to-native, JavaScript-to-WebAssembly, Electron IPC, and preload boundaries; - limitations, residual unknowns, and parser/provider versions. Derived names or reconstructed source must be labeled as inference. Original shipped bytes, validated source maps, imported historical source, runtime observations, and analyst inference must remain distinct authorities. ### Evidence and comparison - Reuse Evidence v2 identities and artifact graph occurrences rather than creating a separate cache. - Anchor every semantic record to the shipped artifact digest and exact location. - Make pagination deterministic for large bundles. - Compare logical modules and functions without treating missing mappings as equality. - Preserve minification, tree-shaking, wrapper, and source-map limitations. - Allow large records to become MCP resources through #99. ### Discovery and routing - Provider capability discovery must state which bundle formats and operations are available. - Opening a JavaScript target must not imply native procedure analysis is supported. - Tool discovery should expose actionable availability through #101 rather than failing unexpectedly after invocation. - Hopper-specific code must not enter the bundle-analysis provider. ## Alternatives Considered - **Artifact inventory only:** establishes provenance but cannot recover module or behavioral structure. - **Treating CodeDB or another source index as the REA provider:** useful for readable source, but does not by itself establish shipped-artifact identity, bundle mappings, package provenance, or Evidence v2 authority. - **Sending JavaScript to Hopper:** wrong provider boundary and cannot make truthful JavaScript semantic claims. - **One bundler-specific parser:** too narrow; bundler detection and adapters should sit behind shared provider contracts. ## Acceptance Criteria - [ ] A provider-neutral bundle-analysis contract exists independently of any specific parser or bundler. - [ ] REA detects supported JavaScript/package bundle families and reports confidence and limitations. - [ ] Representative plain ESM/CommonJS and at least two bundled formats produce deterministic module/chunk inventories. - [ ] Recovered modules, functions, and graph edges cite shipped-artifact digests and exact locations. - [ ] Source maps are integrity-checked before contributing mapped names or source locations. - [ ] Application-owned, dependency, generated, and vendored classifications remain evidence-backed or explicitly inferred. - [ ] Dynamic imports, split chunks, and unresolved loader behavior remain distinguishable from absent edges. - [ ] Electron main, renderer, preload, IPC, native addon, and WebAssembly boundaries are represented when observed. - [ ] Bundle comparison never reports unchanged when either side lacks required semantic coverage. - [ ] Results preserve Evidence v2 authority, confidence, provider, limitations, and residual unknowns. - [ ] CLI and MCP expose equivalent analysis and typed failures. - [ ] Capability discovery distinguishes artifact inventory, bundle semantic analysis, and native procedure analysis. - [ ] Parsing, traversal, decompression, source-map, module, graph, output-byte, and time limits are enforced. - [ ] Tests cover minification, malformed bundles, forged source maps, path traversal, oversized inputs, split chunks, duplicated packages, unsupported wrappers, cancellation, and deterministic pagination. - [ ] Packaged-artifact tests analyze real generated bundles rather than only hand-authored fixtures. ## Non-goals - Claiming recovered source is original source. - Executing arbitrary bundle code during static analysis. - Replacing controlled runtime capture for dynamic behavior. - Making every bundler heuristic part of the domain model. - Treating source-map presence as proof that it matches shipped bytes. - Folding native binary decompilation into the JavaScript provider. ## Relationships - #42 tracks the broader managed-runtime, bytecode, and package-provider family. - #101 governs capability-aware discovery and unavailable-operation explanations. - #99 can expose large bundle records as addressable MCP resources. - #35 and the reference-source work govern historical-source provenance and mapping. - #98 should supply shared progress and cancellation for large bundle analysis. This should become a focused child of #42 if GitHub issue hierarchy is enabled. ## Root capability expansion: source-semantic JavaScript provider A real-world bundled CLI investigation demonstrates the complete missing surface. REA accepted and hashed a shipped JavaScript artifact, but native providers correctly rejected semantic operations. The agent then had no REA-native route for definition/reference navigation, internal function ownership, minified state-machine recovery, or source-bound comparison. This issue must deliver a coherent JavaScript semantic provider, not only bundle identification or text search. ### Required semantic model - Parse JavaScript into an AST without executing it. - Preserve exact artifact byte offsets and generated line/column spans. - Recover chunks, wrappers, logical modules, functions, closures, classes, properties, constants, and package identities where supportable. - Model ESM, CommonJS, bundler registries, re-exports, dynamic imports, require indirection, split chunks, and unresolved loader edges. - Build bounded definition, reference, caller, callee, string-owner, constant-owner, property-read/write, protocol-consumer, and state-transition relationships. - Distinguish syntax, derived structural edges, source-map claims, runtime observations, and analyst hypotheses. - Preserve minifier-generated names as artifact-local identities without claiming they are original. - Represent parse holes, computed properties, eval, opaque wrappers, and unsupported syntax as scoped unknowns. ### Agent-facing operations Provide provider-neutral operations for bundle overview, module/function inventory, one-function inspection, literal/property/protocol queries, definition/references/callers/callees, state reads/writes, cross-version comparison, source-map mapping, and bridge tracing into Electron IPC, native addons, child processes, WebAssembly, HTTP/WebSocket, filesystem, and environment boundaries. ### Source maps, scale, and evidence - Verify maps against exact shipped bytes before granting mapped authority. - Preserve generated and original coordinates together. - Treat sourcesContent, historical source, and inferred mapping as different authorities. - Cache by artifact digest, provider/parser version, analysis profile, and schema. - Bound AST nodes, source bytes, modules, functions, edges, query work, output, time, and memory. - Propagate cancellation/progress and expose large indexes as MCP resources. - Never report unchanged/removed when semantic coverage is incomplete. ### Root completion gate An agent must be able to open a real minified bundle, navigate from a literal to its owning function, inspect callers/callees and state effects, compare it with another version, and receive evidence-linked unknowns for unsupported edges. Tests must use real generated single-file and split-chunk bundles, duplicated packages, ASAR main/preload/renderer graphs, malformed partial parses, dynamic ambiguity, stale maps, native/Wasm bridges, and cross-version minifier renaming.
bsakaguchi 2026-10-15 18:20:50 +00:00
Author
Owner

@morluto commented on GitHub (Jul 20, 2026):

Closing as completed. REA now recovers semantic structure from shipped JavaScript and packaged application bundles without executing them:

  • analyze_javascript_application returns a deterministic JavaScript Application Graph with module/chunk/package identities, import/export/require edges, call relationships, native/Electron/IPC/WebAssembly/managed boundaries, source-map authority, and bounded residual unknowns.
  • trace_application_feature supports literal/property/protocol queries through the graph with direction/depth limits.
  • compare_application_versions and reconcile_javascript_runtime provide cross-version comparison and runtime correlation.
  • The static-analysis pipeline (src/domain/javascriptStaticAnalysis*.ts, src/domain/javascriptSemanticProjection.ts) parses bounded ASTs, resolves bindings, collects callables and module links, detects vendors/bundler registrations, and enforces node/edge/time limits.
  • Tests cover ESM/CommonJS, Webpack/Rspack/Parcel/esbuild/Rollup/Browserify-style bundles, ASAR, Electron main/preload/renderer, native-addon handoffs, minification, malformed sources, source-map handling, and deterministic pagination.

If a dedicated per-function dossier tool is still needed, that can be opened as a focused follow-up.

<!-- gh-comment-id:5023313222 --> @morluto commented on GitHub (Jul 20, 2026): Closing as completed. REA now recovers semantic structure from shipped JavaScript and packaged application bundles without executing them: - `analyze_javascript_application` returns a deterministic JavaScript Application Graph with module/chunk/package identities, import/export/require edges, call relationships, native/Electron/IPC/WebAssembly/managed boundaries, source-map authority, and bounded residual unknowns. - `trace_application_feature` supports literal/property/protocol queries through the graph with direction/depth limits. - `compare_application_versions` and `reconcile_javascript_runtime` provide cross-version comparison and runtime correlation. - The static-analysis pipeline (`src/domain/javascriptStaticAnalysis*.ts`, `src/domain/javascriptSemanticProjection.ts`) parses bounded ASTs, resolves bindings, collects callables and module links, detects vendors/bundler registrations, and enforces node/edge/time limits. - Tests cover ESM/CommonJS, Webpack/Rspack/Parcel/esbuild/Rollup/Browserify-style bundles, ASAR, Electron main/preload/renderer, native-addon handoffs, minification, malformed sources, source-map handling, and deterministic pagination. If a dedicated per-function dossier tool is still needed, that can be opened as a focused follow-up.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#74
No description provided.