morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1605] [Feature] Recover related scripts together in `recover_javascript_sources` #542

New issue

Closed

opened 2026-10-15 18:26:00 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:26:00 +00:00

Owner

Originally created by @pionxzh on GitHub (Oct 10, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1605

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

recover_javascript_sources takes one path. The adapter snapshots it as inputs/bundle.js, and validateModuleSet requires every module's provenance input to equal that one snapshot.

A code-split web app ships a runtime and several chunks. export_web_scripts writes all of them into one directory, and an extracted Electron app usually has separate main, preload, and renderer bundles. Today each file has to be recovered in its own call. A chunk recovered alone cannot see the modules it imports from another chunk, so cross-chunk references stay as numeric require(id) calls with property reads, and the recovered directory has no import edges between them.

Use cases

Small repro (producer webpack@5.101.3, mode: "production", runtimeChunk: "single", a splitChunks group for shared.mjs, concatenateModules: false):

// shared.mjs
export class Cart { constructor() { this.items = []; } add(p) { this.items.push(p); return this; } total() { return this.items.reduce((a, b) => a + b, 0); } }
export const formatPrice = (n) => `$${n.toFixed(2)}`;
export const TAX_RATE = 0.08;
// checkout.mjs
import { Cart, formatPrice, TAX_RATE } from "./shared.mjs";
export function checkout(prices) { const cart = new Cart(); prices.forEach((p) => cart.add(p)); return formatPrice(cart.total() * (1 + TAX_RATE)); }
// entry.mjs
import { formatPrice } from "./shared.mjs";
console.log(formatPrice(1));
import("./checkout.mjs").then(({ checkout }) => console.log(checkout([1, 2])));

This emits runtime.js, main.js, shared.js and 349.chunk.js. I ran Wakaru 1.14.0 with the adapter's exact flags (--unpack=strict --level=standard --provenance --emit-source-map --json).

One call per file, as REA does today. The checkout chunk recovers as:

const n = require(282);
export function checkout(c) {
    const e = new n.Z3;
    c.forEach((c)=>e.add(c));
    return n.$g(e.total() * (1 + n.Lp));
}

All four files in one call:

import { $g, Lp, Z3 } from "./module-282.js";
export function checkout(c) {
    const e = new Z3;
    c.forEach((c)=>e.add(c));
    return $g(e.total() * (1 + Lp));
}

main.js changes the same way (const n = require(282) becomes import { $g } from "./module-282.js").

Then I ran rea analyze-javascript-application (main at 603d04ee) on both recovered directories:

	module import edges	resolved
one call per file	0	–
one call for all files	4	4, all to `module-282.js`

Desired outcome and success criteria

One recovery call accepts several related scripts, such as a files/ directory from export_web_scripts, or explicitly listed chunk files.
Every input is snapshotted and digested separately, and the manifest records each source.

Provenance stays exact: each module's input must be one of the snapshots, and its byte ranges are checked against that input's bytes. Wakaru already reports this per module in multi-input runs:

"module-271.js": {"input": "dist/main.js", "ranges": [[82,192]], "extraction": "structural"},
"module-282.js": {"input": "dist/shared.js", "ranges": [[82,275]], "extraction": "structural"}

The single-file call keeps working unchanged.
analysis_input still points at the recovered directory, so analyze_javascript_application resolves cross-chunk imports.

Authority boundaries and non-goals

No change to the execution boundary: inputs and recovered code are still never executed.
The output directory must still be absent, and publication stays exclusive with rollback.
Not proposing network fetching of missing chunks. A chunk that was not captured stays unresolved, as it does today.
The input and resource limits would need an aggregate rule. That is your call; I don't have a strong preference.

Alternatives considered

One call per file: loses the cross-chunk links shown above.
Concatenating files before recovery: breaks per-file source identity and byte ranges, and is not how the browser loads chunks.

Possible direction

Add paths: string[] next to path (exactly one of them), or let path name a directory. For a directory, Wakaru's <dir> --unpack already scans .js, .mjs and .cjs files recursively and skips hidden entries.
Snapshot each input under inputs/ with its relative path preserved, pass the snapshots to one Wakaru run, and replace the single entry.input === snapshotPath check with a lookup in the snapshot set.
Wakaru's --json and provenance.json are covered by its compatibility policy, so the parser needs no new fields.

I maintain Wakaru and can send a PR if you agree on the input shape.

References

#827, #848 (current adapter)
Wakaru multi-input unpack: https://wakarujs.com/docs/reference/supported-inputs

Originally created by @pionxzh on GitHub (Oct 10, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1605 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation `recover_javascript_sources` takes one `path`. The adapter snapshots it as `inputs/bundle.js`, and `validateModuleSet` requires every module's provenance `input` to equal that one snapshot. A code-split web app ships a runtime and several chunks. `export_web_scripts` writes all of them into one directory, and an extracted Electron app usually has separate main, preload, and renderer bundles. Today each file has to be recovered in its own call. A chunk recovered alone cannot see the modules it imports from another chunk, so cross-chunk references stay as numeric `require(id)` calls with property reads, and the recovered directory has no import edges between them. ### Use cases Small repro (producer webpack@5.101.3, `mode: "production"`, `runtimeChunk: "single"`, a `splitChunks` group for `shared.mjs`, `concatenateModules: false`): ```js // shared.mjs export class Cart { constructor() { this.items = []; } add(p) { this.items.push(p); return this; } total() { return this.items.reduce((a, b) => a + b, 0); } } export const formatPrice = (n) => `$${n.toFixed(2)}`; export const TAX_RATE = 0.08; // checkout.mjs import { Cart, formatPrice, TAX_RATE } from "./shared.mjs"; export function checkout(prices) { const cart = new Cart(); prices.forEach((p) => cart.add(p)); return formatPrice(cart.total() * (1 + TAX_RATE)); } // entry.mjs import { formatPrice } from "./shared.mjs"; console.log(formatPrice(1)); import("./checkout.mjs").then(({ checkout }) => console.log(checkout([1, 2]))); ``` This emits `runtime.js`, `main.js`, `shared.js` and `349.chunk.js`. I ran Wakaru 1.14.0 with the adapter's exact flags (`--unpack=strict --level=standard --provenance --emit-source-map --json`). One call per file, as REA does today. The checkout chunk recovers as: ```js const n = require(282); export function checkout(c) { const e = new n.Z3; c.forEach((c)=>e.add(c)); return n.$g(e.total() * (1 + n.Lp)); } ``` All four files in one call: ```js import { $g, Lp, Z3 } from "./module-282.js"; export function checkout(c) { const e = new Z3; c.forEach((c)=>e.add(c)); return $g(e.total() * (1 + Lp)); } ``` `main.js` changes the same way (`const n = require(282)` becomes `import { $g } from "./module-282.js"`). Then I ran `rea analyze-javascript-application` (main at 603d04ee) on both recovered directories: | | module import edges | resolved | |---|---|---| | one call per file | 0 | – | | one call for all files | 4 | 4, all to `module-282.js` | ### Desired outcome and success criteria - One recovery call accepts several related scripts, such as a `files/` directory from `export_web_scripts`, or explicitly listed chunk files. - Every input is snapshotted and digested separately, and the manifest records each source. - Provenance stays exact: each module's `input` must be one of the snapshots, and its byte ranges are checked against that input's bytes. Wakaru already reports this per module in multi-input runs: ```json "module-271.js": {"input": "dist/main.js", "ranges": [[82,192]], "extraction": "structural"}, "module-282.js": {"input": "dist/shared.js", "ranges": [[82,275]], "extraction": "structural"} ``` - The single-file call keeps working unchanged. - `analysis_input` still points at the recovered directory, so `analyze_javascript_application` resolves cross-chunk imports. ### Authority boundaries and non-goals - No change to the execution boundary: inputs and recovered code are still never executed. - The output directory must still be absent, and publication stays exclusive with rollback. - Not proposing network fetching of missing chunks. A chunk that was not captured stays unresolved, as it does today. - The input and resource limits would need an aggregate rule. That is your call; I don't have a strong preference. ### Alternatives considered - One call per file: loses the cross-chunk links shown above. - Concatenating files before recovery: breaks per-file source identity and byte ranges, and is not how the browser loads chunks. ### Possible direction - Add `paths: string[]` next to `path` (exactly one of them), or let `path` name a directory. For a directory, Wakaru's `<dir> --unpack` already scans `.js`, `.mjs` and `.cjs` files recursively and skips hidden entries. - Snapshot each input under `inputs/` with its relative path preserved, pass the snapshots to one Wakaru run, and replace the single `entry.input === snapshotPath` check with a lookup in the snapshot set. - Wakaru's `--json` and `provenance.json` are covered by its [compatibility policy](https://github.com/pionxzh/wakaru/blob/v1.14.0/docs/cli.md#machine-readable-output-compatibility), so the parser needs no new fields. I maintain Wakaru and can send a PR if you agree on the input shape. ### References - #827, #848 (current adapter) - Wakaru multi-input unpack: https://wakarujs.com/docs/reference/supported-inputs

bsakaguchi

2026-10-15 18:26:00 +00:00

closed this issue
added the
triage

enhancement
labels

bsakaguchi commented

2026-10-15 18:26:01 +00:00

Author

Owner

@JingHao-Leon commented on GitHub (Oct 11, 2026):

Picking this up. Plan: extend recover_javascript_sources to accept multiple paths (or a directory) so a code-split app can be recovered in one call, and relax validateModuleSet's provenance check from a single pinned snapshot path to the set of accepted input snapshots — preserving the per-module provenance values as evidence rather than collapsing them. Will keep the change additive (single-path callers unaffected) and include the webpack splitChunks repro as a test fixture.

@JingHao-Leon commented on GitHub (Oct 11, 2026): Picking this up. Plan: extend `recover_javascript_sources` to accept multiple paths (or a directory) so a code-split app can be recovered in one call, and relax `validateModuleSet`'s provenance check from a single pinned snapshot path to the set of accepted input snapshots — preserving the per-module provenance values as evidence rather than collapsing them. Will keep the change additive (single-path callers unaffected) and include the webpack splitChunks repro as a test fixture.

bsakaguchi commented

2026-10-15 18:26:01 +00:00

Author

Owner

@pionxzh commented on GitHub (Oct 11, 2026):

Thanks for #1686, @JingHao-Leon. I reviewed how the adapter uses Wakaru's provenance: each module binds to its own input snapshot, and its ranges are checked against that input's bytes. It all looks correct to me.

One question about the cap. What does the 16-path limit protect? The 64 MiB total input already bounds memory and time. I noticed WASM glue_paths uses the same cap, so I wondered whether it is a convention or protects something specific here.

I ask because code-split apps often ship more files than that. From what I can measure on the open web, about a quarter of the sites that ship a bundle load more than 17 bundle files from one host, and one in ten load almost 90 or more.

If there is no deeper reason, either option could work:

Raise the cap and keep the 64 MiB total as the real bound.
Accept a directory as path, which matches what export_web_scripts writes. wakaru <dir> --unpack already scans .js, .mjs and .cjs files and skips hidden entries.

@pionxzh commented on GitHub (Oct 11, 2026): Thanks for #1686, @JingHao-Leon. I reviewed how the adapter uses Wakaru's provenance: each module binds to its own input snapshot, and its ranges are checked against that input's bytes. It all looks correct to me. One question about the cap. What does the 16-path limit protect? The 64 MiB total input already bounds memory and time. I noticed WASM `glue_paths` uses the same cap, so I wondered whether it is a convention or protects something specific here. I ask because code-split apps often ship more files than that. From what I can measure on the open web, about a quarter of the sites that ship a bundle load more than 17 bundle files from one host, and one in ten load almost 90 or more. If there is no deeper reason, either option could work: - Raise the cap and keep the 64 MiB total as the real bound. - Accept a directory as `path`, which matches what `export_web_scripts` writes. `wakaru <dir> --unpack` already scans `.js`, `.mjs` and `.cjs` files and skips hidden entries.

bsakaguchi referenced this issue

2026-10-16 21:13:52 +00:00

[PR #542] [MERGED] fix(inspector): preserve lossy Node discovery locations #2063

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:17 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#542

No description provided.

Rows
Columns

[GH-ISSUE #1605] [Feature] Recover related scripts together in recover_javascript_sources #542

Before submitting

Area

Current limitation

Use cases

Desired outcome and success criteria

Authority boundaries and non-goals

Alternatives considered

Possible direction

References

[GH-ISSUE #1605] [Feature] Recover related scripts together in `recover_javascript_sources` #542