[PR #801] [MERGED] feat(web): export captured scripts for JavaScript application analysis #2249

New issue

Closed

opened 2026-10-16 21:16:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:16:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/801
Author: @N0zoM1z0
Created: 10/6/2026
Status: ✅ Merged
Merged: 10/6/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/web-script-export

📝 Commits (3)

e879836 feat(web): export captured scripts for JavaScript application analysis
b1cdfaa test(web): include script export in independent effect audit
b17d0a8 fix(web): preserve capture limitation values in export manifests

📊 Changes

47 files changed (+2158 additions, -35 deletions)

View changed files

📝 .github/workflows/real-browser.yml (+7 -0)
📝 README.md (+21 -14)
📝 README_ar.md (+1 -1)
📝 README_ja.md (+1 -1)
📝 README_ko.md (+1 -1)
📝 README_zh.md (+1 -1)
📝 docs/architecture.mermaid (+1 -1)
📝 docs/browser-observation.md (+4 -0)
📝 docs/product-catalog.json (+16 -8)
📝 docs/testing.md (+7 -0)
📝 docs/verification/managed-conformance-manifest.json (+2 -2)
➕ docs/website-script-export.md (+86 -0)
📝 package.json (+2 -1)
➕ scripts/fixtures/browser-script-site.mjs (+49 -0)
➕ scripts/lib/browser-script-export-assertions.mjs (+70 -0)
➕ scripts/lib/browser-script-export-e2e.mjs (+192 -0)
📝 scripts/lib/catalog-builders.mjs (+8 -1)
📝 scripts/lib/catalog-core.mjs (+1 -0)
➕ scripts/verify-browser-scripts.mjs (+16 -0)
📝 scripts/verify-real-browser.mjs (+6 -0)

...and 27 more files

📄 Description

Summary

Closes #799. Refs #774; follows merged #797.

Add export_web_scripts / rea export-web-scripts to publish the exact JavaScript bytes already retained in a saved website capture for REA’s existing local application analysis.

Independent adapters for passive inspect_web_page Debugger sources and active capture_browser_scenario script response content; saved Evidence is authenticated before publication.
Pure path projection preserves ordinary unambiguous relative URL layouts per origin. Competing versions, query/fragment variants, unavailable competitors, and case/prefix/path collisions stay isolated, with explicit limitations.
Reuse SafeOutputTree for exclusive output, durable byte/digest readback, cancellation and owned-output rollback. Existing output roots and symlinks are preserved and rejected.
Full source-to-file manifest returned inline and persisted: capture/source digests, URLs, frame/script or transaction/event references, coverage, redaction/representation and unavailable reasons. No available bytes => null analysis_input.
Shared typed application workflow, named MCP contract, CLI command, effects audit, provider/family catalog registration and generated documents. No refetch, script execution, or new JavaScript engine.

MCP example

{"capture_path":"/analysis/capture.json","output_directory":"/analysis/site-export"}

Pass result.analysis_input to analyze_javascript_application when non-null. Capture sources first with include_script_sources: true, or scenario capture.network.response_body: true.

Verification

60 relevant focused tests passed across scoped runs across source/path projection, capture identity, filesystem publication/rollback, canonical contracts, SDK-converted input/output schemas and product/CLI catalog facts.
npm run check:fast and npm run docs:check passed; pre-commit/pre-push passed. Eight capture adapter regressions passed again after splitting projection helpers.
Full real Chrome153 verification passed, including active and passive capture → export → existing analysis through CLI and stdio MCP, along with existing popup/network/screenshot checks.
Packed the runtime feature head e879836d and installed into a separate /tmp directory with install scripts disabled. Installed CLI/MCP passed the real four-source workflow: exact bytes, manifest readback, resolved relative import, isolated query variants and zero implicit refetches. Focused process lineage verified no remaining descendants.
Rebased onto latest main894c30ef (#800); upstream Ghidra packaged-bridge changes have no overlap.

The full mixed external-browser lane reports process lineage unavailable for an external browser run token; it is not counted as a full lineage cleanup proof. The separate focused installed workflow has verified lineage.

Scope and interpretation

Export Evidence derives from the saved capture; application analysis describes the actual local exported artifacts. Debugger source and response bytes do not establish execution or a complete website. Browser import maps, root-relative/remote URL semantics, missing dependencies and source-map recovery remain explicit limitations. Redacted and binary bytes are retained exactly.

No real Hopper or Ghidra verification was performed; their provider code is unchanged by this feature.

CI follow-ups

CI exposed an omitted family in the independent contract/effect audit. Added WEB_SCRIPT_TOOL_CONTRACTS and explicit authority assertions; all six audit tests passed.
Reproduced a valid passive capture failing publication in the installed package: the capture accepts empty limitation strings, while the export manifest rejected them. Preserve the producer's string values; all 11 filesystem tests and five advertised SDK-schema tests passed after the correction.
Regenerated output contracts and catalog. Latest reviewed head: b17d0a8a277c11c13f645ca73abed868ff946573. All 20 CI checks passed on this exact head, including the real-browser and all package/install lanes. Merged as 472b72a53068a8e2ec78fe239d687023482daaa7 at 2026-10-06T21:24:32Z.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/801 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/6/2026 **Status:** ✅ Merged **Merged:** 10/6/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/web-script-export` --- ### 📝 Commits (3) - [`e879836`](https://github.com/morluto/rea/commit/e879836d2146943d5b57196994cdbc49c6e78849) feat(web): export captured scripts for JavaScript application analysis - [`b1cdfaa`](https://github.com/morluto/rea/commit/b1cdfaa4eb91f7341087838ad7625f326c137dd0) test(web): include script export in independent effect audit - [`b17d0a8`](https://github.com/morluto/rea/commit/b17d0a8a277c11c13f645ca73abed868ff946573) fix(web): preserve capture limitation values in export manifests ### 📊 Changes **47 files changed** (+2158 additions, -35 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/real-browser.yml` (+7 -0) 📝 `README.md` (+21 -14) 📝 `README_ar.md` (+1 -1) 📝 `README_ja.md` (+1 -1) 📝 `README_ko.md` (+1 -1) 📝 `README_zh.md` (+1 -1) 📝 `docs/architecture.mermaid` (+1 -1) 📝 `docs/browser-observation.md` (+4 -0) 📝 `docs/product-catalog.json` (+16 -8) 📝 `docs/testing.md` (+7 -0) 📝 `docs/verification/managed-conformance-manifest.json` (+2 -2) ➕ `docs/website-script-export.md` (+86 -0) 📝 `package.json` (+2 -1) ➕ `scripts/fixtures/browser-script-site.mjs` (+49 -0) ➕ `scripts/lib/browser-script-export-assertions.mjs` (+70 -0) ➕ `scripts/lib/browser-script-export-e2e.mjs` (+192 -0) 📝 `scripts/lib/catalog-builders.mjs` (+8 -1) 📝 `scripts/lib/catalog-core.mjs` (+1 -0) ➕ `scripts/verify-browser-scripts.mjs` (+16 -0) 📝 `scripts/verify-real-browser.mjs` (+6 -0) _...and 27 more files_ </details> ### 📄 Description ## Summary Closes #799. Refs #774; follows merged #797. Add `export_web_scripts` / `rea export-web-scripts` to publish the exact JavaScript bytes already retained in a saved website capture for REA’s existing local application analysis. - Independent adapters for passive `inspect_web_page` Debugger sources and active `capture_browser_scenario` script response content; saved Evidence is authenticated before publication. - Pure path projection preserves ordinary unambiguous relative URL layouts per origin. Competing versions, query/fragment variants, unavailable competitors, and case/prefix/path collisions stay isolated, with explicit limitations. - Reuse `SafeOutputTree` for exclusive output, durable byte/digest readback, cancellation and owned-output rollback. Existing output roots and symlinks are preserved and rejected. - Full source-to-file manifest returned inline and persisted: capture/source digests, URLs, frame/script or transaction/event references, coverage, redaction/representation and unavailable reasons. No available bytes => null `analysis_input`. - Shared typed application workflow, named MCP contract, CLI command, effects audit, provider/family catalog registration and generated documents. No refetch, script execution, or new JavaScript engine. ## MCP example ```json {"capture_path":"/analysis/capture.json","output_directory":"/analysis/site-export"} ``` Pass `result.analysis_input` to `analyze_javascript_application` when non-null. Capture sources first with `include_script_sources: true`, or scenario `capture.network.response_body: true`. ## Verification - 60 relevant focused tests passed across scoped runs across source/path projection, capture identity, filesystem publication/rollback, canonical contracts, SDK-converted input/output schemas and product/CLI catalog facts. - `npm run check:fast` and `npm run docs:check` passed; pre-commit/pre-push passed. Eight capture adapter regressions passed again after splitting projection helpers. - Full real Chrome153 verification passed, including active and passive capture → export → existing analysis through CLI and stdio MCP, along with existing popup/network/screenshot checks. - Packed the runtime feature head `e879836d` and installed into a separate `/tmp` directory with install scripts disabled. Installed CLI/MCP passed the real four-source workflow: exact bytes, manifest readback, resolved relative import, isolated query variants and zero implicit refetches. Focused process lineage verified no remaining descendants. - Rebased onto latest main894c30ef (#800); upstream Ghidra packaged-bridge changes have no overlap. The full mixed external-browser lane reports process lineage unavailable for an external browser run token; it is not counted as a full lineage cleanup proof. The separate focused installed workflow has verified lineage. ## Scope and interpretation Export Evidence derives from the saved capture; application analysis describes the actual local exported artifacts. Debugger source and response bytes do not establish execution or a complete website. Browser import maps, root-relative/remote URL semantics, missing dependencies and source-map recovery remain explicit limitations. Redacted and binary bytes are retained exactly. No real Hopper or Ghidra verification was performed; their provider code is unchanged by this feature. ## CI follow-ups - CI exposed an omitted family in the independent contract/effect audit. Added `WEB_SCRIPT_TOOL_CONTRACTS` and explicit authority assertions; all six audit tests passed. - Reproduced a valid passive capture failing publication in the installed package: the capture accepts empty limitation strings, while the export manifest rejected them. Preserve the producer's string values; all 11 filesystem tests and five advertised SDK-schema tests passed after the correction. - Regenerated output contracts and catalog. Latest reviewed head: `b17d0a8a277c11c13f645ca73abed868ff946573`. All 20 CI checks passed on this exact head, including the real-browser and all package/install lanes. Merged as `472b72a53068a8e2ec78fe239d687023482daaa7` at 2026-10-06T21:24:32Z. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-16 21:16:16 +00:00