morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #774] [Feature] Website reverse engineering roadmap: network evidence, captured assets, and modular adapters #339

New issue

Open

opened 2026-10-15 18:24:04 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:24:04 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/774

Outcome

Make website reverse engineering a coherent REA workflow: observe selected interactions and network transactions, retain the actual shipped assets, analyze them with existing JavaScript tools, and compare evidence across explicitly selected deployments.

This consolidates the next implementation steps under the broader #39 capability direction. It does not replace #39 or imply that unobserved browser behavior or non-shipped server source has been reconstructed.

Research: 2026-10-06, main 9622b4b4cf0b2eb92305afd52fd67e932d57c753; implementation baseline refreshed to 554e9b16. Source/contracts, open issues/PRs and primary upstream documentation were inspected. The research did not install upstream tools, benchmark them, or run new live-site verification.

Foundations already shipped

Passive CDP page/script/resource/network/console/storage/worker observation, optional JSON body shapes, screenshots, source maps and bundle findings.
Playwright launch/connect scenarios with selected actions, initial/per-step/final artifacts and network/WebSocket/popup events.
Local directory/ASAR JavaScript structural and semantic graphs, feature/semantic tracing, export-shape and version comparisons.
Static/runtime reconciliation with exact captured-source digests or explicitly weaker URL/path mappings.
Shared CLI/MCP contracts, Evidence provenance, owned-process cleanup and real-browser fixture verification.

Relevant sources: docs/browser-observation.md, docs/browser-scenario-contract.md, docs/javascript-artifact-reconstruction.md, docs/javascript-runtime-reconciliation.md, src/browser/PlaywrightScenarioEvents.ts, src/domain/webBundleAnalysis.ts, and src/domain/javascriptRuntimeReconciliationParsing.ts.

First implementation batches

1. Network transaction identity and selected content evidence

Question: which observed request produced this response, and what actual data was exchanged?

Active scenario events currently preserve URL/method/status/header names, but not request identity, content or initiator locations. Two parallel requests to the same URL cannot be paired exactly from those fields. Passive CDP already supplies IDs/initiators and optional value-free JSON shapes; extend/reuse existing capabilities rather than add another browser controller.

Preserve capture-scoped transaction identity across request, response, failure, retry and redirect observations.
Allow explicit content capture with exact representation, MIME/encoding, byte count, SHA-256 and unavailable/unfinished/failure states. Browser-decoded bodies must not be described as raw compressed wire bytes.
Preserve ordinary selected local values; remove transport authentication credentials and explicitly declared secrets. Keep narrower metadata/shape-only selection available.
Preserve producer source/initiator locations where available; do not infer causality from step timing or URL equality.
Cover same-URL concurrency, redirects, delayed completion, failure/cancellation, binary/text/JSON and popup attribution with deterministic regressions and real-browser CLI/MCP verification.

2. Captured website assets into existing JavaScript analysis

Question: which captured module implements this feature, and what changed between deployments?

analyze_web_bundle returns bundle findings and source summaries; advanced application workflows require authenticated JavaScript application-analysis Evidence with structural and semantic graphs. These are distinct producer contracts.

Materialize explicitly selected captured assets with content identity, source capture references and URL-to-artifact mapping; retain missing/lazy assets as unknowns.
Reuse artifact inventory/publication and the existing JavaScript projector rather than create a second analyzer.
Admit capture-derived graph provenance truthfully. Do not relabel live capture as the existing shipped-artifact producer.
Make existing trace/compare/reconcile workflows usable on the resulting artifact/graph.
Verify byte identity, repeated URL variants, generated/original source distinction, absent source-map content, incomplete captures and offline reuse. Refetching the same URL is not proof of the same captured bytes.

Organization and extension boundaries

Organize implementation around named responsibilities: provider-neutral contracts/semantics, browser capture adapters, shared application workflows and artifact publication. Extract network transaction/content collection from the general scenario event collector. Keep Playwright/CDP and upstream formats outside domain/application code. Share lifecycle, cancellation, redaction and artifact identity primitives; avoid a repository-wide rename-only refactor.

Optional integrations must fail independently, identify their engine/version/coverage, and preserve target-free startup. Pin upstream source or release/checksum, retain upstream code unchanged when vendored, and put REA behavior in adapters. Do not globally install or upgrade browsers, Node/toolchains, proxies or trust-store settings.

This follows the broader modularization direction in #740 without requiring that entire roadmap first.

Reusable upstream references

Upstream	Useful capability	Integration boundary
Wakaru	Bundle splitting and readable syntax recovery; output source maps	Optional static recovery adapter; current Apache-2.0, Rust/standalone binaries. Rollup/Vite splitting is heuristic; not a general deobfuscator.
webcrack	webpack/browserify recovery and obfuscator.io decoding	MIT; Node 22/24 and isolated-vm. Some deobfuscation evaluates code, so it cannot inherit an execution-free claim.
Chrome DevTools MCP	Request content, source-mapped console stacks, CSS and profiling evidence	Apache-2.0; borrow representations/adapters. REA already uses CDP. Disable upstream telemetry/CrUX access for a local-only integration.
Playwright MCP	Selected request/body inspection and agent-facing browser results	Reuse existing Playwright integration; avoid duplicate controllers and redundant permission flags.
mitmproxy	Saved-flow/HTTP/WebSocket evidence	MIT; start with caller-provided offline flow/HAR import. Live proxy ownership is separate.
Browsertrix Crawler	Web asset archiving and WARC/WACZ experience	AGPL-3.0-or-later; prefer format interoperability or an optional caller-installed adapter. Not a mandatory Docker/crawler dependency.
webappanalyzer	Maintained Wappalyzer technology patterns	GPL-3.0; evaluate redistribution before copying the database. Pattern matches remain inferences.
rrweb	DOM/session event timelines	Optional import/instrumentation; not source or backend reconstruction.
WABT	WASM validation, imports/exports, sections and WAT	Apache-2.0; optional artifact inspection adapter, preserving exact binary identity and derived-output authority.

Upstream documentation: Wakaru supported inputs, source maps, webcrack evaluation options, DevTools tool reference, mitmproxy examples.

Subsequent focused capabilities

Optional JavaScript recovery adapter with original/derived digests, transform options, warnings and source mappings; verify exact supported bundlers on pinned upstream versions.
Selected execution coverage/source attribution: loaded scripts are not executed handlers; window coverage does not prove causality or global absence.
Offline traffic imports and selected protocol decoding with entry-level provenance and explicit unsupported content.
WASM artifact inspection and evidence-backed JS glue relationships.
Selected element/CSS rule provenance and documented framework payload inspection when an analyst workflow needs them.

Existing work to link, not duplicate

#623: incremental canonical hashing has been fixed via #646; whole-result formatter/transport limits remain separate.
#746: large application latency and unreadable-output diagnostics, still requiring current-main reproduction.
#723: MCP discovery/result repetition and usable complete Evidence.
#725: property presence separate from unknown values/variant pairing.
#64/#375: Electron main/preload/IPC/process lifecycle, distinct from website capture.
Open PR snapshot includes #763 cookie URL fingerprints and #752/#756/#758/#764 semantic corrections; do not duplicate contributor fixes.

Delivery and acceptance

Deliver independently reviewable PRs in dependency order: network evidence, then asset-to-analysis integration. For each: focused regression tests, relevant static/generated-contract checks, real local-browser CLI/MCP and packaged-artifact proof, CI, final source/contract/lifecycle review, and merge only at the verified head. Keep CPU/memory bounded and avoid repeating unrelated full local gates.

Retain selected target/action control, truthful observed/derived/inferred/unknown states, evidence identity and explicit incomplete coverage. The request itself expresses intent. Reuse ordinary scripts/local fixture servers; do not resurrect the replay/interception orchestration removed in #572.

Delivery progress

Network lifecycle and selected request/response/header evidence: #797 (merged, all CI passed).
Captured website scripts → existing local JavaScript analysis: #799 closed by #801 (merged, all 20 CI checks passed). Real active/passive and installed-package CLI/MCP verification passed.

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/774 ## Outcome Make website reverse engineering a coherent REA workflow: observe selected interactions and network transactions, retain the actual shipped assets, analyze them with existing JavaScript tools, and compare evidence across explicitly selected deployments. This consolidates the next implementation steps under the broader #39 capability direction. It does not replace #39 or imply that unobserved browser behavior or non-shipped server source has been reconstructed. Research: 2026-10-06, main `9622b4b4cf0b2eb92305afd52fd67e932d57c753`; implementation baseline refreshed to `554e9b16`. Source/contracts, open issues/PRs and primary upstream documentation were inspected. The research did not install upstream tools, benchmark them, or run new live-site verification. ## Foundations already shipped - Passive CDP page/script/resource/network/console/storage/worker observation, optional JSON body shapes, screenshots, source maps and bundle findings. - Playwright launch/connect scenarios with selected actions, initial/per-step/final artifacts and network/WebSocket/popup events. - Local directory/ASAR JavaScript structural and semantic graphs, feature/semantic tracing, export-shape and version comparisons. - Static/runtime reconciliation with exact captured-source digests or explicitly weaker URL/path mappings. - Shared CLI/MCP contracts, Evidence provenance, owned-process cleanup and real-browser fixture verification. Relevant sources: `docs/browser-observation.md`, `docs/browser-scenario-contract.md`, `docs/javascript-artifact-reconstruction.md`, `docs/javascript-runtime-reconciliation.md`, `src/browser/PlaywrightScenarioEvents.ts`, `src/domain/webBundleAnalysis.ts`, and `src/domain/javascriptRuntimeReconciliationParsing.ts`. ## First implementation batches ### 1. Network transaction identity and selected content evidence **Question:** which observed request produced this response, and what actual data was exchanged? Active scenario events currently preserve URL/method/status/header names, but not request identity, content or initiator locations. Two parallel requests to the same URL cannot be paired exactly from those fields. Passive CDP already supplies IDs/initiators and optional value-free JSON shapes; extend/reuse existing capabilities rather than add another browser controller. - [ ] Preserve capture-scoped transaction identity across request, response, failure, retry and redirect observations. - [ ] Allow explicit content capture with exact representation, MIME/encoding, byte count, SHA-256 and unavailable/unfinished/failure states. Browser-decoded bodies must not be described as raw compressed wire bytes. - [ ] Preserve ordinary selected local values; remove transport authentication credentials and explicitly declared secrets. Keep narrower metadata/shape-only selection available. - [ ] Preserve producer source/initiator locations where available; do not infer causality from step timing or URL equality. - [ ] Cover same-URL concurrency, redirects, delayed completion, failure/cancellation, binary/text/JSON and popup attribution with deterministic regressions and real-browser CLI/MCP verification. ### 2. Captured website assets into existing JavaScript analysis **Question:** which captured module implements this feature, and what changed between deployments? `analyze_web_bundle` returns bundle findings and source summaries; advanced application workflows require authenticated JavaScript application-analysis Evidence with structural and semantic graphs. These are distinct producer contracts. - [ ] Materialize explicitly selected captured assets with content identity, source capture references and URL-to-artifact mapping; retain missing/lazy assets as unknowns. - [ ] Reuse artifact inventory/publication and the existing JavaScript projector rather than create a second analyzer. - [ ] Admit capture-derived graph provenance truthfully. Do not relabel live capture as the existing shipped-artifact producer. - [ ] Make existing trace/compare/reconcile workflows usable on the resulting artifact/graph. - [ ] Verify byte identity, repeated URL variants, generated/original source distinction, absent source-map content, incomplete captures and offline reuse. Refetching the same URL is not proof of the same captured bytes. ## Organization and extension boundaries Organize implementation around named responsibilities: provider-neutral contracts/semantics, browser capture adapters, shared application workflows and artifact publication. Extract network transaction/content collection from the general scenario event collector. Keep Playwright/CDP and upstream formats outside domain/application code. Share lifecycle, cancellation, redaction and artifact identity primitives; avoid a repository-wide rename-only refactor. Optional integrations must fail independently, identify their engine/version/coverage, and preserve target-free startup. Pin upstream source or release/checksum, retain upstream code unchanged when vendored, and put REA behavior in adapters. Do not globally install or upgrade browsers, Node/toolchains, proxies or trust-store settings. This follows the broader modularization direction in #740 without requiring that entire roadmap first. ## Reusable upstream references | Upstream | Useful capability | Integration boundary | | --- | --- | --- | | [Wakaru](https://github.com/pionxzh/wakaru) | Bundle splitting and readable syntax recovery; output source maps | Optional static recovery adapter; current Apache-2.0, Rust/standalone binaries. Rollup/Vite splitting is heuristic; not a general deobfuscator. | | [webcrack](https://github.com/j4k0xb/webcrack) | webpack/browserify recovery and obfuscator.io decoding | MIT; Node 22/24 and isolated-vm. Some deobfuscation evaluates code, so it cannot inherit an execution-free claim. | | [Chrome DevTools MCP](https://github.com/ChromeDevTools/chrome-devtools-mcp) | Request content, source-mapped console stacks, CSS and profiling evidence | Apache-2.0; borrow representations/adapters. REA already uses CDP. Disable upstream telemetry/CrUX access for a local-only integration. | | [Playwright MCP](https://github.com/microsoft/playwright-mcp) | Selected request/body inspection and agent-facing browser results | Reuse existing Playwright integration; avoid duplicate controllers and redundant permission flags. | | [mitmproxy](https://github.com/mitmproxy/mitmproxy) | Saved-flow/HTTP/WebSocket evidence | MIT; start with caller-provided offline flow/HAR import. Live proxy ownership is separate. | | [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) | Web asset archiving and WARC/WACZ experience | AGPL-3.0-or-later; prefer format interoperability or an optional caller-installed adapter. Not a mandatory Docker/crawler dependency. | | [webappanalyzer](https://github.com/enthec/webappanalyzer) | Maintained Wappalyzer technology patterns | GPL-3.0; evaluate redistribution before copying the database. Pattern matches remain inferences. | | [rrweb](https://github.com/rrweb-io/rrweb) | DOM/session event timelines | Optional import/instrumentation; not source or backend reconstruction. | | [WABT](https://github.com/WebAssembly/wabt) | WASM validation, imports/exports, sections and WAT | Apache-2.0; optional artifact inspection adapter, preserving exact binary identity and derived-output authority. | Upstream documentation: [Wakaru supported inputs](https://wakarujs.com/docs/reference/supported-inputs), [source maps](https://wakarujs.com/docs/guides/source-maps), [webcrack evaluation options](https://webcrack.netlify.app/docs/guide/api.html), [DevTools tool reference](https://github.com/ChromeDevTools/chrome-devtools-mcp/blob/main/docs/tool-reference.md), [mitmproxy examples](https://docs.mitmproxy.org/stable/addons/examples/). ## Subsequent focused capabilities - [ ] Optional JavaScript recovery adapter with original/derived digests, transform options, warnings and source mappings; verify exact supported bundlers on pinned upstream versions. - [ ] Selected execution coverage/source attribution: loaded scripts are not executed handlers; window coverage does not prove causality or global absence. - [ ] Offline traffic imports and selected protocol decoding with entry-level provenance and explicit unsupported content. - [ ] WASM artifact inspection and evidence-backed JS glue relationships. - [ ] Selected element/CSS rule provenance and documented framework payload inspection when an analyst workflow needs them. ## Existing work to link, not duplicate - #623: incremental canonical hashing has been fixed via #646; whole-result formatter/transport limits remain separate. - #746: large application latency and unreadable-output diagnostics, still requiring current-main reproduction. - #723: MCP discovery/result repetition and usable complete Evidence. - #725: property presence separate from unknown values/variant pairing. - #64/#375: Electron main/preload/IPC/process lifecycle, distinct from website capture. - Open PR snapshot includes #763 cookie URL fingerprints and #752/#756/#758/#764 semantic corrections; do not duplicate contributor fixes. ## Delivery and acceptance Deliver independently reviewable PRs in dependency order: network evidence, then asset-to-analysis integration. For each: focused regression tests, relevant static/generated-contract checks, real local-browser CLI/MCP and packaged-artifact proof, CI, final source/contract/lifecycle review, and merge only at the verified head. Keep CPU/memory bounded and avoid repeating unrelated full local gates. Retain selected target/action control, truthful observed/derived/inferred/unknown states, evidence identity and explicit incomplete coverage. The request itself expresses intent. Reuse ordinary scripts/local fixture servers; do not resurrect the replay/interception orchestration removed in #572. ## Delivery progress - [x] Network lifecycle and selected request/response/header evidence: #797 (merged, all CI passed). - [x] Captured website scripts → existing local JavaScript analysis: #799 closed by #801 (merged, all 20 CI checks passed). Real active/passive and installed-package CLI/MCP verification passed.

bsakaguchi added the

labels

2026-10-15 18:24:04 +00:00

bsakaguchi commented

2026-10-15 18:24:05 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 7, 2026):

Website implementation progress and handoff

Merged and verified:

#825: form destination evidence and JavaScript mutation uncertainty fixes.
#848: Wakaru module recovery adapter.
#879: captured website module-import resolution.
#897: captured source-map/source-location evidence.
#917: runtime execution attribution and event-listener inspection.

Current increment being closed out: offline HAR/native mitmproxy inspection under #882. It has separate format adapters, a shared application port, inline CLI/MCP Evidence and bounded owned decoder processes. Current focused verification passed 39 tests / 8 files; real upstream mitmproxy 12.2.3 fixtures passed 14 public cases in both the working build and an isolated installed npm package. Cancellation, opaque extensions and verifier failure cleanup were also checked. Ready-for-review PR: #992, rebased onto current main including #963. PR CI and final review/merge are pending; verification above was rerun on this head.

Remaining website work available for focused follow-ups:

#882: live selected binary WebSocket/SSE payload evidence and corresponding real-browser coverage. The offline increment will not close the entire issue.
#883: worker/service-worker source and runtime ownership.
#884: WABT/WebAssembly artifact inspection and website glue evidence.

After closing the current historical-capture increment, our implementation work will move to #969 (modular binary diagnostics/debugger adapters using pwntools/pwndbg) and #972 (modular EVM bytecode analysis/execution evidence). The remaining website issues stay open for contributors to take over. Preserve observed/derived/unknown distinctions, upstream codecs/providers, exact source/artifact identities, shared CLI/MCP contracts and matching real-provider verification in follow-ups.

@N0zoM1z0 commented on GitHub (Oct 7, 2026): ## Website implementation progress and handoff Merged and verified: - #825: form destination evidence and JavaScript mutation uncertainty fixes. - #848: Wakaru module recovery adapter. - #879: captured website module-import resolution. - #897: captured source-map/source-location evidence. - #917: runtime execution attribution and event-listener inspection. **Current increment being closed out:** offline HAR/native mitmproxy inspection under #882. It has separate format adapters, a shared application port, inline CLI/MCP Evidence and bounded owned decoder processes. Current focused verification passed 39 tests / 8 files; real upstream mitmproxy 12.2.3 fixtures passed 14 public cases in both the working build and an isolated installed npm package. Cancellation, opaque extensions and verifier failure cleanup were also checked. Ready-for-review PR: #992, rebased onto current main including #963. PR CI and final review/merge are pending; verification above was rerun on this head. **Remaining website work available for focused follow-ups:** - #882: live selected binary WebSocket/SSE payload evidence and corresponding real-browser coverage. The offline increment will not close the entire issue. - #883: worker/service-worker source and runtime ownership. - #884: WABT/WebAssembly artifact inspection and website glue evidence. After closing the current historical-capture increment, our implementation work will move to **#969 (modular binary diagnostics/debugger adapters using pwntools/pwndbg)** and **#972 (modular EVM bytecode analysis/execution evidence)**. The remaining website issues stay open for contributors to take over. Preserve observed/derived/unknown distinctions, upstream codecs/providers, exact source/artifact identities, shared CLI/MCP contracts and matching real-provider verification in follow-ups.

bsakaguchi commented

2026-10-15 18:24:05 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 7, 2026):

Website work handoff: #992 is merged after 29/29 final-head CI checks, a final head-bound review, all review findings addressed, 130 focused tests and 144 fresh-installed real CLI/MCP cases (plus seven native failure classifications). Historical HAR/native capture evidence now joins the previously merged network/assets, Wakaru, module-resolution, source-map and runtime-attribution increments.

Remaining independently actionable website gaps:

#882: live binary WebSocket/SSE retention; historical capture inspection does not close the live-capture requirement.
#883: workers/service workers, source/runtime ownership and attribution.
#884: WASM/WABT artifact inspection and JS glue relationships.
#1036: preserve prototype-named JSON properties through the shared typed Evidence boundary; historical captures currently return an explicit unsupported representation rather than silently lose data.

I am handing off further website expansion and moving to #969 general modular native binary diagnostics/debugger integrations (pwntools/pwndbg; CTF is only one showcase), followed by #972 EVM bytecode capabilities. The first native increment is explicit offline ELF layout/static mitigation evidence via unchanged upstream adapters, with recorded cores and GDB/pwndbg runtime authority handled separately. These upcoming capabilities are not claimed as shipped until their own installed CLI/MCP tests, review and CI pass.

@N0zoM1z0 commented on GitHub (Oct 7, 2026): Website work handoff: #992 is merged after 29/29 final-head CI checks, a final head-bound review, all review findings addressed, 130 focused tests and 144 fresh-installed real CLI/MCP cases (plus seven native failure classifications). Historical HAR/native capture evidence now joins the previously merged network/assets, Wakaru, module-resolution, source-map and runtime-attribution increments. Remaining independently actionable website gaps: - #882: live binary WebSocket/SSE retention; historical capture inspection does not close the live-capture requirement. - #883: workers/service workers, source/runtime ownership and attribution. - #884: WASM/WABT artifact inspection and JS glue relationships. - #1036: preserve prototype-named JSON properties through the shared typed Evidence boundary; historical captures currently return an explicit unsupported representation rather than silently lose data. I am handing off further website expansion and moving to #969 general modular native binary diagnostics/debugger integrations (pwntools/pwndbg; CTF is only one showcase), followed by #972 EVM bytecode capabilities. The first native increment is explicit offline ELF layout/static mitigation evidence via unchanged upstream adapters, with recorded cores and GDB/pwndbg runtime authority handled separately. These upcoming capabilities are not claimed as shipped until their own installed CLI/MCP tests, review and CI pass.

bsakaguchi commented

2026-10-15 18:24:05 +00:00

Author

Owner

@rozaqabdul656 commented on GitHub (Oct 8, 2026):

I'd like to help with the first implementation batch: network transaction identity and selected content evidence.

To keep this reviewable, I propose starting with a focused slice:

preserve capture-scoped transaction identity across request, response, failure, redirect and retry events;
pair concurrent requests to the same URL correctly;
retain producer/initiator locations where available;
add deterministic regressions for same-URL concurrency, redirects and failed requests.

I would leave broader body capture, redaction policy and captured-asset JavaScript analysis for follow-up PRs unless maintainers prefer a different boundary.

Is this slice available, and does this contract fit the current browser observation architecture?

@rozaqabdul656 commented on GitHub (Oct 8, 2026): I'd like to help with the first implementation batch: network transaction identity and selected content evidence. To keep this reviewable, I propose starting with a focused slice: - preserve capture-scoped transaction identity across request, response, failure, redirect and retry events; - pair concurrent requests to the same URL correctly; - retain producer/initiator locations where available; - add deterministic regressions for same-URL concurrency, redirects and failed requests. I would leave broader body capture, redaction policy and captured-asset JavaScript analysis for follow-up PRs unless maintainers prefer a different boundary. Is this slice available, and does this contract fit the current browser observation architecture?

bsakaguchi referenced this issue

2026-10-15 18:28:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:03 +00:00

bsakaguchi referenced this issue

2026-10-31 00:56:49 +00:00

[PR #1942] [MERGED] fix(verification): align Linux Hopper setup lane with current contracts #4925