[PR #129] [MERGED] feat: add passive website reverse engineering #1851

New issue

Closed

opened 2026-10-16 21:11:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:11:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/129
Author: @N0zoM1z0
Created: 7/14/2026
Status: ✅ Merged
Merged: 7/14/2026
Merged by: @morluto

Base: main ← Head: feat/web-reverse-engineering

📝 Commits (5)

2c1ceab feat: add passive website reverse engineering
5279d77 fix(browser): scope CDP events and fail closed
7d5621d test(cli): allow cold-start integration timing
8481be9 fix(browser): drop disallowed redirect evidence
dfc06cf fix(browser): scope workers and binary frame sizes

📊 Changes

73 files changed (+5559 additions, -188 deletions)

View changed files

📝 AGENTS.md (+9 -3)
📝 CONTRIBUTING.md (+2 -2)
📝 README.md (+30 -12)
📝 README_ar.md (+11 -10)
📝 README_ja.md (+4 -3)
📝 README_ko.md (+4 -3)
📝 README_zh.md (+4 -3)
📝 docs/architecture.mermaid (+10 -3)
➕ docs/browser-observation.md (+121 -0)
📝 docs/error-contract.schema.json (+6 -0)
📝 docs/mcp-prompts.md (+1 -1)
📝 package.json (+1 -0)
📝 scripts/rea.mjs (+1 -1)
📝 scripts/verify-package.mjs (+4 -0)
📝 scripts/verify-published-package.mjs (+2 -2)
➕ scripts/verify-real-browser.mjs (+286 -0)
📝 scripts/verify-real-hopper.mjs (+1 -1)
📝 skills/rea-analysis/SKILL.md (+18 -3)
➕ src/application/BrowserEvidence.ts (+61 -0)
➕ src/application/BrowserObservationPort.ts (+23 -0)

...and 53 more files

📄 Description

Summary

This PR adds provider-neutral, passive website reverse engineering through a user-owned Chrome-family browser's Chrome DevTools Protocol endpoint.

Add list_browser_targets and inspect_web_page MCP tools, with equivalent list-browser-targets and inspect-web-page CLI commands.
Return deterministic Evidence v2 records for browser discovery and page inspection, and retain MCP inspection evidence as session resources.
Capture bounded page/frame metadata, sanitized URLs, DOM structure, accessibility nodes, scripts, resources, workers, network metadata, console metadata, WebSocket metadata, and optional storage key names or script sources.
Add the browser_observe permission capability, exact-origin policy scopes, configuration parsing, availability reporting, catalog identity, documentation, and package verification coverage.
Increase the public MCP inventory from 68 to 70 tools without changing the existing Hopper, native, artifact, or session boundaries.

Motivation

REA currently has strong native and artifact analysis capabilities but no shipped provider for inspecting web applications. This implementation establishes a conservative first web-analysis boundary: it observes an already-running browser owned by the user, keeps CDP-specific mechanics in an adapter, and exposes the same evidence contract through CLI and MCP.

The initial scope is intentionally passive. It provides useful structural and runtime evidence without turning REA into a general-purpose remote-control or arbitrary JavaScript execution surface.

Behavior and security model

Browser observation is disabled by default and requires all three configuration inputs:

REA_BROWSER_OBSERVE_ENABLED=true
REA_BROWSER_CDP_ENDPOINTS_JSON='["http://127.0.0.1:9222"]'
REA_BROWSER_ALLOWED_ORIGINS_JSON='["http://127.0.0.1:3000"]'

CDP endpoints must be literal loopback HTTP URLs with explicit ports; hostnames, remote endpoints, credentials, query strings, and fragments are rejected.
Page access uses exact HTTP(S) origins. Target discovery excludes pages outside the configured and request-approved origin intersection.
Every operation requires explicit approval through the existing permission authority. Policy explanations include endpoint/origin and loopback-or-external network scopes.
Inspection never navigates, clicks, types, closes a page, closes the browser, or calls Runtime.evaluate.
URL credentials are removed, query values are replaced with redaction markers, and fragments are discarded.
Results never retain HTTP headers, request/response bodies, cookies, storage values, console argument values, or WebSocket payloads.
Script source capture and storage key-name capture are independently opt-in. Source maps are reported as metadata but are not fetched.
URL-less script and console observations are omitted when their origin cannot be proven.
Every retained collection and transport payload is bounded. CDP requests support cancellation and timeouts, malformed unsolicited messages poison the connection, and pending-request/concurrency limits fail closed.
Main-frame origin is checked before attach-sensitive capture, immediately before capture, during capture, and after capture. Navigation invalidates and discards the result, including same-origin target changes.
Limitations explicitly report attach-time visibility, unsupported optional CDP domains, truncation, omitted cross-origin data, and observations unavailable before attachment.

Architecture and contracts

Domain schemas and URL/origin normalization live in src/domain/browserObservation.ts.
Application authorization and Evidence v2 composition remain provider-neutral.
HTTP/WebSocket discovery, CDP correlation, capture, sanitization, and protocol handling live under src/browser/.
MCP and CLI adapters consume the same application operations and public schemas.
The target-free server can expose all 70 tools without a Hopper target; browser availability is reported independently from native-analysis availability.
docs/browser-observation.md documents setup, output categories, explicit limits, privacy behavior, and the threat model.

Verification

npm run check
- TypeScript strict typecheck passed.
- oxlint completed with no errors.
- Prettier check passed.
- 84 test files and 559 tests passed.
- Coverage: 79.83% statements, 69.38% branches, 81.99% functions, 82.06% lines.
npx vitest run tests/browserCli.test.ts tests/browserMcp.test.ts --coverage.enabled=false --retry=0
- 4/4 CLI and MCP parity tests passed without retries.
npm run verify:package
- Verified the packed CLI, setup transaction, backups/readback/recovery, skill, 70-tool target-free MCP server, 6 prompts, evidence resources, and target lifecycle in isolation.
REA_BROWSER_EXECUTABLE=/tmp/rea-browser-verification/chrome-linux64/chrome npm run verify:browser
- Passed against Chrome 150.0.7871.115 with real HTTP/WebSocket CDP.
- Observed 13 DOM nodes, 10 accessibility nodes, 1 script, 6 network requests, 7 console events, and 12 WebSocket events while verifying sensitive values were absent.
npm run knip
npm run jscpd (0.93% duplicated lines, below the configured threshold)
npm run scan:todos
git diff --check
Staged path, large-blob, symlink, binary, and credential-signature checks passed.

Real Hopper verification was not performed because this environment is Linux and has no real Hopper installation. Existing Hopper behavior was exercised by the full test suite and isolated package verifier; this PR does not route browser work through Hopper.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/129 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 7/14/2026 **Status:** ✅ Merged **Merged:** 7/14/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `feat/web-reverse-engineering` --- ### 📝 Commits (5) - [`2c1ceab`](https://github.com/morluto/rea/commit/2c1ceabbf1f117daafe166e1134b47737b38e1f6) feat: add passive website reverse engineering - [`5279d77`](https://github.com/morluto/rea/commit/5279d7755c74df579028ce51b4bc53b089bb2749) fix(browser): scope CDP events and fail closed - [`7d5621d`](https://github.com/morluto/rea/commit/7d5621dfbc4333f99846cbe52227d3b4f53ee0cd) test(cli): allow cold-start integration timing - [`8481be9`](https://github.com/morluto/rea/commit/8481be9c9709b13f572fefefcf8508d9e2605386) fix(browser): drop disallowed redirect evidence - [`dfc06cf`](https://github.com/morluto/rea/commit/dfc06cf0ad4a39719ef74846cb83f05e7b6f6289) fix(browser): scope workers and binary frame sizes ### 📊 Changes **73 files changed** (+5559 additions, -188 deletions) <details> <summary>View changed files</summary> 📝 `AGENTS.md` (+9 -3) 📝 `CONTRIBUTING.md` (+2 -2) 📝 `README.md` (+30 -12) 📝 `README_ar.md` (+11 -10) 📝 `README_ja.md` (+4 -3) 📝 `README_ko.md` (+4 -3) 📝 `README_zh.md` (+4 -3) 📝 `docs/architecture.mermaid` (+10 -3) ➕ `docs/browser-observation.md` (+121 -0) 📝 `docs/error-contract.schema.json` (+6 -0) 📝 `docs/mcp-prompts.md` (+1 -1) 📝 `package.json` (+1 -0) 📝 `scripts/rea.mjs` (+1 -1) 📝 `scripts/verify-package.mjs` (+4 -0) 📝 `scripts/verify-published-package.mjs` (+2 -2) ➕ `scripts/verify-real-browser.mjs` (+286 -0) 📝 `scripts/verify-real-hopper.mjs` (+1 -1) 📝 `skills/rea-analysis/SKILL.md` (+18 -3) ➕ `src/application/BrowserEvidence.ts` (+61 -0) ➕ `src/application/BrowserObservationPort.ts` (+23 -0) _...and 53 more files_ </details> ### 📄 Description ## Summary This PR adds provider-neutral, passive website reverse engineering through a user-owned Chrome-family browser's Chrome DevTools Protocol endpoint. - Add `list_browser_targets` and `inspect_web_page` MCP tools, with equivalent `list-browser-targets` and `inspect-web-page` CLI commands. - Return deterministic Evidence v2 records for browser discovery and page inspection, and retain MCP inspection evidence as session resources. - Capture bounded page/frame metadata, sanitized URLs, DOM structure, accessibility nodes, scripts, resources, workers, network metadata, console metadata, WebSocket metadata, and optional storage key names or script sources. - Add the `browser_observe` permission capability, exact-origin policy scopes, configuration parsing, availability reporting, catalog identity, documentation, and package verification coverage. - Increase the public MCP inventory from 68 to 70 tools without changing the existing Hopper, native, artifact, or session boundaries. ## Motivation REA currently has strong native and artifact analysis capabilities but no shipped provider for inspecting web applications. This implementation establishes a conservative first web-analysis boundary: it observes an already-running browser owned by the user, keeps CDP-specific mechanics in an adapter, and exposes the same evidence contract through CLI and MCP. The initial scope is intentionally passive. It provides useful structural and runtime evidence without turning REA into a general-purpose remote-control or arbitrary JavaScript execution surface. ## Behavior and security model Browser observation is disabled by default and requires all three configuration inputs: ```bash REA_BROWSER_OBSERVE_ENABLED=true REA_BROWSER_CDP_ENDPOINTS_JSON='["http://127.0.0.1:9222"]' REA_BROWSER_ALLOWED_ORIGINS_JSON='["http://127.0.0.1:3000"]' ``` - CDP endpoints must be literal loopback HTTP URLs with explicit ports; hostnames, remote endpoints, credentials, query strings, and fragments are rejected. - Page access uses exact HTTP(S) origins. Target discovery excludes pages outside the configured and request-approved origin intersection. - Every operation requires explicit approval through the existing permission authority. Policy explanations include endpoint/origin and loopback-or-external network scopes. - Inspection never navigates, clicks, types, closes a page, closes the browser, or calls `Runtime.evaluate`. - URL credentials are removed, query values are replaced with redaction markers, and fragments are discarded. - Results never retain HTTP headers, request/response bodies, cookies, storage values, console argument values, or WebSocket payloads. - Script source capture and storage key-name capture are independently opt-in. Source maps are reported as metadata but are not fetched. - URL-less script and console observations are omitted when their origin cannot be proven. - Every retained collection and transport payload is bounded. CDP requests support cancellation and timeouts, malformed unsolicited messages poison the connection, and pending-request/concurrency limits fail closed. - Main-frame origin is checked before attach-sensitive capture, immediately before capture, during capture, and after capture. Navigation invalidates and discards the result, including same-origin target changes. - Limitations explicitly report attach-time visibility, unsupported optional CDP domains, truncation, omitted cross-origin data, and observations unavailable before attachment. ## Architecture and contracts - Domain schemas and URL/origin normalization live in `src/domain/browserObservation.ts`. - Application authorization and Evidence v2 composition remain provider-neutral. - HTTP/WebSocket discovery, CDP correlation, capture, sanitization, and protocol handling live under `src/browser/`. - MCP and CLI adapters consume the same application operations and public schemas. - The target-free server can expose all 70 tools without a Hopper target; browser availability is reported independently from native-analysis availability. - `docs/browser-observation.md` documents setup, output categories, explicit limits, privacy behavior, and the threat model. ## Verification - `npm run check` - TypeScript strict typecheck passed. - oxlint completed with no errors. - Prettier check passed. - 84 test files and 559 tests passed. - Coverage: 79.83% statements, 69.38% branches, 81.99% functions, 82.06% lines. - `npx vitest run tests/browserCli.test.ts tests/browserMcp.test.ts --coverage.enabled=false --retry=0` - 4/4 CLI and MCP parity tests passed without retries. - `npm run verify:package` - Verified the packed CLI, setup transaction, backups/readback/recovery, skill, 70-tool target-free MCP server, 6 prompts, evidence resources, and target lifecycle in isolation. - `REA_BROWSER_EXECUTABLE=/tmp/rea-browser-verification/chrome-linux64/chrome npm run verify:browser` - Passed against Chrome 150.0.7871.115 with real HTTP/WebSocket CDP. - Observed 13 DOM nodes, 10 accessibility nodes, 1 script, 6 network requests, 7 console events, and 12 WebSocket events while verifying sensitive values were absent. - `npm run knip` - `npm run jscpd` (0.93% duplicated lines, below the configured threshold) - `npm run scan:todos` - `git diff --check` - Staged path, large-blob, symlink, binary, and credential-signature checks passed. Real Hopper verification was not performed because this environment is Linux and has no real Hopper installation. Existing Hopper behavior was exercised by the full test suite and isolated package verifier; this PR does not route browser work through Hopper. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-16 21:11:48 +00:00