[PR #797] [MERGED] feat(browser): capture correlated network content and lifecycle evidence #2246

New issue

Closed

opened 2026-10-16 21:16:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:16:13 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/797
Author: @N0zoM1z0
Created: 10/6/2026
Status: ✅ Merged
Merged: 10/6/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/website-network-evidence

📝 Commits (3)

ead8009 feat(browser): capture correlated network content and lifecycle evidence
c6affdf fix(browser): align capture defaults and redact encoded media metadata
8971a54 test(browser): verify polling capture cutoff completeness

📊 Changes

31 files changed (+2090 additions, -80 deletions)

View changed files

📝 .github/workflows/real-browser.yml (+3 -0)
📝 docs/browser-scenario-contract.md (+77 -0)
📝 docs/product-catalog.json (+2 -2)
📝 docs/testing.md (+8 -0)
📝 docs/verification/managed-conformance-manifest.json (+2 -2)
📝 package.json (+1 -0)
➕ scripts/fixtures/browser-network-site.mjs (+109 -0)
➕ scripts/lib/browser-network-assertions.mjs (+156 -0)
➕ scripts/lib/browser-network-e2e.mjs (+192 -0)
📝 scripts/lib/browser-scenario-verifier.mjs (+11 -2)
➕ scripts/verify-browser-network.mjs (+16 -0)
📝 scripts/verify-real-browser.mjs (+7 -0)
📝 skills/reverse-engineer-anything/SKILL.md (+1 -1)
📝 src/browser/BrowserScenario.test.ts (+5 -0)
📝 src/browser/BrowserScenarioSecrets.ts (+55 -8)
📝 src/browser/PlaywrightBrowserScenarioProvider.ts (+5 -1)
📝 src/browser/PlaywrightScenarioEvents.ts (+61 -54)
📝 src/browser/PlaywrightScenarioSession.ts (+7 -1)
➕ src/browser/network/PlaywrightNetworkContent.ts (+120 -0)
➕ src/browser/network/PlaywrightScenarioNetwork.test.ts (+532 -0)

...and 11 more files

📄 Description

Summary

First implementation batch for the website reverse-engineering roadmap. Refs #774; the roadmap remains open for captured assets and later integrations.

Correlate request, response, completion, failure, and cutoff observations by capture-scoped request identity, including observed redirect predecessors.
Select request bytes, response bytes, and header values independently with capture.network. Defaults keep metadata alone; selected content also enables network observation.
Keep network readers and transaction collection in src/browser/network/, separate from event orchestration and provider-neutral evidence schemas.
Return exact retained base64 bytes, count, SHA-256, media type, redaction status, and source event references. Preserve duplicate headers; remove credential header values and caller-declared secret representations without decoding unrelated binary bytes.
Drain completed reads before cleanup, bound each read to five seconds, honor cancellation, and mark unfinished/unavailable responses explicitly. No refetch substitutes for the original response.
Compare content under matching selection and source-step attribution. Selection differences and unknown legacy network coverage return unknown with recovery guidance.

Example (CLI and MCP use the same object)

{
  "browser": { "mode": "launch", "executable_path": "/absolute/path/to/chromium" },
  "start_url": { "url": "http://127.0.0.1:3000" },
  "actions": [{ "step_id": "ready", "action": "wait_for", "locator": { "kind": "css", "selector": "#done" }, "state": "visible", "timeout_ms": 10000 }],
  "capture": { "network": { "request_body": true, "response_body": true, "header_values": true } }
}

Content records link to observed request/response events through transaction_id and source_event_sequence. Request bytes reflect what Playwright exposes; not_exposed does not prove absence or multipart file completeness. Responses contain browser-decoded bytes. Playwright scenario capture has no initiator-stack API; passive CDP retains its separate initiator evidence.

Verification

Linux x64, locally installed Chrome 153.0.8010.12, Node 24.18.0. Commands ran with one CPU and a bounded Node heap.

Focused source suite: 41 tests / 6 files covering schema integrity, concurrent same-URL association, redirects, binary/encoded-secret redaction, duplicate headers, independent selection, cutoff, cancellation, timeout/late completion, failures, empty/unexposed bytes, and comparison coverage.
npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/browserScenarioMcp.test.ts tests/boundary/cli/browserCli.test.ts: 8 tests passed, including every advertised canonical JSON Schema.
npm run check:fast, npm run docs:check, and git diff --check: passed.
Full real browser lane: passed passive analysis/source maps, launch/attach lifecycle, popup/shared-context isolation, and real PNG >8 MiB. New network oracle proved CLI/MCP content and connected-browser survival.
Final network oracle also passed against an npm-packed, separately installed package under /tmp, with lifecycle scripts disabled. This checked the final selection metadata, actual binary and gzip-decoded bytes, form/encoded-header redaction, redirects, streaming cutoff, no refetches, and profile cleanup through CLI/MCP.
The server-authored marker oracle rejects deliberately swapped same-URL response IDs. A form-encoding regression was observed failing before the redactor fix and passes afterwards.

The focused installed-package verifier observed zero live descendants at completion. The mixed full browser verifier's in-run lineage report was explicitly unavailable because a Chrome descendant omitted its run token; that recorded PID was gone after cleanup. These reports establish different observations.

Real Hopper/Ghidra verification was not run for this browser change.

CI follow-up

Corrected complete documented request and normalized capture-default assertions caught by CI; 38 focused tests passed.
Encoded body media-type metadata regression now passes; collector coverage is 15 tests.
Updated the continuously polling fixture oracle to accept only explicit request-unfinished cutoff coverage, reject other gaps, and retain detailed failure states. Full real browser verification passed again (launch and external attachment, CLI/MCP, network content, popups, screenshots).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/797 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/6/2026 **Status:** ✅ Merged **Merged:** 10/6/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/website-network-evidence` --- ### 📝 Commits (3) - [`ead8009`](https://github.com/morluto/rea/commit/ead80094b9f5fba6995e13fa31f2b4264c61c9ad) feat(browser): capture correlated network content and lifecycle evidence - [`c6affdf`](https://github.com/morluto/rea/commit/c6affdf4be9a06abb36b38a639dec6520cc24779) fix(browser): align capture defaults and redact encoded media metadata - [`8971a54`](https://github.com/morluto/rea/commit/8971a54ae84f44a7e66133bd5a072762cc1fef21) test(browser): verify polling capture cutoff completeness ### 📊 Changes **31 files changed** (+2090 additions, -80 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/real-browser.yml` (+3 -0) 📝 `docs/browser-scenario-contract.md` (+77 -0) 📝 `docs/product-catalog.json` (+2 -2) 📝 `docs/testing.md` (+8 -0) 📝 `docs/verification/managed-conformance-manifest.json` (+2 -2) 📝 `package.json` (+1 -0) ➕ `scripts/fixtures/browser-network-site.mjs` (+109 -0) ➕ `scripts/lib/browser-network-assertions.mjs` (+156 -0) ➕ `scripts/lib/browser-network-e2e.mjs` (+192 -0) 📝 `scripts/lib/browser-scenario-verifier.mjs` (+11 -2) ➕ `scripts/verify-browser-network.mjs` (+16 -0) 📝 `scripts/verify-real-browser.mjs` (+7 -0) 📝 `skills/reverse-engineer-anything/SKILL.md` (+1 -1) 📝 `src/browser/BrowserScenario.test.ts` (+5 -0) 📝 `src/browser/BrowserScenarioSecrets.ts` (+55 -8) 📝 `src/browser/PlaywrightBrowserScenarioProvider.ts` (+5 -1) 📝 `src/browser/PlaywrightScenarioEvents.ts` (+61 -54) 📝 `src/browser/PlaywrightScenarioSession.ts` (+7 -1) ➕ `src/browser/network/PlaywrightNetworkContent.ts` (+120 -0) ➕ `src/browser/network/PlaywrightScenarioNetwork.test.ts` (+532 -0) _...and 11 more files_ </details> ### 📄 Description ## Summary First implementation batch for the website reverse-engineering roadmap. Refs #774; the roadmap remains open for captured assets and later integrations. - Correlate request, response, completion, failure, and cutoff observations by capture-scoped request identity, including observed redirect predecessors. - Select request bytes, response bytes, and header values independently with `capture.network`. Defaults keep metadata alone; selected content also enables network observation. - Keep network readers and transaction collection in `src/browser/network/`, separate from event orchestration and provider-neutral evidence schemas. - Return exact retained base64 bytes, count, SHA-256, media type, redaction status, and source event references. Preserve duplicate headers; remove credential header values and caller-declared secret representations without decoding unrelated binary bytes. - Drain completed reads before cleanup, bound each read to five seconds, honor cancellation, and mark unfinished/unavailable responses explicitly. No refetch substitutes for the original response. - Compare content under matching selection and source-step attribution. Selection differences and unknown legacy network coverage return `unknown` with recovery guidance. ## Example (CLI and MCP use the same object) ```json { "browser": { "mode": "launch", "executable_path": "/absolute/path/to/chromium" }, "start_url": { "url": "http://127.0.0.1:3000" }, "actions": [{ "step_id": "ready", "action": "wait_for", "locator": { "kind": "css", "selector": "#done" }, "state": "visible", "timeout_ms": 10000 }], "capture": { "network": { "request_body": true, "response_body": true, "header_values": true } } } ``` Content records link to observed request/response events through `transaction_id` and `source_event_sequence`. Request bytes reflect what Playwright exposes; `not_exposed` does not prove absence or multipart file completeness. Responses contain browser-decoded bytes. Playwright scenario capture has no initiator-stack API; passive CDP retains its separate initiator evidence. ## Verification Linux x64, locally installed Chrome 153.0.8010.12, Node 24.18.0. Commands ran with one CPU and a bounded Node heap. - Focused source suite: **41 tests / 6 files** covering schema integrity, concurrent same-URL association, redirects, binary/encoded-secret redaction, duplicate headers, independent selection, cutoff, cancellation, timeout/late completion, failures, empty/unexposed bytes, and comparison coverage. - `npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/browserScenarioMcp.test.ts tests/boundary/cli/browserCli.test.ts`: **8 tests passed**, including every advertised canonical JSON Schema. - `npm run check:fast`, `npm run docs:check`, and `git diff --check`: passed. - Full real browser lane: passed passive analysis/source maps, launch/attach lifecycle, popup/shared-context isolation, and real PNG >8 MiB. New network oracle proved CLI/MCP content and connected-browser survival. - Final network oracle also passed against an npm-packed, separately installed package under `/tmp`, with lifecycle scripts disabled. This checked the final selection metadata, actual binary and gzip-decoded bytes, form/encoded-header redaction, redirects, streaming cutoff, no refetches, and profile cleanup through CLI/MCP. - The server-authored marker oracle rejects deliberately swapped same-URL response IDs. A form-encoding regression was observed failing before the redactor fix and passes afterwards. The focused installed-package verifier observed zero live descendants at completion. The mixed full browser verifier's in-run lineage report was explicitly unavailable because a Chrome descendant omitted its run token; that recorded PID was gone after cleanup. These reports establish different observations. Real Hopper/Ghidra verification was not run for this browser change. ### CI follow-up - Corrected complete documented request and normalized capture-default assertions caught by CI; 38 focused tests passed. - Encoded body media-type metadata regression now passes; collector coverage is 15 tests. - Updated the continuously polling fixture oracle to accept only explicit request-unfinished cutoff coverage, reject other gaps, and retain detailed failure states. Full real browser verification passed again (launch and external attachment, CLI/MCP, network content, popups, screenshots). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-16 21:16:13 +00:00