morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #2045] [CLOSED] feat(browser): record observe_web_session traffic as HAR #4986

New issue

Closed

opened 2026-10-31 00:57:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:57:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2045
Author: @Hyped01
Created: 10/12/2026
Status: ❌ Closed

Base: main ← Head: feat/session-har-1-core

📝 Commits (2)

6b912ce fix(browser): admit browser-serialized HAR request URLs
fba05f5 feat(browser): record observe_web_session traffic as HAR

📊 Changes

20 files changed (+2191 additions, -36 deletions)

View changed files

📝 docs/browser-observation.md (+70 -1)
📝 docs/web-network-captures.md (+6 -0)
📝 src/application/BrowserEvidence.ts (+3 -0)
📝 src/browser/CdpObservationSession.ts (+75 -17)
➕ src/browser/CdpSessionHar.ts (+295 -0)
➕ src/browser/CdpSessionNetwork.ts (+443 -0)
➕ src/browser/CdpSessionResult.ts (+60 -0)
📝 src/browser/history/CaptureRedaction.ts (+2 -7)
📝 src/browser/history/HarSchema.ts (+28 -1)
📝 src/browser/network/PlaywrightNetworkContent.ts (+2 -8)
📝 src/browser/network/PlaywrightScenarioNetwork.test.ts (+44 -0)
➕ src/browser/network/TransportCredentials.ts (+7 -0)
📝 src/cli/browserCommands.ts (+10 -1)
📝 src/cliObservationOptions.ts (+12 -0)
📝 src/contracts/browserToolContracts.ts (+11 -1)
📝 src/domain/browserSession.ts (+67 -0)
➕ tests/boundary/browser/cdpObservationSessionNetwork.test.ts (+806 -0)
📝 tests/boundary/browser/historicalCaptureDecoder.test.ts (+44 -0)
📝 tests/boundary/cli/browserCli.test.ts (+130 -0)
📝 tests/boundary/mcp/browserMcp.test.ts (+76 -0)

📄 Description

Stacked on #2044. Only the top commit, fba05f5e1, is new here; review it alone, or after #2044 merges.

Summary

Refs #774 (batch 1, "Network transaction identity and selected content evidence"). observe_web_session can now record the selected page's in-scope HTTP requests and WebSocket connections as one HAR 1.2 log, across main-frame navigations. inspect_web_network_capture can then inspect that log offline.

Recording is opt-in through network (CLI --network, with --header-values). Without it, the result and the Evidence parameters are unchanged.

Problem and expected behavior

inspect_web_page fails closed with target_changed when the page navigates during its window, and observe_web_session keeps navigation metadata only. Login, checkout and other multi-page flows therefore had no network evidence. With network selected, the session result gains network.har: every in-scope request, redirect hop, failure, WebSocket handshake and frame, with completeness and limitations reported.

Change and scope

Recorder. CdpSessionNetwork records requests, redirect hops, failures, WebSocket handshakes and frames. CDP reports a hop's response together with the next request. Entries are keyed by target and request ID.
Scope. The recorder reuses the timeline's origin boundary. Recording stops right after the event that ends the timeline, so an out-of-scope document redirect still completes its in-scope hop.
Headers are retained only with header_values:
- Authorization, Proxy-Authorization, Cookie and Set-Cookie never are, by the shared rules from #2044.
- URL-valued headers follow the request scope: Location, Content-Location, Referer, Origin, Link and Refresh. One that points outside the selected origins, or whose target cannot be parsed, is omitted and counted.
- Link is read as RFC 8288 link-values, including anchor parameters. A Link is omitted when it does not parse, has a < inside a target, or has an anchor value REA cannot read.
- Refresh follows the HTML declarative-refresh steps, with or without url=. Its delay is matched without backtracking. A quoted URL whose quote appears again after its closing quote is omitted, because Chromium reads to the last matching quote.
- Userinfo in a retained header is judged on the target resolved against the request URL. It is located with src/domain/recordedTransportUrl.ts, the repository's transport-URL parser, and cut at its parsed offset only when the cut leaves exactly the same target. A spelling the parser reads as ambiguous, such as http:user:pass@host/, or a cut that would change the target, omits the header.
HAR 1.2 conformance:
- send, wait and receive are non-negative, and time is their sum;
- a response still loading says so;
- a request whose body is not retained has bodySize -1.
Coverage. coverage_started_at is taken once the browser has processed Network.enable.
Size. har_bytes reports the log's size. This PR does not check it against inspect_web_network_capture's limits; #2047 adds the input-budget limitation. The decoder's 96 MiB reply limit is documented, not estimated: a log dominated by WebSocket frames can exceed it below 32 MiB.
Separate targets. Chrome runs the page's workers and cross-site iframes out of process, as separate targets. This PR does not attach them, so their traffic is not recorded; an always-present limitation, the tool description and the docs say so. #2048 adds workers and flags each out-of-process iframe of the page. Without network, the navigation-only limitation says network was not selected.
Not included: declared-value redaction (#2046), bodies and payloads (#2047), and workers (#2048).

Contract and boundary impact

Semantic owner and earliest changed stage: src/browser/CdpSessionNetwork.ts (recorder) and src/browser/CdpSessionHar.ts (projection).
CLI and MCP/tool-catalog contract: an additive optional network input and a network output, plus one new contract example.
Provider, bridge, target-format, or platform compatibility: CDP only; the HAR is admissible to inspect_web_network_capture.
Evidence, artifact, provenance, or reconstruction contract: observe_web_session Evidence parameters gain network only when it is selected.
Process execution, authorization, cleanup, or containment impact: none beyond Network.enable on the page session, which was already sent.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Evidence and regression coverage

Tests added or updated:
- tests/boundary/browser/cdpObservationSessionNetwork.test.ts covers:
  - navigation tolerance with an offline decode round trip;
  - out-of-scope requests, sockets and redirect targets;
  - a document redirect leaving scope;
  - recording stopping at the timeline's end;
  - URL-valued header spellings: foreign Referer and Origin, bare and malformed Link, quoted < and quoted-pairs, anchor params, a Refresh with and without url=, scheme-relative and backslash userinfo, a backslash retarget, two credentialed targets in one header, and a 200,000-digit Refresh delay;
  - browser-serialized URLs, including a bare % and a query backslash, accepted by the decoder;
  - HAR 1.2 timings;
  - a quoted Refresh URL that Chromium reads to a later quote;
  - navigation-only sessions staying unchanged.
- CLI flag mapping, including --network alone keeping every header out, and MCP Evidence: browserCli.test.ts and browserMcp.test.ts.
Mutation checks: reverting each header rule, the last-first splice order, the Link tokenizer's fail-closed branches, the Refresh delay pattern, the repeated-quote rule or the iframe limitation fails a test.
Remaining proof gaps:
- cross-site iframe traffic is not recorded, as stated above;
- Chrome reports an out-of-process iframe's document request on the page session but its loadingFinished on the iframe's own target, so with the iframe's origin selected that entry says it had not finished loading.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Host: macOS 27.0.1, Node 24.20.0, Chrome 154 (headless, isolated profile).

Per-commit gate: turbo run typecheck lint format:check knip metadata:check docs:check:built mcp-catalog:check:built, npm run verify:architecture-guards, npm run verify:test-discovery and npx jscpd src/ tests/ all pass.
npx vitest run on 11 browser, CLI, MCP and process boundary files: 74 passed, 2 skipped.
exposes CLI-equivalent Evidence results and session tools, an existing MCP test, times out only under heavy machine load. It fails the same way on unmodified main.
Cold review with adversarial verification: the final round reported no findings.

Compatibility, safety, and release

Breaking changes or migration steps: none; recording is opt-in.
Package or release metadata impact: none.
Security, privacy, process, or containment review: credentials and out-of-scope URL targets fail closed and are counted. The HAR is sensitive local evidence, as the docs say.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2045 **Author:** [@Hyped01](https://github.com/Hyped01) **Created:** 10/12/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/session-har-1-core` --- ### 📝 Commits (2) - [`6b912ce`](https://github.com/morluto/rea/commit/6b912ce5b90e9b4a395570af0060c5efee9ab44a) fix(browser): admit browser-serialized HAR request URLs - [`fba05f5`](https://github.com/morluto/rea/commit/fba05f5e123ee1d1893689e2d6b265ea618459ac) feat(browser): record observe_web_session traffic as HAR ### 📊 Changes **20 files changed** (+2191 additions, -36 deletions) <details> <summary>View changed files</summary> 📝 `docs/browser-observation.md` (+70 -1) 📝 `docs/web-network-captures.md` (+6 -0) 📝 `src/application/BrowserEvidence.ts` (+3 -0) 📝 `src/browser/CdpObservationSession.ts` (+75 -17) ➕ `src/browser/CdpSessionHar.ts` (+295 -0) ➕ `src/browser/CdpSessionNetwork.ts` (+443 -0) ➕ `src/browser/CdpSessionResult.ts` (+60 -0) 📝 `src/browser/history/CaptureRedaction.ts` (+2 -7) 📝 `src/browser/history/HarSchema.ts` (+28 -1) 📝 `src/browser/network/PlaywrightNetworkContent.ts` (+2 -8) 📝 `src/browser/network/PlaywrightScenarioNetwork.test.ts` (+44 -0) ➕ `src/browser/network/TransportCredentials.ts` (+7 -0) 📝 `src/cli/browserCommands.ts` (+10 -1) 📝 `src/cliObservationOptions.ts` (+12 -0) 📝 `src/contracts/browserToolContracts.ts` (+11 -1) 📝 `src/domain/browserSession.ts` (+67 -0) ➕ `tests/boundary/browser/cdpObservationSessionNetwork.test.ts` (+806 -0) 📝 `tests/boundary/browser/historicalCaptureDecoder.test.ts` (+44 -0) 📝 `tests/boundary/cli/browserCli.test.ts` (+130 -0) 📝 `tests/boundary/mcp/browserMcp.test.ts` (+76 -0) </details> ### 📄 Description > Stacked on #2044. Only the top commit, `fba05f5e1`, is new here; review it alone, or after #2044 merges. ## Summary Refs #774 (batch 1, "Network transaction identity and selected content evidence"). `observe_web_session` can now record the selected page's in-scope HTTP requests and WebSocket connections as one HAR 1.2 log, across main-frame navigations. `inspect_web_network_capture` can then inspect that log offline. Recording is opt-in through `network` (CLI `--network`, with `--header-values`). Without it, the result and the Evidence parameters are unchanged. ## Problem and expected behavior `inspect_web_page` fails closed with `target_changed` when the page navigates during its window, and `observe_web_session` keeps navigation metadata only. Login, checkout and other multi-page flows therefore had no network evidence. With `network` selected, the session result gains `network.har`: every in-scope request, redirect hop, failure, WebSocket handshake and frame, with completeness and limitations reported. ## Change and scope - **Recorder.** `CdpSessionNetwork` records requests, redirect hops, failures, WebSocket handshakes and frames. CDP reports a hop's response together with the next request. Entries are keyed by target and request ID. - **Scope.** The recorder reuses the timeline's origin boundary. Recording stops right after the event that ends the timeline, so an out-of-scope document redirect still completes its in-scope hop. - **Headers** are retained only with `header_values`: - `Authorization`, `Proxy-Authorization`, `Cookie` and `Set-Cookie` never are, by the shared rules from #2044. - URL-valued headers follow the request scope: `Location`, `Content-Location`, `Referer`, `Origin`, `Link` and `Refresh`. One that points outside the selected origins, or whose target cannot be parsed, is omitted and counted. - `Link` is read as RFC 8288 link-values, including `anchor` parameters. A Link is omitted when it does not parse, has a `<` inside a target, or has an anchor value REA cannot read. - `Refresh` follows the HTML declarative-refresh steps, with or without `url=`. Its delay is matched without backtracking. A quoted URL whose quote appears again after its closing quote is omitted, because Chromium reads to the last matching quote. - Userinfo in a retained header is judged on the target resolved against the request URL. It is located with `src/domain/recordedTransportUrl.ts`, the repository's transport-URL parser, and cut at its parsed offset only when the cut leaves exactly the same target. A spelling the parser reads as ambiguous, such as `http:user:pass@host/`, or a cut that would change the target, omits the header. - **HAR 1.2 conformance:** - `send`, `wait` and `receive` are non-negative, and `time` is their sum; - a response still loading says so; - a request whose body is not retained has `bodySize` `-1`. - **Coverage.** `coverage_started_at` is taken once the browser has processed `Network.enable`. - **Size.** `har_bytes` reports the log's size. This PR does not check it against `inspect_web_network_capture`'s limits; #2047 adds the input-budget limitation. The decoder's 96 MiB reply limit is documented, not estimated: a log dominated by WebSocket frames can exceed it below 32 MiB. - **Separate targets.** Chrome runs the page's workers and cross-site iframes out of process, as separate targets. This PR does not attach them, so their traffic is not recorded; an always-present limitation, the tool description and the docs say so. #2048 adds workers and flags each out-of-process iframe of the page. Without `network`, the navigation-only limitation says network was not selected. - Not included: declared-value redaction (#2046), bodies and payloads (#2047), and workers (#2048). ## Contract and boundary impact - Semantic owner and earliest changed stage: `src/browser/CdpSessionNetwork.ts` (recorder) and `src/browser/CdpSessionHar.ts` (projection). - CLI and MCP/tool-catalog contract: an additive optional `network` input and a `network` output, plus one new contract example. - Provider, bridge, target-format, or platform compatibility: CDP only; the HAR is admissible to `inspect_web_network_capture`. - Evidence, artifact, provenance, or reconstruction contract: `observe_web_session` Evidence parameters gain `network` only when it is selected. - Process execution, authorization, cleanup, or containment impact: none beyond `Network.enable` on the page session, which was already sent. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. ## Evidence and regression coverage - Tests added or updated: - `tests/boundary/browser/cdpObservationSessionNetwork.test.ts` covers: - navigation tolerance with an offline decode round trip; - out-of-scope requests, sockets and redirect targets; - a document redirect leaving scope; - recording stopping at the timeline's end; - URL-valued header spellings: foreign `Referer` and `Origin`, bare and malformed `Link`, quoted `<` and quoted-pairs, `anchor` params, a `Refresh` with and without `url=`, scheme-relative and backslash userinfo, a backslash retarget, two credentialed targets in one header, and a 200,000-digit `Refresh` delay; - browser-serialized URLs, including a bare `%` and a query backslash, accepted by the decoder; - HAR 1.2 timings; - a quoted Refresh URL that Chromium reads to a later quote; - navigation-only sessions staying unchanged. - CLI flag mapping, including `--network` alone keeping every header out, and MCP Evidence: `browserCli.test.ts` and `browserMcp.test.ts`. - Mutation checks: reverting each header rule, the last-first splice order, the Link tokenizer's fail-closed branches, the Refresh delay pattern, the repeated-quote rule or the iframe limitation fails a test. - Remaining proof gaps: - cross-site iframe traffic is not recorded, as stated above; - Chrome reports an out-of-process iframe's document request on the page session but its `loadingFinished` on the iframe's own target, so with the iframe's origin selected that entry says it had not finished loading. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Host: macOS 27.0.1, Node 24.20.0, Chrome 154 (headless, isolated profile). - Per-commit gate: `turbo run typecheck lint format:check knip metadata:check docs:check:built mcp-catalog:check:built`, `npm run verify:architecture-guards`, `npm run verify:test-discovery` and `npx jscpd src/ tests/` all pass. - `npx vitest run` on 11 browser, CLI, MCP and process boundary files: 74 passed, 2 skipped. - `exposes CLI-equivalent Evidence results and session tools`, an existing MCP test, times out only under heavy machine load. It fails the same way on unmodified `main`. - Cold review with adversarial verification: the final round reported no findings. ## Compatibility, safety, and release - Breaking changes or migration steps: none; recording is opt-in. - Package or release metadata impact: none. - Security, privacy, process, or containment review: credentials and out-of-scope URL targets fail closed and are counted. The HAR is sensitive local evidence, as the docs say. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:57:43 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:42:46 +00:00

[PR #5031] [MERGED] feat(capture): decode selected protobuf and gRPC bodies #6746

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4986

No description provided.

Rows
Columns