morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #4456] fix(browser): record dedicated worker requests in page inspection #6253

New issue

Open

opened 2026-10-31 01:13:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:13:03 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4456
Author: @Pharox24
Created: 10/16/2026
Status: 🔄 Open

Base: main ← Head: fix/inspect-web-page-worker-network

📝 Commits (10+)

546850a fix(browser): record dedicated worker requests in page inspection
4b894f0 fix(browser): drain worker commands before the final page check
c0707bc fix(browser): bound, nest, and classify worker network observation
080b7a2 fix(browser): close worker admission before the final page check
97a7b01 fix(browser): report budget-omitted worker WebSockets in their sections
a03fe17 fix(browser): release refused workers and abandon detached body reads
fb99fff fix(browser): bound worker releases and frames, report ID collisions
b0972bb fix(browser): spend worker budgets only on retained traffic
553c221 fix(browser): bound worker frame bytes and keep page events off worker records
d01d8eb fix(browser): bound retained worker request bytes

📊 Changes

14 files changed (+2533 additions, -45 deletions)

View changed files

📝 docs/browser-observation.md (+25 -0)
📝 src/browser/CdpCaptureEventHandlers.ts (+37 -20)
➕ src/browser/CdpCaptureEventHelpers.test.ts (+37 -0)
📝 src/browser/CdpCaptureEventHelpers.ts (+14 -3)
📝 src/browser/CdpCaptureEvents.ts (+7 -2)
📝 src/browser/CdpPageCapture.ts (+90 -18)
➕ src/browser/CdpPageCaptureWorkerNetwork.test.ts (+1003 -0)
➕ src/browser/CdpPageCaptureWorkerNetwork.ts (+717 -0)
📝 src/browser/CdpPageCaptureWorkers.ts (+4 -1)
📝 src/browser/CdpSafeMetadata.ts (+33 -0)
📝 tests/boundary/browser/cdpBrowserProvider.network.test.ts (+562 -1)
📝 tests/fixtures/fakeCdpBrowser.ts (+1 -0)
📝 tests/fixtures/fakeCdpBrowserResults.ts (+1 -0)
📝 tests/fixtures/fakeCdpBrowserTypes.ts (+2 -0)

📄 Description

Summary

inspect_web_page / rea inspect-web-page (and analyze_web_bundle, which shares the capture) never recorded network requests made by the page's dedicated Web Workers, while still listing those workers as attached. An empty or page-only network.requests could be read as "the app made no other requests". This PR observes dedicated-worker network activity during the existing observation window.

Fixes #4425

Problem and expected behavior

Chrome reports a dedicated worker's requests on the worker target, not on the page target that created it. capturePage sent Network.enable only on the page session (src/browser/CdpPageCapture.ts, enableObservationDomains) and its event listener returned early for any other sessionId. Workers were only enumerated afterwards through Target.getTargets, whose attached flag reflects any attached client (for example an open DevTools window), not REA observing the worker.

Expected (from the issue): worker requests appear in normalized_result.network.requests, or coverage states explicitly that worker network activity was not observed.

Change and scope

New CdpWorkerNetworkObservation (src/browser/CdpPageCaptureWorkerNetwork.ts), owned by one capturePage call:
- After the page's own domains are enabled, sends Target.setAutoAttach { autoAttach: true, waitForDebuggerOnStart: false, flatten: true, filter: [{ type: "worker" }] } on the page session. Workers are never paused; existing workers attach immediately and new ones attach as they start.
- For each attached dedicated worker whose URL passes the existing origin policy, enables only Network on that worker session, then sends it the same non-pausing Target.setAutoAttach so workers it starts are observed too. Workers on other origins are not enabled and are recorded as network_requests exclusions (disallowed_origin / unsupported_url / unattributed_origin, count null).
- Forwards only Network.* events from admitted worker sessions into the existing CdpCaptureEvents, so URL policy, redaction, body-shape rules and completeness accounting are unchanged.
- A request belongs to the session that started it (requestWillBeSent / webSocketCreated). Real Chrome announces a new worker's own script on the page session and completes it on the worker session with encodedDataLength: -1. Without this rule that completion was counted as invalid_protocol_value and marked network_requests unavailable. With it, the page-owned record is unchanged from current behaviour.
- JSON response bodies (include_json_body_shapes) for worker requests are read from the worker session. If that worker has already detached (or detaches mid-read, -32001), the body is reported unavailable with a limitation instead of failing the capture.
- One capture admits at most 256 worker sessions and tracks at most 50,000 worker requests (CDP_WORKER_NETWORK_LIMITS, matching the script metadata record bound). Omitted workers and requests count as resource_budget_exhausted, so network_requests is reported truncated; an omitted worker WebSocket counts under websocket_connections and marks websocket_frames truncated. A whole unobserved worker (other origin, over the limit, unavailable auto-attach, or a live command rejection) is recorded under all three sections with an unknown count, and up to 256 refused sessions per capture, during the admission window, are released with Target.detachFromTarget through their parent session; real Chrome keeps that worker running. Turning auto-attach off at capacity was rejected because real Chrome then also detaches admitted workers. Admitted workers retain at most 50,000 WebSocket frames, with later frames counted in dropped_events. Only traffic the origin policy keeps counts toward the frame and request budgets, and with include_websocket_shapes worker frame payload is also capped at 8 MiB before shape inference. Worker request and WebSocket URLs, plus request bodies with include_json_body_shapes, are capped at 8 MiB before ingestion; a URL the origin policy rejects costs nothing, WebSocket URLs use the same admittedWebSocketUrl check as the WebSocket handler, and a request whose redirect hop is over budget stops receiving later events so hops cannot mix. Request bodies are charged only when they get a JSON shape, every hop of a chain the origin policy drops is refunded, and worker JSON response bodies are charged against their own 8 MiB budget before shape inference. Nested worker sessions record their parent, and a parent's detach aborts its descendants. Worker response MIME types and every header value metadata keeps are capped at 8 MiB before metadata parsing, request hops are charged for every string the record keeps (method, type, initiator and redirect details), the frame shape budget applies only to text frames, rejected request hops are refunded, and malformed body envelopes are left to the body parser uncharged. Worker attached values come from a Target.getTargets snapshot taken before auto-attach, so REA's own attachment is not reported as target evidence. Page-session Network events are checked against worker request ownership before ingestion, so a page start on a worker-owned ID is reported rather than merged. A malformed attachment that still names a session is released. A worker request or WebSocket whose ID another session already owns is reported as invalid_protocol_value rather than merged or dropped silently.
- Only a worker session that already detached, or a -32001 session-not-found reply, counts as worker exit and stays under the existing attach-limited network_requests coverage. Any other rejection from a live worker records a provider_unavailable exclusion. Chrome never answers a command pending on a session that detaches, so each worker's commands, including its JSON body reads, abort when its session detaches instead of waiting for the 20 s CDP deadline; an abandoned body read is reported as unavailable body coverage. Transport and cancellation failures still fail the capture (settle(), which runs before the final navigation and origin check and first closes worker admission, so a worker attaching during that check is not enabled), and close() waits for issued worker commands before the page session is released.
Browsers without Target.setAutoAttach (-32601) keep page capture working and report network_requests as provider_unavailable plus the standard optional-command limitation.
One limitation line is added when auto-attach succeeds: requests a worker sends before REA reaches it, and service/shared worker requests, are unavailable.
docs/browser-observation.md documents the coverage.

Not included, intentionally:

Service workers and shared workers (the filter selects worker only).
A per-request worker target_id field. The issue suggests one, but it would change the WebPageInspection contract and generated catalog, so it is left for a separate decision.
Pausing workers (waitForDebuggerOnStart: true) to close the start-up race. That would give REA execution control over the target, so short-lived workers can be partially observed and the limitation says so.
observe_web_session (navigation timeline only) and Electron inspection paths are unchanged.

Contract and boundary impact

Semantic owner and earliest changed stage: CDP browser adapter, capturePage event routing (src/browser/CdpPageCapture.ts, new src/browser/CdpPageCaptureWorkerNetwork.ts).
Contribution scope fit and linked maintainer decision: fix within the existing passive inspect_web_page boundary for the already-authorized page target; no new provider. Auto-attaching the page's own dedicated workers is a new CDP attachment, so maintainers may want to confirm it fits the current boundary.
CLI and MCP/tool-catalog contract: no schema or catalog change. network.requests can now include dedicated-worker requests, and one limitation string is added.
Provider, bridge, target-format, or platform compatibility: needs Target.setAutoAttach with flatten (current Chrome). If unsupported, the capture degrades to provider_unavailable for worker network. Source-derived, not run: a browser that ignores filter may also auto-attach frames; those are neither enabled nor ingested.
Evidence, artifact, provenance, or reconstruction contract: unchanged; worker records go through the same normalization and completeness ledger.
Target selection, authority, upstream behavior, and cleanup details: only dedicated workers auto-attached under the selected page session or under an admitted worker; workers receive only Network.enable and Target.setAutoAttach, no Runtime/Debugger/Page domains, no pausing, no script evaluation. Auto-attached sessions are children of the page session REA already detaches, and the operation's socket is closed by CdpTargetSessionOwner.finish (source-derived).
Generated metadata, package, or installation impact: none.

Expected effects:

Read existing files or captured data.
Launch or attach to a process. (CDP auto-attach to the inspected page's dedicated worker targets; nothing is launched.)
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated: tests/boundary/browser/cdpBrowserProvider.network.test.ts, new dedicated worker network blocks (4 tests) through the real CdpBrowserProvider and the fake CDP WebSocket browser:
- worker request recorded; disallowed-origin worker not enabled, its events not retained and its origin absent from output; exact Target.setAutoAttach parameters; the only worker-session command is Network.enable; a page-announced worker script completed on the worker session with encodedDataLength: -1 keeps encoded_data_length: null and does not mark network_requests unavailable;
- worker JSON body read with the worker sessionId;
- Target.setAutoAttach unsupported, so provider_unavailable exclusion and limitation;
- a main-frame navigation delivered while a worker Network.enable reply is still pending reports target_changed (added for the review finding below).
- a worker started by an admitted worker is attached through that worker's session and its request recorded;
- a live worker rejecting Network.enable (-32601) reports provider_unavailable, while a gone session (-32001) does not;
- a worker command Chrome leaves unanswered when the worker detaches is abandoned instead of timing out the capture;
- workers are admitted only from auto-attach until the final page check; one attaching during the final frame read is only detached;
- a worker JSON body read pending when the worker detaches is reported unavailable instead of timing out the capture (failed with timeout before a03fe17);
- class tests: refused-worker releases capped and none after admission closes; worker WebSocket frames capped with dropped_events; page- and worker-owned request ID collisions reported; excluded-origin frames and requests do not spend the budgets; frame payload byte budget with shapes; page events on a worker-owned ID reported and kept off the worker record; malformed attachment with a session released; worker request bytes capped with excluded URLs keeping their reason; an over-budget redirect hop freezes the earlier hop; allowed worker WebSocket URLs count against the byte budget; dropped redirect chains refunded; non-JSON bodies not charged; nested commands abandoned on parent detach; worker JSON response shapes past 8 MiB omitted as resource_budget_exhausted (boundary test); expanding response metadata bounded; pre-attach worker attached state reported; rejected hops refunded; malformed body envelopes uncharged; long methods, initiator URLs, MIME types and kept header values charged; binary frames outside the shape budget; a JSON response stripped by the metadata budget records json_body_shapes as resource_budget_exhausted (97144b7); base64 bodies sized from their text without decoding, and worker attachments without a target ID released as malformed (98ba117).
- src/browser/CdpCaptureEventHelpers.test.ts: decodeBase64 canonical and non-canonical cases and a 12 MiB payload. The old grouped base64 pattern overflowed the stack on payloads of several megabytes, on the page path as well; it is replaced with an equivalent linear pattern.
- src/browser/CdpPageCaptureWorkerNetwork.test.ts: worker and worker-request budgets, each counting omitted items as resource_budget_exhausted in the matching section, including an omitted WebSocket.
- tests/fixtures/fakeCdpBrowserResults.ts answers Target.setAutoAttach like Chrome; holdReplyFor in the fake lets a test record a command and send its reply itself.
Base reproduction (executed): with only the test and fixture changes on base c8848648, the new block fails (3 failed | 14 passed). The -1 cross-session assertion was added after a real-Chrome run surfaced it, and it failed (1 failed | 16 passed) until the ownership rule was added. The navigation test returned ok: true (a mixed inspection) on 546850a and passes after 4b894f0 moved settle() ahead of the final page check.

Real Chrome (executed): headless Chrome 155.0.8059.80, macOS, local http://127.0.0.1 page, 8 s window, built CLI rea inspect-web-page … --format json. The page fetches every second from the page, a long-lived blob worker, a same-origin script worker, and a new short-lived blob worker per round. Distinct markers recorded:

build	page	long-lived blob worker	script worker	new short-lived workers	`unavailable_sections`
base `c8848648`	8	0	0	0	`[]`
this PR, run 1	8	8	8	8	`[]`
this PR, run 2	8	8	8	7	`[]`
this PR `4b894f0`, run 3	9	8	8	8	`[]`

For c0707bc the harness also starts a worker from another worker 1.5 s after load. Four consecutive runs recorded 8 page / 8 long-lived / 8 script / 8 short-lived / 8 nested-worker markers, each with unavailable_sections: [] and no network_requests exclusions. Three further runs on 080b7a2 recorded the same (one run recorded 9 long-lived-worker markers), and three on a03fe17 did too (one run recorded 7 short-lived-worker markers, the documented start-up race). Six runs on fb99fff recorded 8 page / 8–9 long-lived / 8 script / 6–8 short-lived / 8 nested markers with no network_requests exclusions. Three back-to-back runs of a03fe17 under the same machine load recorded 7–8 short-lived markers. The change does not touch admitted-worker setup, so the short-lived spread is the start-up race under load. During this work one run failed with provider_timeout: CDP command Target.setAutoAttach exceeded its 20000 ms response deadline. A raw CDP trace showed Chrome had detached that short-lived worker 14 ms after the command and never replied; the per-worker abort fixes it, and the boundary test above reproduces it.

Run 2 missed one short-lived worker whose fetch ran before Network.enable reached it; that is the documented start-up race.

User-visible output: worker requests appear as ordinary network.requests entries. Measured JSON bytes, not tokens, for the same page and window: normalized_result 14,463 to 33,773 bytes, network 6,631 to 16,787 bytes (16 to 40 requests). The growth is the newly observed worker requests and their response metadata. Elapsed time was not separately measured; the fixed observation window dominates it.

Representative entry from the real-Chrome run on this branch (run 1 in the table above): a request from the long-lived blob worker, now in normalized_result.network.requests. Its initiator.url is the worker's own blob: script, so the record shows the request came from the worker:

{
  "request_id": "47880.34",
  "url": "http://127.0.0.1:54325/ping?rea_worker_long_6",
  "origin": "http://127.0.0.1:54325",
  "method": "GET",
  "resource_type": "Fetch",
  "status": 200,
  "mime_type": "application/json",
  "encoded_data_length": 156,
  "redirects": [],
  "initiator": {
    "type": "script",
    "url": "blob:http://127.0.0.1:54325/50c27f5d-cf48-491c-ba22-61f193fc288b",
    "line": 0,
    "column": 29
  },
  "body_shapes": {
    "status": "not_approved",
    "request": null,
    "response": null
  }
}

Limitation added to normalized_result.limitations when auto-attach succeeds:

Dedicated worker network activity is observed from when REA attaches to each worker; requests a worker sends before then, and service or shared worker requests, are unavailable.

Remaining proof gaps: Chrome on Linux/Windows, Edge, and other Chromium builds not run; service/shared workers out of scope; no per-request worker attribution; mid-read worker detach (-32001) during a response-body read is source-derived, not exercised by a test; the 50,000 worker-request budget is exercised at the class level rather than through the WebSocket fake.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run test:focused -- tests/boundary/browser/cdpBrowserProvider.network.test.ts src/browser/CdpPageCaptureWorkerNetwork.test.ts src/browser/CdpPageCaptureWorkers.test.ts: 47 passed (on 599aa65). Full browser suite on 63bdcaf: 943 passed, 6 skipped, 0 failed. Earlier, on 599aa65, two load timeouts appeared while unrelated processes held the CPU; alternating browserMcp.test.ts runs then took 16.6 s and 19.8 s against 17.9 s and a 20.2 s timeout on 3892424. Real Chrome: pre-existing workers now report attached: false instead of REA's own true.
npm run test:focused -- all tests/boundary/browser/*.test.ts, src/browser/*.test.ts, tests/boundary/cli/browserCli.test.ts, tests/boundary/mcp/browserMcp.test.ts: 906 passed, 6 skipped, 1 failed on 080b7a2 (905 / 1 failed on c0707bc). In both runs the failure was browserMcp.test.ts > exposes CLI-equivalent Evidence results and session tools reaching its 20 s timeout while unrelated processes held the CPU. Its time is dominated by the initial binary_session status call, and browser calls finish in under 1 s. Alternating runs under the same load passed in 10.7 s and 11.5 s on c0707bc versus 12.1 s and 12.6 s on 4b894f0. On 080b7a2 it passed alone in 19.5 s and 18.1 s under heavier load. On 4b894f0 the full run was 901 passed, 6 skipped. Earlier runs on 546850a saw historicalWebCapture.test.ts > excludes normalized HAR numbers through CLI and MCP time out at 30 s under full parallel load. That HAR path does not use the CDP capture, and run alone it passes (7 passed).
npm run check:fast: passed.
npm run check: passed (architecture guards across 1456 source files).
npm run docs:check: passed.
npm run verify:test-discovery: discovered all 1417 test files exactly once (on 080b7a2).
Real headless Chrome 155 reproduction above (base vs. this branch).

Compatibility, safety, and release

Breaking changes or migration steps: none. Outputs gain worker requests and one limitation string.
Real Hopper/Ghidra, browser, or OS coverage: real Chrome 155 on macOS only.
Package or release metadata impact: none.
Security, privacy, process, or containment review: worker events pass the same origin policy and redaction as page events. Disallowed-origin workers are not enabled and their URLs are not retained. Workers are never paused, and no script runs in them.

Author readiness

Human owner: @Pharox24

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4456 **Author:** [@Pharox24](https://github.com/Pharox24) **Created:** 10/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/inspect-web-page-worker-network` --- ### 📝 Commits (10+) - [`546850a`](https://github.com/morluto/rea/commit/546850a6c04c4e1e86287bc3fcebb2f399a9fae9) fix(browser): record dedicated worker requests in page inspection - [`4b894f0`](https://github.com/morluto/rea/commit/4b894f0a326a87a0cc73b322aea8f0ad28f800d4) fix(browser): drain worker commands before the final page check - [`c0707bc`](https://github.com/morluto/rea/commit/c0707bc041e788891cc52966a48ee16f9879b90b) fix(browser): bound, nest, and classify worker network observation - [`080b7a2`](https://github.com/morluto/rea/commit/080b7a23710fa7a369ac21ade13cce0b12ec3971) fix(browser): close worker admission before the final page check - [`97a7b01`](https://github.com/morluto/rea/commit/97a7b01d136ab32e9d985c0cb0d1d375ae6e7377) fix(browser): report budget-omitted worker WebSockets in their sections - [`a03fe17`](https://github.com/morluto/rea/commit/a03fe171be3caf645b601b3752a5cd53a6cc86f7) fix(browser): release refused workers and abandon detached body reads - [`fb99fff`](https://github.com/morluto/rea/commit/fb99fff6951a5d3209cc545506b484fa67c50ee9) fix(browser): bound worker releases and frames, report ID collisions - [`b0972bb`](https://github.com/morluto/rea/commit/b0972bbffcdd9488f170fd05188213c085a34db7) fix(browser): spend worker budgets only on retained traffic - [`553c221`](https://github.com/morluto/rea/commit/553c2214fc0dfc5af9f7b89e9a220ab0e57a2ed7) fix(browser): bound worker frame bytes and keep page events off worker records - [`d01d8eb`](https://github.com/morluto/rea/commit/d01d8ebcca43c57eef4b1183fe0e16491edaebc9) fix(browser): bound retained worker request bytes ### 📊 Changes **14 files changed** (+2533 additions, -45 deletions) <details> <summary>View changed files</summary> 📝 `docs/browser-observation.md` (+25 -0) 📝 `src/browser/CdpCaptureEventHandlers.ts` (+37 -20) ➕ `src/browser/CdpCaptureEventHelpers.test.ts` (+37 -0) 📝 `src/browser/CdpCaptureEventHelpers.ts` (+14 -3) 📝 `src/browser/CdpCaptureEvents.ts` (+7 -2) 📝 `src/browser/CdpPageCapture.ts` (+90 -18) ➕ `src/browser/CdpPageCaptureWorkerNetwork.test.ts` (+1003 -0) ➕ `src/browser/CdpPageCaptureWorkerNetwork.ts` (+717 -0) 📝 `src/browser/CdpPageCaptureWorkers.ts` (+4 -1) 📝 `src/browser/CdpSafeMetadata.ts` (+33 -0) 📝 `tests/boundary/browser/cdpBrowserProvider.network.test.ts` (+562 -1) 📝 `tests/fixtures/fakeCdpBrowser.ts` (+1 -0) 📝 `tests/fixtures/fakeCdpBrowserResults.ts` (+1 -0) 📝 `tests/fixtures/fakeCdpBrowserTypes.ts` (+2 -0) </details> ### 📄 Description ## Summary `inspect_web_page` / `rea inspect-web-page` (and `analyze_web_bundle`, which shares the capture) never recorded network requests made by the page's dedicated Web Workers, while still listing those workers as attached. An empty or page-only `network.requests` could be read as "the app made no other requests". This PR observes dedicated-worker network activity during the existing observation window. Fixes #4425 ## Problem and expected behavior Chrome reports a dedicated worker's requests on the worker target, not on the page target that created it. `capturePage` sent `Network.enable` only on the page session (`src/browser/CdpPageCapture.ts`, `enableObservationDomains`) and its event listener returned early for any other `sessionId`. Workers were only enumerated afterwards through `Target.getTargets`, whose `attached` flag reflects any attached client (for example an open DevTools window), not REA observing the worker. Expected (from the issue): worker requests appear in `normalized_result.network.requests`, or coverage states explicitly that worker network activity was not observed. ## Change and scope - New `CdpWorkerNetworkObservation` (`src/browser/CdpPageCaptureWorkerNetwork.ts`), owned by one `capturePage` call: - After the page's own domains are enabled, sends `Target.setAutoAttach { autoAttach: true, waitForDebuggerOnStart: false, flatten: true, filter: [{ type: "worker" }] }` on the page session. Workers are **never paused**; existing workers attach immediately and new ones attach as they start. - For each attached dedicated worker whose URL passes the existing origin policy, enables **only** `Network` on that worker session, then sends it the same non-pausing `Target.setAutoAttach` so workers it starts are observed too. Workers on other origins are not enabled and are recorded as `network_requests` exclusions (`disallowed_origin` / `unsupported_url` / `unattributed_origin`, count `null`). - Forwards only `Network.*` events from admitted worker sessions into the existing `CdpCaptureEvents`, so URL policy, redaction, body-shape rules and completeness accounting are unchanged. - A request belongs to the session that started it (`requestWillBeSent` / `webSocketCreated`). Real Chrome announces a new worker's own script on the **page** session and completes it on the **worker** session with `encodedDataLength: -1`. Without this rule that completion was counted as `invalid_protocol_value` and marked `network_requests` unavailable. With it, the page-owned record is unchanged from current behaviour. - JSON response bodies (`include_json_body_shapes`) for worker requests are read from the worker session. If that worker has already detached (or detaches mid-read, `-32001`), the body is reported unavailable with a limitation instead of failing the capture. - One capture admits at most 256 worker sessions and tracks at most 50,000 worker requests (`CDP_WORKER_NETWORK_LIMITS`, matching the script metadata record bound). Omitted workers and requests count as `resource_budget_exhausted`, so `network_requests` is reported truncated; an omitted worker WebSocket counts under `websocket_connections` and marks `websocket_frames` truncated. A whole unobserved worker (other origin, over the limit, unavailable auto-attach, or a live command rejection) is recorded under all three sections with an unknown count, and up to 256 refused sessions per capture, during the admission window, are released with `Target.detachFromTarget` through their parent session; real Chrome keeps that worker running. Turning auto-attach off at capacity was rejected because real Chrome then also detaches admitted workers. Admitted workers retain at most 50,000 WebSocket frames, with later frames counted in `dropped_events`. Only traffic the origin policy keeps counts toward the frame and request budgets, and with `include_websocket_shapes` worker frame payload is also capped at 8 MiB before shape inference. Worker request and WebSocket URLs, plus request bodies with `include_json_body_shapes`, are capped at 8 MiB before ingestion; a URL the origin policy rejects costs nothing, WebSocket URLs use the same `admittedWebSocketUrl` check as the WebSocket handler, and a request whose redirect hop is over budget stops receiving later events so hops cannot mix. Request bodies are charged only when they get a JSON shape, every hop of a chain the origin policy drops is refunded, and worker JSON response bodies are charged against their own 8 MiB budget before shape inference. Nested worker sessions record their parent, and a parent's detach aborts its descendants. Worker response MIME types and every header value metadata keeps are capped at 8 MiB before metadata parsing, request hops are charged for every string the record keeps (method, type, initiator and redirect details), the frame shape budget applies only to text frames, rejected request hops are refunded, and malformed body envelopes are left to the body parser uncharged. Worker `attached` values come from a `Target.getTargets` snapshot taken before auto-attach, so REA's own attachment is not reported as target evidence. Page-session Network events are checked against worker request ownership before ingestion, so a page start on a worker-owned ID is reported rather than merged. A malformed attachment that still names a session is released. A worker request or WebSocket whose ID another session already owns is reported as `invalid_protocol_value` rather than merged or dropped silently. - Only a worker session that already detached, or a `-32001` session-not-found reply, counts as worker exit and stays under the existing attach-limited `network_requests` coverage. Any other rejection from a live worker records a `provider_unavailable` exclusion. Chrome never answers a command pending on a session that detaches, so each worker's commands, including its JSON body reads, abort when its session detaches instead of waiting for the 20 s CDP deadline; an abandoned body read is reported as unavailable body coverage. Transport and cancellation failures still fail the capture (`settle()`, which runs before the final navigation and origin check and first closes worker admission, so a worker attaching during that check is not enabled), and `close()` waits for issued worker commands before the page session is released. - Browsers without `Target.setAutoAttach` (`-32601`) keep page capture working and report `network_requests` as `provider_unavailable` plus the standard optional-command limitation. - One limitation line is added when auto-attach succeeds: requests a worker sends before REA reaches it, and service/shared worker requests, are unavailable. - `docs/browser-observation.md` documents the coverage. Not included, intentionally: - Service workers and shared workers (the filter selects `worker` only). - A per-request worker `target_id` field. The issue suggests one, but it would change the `WebPageInspection` contract and generated catalog, so it is left for a separate decision. - Pausing workers (`waitForDebuggerOnStart: true`) to close the start-up race. That would give REA execution control over the target, so short-lived workers can be partially observed and the limitation says so. - `observe_web_session` (navigation timeline only) and Electron inspection paths are unchanged. ## Contract and boundary impact - Semantic owner and earliest changed stage: CDP browser adapter, `capturePage` event routing (`src/browser/CdpPageCapture.ts`, new `src/browser/CdpPageCaptureWorkerNetwork.ts`). - Contribution scope fit and linked maintainer decision: fix within the existing passive `inspect_web_page` boundary for the already-authorized page target; no new provider. Auto-attaching the page's own dedicated workers is a new CDP attachment, so maintainers may want to confirm it fits the current boundary. - CLI and MCP/tool-catalog contract: no schema or catalog change. `network.requests` can now include dedicated-worker requests, and one limitation string is added. - Provider, bridge, target-format, or platform compatibility: needs `Target.setAutoAttach` with `flatten` (current Chrome). If unsupported, the capture degrades to `provider_unavailable` for worker network. Source-derived, not run: a browser that ignores `filter` may also auto-attach frames; those are neither enabled nor ingested. - Evidence, artifact, provenance, or reconstruction contract: unchanged; worker records go through the same normalization and completeness ledger. - Target selection, authority, upstream behavior, and cleanup details: only dedicated workers auto-attached under the selected page session or under an admitted worker; workers receive only `Network.enable` and `Target.setAutoAttach`, no `Runtime`/`Debugger`/`Page` domains, no pausing, no script evaluation. Auto-attached sessions are children of the page session REA already detaches, and the operation's socket is closed by `CdpTargetSessionOwner.finish` (source-derived). - Generated metadata, package, or installation impact: none. Expected effects: - [ ] Read existing files or captured data. - [x] Launch or attach to a process. (CDP auto-attach to the inspected page's dedicated worker targets; nothing is launched.) - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: `tests/boundary/browser/cdpBrowserProvider.network.test.ts`, new `dedicated worker network` blocks (4 tests) through the real `CdpBrowserProvider` and the fake CDP WebSocket browser: - worker request recorded; disallowed-origin worker not enabled, its events not retained and its origin absent from output; exact `Target.setAutoAttach` parameters; the only worker-session command is `Network.enable`; a page-announced worker script completed on the worker session with `encodedDataLength: -1` keeps `encoded_data_length: null` and does not mark `network_requests` unavailable; - worker JSON body read with the worker `sessionId`; - `Target.setAutoAttach` unsupported, so `provider_unavailable` exclusion and limitation; - a main-frame navigation delivered while a worker `Network.enable` reply is still pending reports `target_changed` (added for the review finding below). - a worker started by an admitted worker is attached through that worker's session and its request recorded; - a live worker rejecting `Network.enable` (`-32601`) reports `provider_unavailable`, while a gone session (`-32001`) does not; - a worker command Chrome leaves unanswered when the worker detaches is abandoned instead of timing out the capture; - workers are admitted only from auto-attach until the final page check; one attaching during the final frame read is only detached; - a worker JSON body read pending when the worker detaches is reported unavailable instead of timing out the capture (failed with `timeout` before `a03fe17`); - class tests: refused-worker releases capped and none after admission closes; worker WebSocket frames capped with `dropped_events`; page- and worker-owned request ID collisions reported; excluded-origin frames and requests do not spend the budgets; frame payload byte budget with shapes; page events on a worker-owned ID reported and kept off the worker record; malformed attachment with a session released; worker request bytes capped with excluded URLs keeping their reason; an over-budget redirect hop freezes the earlier hop; allowed worker WebSocket URLs count against the byte budget; dropped redirect chains refunded; non-JSON bodies not charged; nested commands abandoned on parent detach; worker JSON response shapes past 8 MiB omitted as `resource_budget_exhausted` (boundary test); expanding response metadata bounded; pre-attach worker `attached` state reported; rejected hops refunded; malformed body envelopes uncharged; long methods, initiator URLs, MIME types and kept header values charged; binary frames outside the shape budget; a JSON response stripped by the metadata budget records `json_body_shapes` as `resource_budget_exhausted` (`97144b7`); base64 bodies sized from their text without decoding, and worker attachments without a target ID released as malformed (`98ba117`). - `src/browser/CdpCaptureEventHelpers.test.ts`: `decodeBase64` canonical and non-canonical cases and a 12 MiB payload. The old grouped base64 pattern overflowed the stack on payloads of several megabytes, on the page path as well; it is replaced with an equivalent linear pattern. - `src/browser/CdpPageCaptureWorkerNetwork.test.ts`: worker and worker-request budgets, each counting omitted items as `resource_budget_exhausted` in the matching section, including an omitted WebSocket. - `tests/fixtures/fakeCdpBrowserResults.ts` answers `Target.setAutoAttach` like Chrome; `holdReplyFor` in the fake lets a test record a command and send its reply itself. - Base reproduction (executed): with only the test and fixture changes on base `c8848648`, the new block fails (`3 failed | 14 passed`). The `-1` cross-session assertion was added after a real-Chrome run surfaced it, and it failed (`1 failed | 16 passed`) until the ownership rule was added. The navigation test returned `ok: true` (a mixed inspection) on `546850a` and passes after `4b894f0` moved `settle()` ahead of the final page check. - Real Chrome (executed): headless Chrome 155.0.8059.80, macOS, local `http://127.0.0.1` page, 8 s window, built CLI `rea inspect-web-page … --format json`. The page fetches every second from the page, a long-lived blob worker, a same-origin script worker, and a new short-lived blob worker per round. Distinct markers recorded: | build | page | long-lived blob worker | script worker | new short-lived workers | `unavailable_sections` | |---|---|---|---|---|---| | base `c8848648` | 8 | 0 | 0 | 0 | `[]` | | this PR, run 1 | 8 | 8 | 8 | 8 | `[]` | | this PR, run 2 | 8 | 8 | 8 | 7 | `[]` | | this PR `4b894f0`, run 3 | 9 | 8 | 8 | 8 | `[]` | For `c0707bc` the harness also starts a worker from another worker 1.5 s after load. Four consecutive runs recorded 8 page / 8 long-lived / 8 script / 8 short-lived / 8 nested-worker markers, each with `unavailable_sections: []` and no `network_requests` exclusions. Three further runs on `080b7a2` recorded the same (one run recorded 9 long-lived-worker markers), and three on `a03fe17` did too (one run recorded 7 short-lived-worker markers, the documented start-up race). Six runs on `fb99fff` recorded 8 page / 8–9 long-lived / 8 script / 6–8 short-lived / 8 nested markers with no `network_requests` exclusions. Three back-to-back runs of `a03fe17` under the same machine load recorded 7–8 short-lived markers. The change does not touch admitted-worker setup, so the short-lived spread is the start-up race under load. During this work one run failed with `provider_timeout: CDP command Target.setAutoAttach exceeded its 20000 ms response deadline`. A raw CDP trace showed Chrome had detached that short-lived worker 14 ms after the command and never replied; the per-worker abort fixes it, and the boundary test above reproduces it. Run 2 missed one short-lived worker whose fetch ran before `Network.enable` reached it; that is the documented start-up race. - User-visible output: worker requests appear as ordinary `network.requests` entries. Measured JSON bytes, not tokens, for the same page and window: `normalized_result` 14,463 to 33,773 bytes, `network` 6,631 to 16,787 bytes (16 to 40 requests). The growth is the newly observed worker requests and their response metadata. Elapsed time was not separately measured; the fixed observation window dominates it. Representative entry from the real-Chrome run on this branch (run 1 in the table above): a request from the long-lived blob worker, now in `normalized_result.network.requests`. Its `initiator.url` is the worker's own `blob:` script, so the record shows the request came from the worker: ```json { "request_id": "47880.34", "url": "http://127.0.0.1:54325/ping?rea_worker_long_6", "origin": "http://127.0.0.1:54325", "method": "GET", "resource_type": "Fetch", "status": 200, "mime_type": "application/json", "encoded_data_length": 156, "redirects": [], "initiator": { "type": "script", "url": "blob:http://127.0.0.1:54325/50c27f5d-cf48-491c-ba22-61f193fc288b", "line": 0, "column": 29 }, "body_shapes": { "status": "not_approved", "request": null, "response": null } } ``` Limitation added to `normalized_result.limitations` when auto-attach succeeds: ```text Dedicated worker network activity is observed from when REA attaches to each worker; requests a worker sends before then, and service or shared worker requests, are unavailable. ``` - Remaining proof gaps: Chrome on Linux/Windows, Edge, and other Chromium builds not run; service/shared workers out of scope; no per-request worker attribution; mid-read worker detach (`-32001`) during a response-body read is source-derived, not exercised by a test; the 50,000 worker-request budget is exercised at the class level rather than through the WebSocket fake. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- tests/boundary/browser/cdpBrowserProvider.network.test.ts src/browser/CdpPageCaptureWorkerNetwork.test.ts src/browser/CdpPageCaptureWorkers.test.ts`: 47 passed (on `599aa65`). Full browser suite on `63bdcaf`: 943 passed, 6 skipped, 0 failed. Earlier, on `599aa65`, two load timeouts appeared while unrelated processes held the CPU; alternating `browserMcp.test.ts` runs then took 16.6 s and 19.8 s against 17.9 s and a 20.2 s timeout on `3892424`. Real Chrome: pre-existing workers now report `attached: false` instead of REA's own `true`. - `npm run test:focused --` all `tests/boundary/browser/*.test.ts`, `src/browser/*.test.ts`, `tests/boundary/cli/browserCli.test.ts`, `tests/boundary/mcp/browserMcp.test.ts`: 906 passed, 6 skipped, 1 failed on `080b7a2` (905 / 1 failed on `c0707bc`). In both runs the failure was `browserMcp.test.ts > exposes CLI-equivalent Evidence results and session tools` reaching its 20 s timeout while unrelated processes held the CPU. Its time is dominated by the initial `binary_session` status call, and browser calls finish in under 1 s. Alternating runs under the same load passed in 10.7 s and 11.5 s on `c0707bc` versus 12.1 s and 12.6 s on `4b894f0`. On `080b7a2` it passed alone in 19.5 s and 18.1 s under heavier load. On `4b894f0` the full run was 901 passed, 6 skipped. Earlier runs on `546850a` saw `historicalWebCapture.test.ts > excludes normalized HAR numbers through CLI and MCP` time out at 30 s under full parallel load. That HAR path does not use the CDP capture, and run alone it passes (7 passed). - `npm run check:fast`: passed. - `npm run check`: passed (architecture guards across 1456 source files). - `npm run docs:check`: passed. - `npm run verify:test-discovery`: discovered all 1417 test files exactly once (on `080b7a2`). - Real headless Chrome 155 reproduction above (base vs. this branch). ## Compatibility, safety, and release - Breaking changes or migration steps: none. Outputs gain worker requests and one limitation string. - Real Hopper/Ghidra, browser, or OS coverage: real Chrome 155 on macOS only. - Package or release metadata impact: none. - Security, privacy, process, or containment review: worker events pass the same origin policy and redaction as page events. Disallowed-origin workers are not enabled and their URLs are not retained. Workers are never paused, and no script runs in them. ## Author readiness Human owner: @Pharox24 - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:13:03 +00:00

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:46:19 +00:00

[PR #6253] [MERGED] fix(doctor): check only the current platform's Hopper launcher candidates #7228