[GH-ISSUE #8140] [Bug] One failed subresource request marks a browser scenario's whole events section missing, so compare_web_captures can only return unknown #4684

New issue

Closed

opened 2026-10-31 00:52:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:52:26 +00:00

Owner

Originally created by @Rinkia on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8140

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Browser and web observation

Observed behavior

I ran capture-browser-scenario with capture.network.response_body: true on a local page that loads one script returning 404. Chromium aborts that script request (net::ERR_ABORTED). For that request, the capture records:

a request-failed event;
a network-content item with body: {"state": "unavailable", "reason": "request-failed"}.

These records are correct. However, the capture's completeness becomes:

{"status": "incomplete", "equality_eligible": false, "missing_sections": ["events"]}

The capture still holds all 51 events and dropped none (events.retained: 51, dropped: 0). Its limitations include "Some selected network content was unavailable".

compare-web-captures then reports the events artifact of every step as unknown ("Artifact coverage is not comparable (missing before, …)"):

Before	After	Result
Server variant 1	Same server, run again	`overall_status: "unknown"` for all 3 steps. With the 404 removed, the same pair is `unchanged`.
Server variant 1	Server variant 2, where the fetched `/api/data` JSON body differs and the 404 script now returns 200	`unknown` for all steps. With the 404 removed on both sides, the same comparison is `changed` at `scenario_start`.

The retained events of the second pair differ: the /api/data response body has a different SHA-256, and one transaction is 200 on one side and failed on the other. Even so, those differences cannot prove a change, because the whole section is classified as missing.

Likely cause

src/browser/network/PlaywrightScenarioNetwork.ts (around lines 716–722) adds the gap for any retained body in state unavailable. That includes bodies whose reason is request-failed, set in failed() just above. A failed request has no response body to read, so this is the observed outcome, not a coverage gap.
globalCompleteness in src/browser/PlaywrightBrowserScenarioProvider.ts then adds "events" to missing_sections whenever eventLimitations() is non-empty.

Expected behavior

A failed request's unavailable body should not count as missing event coverage. Its request-failed event already records the outcome, and the comparison can compare that outcome.

More generally, a per-transaction content gap could leave only that transaction's content unknown. Today it makes the event inventory of the whole capture non-comparable.

Reproduction

Container: Linux x64, Node 24.22.1, bundled Chrome for Testing 145.0.7632.6 in launch mode. Seen on the 6.8.0 release and on main at 45cc93596.

Serve a page from 127.0.0.1:3100 that:
- loads /app.js;
- fetches /api/data;
- loads one <script src="/extra.js"> that returns 404.

Run this scenario twice with rea capture-browser-scenario scenario.json --format json:

{
  "browser": {"mode": "launch", "executable_path": "/usr/local/bin/chromium"},
  "start_url": {"url": "http://127.0.0.1:3100/"},
  "actions": [
    {"step_id": "click", "action": "click", "locator": {"kind": "css", "selector": "#b"}, "timeout_ms": 5000},
    {"step_id": "settle", "action": "wait_for_timeout", "duration_ms": 800}
  ],
  "capture": {"events": ["network", "console"], "network": {"response_body": true}, "at_end": ["dom", "screenshot", "url"]}
}

Check normalized_result.completeness of each capture.
Pass the two normalized_result objects to rea compare-web-captures before.json after.json.
Make /extra.js return 200 and repeat. Both captures are now complete, and the comparison is unchanged.

Additional context

I did not test the not_exposed request-body state, which goes through the same branch, or connect mode.

Originally created by @Rinkia on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8140 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Browser and web observation ### Observed behavior I ran `capture-browser-scenario` with `capture.network.response_body: true` on a local page that loads one script returning 404. Chromium aborts that script request (`net::ERR_ABORTED`). For that request, the capture records: - a `request-failed` event; - a `network-content` item with `body: {"state": "unavailable", "reason": "request-failed"}`. These records are correct. However, the capture's completeness becomes: ```json {"status": "incomplete", "equality_eligible": false, "missing_sections": ["events"]} ``` The capture still holds all 51 events and dropped none (`events.retained: 51, dropped: 0`). Its limitations include "Some selected network content was unavailable". `compare-web-captures` then reports the events artifact of **every** step as `unknown` ("Artifact coverage is not comparable (missing before, …)"): | Before | After | Result | |---|---|---| | Server variant 1 | Same server, run again | `overall_status: "unknown"` for all 3 steps. With the 404 removed, the same pair is `unchanged`. | | Server variant 1 | Server variant 2, where the fetched `/api/data` JSON body differs and the 404 script now returns 200 | `unknown` for all steps. With the 404 removed on both sides, the same comparison is `changed` at `scenario_start`. | The retained events of the second pair differ: the `/api/data` response body has a different SHA-256, and one transaction is 200 on one side and failed on the other. Even so, those differences cannot prove a change, because the whole section is classified as missing. ### Likely cause - `src/browser/network/PlaywrightScenarioNetwork.ts` (around lines 716–722) adds the gap for any retained body in state `unavailable`. That includes bodies whose reason is `request-failed`, set in `failed()` just above. A failed request has no response body to read, so this is the observed outcome, not a coverage gap. - `globalCompleteness` in `src/browser/PlaywrightBrowserScenarioProvider.ts` then adds `"events"` to `missing_sections` whenever `eventLimitations()` is non-empty. ### Expected behavior A failed request's unavailable body should not count as missing event coverage. Its `request-failed` event already records the outcome, and the comparison can compare that outcome. More generally, a per-transaction content gap could leave only that transaction's content `unknown`. Today it makes the event inventory of the whole capture non-comparable. ### Reproduction Container: Linux x64, Node 24.22.1, bundled Chrome for Testing 145.0.7632.6 in launch mode. Seen on the 6.8.0 release and on `main` at `45cc93596`. 1. Serve a page from `127.0.0.1:3100` that: - loads `/app.js`; - fetches `/api/data`; - loads one `<script src="/extra.js">` that returns 404. 2. Run this scenario twice with `rea capture-browser-scenario scenario.json --format json`: ```json { "browser": {"mode": "launch", "executable_path": "/usr/local/bin/chromium"}, "start_url": {"url": "http://127.0.0.1:3100/"}, "actions": [ {"step_id": "click", "action": "click", "locator": {"kind": "css", "selector": "#b"}, "timeout_ms": 5000}, {"step_id": "settle", "action": "wait_for_timeout", "duration_ms": 800} ], "capture": {"events": ["network", "console"], "network": {"response_body": true}, "at_end": ["dom", "screenshot", "url"]} } ``` 3. Check `normalized_result.completeness` of each capture. 4. Pass the two `normalized_result` objects to `rea compare-web-captures before.json after.json`. 5. Make `/extra.js` return 200 and repeat. Both captures are now `complete`, and the comparison is `unchanged`. ### Additional context I did not test the `not_exposed` request-body state, which goes through the same branch, or `connect` mode.

bsakaguchi closed this issue

2026-10-31 00:52:26 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:50 +00:00

[PR #4684] [CLOSED] fix(frida): return canonical device_id in list-frida-devices cli output #6357