[GH-ISSUE #1830] [Bug] HTTP capture comparison treats unavailable response bytes as an observed change #615

New issue

Closed

opened 2026-10-15 18:26:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:26:39 +00:00

Owner

Originally created by @shoyann on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1830

Before submitting

I searched open and closed issues and pull requests for related reports.
The reproduction uses generated local fixture data without credentials or private targets.

Area

Evidence, artifacts, or reconstruction; browser capture comparison (CLI/shared workflow).

Observed behavior

Comparing two captures of the same HTTP response reports network.status: "changed" and a modified request when the only difference is whether Network.loadingFinished was observed. In the earlier capture encoded_data_length is null; in the later capture it is 100. The request URL/method, response status (200), MIME type (application/json), initiator, and redirects are the same.

The result's own limitation says that a changed status proves an observed difference. Missing byte-count evidence does not prove that the response changed.

Reproduction

On a source checkout, build the CLI and use the existing startFakeCdpBrowser fixture with the real CdpBrowserProvider:

Start a loopback CDP HTTP/WebSocket fixture and emit Network.requestWillBeSent for a GET /api/items, followed by Network.responseReceived with status 200 and MIME type application/json.
Capture with inspectPage using observation_ms: 0, without emitting Network.loadingFinished.
Capture the same event sequence again, adding Network.loadingFinished with encodedDataLength: 100. Keep the same server, URL, and response facts.
Compare { before: { inspection: first }, after: { inspection: second } } through compareWebCaptures. The same result is reachable through rea compare-web-captures before.json after.json --json, with each JSON file containing { "inspection": ... }.

The production provider observes the real loopback protocol exchange. This reproduction uses a simulated browser, not native Chrome.

Expected behavior

The network dimension should remain unknown with zero proven changes when the only discrepancy is an unavailable response field. A status change from 200 to 500, or different byte counts observed on both sides, should still be changed. Repeated requests with the same method/URL/type must not be paired by capture-specific request IDs or array order.

Relevant evidence

before: status=200, mime_type=application/json, encoded_data_length=null
after:  status=200, mime_type=application/json, encoded_data_length=100

actual:   network.status=changed, total_changes=1, change=modified
expected: network.status=unknown, total_changes=0

Reverse comparison: also incorrectly changed.
Identical complete responses: unknown (attachment-window limitation).
200 -> 500 with observed fields: changed, as expected.

Impact and scope

This creates false positive changes in HTTP capture comparison and can make the overall comparison changed even when no dimension contains a proven difference. The primary reproduction concerns top-level response metadata, particularly captures that end before a request finishes. It does not require changing the observation provider or its serialized capture schema.

Version and environment

REA source revision: 76e4153 (also reproduced on f178bf1c).
Node.js 24.18.0, npm 11.16.0, WSL Ubuntu x64.
Interface: shared comparison workflow and public CLI.
Earliest failure stage: comparison/projection of observed response facts.

Source-derived analysis

src/domain/webCaptureDiff.ts hashes nullable status, mime_type, and encoded_data_length directly into each grouped request projection. The shared comparator treats a changed fingerprint as a proven modification even when network coverage is incomplete. A fix should compare jointly observed fields while retaining the association between known fields of repeated requests.

I am preparing a focused fix and producer/CLI regression coverage.

Originally created by @shoyann on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1830 ### Before submitting - [x] I searched open and closed issues and pull requests for related reports. - [x] The reproduction uses generated local fixture data without credentials or private targets. ### Area Evidence, artifacts, or reconstruction; browser capture comparison (CLI/shared workflow). ### Observed behavior Comparing two captures of the same HTTP response reports `network.status: "changed"` and a `modified` request when the only difference is whether `Network.loadingFinished` was observed. In the earlier capture `encoded_data_length` is `null`; in the later capture it is `100`. The request URL/method, response status (`200`), MIME type (`application/json`), initiator, and redirects are the same. The result's own limitation says that a changed status proves an observed difference. Missing byte-count evidence does not prove that the response changed. ### Reproduction On a source checkout, build the CLI and use the existing `startFakeCdpBrowser` fixture with the real `CdpBrowserProvider`: 1. Start a loopback CDP HTTP/WebSocket fixture and emit `Network.requestWillBeSent` for a `GET /api/items`, followed by `Network.responseReceived` with status `200` and MIME type `application/json`. 2. Capture with `inspectPage` using `observation_ms: 0`, without emitting `Network.loadingFinished`. 3. Capture the same event sequence again, adding `Network.loadingFinished` with `encodedDataLength: 100`. Keep the same server, URL, and response facts. 4. Compare `{ before: { inspection: first }, after: { inspection: second } }` through `compareWebCaptures`. The same result is reachable through `rea compare-web-captures before.json after.json --json`, with each JSON file containing `{ "inspection": ... }`. The production provider observes the real loopback protocol exchange. This reproduction uses a simulated browser, not native Chrome. ### Expected behavior The network dimension should remain `unknown` with zero proven changes when the only discrepancy is an unavailable response field. A status change from `200` to `500`, or different byte counts observed on both sides, should still be `changed`. Repeated requests with the same method/URL/type must not be paired by capture-specific request IDs or array order. ### Relevant evidence ```text before: status=200, mime_type=application/json, encoded_data_length=null after: status=200, mime_type=application/json, encoded_data_length=100 actual: network.status=changed, total_changes=1, change=modified expected: network.status=unknown, total_changes=0 Reverse comparison: also incorrectly changed. Identical complete responses: unknown (attachment-window limitation). 200 -> 500 with observed fields: changed, as expected. ``` ### Impact and scope This creates false positive changes in HTTP capture comparison and can make the overall comparison `changed` even when no dimension contains a proven difference. The primary reproduction concerns top-level response metadata, particularly captures that end before a request finishes. It does not require changing the observation provider or its serialized capture schema. ### Version and environment - REA source revision: `76e4153` (also reproduced on `f178bf1c`). - Node.js 24.18.0, npm 11.16.0, WSL Ubuntu x64. - Interface: shared comparison workflow and public CLI. - Earliest failure stage: comparison/projection of observed response facts. ### Source-derived analysis `src/domain/webCaptureDiff.ts` hashes nullable `status`, `mime_type`, and `encoded_data_length` directly into each grouped request projection. The shared comparator treats a changed fingerprint as a proven modification even when network coverage is incomplete. A fix should compare jointly observed fields while retaining the association between known fields of repeated requests. I am preparing a focused fix and producer/CLI regression coverage.

bsakaguchi

2026-10-15 18:26:39 +00:00

closed this issue
added the
area: runtime

bug
labels

bsakaguchi commented

2026-10-15 18:26:40 +00:00

Author

Owner

@morluto commented on GitHub (Oct 11, 2026):

Fixed on main in 486b56724. HTTP comparison projects jointly observed response fields before fingerprinting; unavailable fields retain uncertainty, while proved field and complete-inventory differences remain changes. Boundary regressions pass, including missing byte counts in both directions and repeated-request controls. PR #1834 remains open for its additional protocol-driven and public CLI regression coverage.

@morluto commented on GitHub (Oct 11, 2026): Fixed on main in [486b56724](https://github.com/morluto/rea/commit/486b56724c4eaba50719177bcb9eae84d6907297). HTTP comparison projects jointly observed response fields before fingerprinting; unavailable fields retain uncertainty, while proved field and complete-inventory differences remain changes. Boundary regressions pass, including missing byte counts in both directions and repeated-request controls. PR #1834 remains open for its additional protocol-driven and public CLI regression coverage.

bsakaguchi referenced this issue

2026-10-16 21:14:29 +00:00

[PR #615] [MERGED] chore: migrate formatting to Oxfmt #2115

bsakaguchi referenced this issue

2026-10-16 21:14:30 +00:00

[PR #617] [MERGED] fix(repo): keep dependency links out of version control #2117