[PR #992] [MERGED] feat(browser): inspect historical web network captures #2561

New issue

Closed

opened 2026-10-17 21:07:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:07:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/992
Author: @N0zoM1z0
Created: 10/7/2026
Status: ✅ Merged
Merged: 10/7/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/web-network-history

📝 Commits (10+)

3d3158e feat(browser): inspect retained HAR and mitmproxy capture evidence
0a65842 fix(browser): preserve capture boundaries during cancellation and decoding
4626a48 fix(browser): exclude sensitive capture fields across representations
2b24881 Merge remote-tracking branch 'origin/main' into feat/web-network-history
c7bfec0 Merge remote-tracking branch 'origin/main' into feat/web-network-history
0a4827f fix(browser): exclude sensitive capture identities and bound diagnostics
4327179 Merge remote-tracking branch 'origin/main' into feat/web-network-history
5cc681e fix(browser): validate excluded HAR subtrees before projection
0b6b5c0 Merge remote-tracking branch 'origin/main' into feat/web-network-history
f526d5c fix(browser): distinguish capture input limits from decoder failures

📊 Changes

77 files changed (+6315 additions, -47 deletions)

View changed files

➕ .github/workflows/real-web-network-captures.yml (+75 -0)
📝 README.md (+9 -1)
📝 README_ar.md (+1 -1)
📝 README_ja.md (+1 -1)
📝 README_ko.md (+1 -1)
📝 README_zh.md (+1 -1)
➕ bridge/mitmproxy/capture.py (+210 -0)
📝 docs/error-contract.schema.json (+68 -0)
📝 docs/product-catalog.json (+23 -9)
📝 docs/verification/managed-conformance-manifest.json (+2 -2)
➕ docs/web-network-captures.md (+147 -0)
📝 package-lock.json (+36 -6)
📝 package.json (+7 -2)
➕ scripts/fixtures/generate-mitmproxy-capture.py (+151 -0)
📝 scripts/lib/catalog-builders.mjs (+13 -0)
📝 scripts/lib/catalog-core.mjs (+3 -0)
➕ scripts/verify-web-network-captures.mjs (+994 -0)
📝 skills/reverse-engineer-anything/SKILL.md (+8 -3)
➕ src/application/WebNetworkCapturePort.ts (+15 -0)
➕ src/application/WebNetworkCaptureService.test.ts (+340 -0)

...and 57 more files

📄 Description

Summary

Add offline historical web-network inspection through equivalent CLI and MCP operations. Refs #882 and #774; live binary WebSocket/SSE retention remains separate work, so this does not close #882.

Problem and expected behavior

Inspect a caller-selected producer capture without fetching recorded URLs. Return original fields, record identities and locations, exposed exact bytes and numeric literals, artifact identity and inline Evidence. Historical records do not establish browser execution or deployment authenticity.

Change and scope

Pure domain schemas and shared application workflow; replaceable HAR/native adapters under src/browser/history/.
HAR 1.2 uses unchanged locked JSON/schema packages. A strict jsonc-parser visitor rejects all duplicate members before materialization, including equal values and escaped names, while retaining original numeric lexemes and exact nesting-error locations. Native captures use a thin bridge over unchanged mitmproxy 12.2.3 tnetstring decoding, without FlowReader migrations or HAR conversion.
Owned snapshots and process supervision bound decoding, retained diagnostics, deadlines/cancellation and independent cleanup. No listener, target requests, provider installation or host-default changes.
Preserve opaque producer extensions and ordinary URLs/query data. Exclude known transport authentication and caller-marked literals structurally, including sensitive property identities/coordinates, backups, leading header whitespace and non-UTF-8 credential-bearing byte sidecars.
Retain malformed-input constraints, native/provider failures, host read-denial errno and structured failures preceding cleanup. Sensitive coordinates become explicit unknowns rather than invented identities. Extracted HAR records inherit whole-subtree exclusions from log/entries names: null payloads and empty sidecars, retaining original ordinals/count. Literal unknown argument names are distinguished from producer pointers.
Add provider/provenance docs, source and production-boundary regressions, packaged bridge, generated contracts/catalog and conditional real-producer CI.

Contract and boundary impact

Semantic owner/earliest stage: historical domain schema; adapters validate original producer representations before projection and application Evidence.
CLI/MCP: inspect-web-network-capture / inspect_web_network_capture; explicit local path/format, optional original ordinal selection and sensitive literal values. Named contracts and SDK-converted input/output schemas remain valid.
Provider profile: HAR 1.2; bring-your-own mitmdump 12.2.3 native decoding on Linux only. Complete selected provider id/name/version is carried into Evidence and verified at the decoder boundary. Exact upstream provenance/digest documented; mismatched profiles are unsupported.
Evidence: original stable SHA-256/size (explicitly declared digest matches set artifact.sha256 and Evidence.subject to null with an explanation), record sequence, JSON pointers or byte ranges, missing/empty/redacted distinctions, exact producer-byte and numeric sidecars; matching sensitive numeric lexemes, native byte-range coordinates and reported values are excluded with a real-location redaction. Runtime attribution remains unknown. Explicitly sensitive coordinates are omitted; affected record location is { "kind": "unknown", "reason": "explicit-sensitive-value" }.
Error contract: additive access_denied code with unavailable category, selected path, errno, operation and read boundary; retryable artifact_changed / integrity_mismatch for concurrent stable-read changes. IO/provider failures keep their original category. Explicit-sensitive failure projection also covers application-generated errors. Cleanup retains the prior structured projected error, including streams, status and truncation.
Limits/effects: 32 MiB input, 96 MiB complete reply, 64-level nesting and 30-second owned-command deadline. HAR uses a 192 MiB Node old-generation heap; native decoding a 768 MiB Linux address-space limit. These are not aggregate RSS claims. Limits fail without successful partial evidence; cleanup independently completes before return.
Packaging: canonical inventory and generated schema/catalog/skill/conformance metadata updated; native bridge packaged, dependencies locked. No release publishing in this PR.

Evidence and regression coverage

Real fixtures are generated by unchanged mitmproxy FlowWriter/SaveHar. Coverage includes selection/order, exact binary bodies/WebSocket messages/trailers/extensions, large numeric values, string/byte transport URLs and backups, sensitive key/path/coordinate exclusions, malformed native records and omitted subtrees, noncanonical HAR base64, input/depth/reply bounds, access denial, unavailable/version-mismatched engines, cancellation and cleanup.

Before-fix proofs are retained locally: source property/path/diagnostic regressions (9 failures), hidden HAR validation (2), input acquisition/classification (2), decoder failure categories (4), leading whitespace (3), the coordinate/permission/structured-cleanup batch (6), and authored-limitations collisions (4). Real prior installed bridges also demonstrated invalid omitted credential acceptance, wrong native-key classification, and non-UTF-8 credential retention. Follow-ups fix these before merge.

{
  "name": "inspect_web_network_capture",
  "arguments": {
    "capture_path": "/absolute/session.har",
    "format": "har",
    "record_ordinals": [2, 0]
  }
}

rea inspect-web-network-capture /absolute/session.har har --record 2 --record 0 --json

The result includes normalized_result with artifact identity, selected producer records/locations, reported fields, exact byte/numeric sidecars, redactions, limitations and runtime_attribution: "unknown".

Observed, derived, inferred and unknown claims remain distinguishable.
Artifact identity, producer provenance and meaningful failures remain preserved.
Unsupported, unavailable, excluded or incomplete facets remain visible.

Validation performed

Current head 02248703de, including upstream main 6e4535c1:

Final focused regression/contract batch: 130 tests / 11 files passed, including numeric sensitive literals, complete provider identity validation, actual SDK calls and advertised schema dialect validity. Before-fix failures also cover canonicalized byte representations and the ownership preparation deadline. Main integration also passed 191 tests / 8 files across stable readers, artifact inventory and module boundaries. These are separate runs, not additive unique coverage.
Cached build, npm run check:fast, pre-push checks, generated documents and npm run docs:check passed. Five additional targeted repetitions passed all10 Linux PID-normalization cases. A single CI capture failure could not be reproduced; its test now reports the tagged failure instead of masking it with a boolean assertion. Earlier ownership/prelaunch cancellation coverage passed22 tests; initial Node 24.18 uncaught-AbortError CI failure was reproduced and fixed rather than retried blindly.
Final-head installed package passed 144 real CLI/MCP cases, plus seven native failure classifications, using unchanged mitmproxy 12.2.3. Coverage includes numeric declarations, full provider provenance and generated base64/digest declarations. Original artifacts unchanged and owned descendants [] in completed runs.
Deterministic actual-filesystem probes cover unlink-before-open, unlink-after-read, symlink replacement, permission denial and closure after prelaunch cancellation. Initial missing input remains distinct; Windows symlink creation is explicitly skipped.
Tarball includes the native bridge; reference, private-game, cache, planning and nested dependency paths are excluded. No host defaults changed.
Final-head CI and automated review are required merge gates. Previous Apple E2E hit hdiutil Resource temporarily unavailable; the next normal CI run passed that lane. No product timeout or check was weakened.

Compatibility, safety, and release

Additive operation/error/location metadata. Native capture verification is Linux only; this offline increment makes no new real Hopper/Ghidra or live-browser support claim. HAR decoded text does not establish wire bytes, and assembled mitmproxy messages do not establish original wire fragments. Both formats explicitly reject proto members because the current shared JSON schema cannot preserve them; follow-up #1036 records the traced boundary and acceptance. Live binary WS/SSE, workers/WASM and proxy capture are outside this increment. Source-owned fixtures only; no private game data. REA 5.0.0 was published separately from its frozen checkpoint.

Review checklist

Focused outcome and Conventional Commit title.
Related issues linked; remaining work explicitly preserved.
Meaningful observable behavior and failure paths tested.
Owning docs, contracts, package and generated metadata updated.
Representative CLI/MCP request and result fields documented.
Final diff checked for credentials, unrelated changes and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/992 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/7/2026 **Status:** ✅ Merged **Merged:** 10/7/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/web-network-history` --- ### 📝 Commits (10+) - [`3d3158e`](https://github.com/morluto/rea/commit/3d3158eb50615bcfa47241840b5b4513bb7e48a9) feat(browser): inspect retained HAR and mitmproxy capture evidence - [`0a65842`](https://github.com/morluto/rea/commit/0a658429b5f7c56284039102011c427c8382b295) fix(browser): preserve capture boundaries during cancellation and decoding - [`4626a48`](https://github.com/morluto/rea/commit/4626a484673dbfae1e3cca6c71b2629026bdc03a) fix(browser): exclude sensitive capture fields across representations - [`2b24881`](https://github.com/morluto/rea/commit/2b248814a2450d28012f69860d03320ef38f20b2) Merge remote-tracking branch 'origin/main' into feat/web-network-history - [`c7bfec0`](https://github.com/morluto/rea/commit/c7bfec08b2284f49d69353f5f192d1bee0a63750) Merge remote-tracking branch 'origin/main' into feat/web-network-history - [`0a4827f`](https://github.com/morluto/rea/commit/0a4827f32429fb7ed2210e63f77a1916286c28c1) fix(browser): exclude sensitive capture identities and bound diagnostics - [`4327179`](https://github.com/morluto/rea/commit/432717980ed5b30feb4b563e4435d1d6d2795187) Merge remote-tracking branch 'origin/main' into feat/web-network-history - [`5cc681e`](https://github.com/morluto/rea/commit/5cc681edcaf12f792dfdb55c9787bd46fb388a39) fix(browser): validate excluded HAR subtrees before projection - [`0b6b5c0`](https://github.com/morluto/rea/commit/0b6b5c00643952664c96a096f73db0a504529a6f) Merge remote-tracking branch 'origin/main' into feat/web-network-history - [`f526d5c`](https://github.com/morluto/rea/commit/f526d5c287d12be341ef0c4ebdd29bba673289e5) fix(browser): distinguish capture input limits from decoder failures ### 📊 Changes **77 files changed** (+6315 additions, -47 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/real-web-network-captures.yml` (+75 -0) 📝 `README.md` (+9 -1) 📝 `README_ar.md` (+1 -1) 📝 `README_ja.md` (+1 -1) 📝 `README_ko.md` (+1 -1) 📝 `README_zh.md` (+1 -1) ➕ `bridge/mitmproxy/capture.py` (+210 -0) 📝 `docs/error-contract.schema.json` (+68 -0) 📝 `docs/product-catalog.json` (+23 -9) 📝 `docs/verification/managed-conformance-manifest.json` (+2 -2) ➕ `docs/web-network-captures.md` (+147 -0) 📝 `package-lock.json` (+36 -6) 📝 `package.json` (+7 -2) ➕ `scripts/fixtures/generate-mitmproxy-capture.py` (+151 -0) 📝 `scripts/lib/catalog-builders.mjs` (+13 -0) 📝 `scripts/lib/catalog-core.mjs` (+3 -0) ➕ `scripts/verify-web-network-captures.mjs` (+994 -0) 📝 `skills/reverse-engineer-anything/SKILL.md` (+8 -3) ➕ `src/application/WebNetworkCapturePort.ts` (+15 -0) ➕ `src/application/WebNetworkCaptureService.test.ts` (+340 -0) _...and 57 more files_ </details> ### 📄 Description ## Summary Add offline historical web-network inspection through equivalent CLI and MCP operations. Refs #882 and #774; live binary WebSocket/SSE retention remains separate work, so this does not close #882. ## Problem and expected behavior Inspect a caller-selected producer capture without fetching recorded URLs. Return original fields, record identities and locations, exposed exact bytes and numeric literals, artifact identity and inline Evidence. Historical records do not establish browser execution or deployment authenticity. ## Change and scope - Pure domain schemas and shared application workflow; replaceable HAR/native adapters under `src/browser/history/`. - HAR 1.2 uses unchanged locked JSON/schema packages. A strict jsonc-parser visitor rejects all duplicate members before materialization, including equal values and escaped names, while retaining original numeric lexemes and exact nesting-error locations. Native captures use a thin bridge over unchanged mitmproxy 12.2.3 tnetstring decoding, without FlowReader migrations or HAR conversion. - Owned snapshots and process supervision bound decoding, retained diagnostics, deadlines/cancellation and independent cleanup. No listener, target requests, provider installation or host-default changes. - Preserve opaque producer extensions and ordinary URLs/query data. Exclude known transport authentication and caller-marked literals structurally, including sensitive property identities/coordinates, backups, leading header whitespace and non-UTF-8 credential-bearing byte sidecars. - Retain malformed-input constraints, native/provider failures, host read-denial errno and structured failures preceding cleanup. Sensitive coordinates become explicit unknowns rather than invented identities. Extracted HAR records inherit whole-subtree exclusions from log/entries names: null payloads and empty sidecars, retaining original ordinals/count. Literal unknown argument names are distinguished from producer pointers. - Add provider/provenance docs, source and production-boundary regressions, packaged bridge, generated contracts/catalog and conditional real-producer CI. ## Contract and boundary impact - Semantic owner/earliest stage: historical domain schema; adapters validate original producer representations before projection and application Evidence. - CLI/MCP: `inspect-web-network-capture` / `inspect_web_network_capture`; explicit local path/format, optional original ordinal selection and sensitive literal values. Named contracts and SDK-converted input/output schemas remain valid. - Provider profile: HAR 1.2; bring-your-own mitmdump 12.2.3 native decoding on Linux only. Complete selected provider id/name/version is carried into Evidence and verified at the decoder boundary. Exact upstream provenance/digest documented; mismatched profiles are unsupported. - Evidence: original stable SHA-256/size (explicitly declared digest matches set artifact.sha256 and Evidence.subject to null with an explanation), record sequence, JSON pointers or byte ranges, missing/empty/redacted distinctions, exact producer-byte and numeric sidecars; matching sensitive numeric lexemes, native byte-range coordinates and reported values are excluded with a real-location redaction. Runtime attribution remains unknown. Explicitly sensitive coordinates are omitted; affected record location is `{ "kind": "unknown", "reason": "explicit-sensitive-value" }`. - Error contract: additive `access_denied` code with unavailable category, selected path, errno, operation and read boundary; retryable `artifact_changed` / `integrity_mismatch` for concurrent stable-read changes. IO/provider failures keep their original category. Explicit-sensitive failure projection also covers application-generated errors. Cleanup retains the prior structured projected error, including streams, status and truncation. - Limits/effects: 32 MiB input, 96 MiB complete reply, 64-level nesting and 30-second owned-command deadline. HAR uses a 192 MiB Node old-generation heap; native decoding a 768 MiB Linux address-space limit. These are not aggregate RSS claims. Limits fail without successful partial evidence; cleanup independently completes before return. - Packaging: canonical inventory and generated schema/catalog/skill/conformance metadata updated; native bridge packaged, dependencies locked. No release publishing in this PR. ## Evidence and regression coverage Real fixtures are generated by unchanged mitmproxy FlowWriter/SaveHar. Coverage includes selection/order, exact binary bodies/WebSocket messages/trailers/extensions, large numeric values, string/byte transport URLs and backups, sensitive key/path/coordinate exclusions, malformed native records and omitted subtrees, noncanonical HAR base64, input/depth/reply bounds, access denial, unavailable/version-mismatched engines, cancellation and cleanup. Before-fix proofs are retained locally: source property/path/diagnostic regressions (9 failures), hidden HAR validation (2), input acquisition/classification (2), decoder failure categories (4), leading whitespace (3), the coordinate/permission/structured-cleanup batch (6), and authored-limitations collisions (4). Real prior installed bridges also demonstrated invalid omitted credential acceptance, wrong native-key classification, and non-UTF-8 credential retention. Follow-ups fix these before merge. ```json { "name": "inspect_web_network_capture", "arguments": { "capture_path": "/absolute/session.har", "format": "har", "record_ordinals": [2, 0] } } ``` ```sh rea inspect-web-network-capture /absolute/session.har har --record 2 --record 0 --json ``` The result includes `normalized_result` with artifact identity, selected producer records/locations, reported fields, exact byte/numeric sidecars, redactions, limitations and `runtime_attribution: "unknown"`. - [x] Observed, derived, inferred and unknown claims remain distinguishable. - [x] Artifact identity, producer provenance and meaningful failures remain preserved. - [x] Unsupported, unavailable, excluded or incomplete facets remain visible. ## Validation performed Current head **02248703dee53128593645db855250906e303471**, including upstream main **6e4535c1**: - Final focused regression/contract batch: **130 tests / 11 files** passed, including numeric sensitive literals, complete provider identity validation, actual SDK calls and advertised schema dialect validity. Before-fix failures also cover canonicalized byte representations and the ownership preparation deadline. Main integration also passed **191 tests / 8 files** across stable readers, artifact inventory and module boundaries. These are separate runs, not additive unique coverage. - Cached build, `npm run check:fast`, pre-push checks, generated documents and `npm run docs:check` passed. Five additional targeted repetitions passed all10 Linux PID-normalization cases. A single CI capture failure could not be reproduced; its test now reports the tagged failure instead of masking it with a boolean assertion. Earlier ownership/prelaunch cancellation coverage passed22 tests; initial Node 24.18 uncaught-AbortError CI failure was reproduced and fixed rather than retried blindly. - Final-head installed package passed **144 real CLI/MCP cases**, plus **seven native failure classifications**, using unchanged mitmproxy 12.2.3. Coverage includes numeric declarations, full provider provenance and generated base64/digest declarations. Original artifacts unchanged and owned descendants `[]` in completed runs. - Deterministic actual-filesystem probes cover unlink-before-open, unlink-after-read, symlink replacement, permission denial and closure after prelaunch cancellation. Initial missing input remains distinct; Windows symlink creation is explicitly skipped. - Tarball includes the native bridge; reference, private-game, cache, planning and nested dependency paths are excluded. No host defaults changed. - Final-head CI and automated review are required merge gates. Previous Apple E2E hit `hdiutil` Resource temporarily unavailable; the next normal CI run passed that lane. No product timeout or check was weakened. ## Compatibility, safety, and release Additive operation/error/location metadata. Native capture verification is Linux only; this offline increment makes no new real Hopper/Ghidra or live-browser support claim. HAR decoded text does not establish wire bytes, and assembled mitmproxy messages do not establish original wire fragments. Both formats explicitly reject __proto__ members because the current shared JSON schema cannot preserve them; follow-up #1036 records the traced boundary and acceptance. Live binary WS/SSE, workers/WASM and proxy capture are outside this increment. Source-owned fixtures only; no private game data. REA 5.0.0 was published separately from its frozen checkpoint. ## Review checklist - [x] Focused outcome and Conventional Commit title. - [x] Related issues linked; remaining work explicitly preserved. - [x] Meaningful observable behavior and failure paths tested. - [x] Owning docs, contracts, package and generated metadata updated. - [x] Representative CLI/MCP request and result fields documented. - [x] Final diff checked for credentials, unrelated changes and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:07:36 +00:00