[PR #2047] [CLOSED] feat(browser): retain selected bodies and payloads in session HAR #4987

New issue

Closed

opened 2026-10-31 00:57:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:57:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2047
Author: @Hyped01
Created: 10/12/2026
Status: ❌ Closed

Base: main ← Head: feat/session-har-3-content

📝 Commits (4)

6b912ce fix(browser): admit browser-serialized HAR request URLs
fba05f5 feat(browser): record observe_web_session traffic as HAR
43a0e2b feat(browser): redact declared values in session HAR
254e150 feat(browser): retain selected bodies and payloads in session HAR

📊 Changes

31 files changed (+5741 additions, -80 deletions)

View changed files

📝 docs/browser-observation.md (+192 -3)
📝 docs/web-network-captures.md (+6 -0)
📝 src/application/BrowserEvidence.ts (+46 -7)
📝 src/browser/CdpCaptureEventBodyShapes.ts (+2 -1)
📝 src/browser/CdpCaptureEventHandlers.ts (+1 -1)
📝 src/browser/CdpCaptureEventHelpers.ts (+0 -7)
📝 src/browser/CdpCaptureStorageFingerprints.ts (+4 -14)
📝 src/browser/CdpObservationSession.ts (+103 -21)
➕ src/browser/CdpSessionHar.ts (+477 -0)
➕ src/browser/CdpSessionNetwork.ts (+746 -0)
➕ src/browser/CdpSessionResult.ts (+122 -0)
➕ src/browser/SessionRedactor.test.ts (+321 -0)
➕ src/browser/SessionRedactor.ts (+464 -0)
📝 src/browser/history/CaptureRedaction.ts (+2 -7)
📝 src/browser/history/HarCapture.test.ts (+37 -0)
📝 src/browser/history/HarCapture.ts (+3 -7)
📝 src/browser/history/HarSchema.ts (+28 -1)
➕ src/browser/network/Base64.ts (+16 -0)
📝 src/browser/network/PlaywrightNetworkContent.ts (+2 -8)
📝 src/browser/network/PlaywrightScenarioNetwork.test.ts (+44 -0)

...and 11 more files

📄 Description

Stacked on #2046 (and #2045). Only the top commit, 254e150a2, is new here.

Summary

Refs #774. This PR adds request_bodies, response_bodies (none, json or all) and websocket_payloads to observe_web_session's network selection. The CLI flags are --request-bodies, --response-bodies and --websocket-payloads.

Problem and expected behavior

Without content, the session HAR from #2045 shows that a request happened but not what it carried. Selected bodies and payloads should be retained within a fixed budget. Every body that is not retained should say why, and declared values from #2046 must be redacted in content as well.

Change and scope

Response bodies are read with Network.getResponseBody after loading finishes, each within a 5-second deadline. These cases are stated in their entry comments and reported unavailable:
- a read that misses the deadline;
- a body the browser has released (-32000);
- a browser without the method (-32601);
- a body still loading when recording stopped;
- reads still pending when the target ends or disconnects.
Caller cancellation and other protocol failures still fail the observation.
Window. The timeline ends with the window; reads still settling afterwards do not extend it.
Request bodies come from CDP's postDataEntries bytes when they are present. A UTF-8 body is retained as text with its byte size. Any other body is reported by size only, since HAR postData carries text.
Budget. Retained bodies and payloads share a budget equal to inspect_web_network_capture's 32 MiB input budget. A read reserves its base64 worst case while in flight.
Message size.
- A body whose CDP reply could exceed the 64 MiB message limit is not read. Chrome escapes non-ASCII text up to six-fold.
- Selected request bodies are inlined only up to about 7.9 MiB, so subresource request events fit.
- Chrome inlines navigation (form submission) bodies whole regardless. An oversized navigation event ends the window, as the docs say.
Malformed content, including non-canonical base64, is not retained and is counted as invalid_protocol_value.
Redaction in content. Text bodies get the same survival check as other fields. Binary content is redacted on its decoded bytes and also checked on its base64 text; base64 cannot be edited in place, so a hit there empties the field.
Bounded checking. The declared-value check builds each decoding of a body or payload lazily, depth first. It keeps at most one decoding per nesting level alive and stops at the first match, so memory stays bounded for 32 MiB fields while the same decodings are checked. The check is still CPU-bound: seconds for a 32 MiB body, longer for content made mostly of escapes. The docs say so.
One base64 decoder. A single linear decoder in src/browser/network/Base64.ts replaces three copies: the CDP capture helper, the HAR decoder's, and the storage fingerprints'. The old pattern backtracked recursively and threw RangeError on multi-megabyte input, which inspecting a session HAR with a large binary body now exercises.
Withheld content. A body or payload that redaction would empty is withheld, never reported as empty. A body keeps its size, its entry comment says why, and network_bodies counts it. A payload gets data: null, is counted under websocket_frames, and a limitation states how many.
Over-budget limitation. A HAR larger than inspect_web_network_capture's 32 MiB input budget says so. The limitation names every contributor: origins, bodies, WebSocket payloads, header values and the window.

Contract and boundary impact

Semantic owner and earliest changed stage: src/browser/CdpSessionNetwork.ts (reads, budget) and src/browser/CdpSessionHar.ts (content).
CLI and MCP/tool-catalog contract: additive selection options, plus the output fields content_budget_bytes and retained_content_bytes, both described in the schema and the docs.
Provider, bridge, target-format, or platform compatibility: the shared base64 decoder accepts exactly the strings the old patterns accepted. That was verified against the old regex on 200,000 random strings.
Evidence, artifact, provenance, or reconstruction contract: none beyond the selection being recorded.
Process execution, authorization, cleanup, or containment impact: none.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Evidence and regression coverage

Tests added or updated:
- budget exhaustion through concurrent reservations, and wire-size admission;
- released, unfinished, unexposed and malformed bodies, and transport loss;
- an 8 MiB binary body and an 8 MiB binary frame retained;
- an 8 MiB base64 body decoded by decodeHarCapture, and 8 MiB storage fingerprints;
- exact request bytes from postDataEntries, both UTF-8 and binary;
- base64 redaction, including a base64-declared key;
- the read deadline, and abandonment on target crash;
- caller cancellation;
- an oversized navigation event;
- withheld bodies and payloads;
- frames and request bodies reaching the retained-content budget after earlier content used it;
- the over-budget limitation at and just past the boundary.
Mutation checks: each of these fails a test.
- Restoring the backtracking pattern, in each of the three former copies.
- Ignoring postDataEntries.
- A non-fatal UTF-8 decoder.
- The abandon, canonical-base64, unfinished-body and wire-size checks.
- Both base64-text checks.
Remaining proof gaps: Chrome's navigation body handling was verified against Chrome 154 through CDP, not in other browsers.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Per-commit gate: pass.
npx vitest run on 12 browser, CLI, MCP and redactor test files: 121 passed, 2 skipped.
exposes CLI-equivalent Evidence results and session tools, an existing MCP test, times out only under heavy machine load. It fails the same way on unmodified main.
Cold review with adversarial verification: the final round reported no findings.

Compatibility, safety, and release

Breaking changes or migration steps: none; content retention is opt-in.
Package or release metadata impact: none.
Security, privacy, process, or containment review: the budget bounds retained content, and malformed or unreadable content is omitted with a reason rather than failing open.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2047 **Author:** [@Hyped01](https://github.com/Hyped01) **Created:** 10/12/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/session-har-3-content` --- ### 📝 Commits (4) - [`6b912ce`](https://github.com/morluto/rea/commit/6b912ce5b90e9b4a395570af0060c5efee9ab44a) fix(browser): admit browser-serialized HAR request URLs - [`fba05f5`](https://github.com/morluto/rea/commit/fba05f5e123ee1d1893689e2d6b265ea618459ac) feat(browser): record observe_web_session traffic as HAR - [`43a0e2b`](https://github.com/morluto/rea/commit/43a0e2ba067fdd4c811e4a561dc2a09a4ee1da16) feat(browser): redact declared values in session HAR - [`254e150`](https://github.com/morluto/rea/commit/254e150a232a77af16d88e367ee6ac9be9e01ace) feat(browser): retain selected bodies and payloads in session HAR ### 📊 Changes **31 files changed** (+5741 additions, -80 deletions) <details> <summary>View changed files</summary> 📝 `docs/browser-observation.md` (+192 -3) 📝 `docs/web-network-captures.md` (+6 -0) 📝 `src/application/BrowserEvidence.ts` (+46 -7) 📝 `src/browser/CdpCaptureEventBodyShapes.ts` (+2 -1) 📝 `src/browser/CdpCaptureEventHandlers.ts` (+1 -1) 📝 `src/browser/CdpCaptureEventHelpers.ts` (+0 -7) 📝 `src/browser/CdpCaptureStorageFingerprints.ts` (+4 -14) 📝 `src/browser/CdpObservationSession.ts` (+103 -21) ➕ `src/browser/CdpSessionHar.ts` (+477 -0) ➕ `src/browser/CdpSessionNetwork.ts` (+746 -0) ➕ `src/browser/CdpSessionResult.ts` (+122 -0) ➕ `src/browser/SessionRedactor.test.ts` (+321 -0) ➕ `src/browser/SessionRedactor.ts` (+464 -0) 📝 `src/browser/history/CaptureRedaction.ts` (+2 -7) 📝 `src/browser/history/HarCapture.test.ts` (+37 -0) 📝 `src/browser/history/HarCapture.ts` (+3 -7) 📝 `src/browser/history/HarSchema.ts` (+28 -1) ➕ `src/browser/network/Base64.ts` (+16 -0) 📝 `src/browser/network/PlaywrightNetworkContent.ts` (+2 -8) 📝 `src/browser/network/PlaywrightScenarioNetwork.test.ts` (+44 -0) _...and 11 more files_ </details> ### 📄 Description > Stacked on #2046 (and #2045). Only the top commit, `254e150a2`, is new here. ## Summary Refs #774. This PR adds `request_bodies`, `response_bodies` (`none`, `json` or `all`) and `websocket_payloads` to `observe_web_session`'s network selection. The CLI flags are `--request-bodies`, `--response-bodies` and `--websocket-payloads`. ## Problem and expected behavior Without content, the session HAR from #2045 shows that a request happened but not what it carried. Selected bodies and payloads should be retained within a fixed budget. Every body that is not retained should say why, and declared values from #2046 must be redacted in content as well. ## Change and scope - **Response bodies** are read with `Network.getResponseBody` after loading finishes, each within a 5-second deadline. These cases are stated in their entry comments and reported unavailable: - a read that misses the deadline; - a body the browser has released (`-32000`); - a browser without the method (`-32601`); - a body still loading when recording stopped; - reads still pending when the target ends or disconnects. Caller cancellation and other protocol failures still fail the observation. - **Window.** The timeline ends with the window; reads still settling afterwards do not extend it. - **Request bodies** come from CDP's `postDataEntries` bytes when they are present. A UTF-8 body is retained as text with its byte size. Any other body is reported by size only, since HAR `postData` carries text. - **Budget.** Retained bodies and payloads share a budget equal to `inspect_web_network_capture`'s 32 MiB input budget. A read reserves its base64 worst case while in flight. - **Message size.** - A body whose CDP reply could exceed the 64 MiB message limit is not read. Chrome escapes non-ASCII text up to six-fold. - Selected request bodies are inlined only up to about 7.9 MiB, so subresource request events fit. - Chrome inlines navigation (form submission) bodies whole regardless. An oversized navigation event ends the window, as the docs say. - **Malformed content**, including non-canonical base64, is not retained and is counted as `invalid_protocol_value`. - **Redaction in content.** Text bodies get the same survival check as other fields. Binary content is redacted on its decoded bytes and also checked on its base64 text; base64 cannot be edited in place, so a hit there empties the field. - **Bounded checking.** The declared-value check builds each decoding of a body or payload lazily, depth first. It keeps at most one decoding per nesting level alive and stops at the first match, so memory stays bounded for 32 MiB fields while the same decodings are checked. The check is still CPU-bound: seconds for a 32 MiB body, longer for content made mostly of escapes. The docs say so. - **One base64 decoder.** A single linear decoder in `src/browser/network/Base64.ts` replaces three copies: the CDP capture helper, the HAR decoder's, and the storage fingerprints'. The old pattern backtracked recursively and threw `RangeError` on multi-megabyte input, which inspecting a session HAR with a large binary body now exercises. - **Withheld content.** A body or payload that redaction would empty is withheld, never reported as empty. A body keeps its size, its entry comment says why, and `network_bodies` counts it. A payload gets `data: null`, is counted under `websocket_frames`, and a limitation states how many. - **Over-budget limitation.** A HAR larger than `inspect_web_network_capture`'s 32 MiB input budget says so. The limitation names every contributor: origins, bodies, WebSocket payloads, header values and the window. ## Contract and boundary impact - Semantic owner and earliest changed stage: `src/browser/CdpSessionNetwork.ts` (reads, budget) and `src/browser/CdpSessionHar.ts` (content). - CLI and MCP/tool-catalog contract: additive selection options, plus the output fields `content_budget_bytes` and `retained_content_bytes`, both described in the schema and the docs. - Provider, bridge, target-format, or platform compatibility: the shared base64 decoder accepts exactly the strings the old patterns accepted. That was verified against the old regex on 200,000 random strings. - Evidence, artifact, provenance, or reconstruction contract: none beyond the selection being recorded. - Process execution, authorization, cleanup, or containment impact: none. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. ## Evidence and regression coverage - Tests added or updated: - budget exhaustion through concurrent reservations, and wire-size admission; - released, unfinished, unexposed and malformed bodies, and transport loss; - an 8 MiB binary body and an 8 MiB binary frame retained; - an 8 MiB base64 body decoded by `decodeHarCapture`, and 8 MiB storage fingerprints; - exact request bytes from `postDataEntries`, both UTF-8 and binary; - base64 redaction, including a base64-declared key; - the read deadline, and abandonment on target crash; - caller cancellation; - an oversized navigation event; - withheld bodies and payloads; - frames and request bodies reaching the retained-content budget after earlier content used it; - the over-budget limitation at and just past the boundary. - Mutation checks: each of these fails a test. - Restoring the backtracking pattern, in each of the three former copies. - Ignoring `postDataEntries`. - A non-fatal UTF-8 decoder. - The abandon, canonical-base64, unfinished-body and wire-size checks. - Both base64-text checks. - Remaining proof gaps: Chrome's navigation body handling was verified against Chrome 154 through CDP, not in other browsers. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - Per-commit gate: pass. - `npx vitest run` on 12 browser, CLI, MCP and redactor test files: 121 passed, 2 skipped. - `exposes CLI-equivalent Evidence results and session tools`, an existing MCP test, times out only under heavy machine load. It fails the same way on unmodified `main`. - Cold review with adversarial verification: the final round reported no findings. ## Compatibility, safety, and release - Breaking changes or migration steps: none; content retention is opt-in. - Package or release metadata impact: none. - Security, privacy, process, or containment review: the budget bounds retained content, and malformed or unreadable content is omitted with a reason rather than failing open. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:57:43 +00:00