[PR #6012] [MERGED] fix(capture): keep excluded MIME protobuf applicability partial #7120

New issue

Closed

opened 2026-11-01 21:45:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:45:33 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6012
Author: @kaoru0822-kitauji
Created: 10/21/2026
Status: ✅ Merged
Merged: 10/21/2026
Merged by: @kaoru0822-kitauji

Base: main ← Head: fix/5926-redacted-mime-protobuf-coverage

📝 Commits (2)

2e93f06 fix(capture): keep excluded MIME protobuf applicability partial
f2ce3ce fix(capture): compare repeated Content-Type declarations by essence

📊 Changes

4 files changed (+195 additions, -17 deletions)

View changed files

📝 docs/web-network-captures.md (+3 -0)
📝 src/browser/history/protobuf/ProtobufCapture.ts (+48 -17)
📝 tests/acceptance/tools/protobufCapture.test.ts (+36 -0)
📝 tests/boundary/browser/sazWebCapture.test.ts (+108 -0)

📄 Description

What broke: Automatic protobuf framing treated an explicitly excluded (header name or value), conflicting or otherwise unresolved Content-Type as known-absent, so a selected retained body was skipped and protobuf_coverage still reported complete.
Fix: bodyMetadata (the owner of body MIME facts) now reports whether framing applicability is established; an unresolved non-empty body adds a record limitation and keeps coverage partial. HAR uses its required mimeType, falling back only to a visible header value.
Risk: Low. Only selected bodies with unresolved MIME change from complete to partial. Visible, known-absent and known-other MIME types, explicit --protobuf-framing, and credential exclusion behave as before.

Summary

Fixes #5926. With --record, a SAZ response whose Content-Type name or value was excluded through --sensitive-value returned {"status":"complete"} with no protobuf facts and no limitation. The same thing happened when two visible Content-Type headers conflicted. Unresolved applicability now stays visible: coverage is partial and the record carries a limitation. The body bytes and redaction coordinates are unchanged.

Problem and expected behavior

bodyMetadata collapsed an excluded or conflicting Content-Type to undefined, which is the same value it uses for a header that is genuinely absent. The automatic-framing branch then skipped the body without setting incomplete, and the final reduction upgraded coverage to complete. This breaks the rule that unresolved applicability must remain unknown rather than be treated as absent (AGENTS.md).

Change and scope

src/browser/history/protobuf/ProtobufCapture.ts
- bodyMetadata now returns contentTypeKnown, using the same tri-state approach as the existing HTTP-coding checks:
  - A unique visible value selects framing.
  - Absence counts as established only when every header name is known.
  - An excluded or conflicting value counts as unknown.
- HAR: the schema requires a string mimeType, so a non-string value means the declaration was excluded. The existing fallback to a visible header value is kept; without a visible header, applicability is unknown.
- Repeated Content-Type headers now agree when their media-type essences match (the part before ;, compared case-insensitively). The first raw value is still the one reported. bodyMetadata now also chooses the automatic framing, so prepareCaptureProtobuf no longer parses the raw value a second time. The framing lookup ignores inherited object properties.
- When automatic framing finds no framing and applicability is unknown, a non-empty body sets incomplete and adds the limitation "Selected body's MIME metadata is excluded, conflicting or unavailable, …". Empty bodies are skipped because they have no protobuf content under any framing. The limitation avoids the literal Content-Type because explicit-value output redaction would otherwise replace it with [REDACTED] in the name-exclusion case.
docs/web-network-captures.md: one paragraph stating this contract.
Not included:
- Other Content-Type consumers were checked. API-surface grouping and scenario records group by MIME but make no completeness claim, and this is the only capture decoder that picks framing from MIME.
- Credential checks for an unknown declared transport when the caller supplies an override (see Remaining proof gaps).

Scope: an ordinary fix within existing boundaries, so no maintainer scope decision is needed.

Contract and boundary impact

Semantic owner and earliest changed stage: bodyMetadata / prepareCaptureProtobuf in the historical-capture protobuf stage.
Contribution scope fit and linked maintainer decision: an existing boundary; no decision required.
CLI and MCP/tool-catalog contract: no schema change. protobuf_coverage.status and record.limitations values change only for unresolved MIME.
Provider, bridge, target-format, or platform compatibility: SAZ, mitmproxy (native header pairs) and HAR.
Evidence, artifact, provenance, or reconstruction contract: coverage now preserves unresolved applicability. Retained bytes, hashes and redaction coordinates are unchanged.
Target selection, authority, upstream behavior, and cleanup details: not applicable.
Generated metadata, package, or installation impact: none.
Read existing files or captured data.

Evidence and regression coverage

Tests added or updated:
- tests/boundary/browser/sazWebCapture.test.ts: a public CLI test covering:
  - a visible MIME, which decodes 1=1
  - conflicting duplicate headers, which give partial plus a limitation
  - duplicates that share an essence (application/grpc and Application/GRPC ; x=1), which still decode 1=1 with no limitation; this case failed on the first PR head
  - known-absent and application/octet-stream, which stay complete with no limitation
  - name exclusion and value exclusion, which give partial plus a limitation, keep the identical retained body, and keep the header redaction coordinate
  - value exclusion with --protobuf-framing grpc, which is complete and decodes
- tests/acceptance/tools/protobufCapture.test.ts: a HAR MCP case where the mimeType is excluded, giving partial plus a limitation and keeping the body.
Base reproduction: both new tests failed on the base build (6d356b843 dist) with expected [] to deeply equal [StringMatching…] and passed after the fix. The CLI repro from the issue, run on base and on the fix, gave:

Case	Base	Fix
visible application/grpc	complete, 1=1	complete, 1=1
MIME value excluded	complete, none, no limitation	partial, MIME limitation
header name excluded	complete, none, no limitation	partial, MIME limitation
value excluded + `--protobuf-framing grpc`	complete, 1=1	complete, 1=1
name excluded + `--protobuf-framing grpc`	partial, HTTP-coding limitation	unchanged
Content-Type absent	complete	complete
application/octet-stream	complete	complete
conflicting duplicate Content-Type	complete, no limitation	partial, MIME limitation

The retained body sha256 was identical (28e42c4efba6…) in every case.

User-visible CLI/MCP output: "protobuf_coverage":{"status":"partial","selected_records":1,"inspected_records":1} plus the record limitation "Selected body's MIME metadata is excluded, conflicting or unavailable, so automatic protobuf framing applicability is unknown; its retained bytes were not interpreted. Select explicit framing to decode them."
Remaining proof gaps:
- No real Fiddler or mitmproxy export was used; the fixtures are synthetic.
- Performance was not measured separately. The change adds one boolean per selected body and no extra decoding.
- Pre-existing and unchanged: when the declared MIME is unknown and the caller supplies an explicit framing, trailer-credential exclusion checks only that framing, because no declared transport is available.
Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Issue CLI repro (8 cases, above) on base and on the fix: base reproduced the bug; the fix gave the expected results.
npm run test:focused -- tests/boundary/browser/sazWebCapture.test.ts tests/acceptance/tools/protobufCapture.test.ts tests/boundary/browser/historicalWebCapture.test.ts tests/boundary/browser/historicalCaptureDecoder.test.ts tests/boundary/browser/largeBase64Capture.test.ts: 67 passed, 3 skipped (existing platform skips). After the essence follow-up, the SAZ test failed once on an expectation in the new test itself: SAZ keeps the header's leading space in the reported value. With the expectation corrected, I reran sazWebCapture and protobufCapture (38 passed, 1 skipped); the other three files passed in the follow-up run.
Follow-up npm run test:local -- on the protobuf unit tests, WebNetworkCaptureService, protobufCaptureView and HarCapture: 125 passed.
npm run test:focused -- tests/boundary/mcp/analysisViewMcp.test.ts tests/boundary/cli/analysisViewCli.test.ts tests/evaluation/sazWorkflowSemantics.test.ts tests/acceptance/tools/qlogCapture.test.ts: 43 passed, 1 skipped.
npm run test:local -- on the protobuf unit tests, WebNetworkCaptureService, protobufCaptureView, HarCapture, CaptureRedaction and SazMetadata: 161 passed. webNetworkCaptureView: 51 passed.
npm run check:fast: passed (also rerun after the follow-up).
npm run format:check: passed (also rerun after the follow-up).
npm run docs:check: passed.

Compatibility, safety, and release

Breaking changes or migration steps: none. A caller that relied on complete for excluded-MIME bodies now sees partial, which is the documented contract.
Real Hopper/Ghidra, browser, or OS coverage: not applicable. Tested on macOS arm64 only.
Package or release metadata impact: none.
Security, privacy, process, or containment review:
- No new bytes are published, and the MIME is not guessed.
- Credential exclusion still runs with the visible declared transport, including when other header names are hidden; the existing trailer-credential test passes.

Author readiness

Human owner: @kaoru0822-kitauji

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6012 **Author:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Created:** 10/21/2026 **Status:** ✅ Merged **Merged:** 10/21/2026 **Merged by:** [@kaoru0822-kitauji](https://github.com/kaoru0822-kitauji) **Base:** `main` ← **Head:** `fix/5926-redacted-mime-protobuf-coverage` --- ### 📝 Commits (2) - [`2e93f06`](https://github.com/morluto/rea/commit/2e93f066653ec60512542c83342c57e5b337b89a) fix(capture): keep excluded MIME protobuf applicability partial - [`f2ce3ce`](https://github.com/morluto/rea/commit/f2ce3ce53fcba7e1a2f6f477e3afb695a23a3d9e) fix(capture): compare repeated Content-Type declarations by essence ### 📊 Changes **4 files changed** (+195 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `docs/web-network-captures.md` (+3 -0) 📝 `src/browser/history/protobuf/ProtobufCapture.ts` (+48 -17) 📝 `tests/acceptance/tools/protobufCapture.test.ts` (+36 -0) 📝 `tests/boundary/browser/sazWebCapture.test.ts` (+108 -0) </details> ### 📄 Description **What broke:** Automatic protobuf framing treated an explicitly excluded (header name or value), conflicting or otherwise unresolved Content-Type as known-absent, so a selected retained body was skipped and `protobuf_coverage` still reported `complete`. **Fix:** `bodyMetadata` (the owner of body MIME facts) now reports whether framing applicability is established; an unresolved non-empty body adds a record limitation and keeps coverage `partial`. HAR uses its required `mimeType`, falling back only to a visible header value. **Risk:** Low. Only selected bodies with unresolved MIME change from `complete` to `partial`. Visible, known-absent and known-other MIME types, explicit `--protobuf-framing`, and credential exclusion behave as before. ## Summary Fixes #5926. With `--record`, a SAZ response whose `Content-Type` name or value was excluded through `--sensitive-value` returned `{"status":"complete"}` with no protobuf facts and no limitation. The same thing happened when two visible `Content-Type` headers conflicted. Unresolved applicability now stays visible: coverage is `partial` and the record carries a limitation. The body bytes and redaction coordinates are unchanged. ## Problem and expected behavior `bodyMetadata` collapsed an excluded or conflicting Content-Type to `undefined`, which is the same value it uses for a header that is genuinely absent. The automatic-framing branch then skipped the body without setting `incomplete`, and the final reduction upgraded coverage to `complete`. This breaks the rule that unresolved applicability must remain unknown rather than be treated as absent (AGENTS.md). ## Change and scope - `src/browser/history/protobuf/ProtobufCapture.ts` - `bodyMetadata` now returns `contentTypeKnown`, using the same tri-state approach as the existing HTTP-coding checks: - A unique visible value selects framing. - Absence counts as established only when every header name is known. - An excluded or conflicting value counts as unknown. - HAR: the schema requires a string `mimeType`, so a non-string value means the declaration was excluded. The existing fallback to a visible header value is kept; without a visible header, applicability is unknown. - Repeated `Content-Type` headers now agree when their media-type essences match (the part before `;`, compared case-insensitively). The first raw value is still the one reported. `bodyMetadata` now also chooses the automatic framing, so `prepareCaptureProtobuf` no longer parses the raw value a second time. The framing lookup ignores inherited object properties. - When automatic framing finds no framing and applicability is unknown, a non-empty body sets `incomplete` and adds the limitation "Selected body's MIME metadata is excluded, conflicting or unavailable, …". Empty bodies are skipped because they have no protobuf content under any framing. The limitation avoids the literal `Content-Type` because explicit-value output redaction would otherwise replace it with `[REDACTED]` in the name-exclusion case. - `docs/web-network-captures.md`: one paragraph stating this contract. - Not included: - Other Content-Type consumers were checked. API-surface grouping and scenario records group by MIME but make no completeness claim, and this is the only capture decoder that picks framing from MIME. - Credential checks for an *unknown* declared transport when the caller supplies an override (see Remaining proof gaps). Scope: an ordinary fix within existing boundaries, so no maintainer scope decision is needed. ## Contract and boundary impact - Semantic owner and earliest changed stage: `bodyMetadata` / `prepareCaptureProtobuf` in the historical-capture protobuf stage. - Contribution scope fit and linked maintainer decision: an existing boundary; no decision required. - CLI and MCP/tool-catalog contract: no schema change. `protobuf_coverage.status` and `record.limitations` values change only for unresolved MIME. - Provider, bridge, target-format, or platform compatibility: SAZ, mitmproxy (native header pairs) and HAR. - Evidence, artifact, provenance, or reconstruction contract: coverage now preserves unresolved applicability. Retained bytes, hashes and redaction coordinates are unchanged. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata, package, or installation impact: none. - [x] Read existing files or captured data. ## Evidence and regression coverage - Tests added or updated: - `tests/boundary/browser/sazWebCapture.test.ts`: a public CLI test covering: - a visible MIME, which decodes `1=1` - conflicting duplicate headers, which give `partial` plus a limitation - duplicates that share an essence (`application/grpc` and `Application/GRPC ; x=1`), which still decode `1=1` with no limitation; this case failed on the first PR head - known-absent and `application/octet-stream`, which stay `complete` with no limitation - name exclusion and value exclusion, which give `partial` plus a limitation, keep the identical retained body, and keep the header redaction coordinate - value exclusion with `--protobuf-framing grpc`, which is `complete` and decodes - `tests/acceptance/tools/protobufCapture.test.ts`: a HAR MCP case where the `mimeType` is excluded, giving `partial` plus a limitation and keeping the body. - Base reproduction: both new tests failed on the base build (`6d356b843` dist) with `expected [] to deeply equal [StringMatching…]` and passed after the fix. The CLI repro from the issue, run on base and on the fix, gave: | Case | Base | Fix | | --- | --- | --- | | visible application/grpc | complete, 1=1 | complete, 1=1 | | MIME value excluded | **complete**, none, no limitation | partial, MIME limitation | | header name excluded | **complete**, none, no limitation | partial, MIME limitation | | value excluded + `--protobuf-framing grpc` | complete, 1=1 | complete, 1=1 | | name excluded + `--protobuf-framing grpc` | partial, HTTP-coding limitation | unchanged | | Content-Type absent | complete | complete | | application/octet-stream | complete | complete | | conflicting duplicate Content-Type | **complete**, no limitation | partial, MIME limitation | The retained body sha256 was identical (`28e42c4efba6…`) in every case. - User-visible CLI/MCP output: `"protobuf_coverage":{"status":"partial","selected_records":1,"inspected_records":1}` plus the record limitation "Selected body's MIME metadata is excluded, conflicting or unavailable, so automatic protobuf framing applicability is unknown; its retained bytes were not interpreted. Select explicit framing to decode them." - Remaining proof gaps: - No real Fiddler or mitmproxy export was used; the fixtures are synthetic. - Performance was not measured separately. The change adds one boolean per selected body and no extra decoding. - Pre-existing and unchanged: when the declared MIME is unknown and the caller supplies an explicit framing, trailer-credential exclusion checks only that framing, because no declared transport is available. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - Issue CLI repro (8 cases, above) on base and on the fix: base reproduced the bug; the fix gave the expected results. - `npm run test:focused -- tests/boundary/browser/sazWebCapture.test.ts tests/acceptance/tools/protobufCapture.test.ts tests/boundary/browser/historicalWebCapture.test.ts tests/boundary/browser/historicalCaptureDecoder.test.ts tests/boundary/browser/largeBase64Capture.test.ts`: 67 passed, 3 skipped (existing platform skips). After the essence follow-up, the SAZ test failed once on an expectation in the new test itself: SAZ keeps the header's leading space in the reported value. With the expectation corrected, I reran `sazWebCapture` and `protobufCapture` (38 passed, 1 skipped); the other three files passed in the follow-up run. - Follow-up `npm run test:local --` on the protobuf unit tests, `WebNetworkCaptureService`, `protobufCaptureView` and `HarCapture`: 125 passed. - `npm run test:focused -- tests/boundary/mcp/analysisViewMcp.test.ts tests/boundary/cli/analysisViewCli.test.ts tests/evaluation/sazWorkflowSemantics.test.ts tests/acceptance/tools/qlogCapture.test.ts`: 43 passed, 1 skipped. - `npm run test:local --` on the protobuf unit tests, `WebNetworkCaptureService`, `protobufCaptureView`, `HarCapture`, `CaptureRedaction` and `SazMetadata`: 161 passed. `webNetworkCaptureView`: 51 passed. - `npm run check:fast`: passed (also rerun after the follow-up). - `npm run format:check`: passed (also rerun after the follow-up). - `npm run docs:check`: passed. ## Compatibility, safety, and release - Breaking changes or migration steps: none. A caller that relied on `complete` for excluded-MIME bodies now sees `partial`, which is the documented contract. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. Tested on macOS arm64 only. - Package or release metadata impact: none. - Security, privacy, process, or containment review: - No new bytes are published, and the MIME is not guessed. - Credential exclusion still runs with the visible declared transport, including when other header names are hidden; the existing trailer-credential test passes. ## Author readiness Human owner: @kaoru0822-kitauji - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:45:33 +00:00