[PR #5031] [MERGED] feat(capture): decode selected protobuf and gRPC bodies #6746

New issue

Closed

opened 2026-11-01 21:42:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:42:46 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5031
Author: @AayushJB03
Created: 10/18/2026
Status: ✅ Merged
Merged: 10/20/2026
Merged by: @gensokyo-dev

Base: main ← Head: feat/decode-protobuf-capture-bodies

📝 Commits (10+)

6c479c8 feat(capture): decode protobuf and gRPC-web bodies in selected capture records
be19877 Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup
cd14e6d feat(capture): retain bounded protobuf facts and schema interpretations
6c84caa Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup
3aae719 fix(capture): validate retained test diagnostics and preserve dependency notices
e5e3136 fix(capture): sanitize sources before protobuf interpretation failures
3e6651b Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup
6b7e215 fix(capture): honor HTTP whitespace in identity codings
287d9c4 fix(capture): preserve unknown HTTP coding metadata
06ac908 fix(capture): inspect credentials before HTTP coding admission

📊 Changes

35 files changed (+4503 additions, -176 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/runtime-observation.md (+6 -0)
📝 docs/web-network-captures.md (+90 -23)
📝 licenses/README.md (+1 -0)
📝 licenses/npm-dependencies.txt (+6 -1)
📝 licenses/npm-runtime-notices.txt (+249 -1)
➕ licenses/protobuf-es-v2.16.0-LICENSE.txt (+201 -0)
📝 licenses/upstream-sources.json (+5 -0)
📝 package-lock.json (+7 -0)
📝 package.json (+1 -0)
📝 src/application/WebNetworkCaptureService.test.ts (+26 -0)
📝 src/application/WebNetworkCaptureService.ts (+48 -0)
📝 src/browser/history/CaptureFailures.ts (+64 -18)
📝 src/browser/history/CaptureRedaction.ts (+26 -1)
📝 src/browser/history/HistoricalCaptureDecoder.ts (+58 -38)
➕ src/browser/history/protobuf/ProtobufBody.ts (+194 -0)
➕ src/browser/history/protobuf/ProtobufCapture.test.ts (+77 -0)
➕ src/browser/history/protobuf/ProtobufCapture.ts (+501 -0)
➕ src/browser/history/protobuf/ProtobufFraming.ts (+144 -0)
➕ src/browser/history/protobuf/ProtobufSchema.ts (+299 -0)
➕ src/browser/history/protobuf/ProtobufWire.test.ts (+522 -0)

...and 15 more files

📄 Description

Summary

Fixes #4986. Selected historical HTTP bodies expose bounded Protobuf/gRPC facts, optional supplied-schema interpretations and compact field queries through the same CLI/MCP implementation. Continues @AayushJB03's original PR with its commit preserved and follow-up fixes.

Behavior and ownership

Exact uint64 values, occurrence/parent identities, byte ranges and source/body hashes replace recursive guessed trees. Ambiguous UTF-8/nested/packed readings remain candidates. Partial/unknown coverage is explicit.
Existing analysis views provide body/frame/field collections, batch filters, columns and paging. Defaults omit large values; complete retained Evidence remains accessible after source deletion. Immutable validation and a bounded recent-filter cache share work.
Only selected records are interpreted. Raw Protobuf, gRPC and gRPC-web framing stay distinct. Local compiled FileDescriptorSet plus message type optionally adds scalar, packed and nested readings without schema downloads or target execution.
Independent credential inspection precedes HTTP eligibility, schema I/O and fact allocation. Hidden metadata cannot bypass trailer exclusion; unavailable coding metadata plus uninspectable framing fails before retaining unchecked source. Safe source otherwise survives typed later failures. Header names/unknown values and raw spelling remain owned by capture metadata.
Nested schema errors remain local, UTF-8 preserves BOM, packed interpretation yields/cancels, overrides require body coordinates and descriptor errors identify their actual input. Shared capture/process owners retain cleanup authority.

Domain contracts describe facts; browser/history adapters own parsing; shared application workflows serve named CLI/MCP contracts. SDK-converted schemas are validated against AJV2020. Additive options/view variant; authored docs/skill updated. Pinned @bufbuild/protobuf@2.16.0 has verified Apache/BSD notices and locked inventory.

Expected effects: read selected existing files/captured data and launch existing owned capture-decoder workers. No new target execution, injection, mutation or protection-recovery capability requiring a scope decision.

Executed validation

Head 06ac908f5; main snapshot 4668cea24:

npm run check, npm run docs:check: static/architecture/generated-doc/license checks and 5 README tests passed.
Twelve relevant source suites: 392 passed. Ghidra cases qualify source integration, not a new real-provider run.
Focused SAZ, historical decoder, advertised MCP schema and public Protobuf CLI/MCP suites: 42 passed, 1 existing runtime-condition skip. Includes retained/exported Evidence, source removal, malformed input, exclusions, hidden metadata, cancellation, capacity and cleanup.
npm run verify:package: packed install/update, CLI/MCP/Evidence, planned/idempotent setup, backups, uninstall and cleanup passed.
Original overflow and first six review failures reproduced on earlier compiled heads; SAZ whitespace and unknown-coding regressions also reproduced before their fixes. All 9 review threads resolved; reviewer confirmed the final source-admission fix. Codex automated review unavailable due repository quota.
Official Connect Eliza responses: Say 69 bytes/1 data frame, Introduce 230 bytes/4 data frames, each plus trailer. CLI/MCP values, all frame/field ranges and source/body/descriptor hashes matched independent protoc/Python oracles; Evidence IDs matched. Response bytes/headers/status/URL were observed; HAR timing is synthetic. Pinned official schema; descriptor SHA256 375f4946642425dbc53c0fe89467235016ff4dea543be1bf82720bddc948580d.

Example interpretation: field_id: "offset:5", schema_name: "sentence", schema_value: "hello", with mandatory provenance/coordinates, coverage and parent links.

Current-head GitHub CI: 45 checks passed, including all five test shards, Windows/package, native packages and real-application lanes. CI checked merge preview ff8c1be730 (head 06ac908f5 plus main 799e8265b); its only addition over the local qualification snapshot is the upstream Hopper-owner verification fix. All 9 review threads are resolved. A prior-head unchanged Windows fixture EBUSY passed on subsequent heads without an unrelated code change.

Measured completed tasks

Token estimates, latency, CPU, memory and limitations

Baseline a9370a906 retains the original decoder; candidate 06ac908f5 shares main 4668cea24. Both read the same selected UTF-8 candidate with source/body hashes and exact range; eight baseline sidecars were independently audited. Real Say uses complete inspection in both; generated 20,001-field data uses original complete inspection versus summary plus one field page.

Primary task	Original token estimate	Follow-up token estimate
Real Say: used input declarations, arguments, complete replies	5,212	6,422
Generated 20k fields: same accounting	1,380,174	20,551
Generated 20k fields: all catalog input declarations included	1,610,547	238,508

Tokenx estimates count the extra RPC. Actual model usage/rendering is unmeasured; small answers grow. Generated data is a stress case, not observed application scale.

Two fresh-process primary-only runs, complete verification harness including SDK/catalog/schema checks, token estimates and teardown:

Workload/version	Wall s	Family CPU s	Sampled family RSS GiB
Real/original	5.16–5.30	5.06–5.18	0.66–0.66
Real/follow-up	5.35–5.35	5.23–5.23	0.66–0.66
Generated/original	6.60–7.62	6.47–7.30	0.66–0.68
Generated/follow-up	8.94–9.40	8.85–9.30	0.85–0.88

Cold generated-case RPCs regress: original 1.38–1.65 s versus follow-up 2.80–3.04 s for summary plus first page. Smaller context and stronger facts cost initial time/memory; no cold-speedup claim.

Separate retained reads after source removal: original 2,029/733/775 ms; selected fields 10.57/8.39/6.46 ms. These probes are excluded from primary counters. Full probe harness: original 14.01 s wall/13.93 s CPU/0.913 GiB; follow-up 8.99 s/8.89 s/0.870 GiB. Two-RPC real-schema correctness harness: 13.28 s/13.10 s/1.222 GiB; the old decoder has no equivalent schema capability.

Linux/Node24.18, two CPUs, 3 GiB V8 heaps, 8 GiB family guard; no retries/stops. RSS sampled every20ms sums family RSS and can double-count shared pages/miss short peaks. Runtime caches fresh; OS cache/host load uncontrolled. Engine-only costs and post-task sidecar audit unmeasured. Results qualify scoped projection/reuse, not a general provider speedup.

Real traffic verification is Linux/HAR only. Generated SAZ files exercise the production reader, not a Fiddler producer golden. Real native-gRPC, mitmproxy and other-OS protocol coverage remains unverified. Compression, map semantics, extensions and required-field validation remain explicitly unsupported. Supplied schemas do not establish the sender's schema; missing/mismatched interpretations remain visible.

Author readiness

Human owner: @gensokyo-dev

The human owner explicitly authorized this work, delegated the technical review and validation to Codex, directed these declarations to be recorded on their behalf, and requested ready-for-review submission and normal merge. This records that direction; it does not assert a separate, independent human diff review. Codex performed the documented technical review and checks.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5031 **Author:** [@AayushJB03](https://github.com/AayushJB03) **Created:** 10/18/2026 **Status:** ✅ Merged **Merged:** 10/20/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `feat/decode-protobuf-capture-bodies` --- ### 📝 Commits (10+) - [`6c479c8`](https://github.com/morluto/rea/commit/6c479c89a68e4fcbb88a839191273585c401874c) feat(capture): decode protobuf and gRPC-web bodies in selected capture records - [`be19877`](https://github.com/morluto/rea/commit/be19877f21512923794833bade6a365e19fe79de) Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup - [`cd14e6d`](https://github.com/morluto/rea/commit/cd14e6dbc92cfb50baac4e481801c8457bb271cd) feat(capture): retain bounded protobuf facts and schema interpretations - [`6c84caa`](https://github.com/morluto/rea/commit/6c84caa8174993ee4ca6840b81fdfb290bdc2dc8) Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup - [`3aae719`](https://github.com/morluto/rea/commit/3aae719c323a560e6578d64a937f03c5fc73d4b8) fix(capture): validate retained test diagnostics and preserve dependency notices - [`e5e3136`](https://github.com/morluto/rea/commit/e5e3136e0bdb2066295a078daacba7d934ccaaa8) fix(capture): sanitize sources before protobuf interpretation failures - [`3e6651b`](https://github.com/morluto/rea/commit/3e6651bfa801cfbf6141e5af21484e87ee2a802d) Merge remote-tracking branch 'origin/main' into feat/4986-protobuf-followup - [`6b7e215`](https://github.com/morluto/rea/commit/6b7e2150f0e71b467a50025b00f4bf27c9bc82b2) fix(capture): honor HTTP whitespace in identity codings - [`287d9c4`](https://github.com/morluto/rea/commit/287d9c4c91f5935af4333cf40dbeebc5ae5e09b1) fix(capture): preserve unknown HTTP coding metadata - [`06ac908`](https://github.com/morluto/rea/commit/06ac908f5d83aa80cfde92a740a30087ab3f86d0) fix(capture): inspect credentials before HTTP coding admission ### 📊 Changes **35 files changed** (+4503 additions, -176 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/runtime-observation.md` (+6 -0) 📝 `docs/web-network-captures.md` (+90 -23) 📝 `licenses/README.md` (+1 -0) 📝 `licenses/npm-dependencies.txt` (+6 -1) 📝 `licenses/npm-runtime-notices.txt` (+249 -1) ➕ `licenses/protobuf-es-v2.16.0-LICENSE.txt` (+201 -0) 📝 `licenses/upstream-sources.json` (+5 -0) 📝 `package-lock.json` (+7 -0) 📝 `package.json` (+1 -0) 📝 `src/application/WebNetworkCaptureService.test.ts` (+26 -0) 📝 `src/application/WebNetworkCaptureService.ts` (+48 -0) 📝 `src/browser/history/CaptureFailures.ts` (+64 -18) 📝 `src/browser/history/CaptureRedaction.ts` (+26 -1) 📝 `src/browser/history/HistoricalCaptureDecoder.ts` (+58 -38) ➕ `src/browser/history/protobuf/ProtobufBody.ts` (+194 -0) ➕ `src/browser/history/protobuf/ProtobufCapture.test.ts` (+77 -0) ➕ `src/browser/history/protobuf/ProtobufCapture.ts` (+501 -0) ➕ `src/browser/history/protobuf/ProtobufFraming.ts` (+144 -0) ➕ `src/browser/history/protobuf/ProtobufSchema.ts` (+299 -0) ➕ `src/browser/history/protobuf/ProtobufWire.test.ts` (+522 -0) _...and 15 more files_ </details> ### 📄 Description ## Summary Fixes #4986. Selected historical HTTP bodies expose bounded Protobuf/gRPC facts, optional supplied-schema interpretations and compact field queries through the same CLI/MCP implementation. Continues @AayushJB03's original PR with its commit preserved and follow-up fixes. ## Behavior and ownership - Exact uint64 values, occurrence/parent identities, byte ranges and source/body hashes replace recursive guessed trees. Ambiguous UTF-8/nested/packed readings remain candidates. Partial/unknown coverage is explicit. - Existing analysis views provide body/frame/field collections, batch filters, columns and paging. Defaults omit large values; complete retained Evidence remains accessible after source deletion. Immutable validation and a bounded recent-filter cache share work. - Only selected records are interpreted. Raw Protobuf, gRPC and gRPC-web framing stay distinct. Local compiled FileDescriptorSet plus message type optionally adds scalar, packed and nested readings without schema downloads or target execution. - Independent credential inspection precedes HTTP eligibility, schema I/O and fact allocation. Hidden metadata cannot bypass trailer exclusion; unavailable coding metadata plus uninspectable framing fails before retaining unchecked source. Safe source otherwise survives typed later failures. Header names/unknown values and raw spelling remain owned by capture metadata. - Nested schema errors remain local, UTF-8 preserves BOM, packed interpretation yields/cancels, overrides require body coordinates and descriptor errors identify their actual input. Shared capture/process owners retain cleanup authority. Domain contracts describe facts; browser/history adapters own parsing; shared application workflows serve named CLI/MCP contracts. SDK-converted schemas are validated against AJV2020. Additive options/view variant; authored docs/skill updated. Pinned `@bufbuild/protobuf@2.16.0` has verified Apache/BSD notices and locked inventory. Expected effects: read selected existing files/captured data and launch existing owned capture-decoder workers. No new target execution, injection, mutation or protection-recovery capability requiring a scope decision. ## Executed validation Head `06ac908f5`; main snapshot `4668cea24`: - `npm run check`, `npm run docs:check`: static/architecture/generated-doc/license checks and **5 README tests passed**. - Twelve relevant source suites: **392 passed**. Ghidra cases qualify source integration, not a new real-provider run. - Focused SAZ, historical decoder, advertised MCP schema and public Protobuf CLI/MCP suites: **42 passed, 1 existing runtime-condition skip**. Includes retained/exported Evidence, source removal, malformed input, exclusions, hidden metadata, cancellation, capacity and cleanup. - `npm run verify:package`: packed install/update, CLI/MCP/Evidence, planned/idempotent setup, backups, uninstall and cleanup passed. - Original overflow and first six review failures reproduced on earlier compiled heads; SAZ whitespace and unknown-coding regressions also reproduced before their fixes. All **9 review threads resolved**; reviewer confirmed the final source-admission fix. Codex automated review unavailable due repository quota. - Official Connect Eliza responses: Say **69 bytes/1 data frame**, Introduce **230 bytes/4 data frames**, each plus trailer. CLI/MCP values, all frame/field ranges and source/body/descriptor hashes matched independent protoc/Python oracles; Evidence IDs matched. Response bytes/headers/status/URL were observed; HAR timing is synthetic. [Pinned official schema](https://github.com/connectrpc/examples-go/blob/85945ad604d820f4061d6b795092d32587e5d261/proto/connectrpc/eliza/v1/eliza.proto); descriptor SHA256 `375f4946642425dbc53c0fe89467235016ff4dea543be1bf82720bddc948580d`. Example interpretation: `field_id: "offset:5", schema_name: "sentence", schema_value: "hello"`, with mandatory provenance/coordinates, coverage and parent links. Current-head GitHub CI: **45 checks passed**, including all five test shards, Windows/package, native packages and real-application lanes. CI checked merge preview `ff8c1be730` (head `06ac908f5` plus main `799e8265b`); its only addition over the local qualification snapshot is the upstream Hopper-owner verification fix. All 9 review threads are resolved. A prior-head unchanged Windows fixture EBUSY passed on subsequent heads without an unrelated code change. ## Measured completed tasks <details> <summary>Token estimates, latency, CPU, memory and limitations</summary> Baseline `a9370a906` retains the original decoder; candidate `06ac908f5` shares main `4668cea24`. Both read the same selected UTF-8 candidate with source/body hashes and exact range; eight baseline sidecars were independently audited. Real Say uses complete inspection in both; generated 20,001-field data uses original complete inspection versus summary plus one field page. | Primary task | Original token estimate | Follow-up token estimate | | --- | ---: | ---: | | Real Say: used input declarations, arguments, complete replies | 5,212 | 6,422 | | Generated 20k fields: same accounting | 1,380,174 | 20,551 | | Generated 20k fields: all catalog input declarations included | 1,610,547 | 238,508 | Tokenx estimates count the extra RPC. Actual model usage/rendering is unmeasured; small answers grow. Generated data is a stress case, not observed application scale. Two fresh-process primary-only runs, complete verification harness including SDK/catalog/schema checks, token estimates and teardown: | Workload/version | Wall s | Family CPU s | Sampled family RSS GiB | | --- | ---: | ---: | ---: | | Real/original | 5.16–5.30 | 5.06–5.18 | 0.66–0.66 | | Real/follow-up | 5.35–5.35 | 5.23–5.23 | 0.66–0.66 | | Generated/original | 6.60–7.62 | 6.47–7.30 | 0.66–0.68 | | Generated/follow-up | 8.94–9.40 | 8.85–9.30 | 0.85–0.88 | Cold generated-case RPCs regress: original 1.38–1.65 s versus follow-up 2.80–3.04 s for summary plus first page. Smaller context and stronger facts cost initial time/memory; no cold-speedup claim. Separate retained reads after source removal: original 2,029/733/775 ms; selected fields 10.57/8.39/6.46 ms. These probes are excluded from primary counters. Full probe harness: original 14.01 s wall/13.93 s CPU/0.913 GiB; follow-up 8.99 s/8.89 s/0.870 GiB. Two-RPC real-schema correctness harness: 13.28 s/13.10 s/1.222 GiB; the old decoder has no equivalent schema capability. Linux/Node24.18, two CPUs, 3 GiB V8 heaps, 8 GiB family guard; no retries/stops. RSS sampled every20ms sums family RSS and can double-count shared pages/miss short peaks. Runtime caches fresh; OS cache/host load uncontrolled. Engine-only costs and post-task sidecar audit unmeasured. Results qualify scoped projection/reuse, not a general provider speedup. </details> Real traffic verification is Linux/HAR only. Generated SAZ files exercise the production reader, not a Fiddler producer golden. Real native-gRPC, mitmproxy and other-OS protocol coverage remains unverified. Compression, map semantics, extensions and required-field validation remain explicitly unsupported. Supplied schemas do not establish the sender's schema; missing/mismatched interpretations remain visible. ## Author readiness Human owner: @gensokyo-dev The human owner explicitly authorized this work, delegated the technical review and validation to Codex, directed these declarations to be recorded on their behalf, and requested ready-for-review submission and normal merge. This records that direction; it does not assert a separate, independent human diff review. Codex performed the documented technical review and checks. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:42:46 +00:00