morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5926] [Bug] Redacted Content-Type metadata still yields complete protobuf coverage #3601

New issue

Closed

opened 2026-10-31 00:42:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:17 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5926

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public CLI reports complete protobuf coverage for a selected SAZ response after its Content-Type name or value is explicitly redacted. The body remains retained, but no protobuf facts or body limitation explain that automatic framing could not establish its applicability:

{"status":"complete","selected_records":1,"inspected_records":1}

I executed this with a synthetic seven-byte uncompressed gRPC body. Without redaction, the same body decodes completely as field 1 = 1. With either MIME redaction, the command exits 0, retains the identical body bytes and hash, emits no record.protobuf, and leaves record.limitations empty. The header's own redaction coordinate is correctly retained.

Reproduction

I ran eight sequential CLI cases using a ZIP containing raw/1_s.txt. Its HTTP response is:

HTTP/1.1 200 OK\r\n
Content-Type: application/grpc\r\n
Content-Length: 7\r\n
\r\n

Append the literal bytes 00 00 00 00 02 08 01 after the header separator. These are a complete gRPC data frame with the protobuf payload 08 01.

Run the built repository CLI on that archive:

node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --json
node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --sensitive-value application/grpc --json
node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --sensitive-value Content-Type --json

The first call decodes field 1 = 1. Both redaction calls return the complete coverage object above with no protobuf decoding or record limitation. Each case reproduced once.

The executable generator and comparison used for the eight calls are included below. Save it outside the checkout and run node reproduce.mjs "$REA_CHECKOUT"; it writes fixtures and complete results next to the script. It uses the checkout's installed ZIP dependency and built CLI.

Expected behavior

When explicitly excluded MIME metadata leaves body applicability unresolved, preserve that uncertainty in protobuf coverage, for example with status: "partial" and a body limitation. This does not require guessing the content type or attempting to decode every unlabelled body.

The repository requires that unresolved applicability remain unknown rather than be treated as absent. See AGENTS.md, lines 19–21. Automatic decoding is scoped to recognized MIME types, while the capture documentation says protobuf coverage preserves partial decoding.

The hidden-header-name case also makes HTTP coding metadata unknown. The specific coding contract requires retaining that uncertainty. Explicit framing on this same case correctly returns partial with a coding limitation; automatic framing bypasses that qualification.

Impact and scope

This is a bounded coverage-reporting defect: complete coverage conceals unresolved applicability for an explicitly selected retained body. It affects the tested default automatic-framing path after Content-Type name/value exclusion. Original capture bytes and retained body bytes were unchanged in all eight cases.

For MIME-value redaction, supplying the known framing with --protobuf-framing grpc restored complete decoding in the executed fixture. For header-name redaction, the same override instead returned partial and an HTTP-coding limitation, because header names remained unknown. It did not restore decoding.

I did not test other capture formats, a real Fiddler export, or a credential-bearing body. No security exposure or source-byte loss is claimed.

REA version

5241597594; package metadata rea-agents 6.7.0.

Runtime and platform

macOS 27.0.1 arm64, Node v26.12.0. The repository-pinned Node 24.18.0 runtime was not exercised.

Target and artifact context

Synthetic Fiddler SAZ ZIP containing one HTTP response and an uncompressed seven-byte gRPC body. Offline file inspection; no live target or external provider.

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

Executed results, all exit 0 and all with unchanged retained body bytes/hash:

Input/selection	Protobuf facts	Coverage	Record limitation
Visible application/grpc	Complete body, field 1 = 1	complete, 1/1	None
MIME value explicitly excluded	None	complete, 1/1	None
Header name explicitly excluded	None	complete, 1/1	None
MIME value excluded, explicit grpc framing	Complete body, field 1 = 1	complete, 1/1	None
Header name excluded, explicit grpc framing	None	partial, 1/1	HTTP coding unknown
Content-Type genuinely absent	None	complete, 1/1	None
Known application/octet-stream	None	complete, 1/1	None
Content-Encoding identity value excluded, MIME visible	None	partial, 1/1	HTTP coding unknown

The known-absent and known-other-MIME controls show why the issue concerns excluded applicability rather than requiring protobuf decoding of all retained bodies.

Typed outcome or recovery expected

Retain the source bytes and header redaction provenance, and qualify incomplete protobuf applicability with partial coverage and a limitation. An unavailable result is appropriate only where the existing source-inspection contract requires it; it is not required merely because a MIME value is hidden.

Analysis or suspected cause

Source-derived explanation: body metadata collapses a redacted content type to the same undefined value as an absent one. A hidden header name also makes HTTP coding and identity unknown.

The automatic-framing branch skips undefined framing without setting incomplete. It therefore never reaches the unknown-identity qualification. The final reduction counts the record as inspected and upgrades coverage to complete. Counting the visited record is reasonable; discarding unresolved applicability before deciding completeness is the problem.

Additional context

The relevant production lines remain unchanged in checked upstream main aa53c065387c58b4189d82c47f1f3445fc505e30. This report concerns eligibility before decoding, separate from compressed-frame/body status aggregation and malformed gRPC-web trailer validation.

The reproduction uses actual CLI subprocesses and synthetic source bytes. MCP, other platforms, the published package, performance, and real-producer export compatibility were not exercised.

Executed fixture generator and CLI runner:

import { createRequire } from 'node:module';
import { writeFileSync, readFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createHash } from 'node:crypto';

const checkout = process.argv[2];
const out = dirname(fileURLToPath(import.meta.url));
const require = createRequire(join(checkout, 'package.json'));
const { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } = require('@zip.js/zip.js');
const body = Buffer.from([0, 0, 0, 0, 2, 8, 1]);
const variants = [
  ['control', 'Content-Type: application/grpc\r\n', []],
  ['mime-value', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'application/grpc']],
  ['mime-name', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'Content-Type']],
  ['mime-value-override', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'application/grpc', '--protobuf-framing', 'grpc']],
  ['mime-name-override', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'Content-Type', '--protobuf-framing', 'grpc']],
  ['mime-absent', '', []],
  ['mime-other', 'Content-Type: application/octet-stream\r\n', []],
  ['coding-value', 'Content-Type: application/grpc\r\nContent-Encoding: identity\r\n', ['--sensitive-value', 'identity']],
];
const summary = [];
for (const [name, headers, extra] of variants) {
  const writer = new ZipWriter(new Uint8ArrayWriter(), {useWebWorkers:false});
  const response = Buffer.concat([Buffer.from('HTTP/1.1 200 OK\r\n' + headers + 'Content-Length: 7\r\n\r\n'), body]);
  await writer.add('raw/1_s.txt', new Uint8ArrayReader(response));
  const path = join(out, name + '.saz');
  const input = Buffer.from(await writer.close());
  writeFileSync(path, input);
  const args = [join(checkout, 'scripts/rea.mjs'), 'inspect-web-network-capture', path, 'saz', '--record', '0', '--json', ...extra];
  let stdout, stderr = '', exitCode = 0;
  try { stdout = execFileSync(process.execPath, args, {encoding:'utf8', timeout:40000, maxBuffer:8*1024*1024}); }
  catch(error) { stdout = error.stdout; stderr = String(error.stderr); exitCode = error.status; }
  writeFileSync(join(out,name+'.stdout.json'),stdout);
  writeFileSync(join(out,name+'.stderr.txt'),stderr);
  const result = JSON.parse(stdout), capture = result.normalized_result;
  const record = capture?.records?.[0];
  const retained = record?.binary_fields.find(f => f.pointer === '/response/body');
  summary.push({name,exitCode,coverage:capture?.protobuf_coverage,headers:record?.reported?.response?.headers,protobuf:record?.protobuf ?? null,redactions:record?.redactions,recordLimitations:record?.limitations,captureLimitations:capture?.limitations,bodyState:retained?.state,bodyUnchanged:retained?.content_base64 === body.toString('base64') && retained?.sha256 === createHash('sha256').update(body).digest('hex'),inputUnchanged:readFileSync(path).equals(input),error:result.code});
}
writeFileSync(join(out,'summary.json'),JSON.stringify(summary,null,2)+'\n');
console.log(JSON.stringify(summary,null,2));

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5926 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The public CLI reports complete protobuf coverage for a selected SAZ response after its `Content-Type` name or value is explicitly redacted. The body remains retained, but no protobuf facts or body limitation explain that automatic framing could not establish its applicability: ```json {"status":"complete","selected_records":1,"inspected_records":1} ``` I executed this with a synthetic seven-byte uncompressed gRPC body. Without redaction, the same body decodes completely as field 1 = 1. With either MIME redaction, the command exits 0, retains the identical body bytes and hash, emits no `record.protobuf`, and leaves `record.limitations` empty. The header's own redaction coordinate is correctly retained. ### Reproduction I ran eight sequential CLI cases using a ZIP containing `raw/1_s.txt`. Its HTTP response is: ```text HTTP/1.1 200 OK\r\n Content-Type: application/grpc\r\n Content-Length: 7\r\n \r\n ``` Append the literal bytes `00 00 00 00 02 08 01` after the header separator. These are a complete gRPC data frame with the protobuf payload `08 01`. Run the built repository CLI on that archive: ```sh node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --json node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --sensitive-value application/grpc --json node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" saz --record 0 --sensitive-value Content-Type --json ``` The first call decodes field 1 = 1. Both redaction calls return the complete coverage object above with no protobuf decoding or record limitation. Each case reproduced once. The executable generator and comparison used for the eight calls are included below. Save it outside the checkout and run `node reproduce.mjs "$REA_CHECKOUT"`; it writes fixtures and complete results next to the script. It uses the checkout's installed ZIP dependency and built CLI. ### Expected behavior When explicitly excluded MIME metadata leaves body applicability unresolved, preserve that uncertainty in protobuf coverage, for example with `status: "partial"` and a body limitation. This does not require guessing the content type or attempting to decode every unlabelled body. The repository requires that unresolved applicability remain unknown rather than be treated as absent. See [AGENTS.md, lines 19–21](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/AGENTS.md#L19-L21). Automatic decoding is scoped to recognized MIME types, while [the capture documentation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/web-network-captures.md#L208-L214) says protobuf coverage preserves partial decoding. The hidden-header-name case also makes HTTP coding metadata unknown. The [specific coding contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/web-network-captures.md#L179-L186) requires retaining that uncertainty. Explicit framing on this same case correctly returns partial with a coding limitation; automatic framing bypasses that qualification. ### Impact and scope This is a bounded coverage-reporting defect: complete coverage conceals unresolved applicability for an explicitly selected retained body. It affects the tested default automatic-framing path after Content-Type name/value exclusion. Original capture bytes and retained body bytes were unchanged in all eight cases. For MIME-value redaction, supplying the known framing with `--protobuf-framing grpc` restored complete decoding in the executed fixture. For header-name redaction, the same override instead returned partial and an HTTP-coding limitation, because header names remained unknown. It did not restore decoding. I did not test other capture formats, a real Fiddler export, or a credential-bearing body. No security exposure or source-byte loss is claimed. ### REA version 52415975942a4b9c53a817915b313ccc404f280e; package metadata rea-agents 6.7.0. ### Runtime and platform macOS 27.0.1 arm64, Node v26.12.0. The repository-pinned Node 24.18.0 runtime was not exercised. ### Target and artifact context Synthetic Fiddler SAZ ZIP containing one HTTP response and an uncompressed seven-byte gRPC body. Offline file inspection; no live target or external provider. ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence Executed results, all exit 0 and all with unchanged retained body bytes/hash: | Input/selection | Protobuf facts | Coverage | Record limitation | | --- | --- | --- | --- | | Visible application/grpc | Complete body, field 1 = 1 | complete, 1/1 | None | | MIME value explicitly excluded | None | complete, 1/1 | None | | Header name explicitly excluded | None | complete, 1/1 | None | | MIME value excluded, explicit grpc framing | Complete body, field 1 = 1 | complete, 1/1 | None | | Header name excluded, explicit grpc framing | None | partial, 1/1 | HTTP coding unknown | | Content-Type genuinely absent | None | complete, 1/1 | None | | Known application/octet-stream | None | complete, 1/1 | None | | Content-Encoding identity value excluded, MIME visible | None | partial, 1/1 | HTTP coding unknown | The known-absent and known-other-MIME controls show why the issue concerns excluded applicability rather than requiring protobuf decoding of all retained bodies. ### Typed outcome or recovery expected Retain the source bytes and header redaction provenance, and qualify incomplete protobuf applicability with partial coverage and a limitation. An unavailable result is appropriate only where the existing source-inspection contract requires it; it is not required merely because a MIME value is hidden. ### Analysis or suspected cause Source-derived explanation: [body metadata](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufCapture.ts#L143-L189) collapses a redacted content type to the same undefined value as an absent one. A hidden header name also makes HTTP coding and identity unknown. The [automatic-framing branch](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufCapture.ts#L245-L270) skips undefined framing without setting `incomplete`. It therefore never reaches the [unknown-identity qualification](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufCapture.ts#L320-L342). The [final reduction](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufCapture.ts#L492-L501) counts the record as inspected and upgrades coverage to complete. Counting the visited record is reasonable; discarding unresolved applicability before deciding completeness is the problem. ### Additional context The relevant production lines remain unchanged in checked upstream main `aa53c065387c58b4189d82c47f1f3445fc505e30`. This report concerns eligibility before decoding, separate from compressed-frame/body status aggregation and malformed gRPC-web trailer validation. The reproduction uses actual CLI subprocesses and synthetic source bytes. MCP, other platforms, the published package, performance, and real-producer export compatibility were not exercised. Executed fixture generator and CLI runner: ```javascript import { createRequire } from 'node:module'; import { writeFileSync, readFileSync } from 'node:fs'; import { execFileSync } from 'node:child_process'; import { join, dirname } from 'node:path'; import { fileURLToPath } from 'node:url'; import { createHash } from 'node:crypto'; const checkout = process.argv[2]; const out = dirname(fileURLToPath(import.meta.url)); const require = createRequire(join(checkout, 'package.json')); const { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } = require('@zip.js/zip.js'); const body = Buffer.from([0, 0, 0, 0, 2, 8, 1]); const variants = [ ['control', 'Content-Type: application/grpc\r\n', []], ['mime-value', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'application/grpc']], ['mime-name', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'Content-Type']], ['mime-value-override', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'application/grpc', '--protobuf-framing', 'grpc']], ['mime-name-override', 'Content-Type: application/grpc\r\n', ['--sensitive-value', 'Content-Type', '--protobuf-framing', 'grpc']], ['mime-absent', '', []], ['mime-other', 'Content-Type: application/octet-stream\r\n', []], ['coding-value', 'Content-Type: application/grpc\r\nContent-Encoding: identity\r\n', ['--sensitive-value', 'identity']], ]; const summary = []; for (const [name, headers, extra] of variants) { const writer = new ZipWriter(new Uint8ArrayWriter(), {useWebWorkers:false}); const response = Buffer.concat([Buffer.from('HTTP/1.1 200 OK\r\n' + headers + 'Content-Length: 7\r\n\r\n'), body]); await writer.add('raw/1_s.txt', new Uint8ArrayReader(response)); const path = join(out, name + '.saz'); const input = Buffer.from(await writer.close()); writeFileSync(path, input); const args = [join(checkout, 'scripts/rea.mjs'), 'inspect-web-network-capture', path, 'saz', '--record', '0', '--json', ...extra]; let stdout, stderr = '', exitCode = 0; try { stdout = execFileSync(process.execPath, args, {encoding:'utf8', timeout:40000, maxBuffer:8*1024*1024}); } catch(error) { stdout = error.stdout; stderr = String(error.stderr); exitCode = error.status; } writeFileSync(join(out,name+'.stdout.json'),stdout); writeFileSync(join(out,name+'.stderr.txt'),stderr); const result = JSON.parse(stdout), capture = result.normalized_result; const record = capture?.records?.[0]; const retained = record?.binary_fields.find(f => f.pointer === '/response/body'); summary.push({name,exitCode,coverage:capture?.protobuf_coverage,headers:record?.reported?.response?.headers,protobuf:record?.protobuf ?? null,redactions:record?.redactions,recordLimitations:record?.limitations,captureLimitations:capture?.limitations,bodyState:retained?.state,bodyUnchanged:retained?.content_base64 === body.toString('base64') && retained?.sha256 === createHash('sha256').update(body).digest('hex'),inputUnchanged:readFileSync(path).equals(input),error:result.code}); } writeFileSync(join(out,'summary.json'),JSON.stringify(summary,null,2)+'\n'); console.log(JSON.stringify(summary,null,2)); ```

bsakaguchi closed this issue

2026-10-31 00:42:18 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:53 +00:00

[PR #3601] [CLOSED] fix(process): report cleanup states when capture preparation fails #5848