morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6749] [Bug] gRPC-web trailers lose grpc-status when another value contains HTTP obs-text #4110

New issue

Closed

opened 2026-10-31 00:46:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6749

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Inspecting a retained gRPC-web body containing grpc-status: 0 followed by a custom trailer value with octet E9 makes REA discard every decoded trailer field. The trailer has status: "partial", examined_bytes: 0, and error gRPC-web trailer header block is not valid UTF-8. The earlier ASCII status and later ASCII field are both absent. Original body bytes and the preceding protobuf field remain intact.

This reproduced through the shared source capture application/subprocess and retained-view boundaries for binary and text gRPC-web. ASCII and UTF-8 controls decoded completely. Built CLI, installed-package and MCP transport execution were not tested.

Reproduction

The investigation executed the production WebNetworkCaptureService / HistoricalCaptureDecoder / NodeCaptureProcess.ts path against self-authored HAR files. The following checkout-local presentation preserves that path and the exact failing body; temporary paths and output logging are simplified. Locked dependencies must be installed, and node below denotes the tested runtime. The worker launcher selects the source entrypoint with the locked tsx loader instead of a compiled entrypoint.

node --import tsx --input-type=module <<'JS'
import { mkdtemp, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { HistoricalCaptureDecoder } from './src/browser/history/HistoricalCaptureDecoder.ts';
import { HAR_CAPTURE_PROVIDER_IDENTITY } from './src/browser/history/CaptureRelease.ts';
import { WebNetworkCaptureService } from './src/application/WebNetworkCaptureService.ts';
import { PrivateRuntimeRoot } from './src/process/PrivateRuntimeRoot.ts';

const repo = process.cwd();
const work = await mkdtemp(join(tmpdir(), 'rea-trailer-'));
const body = Buffer.from(
  '00000000020801800000002d' +
  '677270632d7374617475733a20300d0a' +
  '782d6e6f74653a20636166e90d0a' +
  '782d61667465723a206b6570740d0a', 'hex');
const capturePath = join(work, 'trailer.har');
await writeFile(capturePath, JSON.stringify({log: {
  version: '1.2', creator: {name: 'synthetic', version: '1'},
  entries: [{
    startedDateTime: new Date().toISOString(), time: 0,
    request: {
      method: 'POST', url: 'https://example.invalid/rpc',
      httpVersion: 'HTTP/1.1', headers: [], cookies: [], queryString: [],
      headersSize: -1, bodySize: 0
    },
    response: {
      status: 200, statusText: 'OK', httpVersion: 'HTTP/1.1',
      headers: [], cookies: [], redirectURL: '',
      headersSize: -1, bodySize: body.length,
      content: {size: body.length, mimeType: 'application/grpc-web+proto',
        text: body.toString('base64'), encoding: 'base64'}
    },
    cache: {}, timings: {send: 0, wait: 0, receive: 0}
  }]
}}));
const decoder = new HistoricalCaptureDecoder([{
  format: 'har', identity: HAR_CAPTURE_PROVIDER_IDENTITY,
  command: async requestPath => ({
    command: process.execPath,
    arguments: ['--import', join(repo, 'node_modules/tsx/dist/loader.mjs'),
      join(repo, 'src/browser/history/NodeCaptureProcess.ts'), requestPath, 'har']
  })
}], {PATH: process.env.PATH, HOME: work, TMPDIR: work, TSX_DISABLE_CACHE: '1'},
  () => PrivateRuntimeRoot.create({parent: work, prefix: 'capture-'}));
const service = new WebNetworkCaptureService(decoder);
try {
  const result = await service.inspect({
    capture_path: capturePath, format: 'har', record_ordinals: [0]
  });
  if (!result.ok) throw result.error;
  const report = result.value.normalized_result;
  console.log(JSON.stringify({
    coverage: report.protobuf_coverage,
    body: report.records[0].protobuf[0]
  }, null, 2));
} finally {
  await service.close();
}
JS

The 57-byte body has a protobuf data frame containing field 1 = 1, then an uncompressed terminal trailer frame. Its payload is grpc-status: 0\r\nx-note: caf + E9 + \r\nx-after: kept\r\n.

Each non-UTF-8 case ran once: the binary fixture, the same payload in gRPC-web-text, and an executed reduction containing only the preceding status and x-note: + E9. All three lost every decoded trailer field. Separate ASCII and UTF-8 controls passed.

Expected behavior

Preserve unrelated ASCII trailer facts, including grpc-status: 0, when another value contains HTTP-valid opaque octets. This is a receiver-side decoding report: E9 is outside gRPC's preferred sender-side ASCII metadata grammar, but that does not justify losing the entire status block or require interpreting arbitrary bytes as a particular Unicode charset.

The gRPC-Web framing contract uses an HTTP/1 field block. RFC 9110 section 5.5 permits obs-text octets 80–FF as opaque field content. The native gRPC receiving rule permits accepting or discarding the individual HTTP-valid non-ASCII value while requiring tolerance on receipt.

REA's parseGrpcWebTrailers explicitly describes preserving the valid prefix of this field block.

Impact and scope

The structured RPC status and all custom trailer facts disappear for the affected captures, and body/coverage status becomes partial. Original capture/body bytes and previously decoded protobuf data remain available. Verified only on the stated self-authored fixtures; no real producer prevalence, live RPC failure, source-byte loss or security consequence is claimed.

A separately executed manual inspection of the retained bytes recovered status 0 and both custom fields for this small fixture. That is a byte-inspection workaround, not an in-product override or a general charset assertion.

REA version

Source revision bc411c24befceced5a9ea1ca1c1ce123fd02bf48, manifest version 6.7.0. The published npm artifact was not executed.

Runtime and platform

Node.js v24.22.1; Linux x64; locked tsx 4.23.15 and protobuf-es 2.16.0. No browser, proxy listener or remote target was used.

Target and artifact context

Self-authored HAR 1.2, with one selected response body using application/grpc-web+proto and a base64 HAR envelope. A second tested fixture used application/grpc-web-text+proto. All trailer field names were lowercase; values contained no forbidden controls.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

Selected observed binary-case result:

{
  "coverage": {"status": "partial", "selected_records": 1, "inspected_records": 1},
  "body_status": "partial",
  "decoded_bytes": 57,
  "examined_bytes": 12,
  "trailer": {
    "index": 1, "kind": "trailer", "offset": 7,
    "payload_offset": 12, "bytes": 45, "compressed": false,
    "status": "partial", "examined_bytes": 0,
    "error": {"offset": 12, "reason": "gRPC-web trailer header block is not valid UTF-8"}
  }
}

The trailer's trailers property is absent. The retained frame view explicitly selecting that property carries the same absence and error. ASCII cafe and UTF-8 café controls return all three fields, complete coverage, and fully examined bodies.

Evidence provenance: observed application/subprocess results and saved retained-view outputs; protocol applicability and diagnosis are source-derived.
Local verification: reproduced.
Reproduction completeness: complete for the specified source boundaries and fixtures. CLI/MCP and real producer exports remain untested.

Typed outcome or recovery expected

Keep the unaffected ASCII status and metadata fields as structured facts and preserve original opaque bytes. The legal HTTP octet should not trigger a whole-block UTF-8 prerequisite. Preserve existing protobuf facts and forbidden-control validation.

Analysis or suspected cause

Source-derived: ProtobufTrailers.ts fatally decodes the entire block as UTF-8 before parsing any lines. On E9 it returns zero examined bytes and an empty field list. ProtobufBody.ts then omits trailer facts for that error kind. No fix was made.

Additional context

Related closed #5386 concerns forbidden C0/DEL values being incorrectly accepted as complete. This input instead uses HTTP-valid, receiver-tolerated obs-text and loses unrelated valid ASCII fields; fixing it can preserve the existing control rejection. Binary/text framing and retained views share one failure owner and are not separate reports.

All runtime verification and evidence review finished before this draft was written. No reproduction was rerun afterward.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Shared application/subprocess observations show affected recorded captures losing structured RPC status and all trailer metadata, including otherwise valid ASCII fields. Original bytes and preceding protobuf facts remain available; real-world prevalence is unmeasured.
Workaround status: available
Workaround basis: Manual byte-preserving inspection of the retained source body recovered status 0 and both metadata fields in the tested small fixture, without modifying the input. This does not repair the structured REA result.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6749 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Inspecting a retained gRPC-web body containing `grpc-status: 0` followed by a custom trailer value with octet E9 makes REA discard every decoded trailer field. The trailer has `status: "partial"`, `examined_bytes: 0`, and error `gRPC-web trailer header block is not valid UTF-8`. The earlier ASCII status and later ASCII field are both absent. Original body bytes and the preceding protobuf field remain intact. This reproduced through the shared source capture application/subprocess and retained-view boundaries for binary and text gRPC-web. ASCII and UTF-8 controls decoded completely. Built CLI, installed-package and MCP transport execution were not tested. ## Reproduction The investigation executed the production `WebNetworkCaptureService` / `HistoricalCaptureDecoder` / `NodeCaptureProcess.ts` path against self-authored HAR files. The following checkout-local presentation preserves that path and the exact failing body; temporary paths and output logging are simplified. Locked dependencies must be installed, and `node` below denotes the tested runtime. The worker launcher selects the source entrypoint with the locked tsx loader instead of a compiled entrypoint. ```sh node --import tsx --input-type=module <<'JS' import { mkdtemp, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { HistoricalCaptureDecoder } from './src/browser/history/HistoricalCaptureDecoder.ts'; import { HAR_CAPTURE_PROVIDER_IDENTITY } from './src/browser/history/CaptureRelease.ts'; import { WebNetworkCaptureService } from './src/application/WebNetworkCaptureService.ts'; import { PrivateRuntimeRoot } from './src/process/PrivateRuntimeRoot.ts'; const repo = process.cwd(); const work = await mkdtemp(join(tmpdir(), 'rea-trailer-')); const body = Buffer.from( '00000000020801800000002d' + '677270632d7374617475733a20300d0a' + '782d6e6f74653a20636166e90d0a' + '782d61667465723a206b6570740d0a', 'hex'); const capturePath = join(work, 'trailer.har'); await writeFile(capturePath, JSON.stringify({log: { version: '1.2', creator: {name: 'synthetic', version: '1'}, entries: [{ startedDateTime: new Date().toISOString(), time: 0, request: { method: 'POST', url: 'https://example.invalid/rpc', httpVersion: 'HTTP/1.1', headers: [], cookies: [], queryString: [], headersSize: -1, bodySize: 0 }, response: { status: 200, statusText: 'OK', httpVersion: 'HTTP/1.1', headers: [], cookies: [], redirectURL: '', headersSize: -1, bodySize: body.length, content: {size: body.length, mimeType: 'application/grpc-web+proto', text: body.toString('base64'), encoding: 'base64'} }, cache: {}, timings: {send: 0, wait: 0, receive: 0} }] }})); const decoder = new HistoricalCaptureDecoder([{ format: 'har', identity: HAR_CAPTURE_PROVIDER_IDENTITY, command: async requestPath => ({ command: process.execPath, arguments: ['--import', join(repo, 'node_modules/tsx/dist/loader.mjs'), join(repo, 'src/browser/history/NodeCaptureProcess.ts'), requestPath, 'har'] }) }], {PATH: process.env.PATH, HOME: work, TMPDIR: work, TSX_DISABLE_CACHE: '1'}, () => PrivateRuntimeRoot.create({parent: work, prefix: 'capture-'})); const service = new WebNetworkCaptureService(decoder); try { const result = await service.inspect({ capture_path: capturePath, format: 'har', record_ordinals: [0] }); if (!result.ok) throw result.error; const report = result.value.normalized_result; console.log(JSON.stringify({ coverage: report.protobuf_coverage, body: report.records[0].protobuf[0] }, null, 2)); } finally { await service.close(); } JS ``` The 57-byte body has a protobuf data frame containing field 1 = 1, then an uncompressed terminal trailer frame. Its payload is `grpc-status: 0\r\nx-note: caf` + E9 + `\r\nx-after: kept\r\n`. Each non-UTF-8 case ran once: the binary fixture, the same payload in gRPC-web-text, and an executed reduction containing only the preceding status and `x-note: ` + E9. All three lost every decoded trailer field. Separate ASCII and UTF-8 controls passed. ## Expected behavior Preserve unrelated ASCII trailer facts, including `grpc-status: 0`, when another value contains HTTP-valid opaque octets. This is a receiver-side decoding report: E9 is outside gRPC's preferred sender-side ASCII metadata grammar, but that does not justify losing the entire status block or require interpreting arbitrary bytes as a particular Unicode charset. The [gRPC-Web framing contract](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md#protocol-differences-vs-grpc-over-http2) uses an HTTP/1 field block. [RFC 9110 section 5.5](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.5) permits obs-text octets 80–FF as opaque field content. The [native gRPC receiving rule](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md#requests) permits accepting or discarding the individual HTTP-valid non-ASCII value while requiring tolerance on receipt. REA's `parseGrpcWebTrailers` explicitly describes preserving the valid prefix of this field block. ## Impact and scope The structured RPC status and all custom trailer facts disappear for the affected captures, and body/coverage status becomes partial. Original capture/body bytes and previously decoded protobuf data remain available. Verified only on the stated self-authored fixtures; no real producer prevalence, live RPC failure, source-byte loss or security consequence is claimed. A separately executed manual inspection of the retained bytes recovered status 0 and both custom fields for this small fixture. That is a byte-inspection workaround, not an in-product override or a general charset assertion. ## REA version Source revision `bc411c24befceced5a9ea1ca1c1ce123fd02bf48`, manifest version 6.7.0. The published npm artifact was not executed. ## Runtime and platform Node.js v24.22.1; Linux x64; locked tsx 4.23.15 and protobuf-es 2.16.0. No browser, proxy listener or remote target was used. ## Target and artifact context Self-authored HAR 1.2, with one selected response body using `application/grpc-web+proto` and a base64 HAR envelope. A second tested fixture used `application/grpc-web-text+proto`. All trailer field names were lowercase; values contained no forbidden controls. ## Interface Other ## Earliest failure stage Transport or result projection ## Relevant output or evidence Selected observed binary-case result: ```json { "coverage": {"status": "partial", "selected_records": 1, "inspected_records": 1}, "body_status": "partial", "decoded_bytes": 57, "examined_bytes": 12, "trailer": { "index": 1, "kind": "trailer", "offset": 7, "payload_offset": 12, "bytes": 45, "compressed": false, "status": "partial", "examined_bytes": 0, "error": {"offset": 12, "reason": "gRPC-web trailer header block is not valid UTF-8"} } } ``` The trailer's `trailers` property is absent. The retained frame view explicitly selecting that property carries the same absence and error. ASCII `cafe` and UTF-8 `café` controls return all three fields, complete coverage, and fully examined bodies. - Evidence provenance: observed application/subprocess results and saved retained-view outputs; protocol applicability and diagnosis are source-derived. - Local verification: reproduced. - Reproduction completeness: complete for the specified source boundaries and fixtures. CLI/MCP and real producer exports remain untested. ## Typed outcome or recovery expected Keep the unaffected ASCII status and metadata fields as structured facts and preserve original opaque bytes. The legal HTTP octet should not trigger a whole-block UTF-8 prerequisite. Preserve existing protobuf facts and forbidden-control validation. ## Analysis or suspected cause Source-derived: [ProtobufTrailers.ts](https://github.com/morluto/rea/blob/bc411c24befceced5a9ea1ca1c1ce123fd02bf48/src/browser/history/protobuf/ProtobufTrailers.ts#L14-L33) fatally decodes the entire block as UTF-8 before parsing any lines. On E9 it returns zero examined bytes and an empty field list. `ProtobufBody.ts` then omits trailer facts for that error kind. No fix was made. ## Additional context Related closed [#5386](https://github.com/morluto/rea/issues/5386) concerns forbidden C0/DEL values being incorrectly accepted as complete. This input instead uses HTTP-valid, receiver-tolerated obs-text and loses unrelated valid ASCII fields; fixing it can preserve the existing control rejection. Binary/text framing and retained views share one failure owner and are not separate reports. All runtime verification and evidence review finished before this draft was written. No reproduction was rerun afterward. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Shared application/subprocess observations show affected recorded captures losing structured RPC status and all trailer metadata, including otherwise valid ASCII fields. Original bytes and preceding protobuf facts remain available; real-world prevalence is unmeasured. - Workaround status: available - Workaround basis: Manual byte-preserving inspection of the retained source body recovered status 0 and both metadata fields in the tested small fixture, without modifying the input. This does not repair the structured REA result.

bsakaguchi closed this issue

2026-10-31 00:46:53 +00:00

bsakaguchi referenced this issue

2026-10-31 00:49:54 +00:00

[GH-ISSUE #7457] [Feature] Evidence-backed firmware update comparison with component-level change attribution #4452

bsakaguchi referenced this issue

2026-10-31 00:49:56 +00:00

[GH-ISSUE #7461] [Feature] Evidence-backed firmware update comparison with component-level change attribution #4454

bsakaguchi referenced this issue

2026-11-01 21:12:00 +00:00

[GH-ISSUE #8292] [Feature] Consider moving from binwalk to binwalk-ng #6530