morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5942] [Bug] gRPC-web-text base64 failures report encoded length as decoded_bytes #3612

New issue

Closed

opened 2026-10-31 00:42:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:22 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5942

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public inspect-web-network-capture CLI returns the retained protocol-text length as decoded_bytes when gRPC-web-text base64 preparation fails. For the four ASCII bytes AB==, the result reports four decoded bytes even though it rejects the noncanonical base64 and returns no decoded frames or fields. The invalid alphabet !!!! produces the same incorrect count.

The command correctly reports a partial body, zero examined bytes, and a base64 error. The defect is the accompanying decoded-byte count. Original source bytes and their hashes remain intact.

Reproduction

I ran five sequential calls to the built repository CLI, once per synthetic HAR 1.2 fixture:

node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" har --record 0 --json

Each response has MIME application/grpc-web-text. Its protocol text is itself encoded in a canonical HAR response.content.text base64 envelope; malformed protocol text therefore passes ordinary HAR validation and reaches the gRPC-web-text error path.

Executed cases:

Valid protocol text AAAAAAIIAQ==: 12 retained bytes, 7 decoded and examined bytes, complete status, one data frame, field 1 = 1.
Noncanonical protocol text AB==: 4 retained bytes, incorrectly 4 decoded bytes, zero examined bytes, partial status and no frames or fields.
Invalid alphabet !!!!: the same incorrect four-byte decoded count and base64 error.
Valid chunk followed by an invalid chunk, AAAAAAIIAQ==AB==: 16 retained bytes, incorrectly 16 decoded bytes, zero examined bytes and no frames or fields.
Empty valid text: zero retained/decoded/examined bytes and complete status.

All five commands exited 0. Assertions confirmed the input files and retained body bytes/hashes were unchanged. The complete executed generator and runner are included under Additional context; save it outside the checkout and run node reproduce.mjs "$REA_CHECKOUT". It writes fixtures and results next to the script.

Expected behavior

Keep retained-text length separate from decoded-body length. If preparation returns no decoded body, decoded_bytes should not fall back to the encoded input length. A zero count with the existing partial status and diagnostic would represent the current all-or-nothing failure without adding a new public field type.

The capture documentation distinguishes retained source from decoded body/frame/field facts and assigns gRPC-web-text coordinates to base64-decoded bytes. The body schema and consistency checks compare examined coverage with decoded_bytes. The successful chunk test expects decoded length rather than encoded-text length. The executed valid control confirms this distinction.

This report does not require changing offset_basis: keeping the documented format-wide coordinate basis is compatible with correcting the count. It also does not require recovering fields from a valid prefix after a later chunk fails.

Impact and scope

This is a bounded Evidence accounting defect for selected gRPC-web-text bodies whose protocol base64 is rejected. Consumers reading decoded_bytes receive a positive decoded size when no decoded buffer or facts are available. Body-view rows also preserve this property in the source implementation; a separate view command was not executed.

The partial status and explicit error remain available, so consumers can disregard the count on this failure and use the retained binary descriptor for the encoded source length. No source-byte loss, security exposure, or downstream application failure was observed. Frequency in real captures is unknown.

REA version

5241597594; package metadata rea-agents 6.7.0.

Runtime and platform

macOS 27.0.1 arm64, Node v26.12.0. The repository-pinned Node 24.18.0 runtime was not exercised.

Target and artifact context

Five synthetic local HAR 1.2 files with gRPC-web-text response bodies. Offline inspection; no browser, external provider, target execution or recorded-URL fetch.

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

For protocol text AB==, the observed normalized_result.records[0].protobuf[0] includes:

{
  "pointer": "/response/content/text",
  "framing": "grpc-web-text",
  "offset_basis": "base64-decoded-body",
  "decoded_bytes": 4,
  "status": "partial",
  "examined_bytes": 0,
  "frames": [],
  "fields": [],
  "error": {
    "offset": 0,
    "reason": "gRPC-web text is not canonical base64 or independently padded base64 chunks"
  }
}

The retained binary sidecar separately and correctly reports four bytes. The invalid-alphabet case produces the same projection. These are actual public CLI results, not direct internal-function calls.

Typed outcome or recovery expected

Preserve the partial result, error and retained source descriptor, while returning a decoded-byte count that reflects the absence of an available decoded body.

Analysis or suspected cause

Source-derived cause: the body owner initializes decoded_bytes with body.length. For gRPC-web-text, this is the retained base64 text length. Preparation returns undefined data on malformed/noncanonical text, and the early failure return occurs before result.decoded_bytes = data.length.

The production capture path passes that prepared result into the same owner. The body-view row schema includes the stored count, so the representation error originates before projection. Only the capture CLI was executed here.

Additional context

The relevant body/framing/schema/view source is unchanged in checked upstream main ce1f4b1bbaced57529d609f3007a76de583e11c4.

Issue #5926 concerns automatic framing after MIME redaction; this reproduction leaves MIME available and correctly reports partial coverage. Issue #5386 and PR #5422 concern trailer-value validation after frame decoding. Neither requires the same failure-path byte-count fix.

MCP, the published package, other operating systems, real-producer captures and performance were not exercised.

Executed fixture generator and CLI runner:

import { writeFileSync, readFileSync } from 'node:fs';
import { spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import assert from 'node:assert/strict';

const checkout = process.argv[2];
const out = dirname(fileURLToPath(import.meta.url));
const stamp = spawnSync('date', ['+%Y-%m-%dT%H:%M:%S%z'], {
  encoding: 'utf8', env: {...process.env, TZ:'America/Los_Angeles'},
}).stdout.trim();
const good = Buffer.from([0,0,0,0,2,8,1]).toString('base64');
const variants = [
  ['control', good, 7],
  ['noncanonical', 'AB==', null],
  ['invalid-alphabet', '!!!!', null],
  ['valid-prefix-invalid-suffix', good + 'AB==', null],
  ['empty-control', '', 0],
];
const summary = [];
for (const [name,text,expectedDecoded] of variants) {
  const bytes = Buffer.from(text);
  const content = {size:bytes.length, mimeType:'application/grpc-web-text', encoding:'base64', text:bytes.toString('base64')};
  const fixture = {log:{version:'1.2',creator:{name:'synthetic',version:'1'},entries:[{
    startedDateTime:stamp,time:0,
    request:{method:'POST',url:'https://example.test/rpc',httpVersion:'HTTP/1.1',cookies:[],headers:[],queryString:[],headersSize:-1,bodySize:0},
    response:{status:200,statusText:'OK',httpVersion:'HTTP/1.1',cookies:[],headers:[{name:'Content-Type',value:content.mimeType}],content,redirectURL:'',headersSize:-1,bodySize:bytes.length},
    cache:{},timings:{send:0,wait:0,receive:0},
  }]}};
  const path = join(out, name+'.har');
  const input = JSON.stringify(fixture);
  writeFileSync(path,input);
  const args = [join(checkout,'scripts/rea.mjs'),'inspect-web-network-capture',path,'har','--record','0','--json'];
  const result = spawnSync(process.execPath,args,{encoding:'utf8',timeout:40000,maxBuffer:8*1024*1024});
  writeFileSync(join(out,name+'.stdout.json'),result.stdout);
  writeFileSync(join(out,name+'.stderr.txt'),result.stderr);
  assert.equal(result.status,0,result.stderr);
  const evidence = JSON.parse(result.stdout);
  const capture = evidence.normalized_result;
  const record = capture.records[0];
  const body = record.protobuf[0];
  const retained = record.binary_fields.find(f=>f.pointer==='/response/content/text');
  assert.equal(retained.content_base64,bytes.toString('base64'));
  assert.equal(retained.sha256,createHash('sha256').update(bytes).digest('hex'));
  assert.equal(readFileSync(path,'utf8'),input);
  if (expectedDecoded === null) {
    assert.equal(body.status,'partial');
    assert.equal(body.decoded_bytes,bytes.length);
    assert.equal(body.examined_bytes,0);
    assert.deepEqual(body.frames,[]);
    assert.deepEqual(body.fields,[]);
    assert.match(body.error.reason,/not canonical base64/);
    assert.notEqual(body.decoded_bytes,0);
  } else {
    assert.equal(body.status,'complete');
    assert.equal(body.decoded_bytes,expectedDecoded);
    assert.equal(body.examined_bytes,expectedDecoded);
  }
  summary.push({name,exitCode:result.status,inputText:text,retainedBytes:retained.bytes,coverage:capture.protobuf_coverage,body,bodyUnchanged:true,inputUnchanged:true});
}
writeFileSync(join(out,'summary.json'),JSON.stringify(summary,null,2)+'\n');
console.log(JSON.stringify(summary,null,2));

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5942 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The public `inspect-web-network-capture` CLI returns the retained protocol-text length as `decoded_bytes` when gRPC-web-text base64 preparation fails. For the four ASCII bytes `AB==`, the result reports four decoded bytes even though it rejects the noncanonical base64 and returns no decoded frames or fields. The invalid alphabet `!!!!` produces the same incorrect count. The command correctly reports a partial body, zero examined bytes, and a base64 error. The defect is the accompanying decoded-byte count. Original source bytes and their hashes remain intact. ### Reproduction I ran five sequential calls to the built repository CLI, once per synthetic HAR 1.2 fixture: ```sh node scripts/rea.mjs inspect-web-network-capture "$CAPTURE" har --record 0 --json ``` Each response has MIME `application/grpc-web-text`. Its protocol text is itself encoded in a canonical HAR `response.content.text` base64 envelope; malformed protocol text therefore passes ordinary HAR validation and reaches the gRPC-web-text error path. Executed cases: - Valid protocol text `AAAAAAIIAQ==`: 12 retained bytes, 7 decoded and examined bytes, complete status, one data frame, field 1 = 1. - Noncanonical protocol text `AB==`: 4 retained bytes, incorrectly 4 decoded bytes, zero examined bytes, partial status and no frames or fields. - Invalid alphabet `!!!!`: the same incorrect four-byte decoded count and base64 error. - Valid chunk followed by an invalid chunk, `AAAAAAIIAQ==AB==`: 16 retained bytes, incorrectly 16 decoded bytes, zero examined bytes and no frames or fields. - Empty valid text: zero retained/decoded/examined bytes and complete status. All five commands exited 0. Assertions confirmed the input files and retained body bytes/hashes were unchanged. The complete executed generator and runner are included under Additional context; save it outside the checkout and run `node reproduce.mjs "$REA_CHECKOUT"`. It writes fixtures and results next to the script. ### Expected behavior Keep retained-text length separate from decoded-body length. If preparation returns no decoded body, `decoded_bytes` should not fall back to the encoded input length. A zero count with the existing partial status and diagnostic would represent the current all-or-nothing failure without adding a new public field type. The [capture documentation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/web-network-captures.md#L127-L145) distinguishes retained source from decoded body/frame/field facts and assigns gRPC-web-text coordinates to base64-decoded bytes. The [body schema and consistency checks](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/protobufWire.ts#L82-L118) compare examined coverage with `decoded_bytes`. The [successful chunk test](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufWire.test.ts#L225-L260) expects decoded length rather than encoded-text length. The executed valid control confirms this distinction. This report does not require changing `offset_basis`: keeping the documented format-wide coordinate basis is compatible with correcting the count. It also does not require recovering fields from a valid prefix after a later chunk fails. ### Impact and scope This is a bounded Evidence accounting defect for selected gRPC-web-text bodies whose protocol base64 is rejected. Consumers reading `decoded_bytes` receive a positive decoded size when no decoded buffer or facts are available. Body-view rows also preserve this property in the source implementation; a separate view command was not executed. The partial status and explicit error remain available, so consumers can disregard the count on this failure and use the retained binary descriptor for the encoded source length. No source-byte loss, security exposure, or downstream application failure was observed. Frequency in real captures is unknown. ### REA version 52415975942a4b9c53a817915b313ccc404f280e; package metadata rea-agents 6.7.0. ### Runtime and platform macOS 27.0.1 arm64, Node v26.12.0. The repository-pinned Node 24.18.0 runtime was not exercised. ### Target and artifact context Five synthetic local HAR 1.2 files with gRPC-web-text response bodies. Offline inspection; no browser, external provider, target execution or recorded-URL fetch. ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence For protocol text `AB==`, the observed `normalized_result.records[0].protobuf[0]` includes: ```json { "pointer": "/response/content/text", "framing": "grpc-web-text", "offset_basis": "base64-decoded-body", "decoded_bytes": 4, "status": "partial", "examined_bytes": 0, "frames": [], "fields": [], "error": { "offset": 0, "reason": "gRPC-web text is not canonical base64 or independently padded base64 chunks" } } ``` The retained binary sidecar separately and correctly reports four bytes. The invalid-alphabet case produces the same projection. These are actual public CLI results, not direct internal-function calls. ### Typed outcome or recovery expected Preserve the partial result, error and retained source descriptor, while returning a decoded-byte count that reflects the absence of an available decoded body. ### Analysis or suspected cause Source-derived cause: [the body owner initializes `decoded_bytes` with `body.length`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufBody.ts#L44-L81). For gRPC-web-text, this is the retained base64 text length. [Preparation returns undefined data on malformed/noncanonical text](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufFraming.ts#L40-L104), and the early failure return occurs before `result.decoded_bytes = data.length`. The [production capture path](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/history/protobuf/ProtobufCapture.ts#L437-L485) passes that prepared result into the same owner. The [body-view row schema](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisView/protobufCaptureView.ts#L21-L39) includes the stored count, so the representation error originates before projection. Only the capture CLI was executed here. ### Additional context The relevant body/framing/schema/view source is unchanged in checked upstream main `ce1f4b1bbaced57529d609f3007a76de583e11c4`. Issue #5926 concerns automatic framing after MIME redaction; this reproduction leaves MIME available and correctly reports partial coverage. Issue #5386 and PR #5422 concern trailer-value validation after frame decoding. Neither requires the same failure-path byte-count fix. MCP, the published package, other operating systems, real-producer captures and performance were not exercised. Executed fixture generator and CLI runner: ```javascript import { writeFileSync, readFileSync } from 'node:fs'; import { spawnSync } from 'node:child_process'; import { createHash } from 'node:crypto'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; import assert from 'node:assert/strict'; const checkout = process.argv[2]; const out = dirname(fileURLToPath(import.meta.url)); const stamp = spawnSync('date', ['+%Y-%m-%dT%H:%M:%S%z'], { encoding: 'utf8', env: {...process.env, TZ:'America/Los_Angeles'}, }).stdout.trim(); const good = Buffer.from([0,0,0,0,2,8,1]).toString('base64'); const variants = [ ['control', good, 7], ['noncanonical', 'AB==', null], ['invalid-alphabet', '!!!!', null], ['valid-prefix-invalid-suffix', good + 'AB==', null], ['empty-control', '', 0], ]; const summary = []; for (const [name,text,expectedDecoded] of variants) { const bytes = Buffer.from(text); const content = {size:bytes.length, mimeType:'application/grpc-web-text', encoding:'base64', text:bytes.toString('base64')}; const fixture = {log:{version:'1.2',creator:{name:'synthetic',version:'1'},entries:[{ startedDateTime:stamp,time:0, request:{method:'POST',url:'https://example.test/rpc',httpVersion:'HTTP/1.1',cookies:[],headers:[],queryString:[],headersSize:-1,bodySize:0}, response:{status:200,statusText:'OK',httpVersion:'HTTP/1.1',cookies:[],headers:[{name:'Content-Type',value:content.mimeType}],content,redirectURL:'',headersSize:-1,bodySize:bytes.length}, cache:{},timings:{send:0,wait:0,receive:0}, }]}}; const path = join(out, name+'.har'); const input = JSON.stringify(fixture); writeFileSync(path,input); const args = [join(checkout,'scripts/rea.mjs'),'inspect-web-network-capture',path,'har','--record','0','--json']; const result = spawnSync(process.execPath,args,{encoding:'utf8',timeout:40000,maxBuffer:8*1024*1024}); writeFileSync(join(out,name+'.stdout.json'),result.stdout); writeFileSync(join(out,name+'.stderr.txt'),result.stderr); assert.equal(result.status,0,result.stderr); const evidence = JSON.parse(result.stdout); const capture = evidence.normalized_result; const record = capture.records[0]; const body = record.protobuf[0]; const retained = record.binary_fields.find(f=>f.pointer==='/response/content/text'); assert.equal(retained.content_base64,bytes.toString('base64')); assert.equal(retained.sha256,createHash('sha256').update(bytes).digest('hex')); assert.equal(readFileSync(path,'utf8'),input); if (expectedDecoded === null) { assert.equal(body.status,'partial'); assert.equal(body.decoded_bytes,bytes.length); assert.equal(body.examined_bytes,0); assert.deepEqual(body.frames,[]); assert.deepEqual(body.fields,[]); assert.match(body.error.reason,/not canonical base64/); assert.notEqual(body.decoded_bytes,0); } else { assert.equal(body.status,'complete'); assert.equal(body.decoded_bytes,expectedDecoded); assert.equal(body.examined_bytes,expectedDecoded); } summary.push({name,exitCode:result.status,inputText:text,retainedBytes:retained.bytes,coverage:capture.protobuf_coverage,body,bodyUnchanged:true,inputUnchanged:true}); } writeFileSync(join(out,'summary.json'),JSON.stringify(summary,null,2)+'\n'); console.log(JSON.stringify(summary,null,2)); ```

bsakaguchi closed this issue

2026-10-31 00:42:22 +00:00

bsakaguchi referenced this issue

2026-11-01 21:10:58 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471