[GH-ISSUE #6564] [Bug] environment-compare equates binary bytes with their base64 text #3952

New issue

Closed

opened 2026-10-31 00:45:33 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:33 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6564

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed production compareEnvironment call reports agreement: 1, equal: true, and no differences when the environment's response contains the single byte ff and the live probe's response contains the four UTF-8 bytes 2f 77 3d 3d (/w==). Both responses have status 200 and application/octet-stream; the binary response is retained as HAR base64 /w==, while the probe retains that spelling as literal text. Every normalization count is zero. The response bodies are complete, recorded, and below the replay limit.

This is an application-boundary reproduction with authenticated fixture Evidence and the actual manifest/state projection, source loading, and comparison service. CLI/MCP transport, browser recording, and full HAR ingestion were not run.

Reproduction

Executed once against the source revision below using Node plus tsx 4.23.15. This is the substantive scratch probe, with the checkout location supplied as REA_SOURCE_ROOT instead of a private absolute path. It uses the repository's existing fixture constructors and production application modules; no target application executes.

import { createHash } from 'node:crypto';
const root = process.env.REA_SOURCE_ROOT;
const { analysisViewCaptureEvidence, environmentManifestCaptureFixture } = await import(`${root}/tests/fixtures/analysisView.ts`);
const { environmentStateServeFixture, environmentServeSource } = await import(`${root}/tests/fixtures/environmentServe.ts`);
const { compareEnvironment } = await import(`${root}/src/application/environment/environmentCompare.ts`);
function capture(encoding) {
  const reported = { request: { method: 'GET', url: 'https://example.test/api/blob' }, response: { status: 200, content: { mimeType: 'application/octet-stream', text: '/w==', ...(encoding ? { encoding } : {}) } } };
  const value = environmentManifestCaptureFixture([reported]);
  if (encoding) value.records[0].binary_fields = [{ pointer: '/response/content/text', representation: 'har-base64-content', state: 'retained', content_base64: '/w==', bytes: 1, sha256: createHash('sha256').update(Buffer.from([255])).digest('hex') }];
  return analysisViewCaptureEvidence(value);
}
const baseline = capture('base64');
const fixture = environmentStateServeFixture(baseline);
const source = environmentServeSource(fixture);
const result = compareEnvironment({ source, captureValue: baseline, probes: [{ label: 'probe', value: capture(undefined) }], operation: 'environment-compare' });
if (!result.ok) throw result.error;
console.log(JSON.stringify(result.value));

With the checkout's locked dependencies available, an equivalent invocation is REA_SOURCE_ROOT=/tmp/rea-example npx tsx /tmp/rea-example/probe.mjs. The local execution used a scratch loader to resolve checkout imports against an already installed dependency set; this command spelling was not separately executed. A binary-versus-identical-binary control agreed, and a plain-a versus plain-b text control correctly reported a value difference.

Expected behavior

Different retained response bytes should produce a body difference. Base64 is an evidence representation of bytes, not literal response text: the documented replay contract retains body_encoding specifically to distinguish those cases. Identical binary responses should continue to agree, and a HAR base64 body that decodes to the same valid UTF-8 text should still compare by its decoded text.

Impact and scope

P2: fidelity reports can claim complete agreement for a changed binary API response. The observed case contains one request, no identifiers, no generated values, no clocks, no truncation, and no unobserved bodies. This changes the analysis result; it does not mutate the input Evidence. Real-world frequency and first affected revision were not established. Manual comparison of retained bytes and encoding metadata is an available workaround.

REA version

Current main source 6bb4f4bf7d64e5d2f47479d46225a5383b57cdad. The environment-compare implementation is absent from verified npm release 6.7.0's tag rea-agents-6.7.0 at 27dc42d10249f0410904a75c0997842eb03b931c; this report makes no published-6.7.0 runtime claim.

Runtime and platform

Node.js v26.12.1; Linux x86_64; tsx 4.23.15. In-process production application service with synthetic retained-capture Evidence; no external analysis provider.

Target and artifact context

A one-record HAR-shaped capture Evidence for GET https://example.test/api/blob; status 200, application/octet-stream, complete binary byte ff versus complete literal UTF-8 text /w==.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

The executed mismatching case returned this subset:

{"summary":{"steps":1,"equal":1,"agreement":1,"normalised":{"ids":0,"clocks":0,"generated":0,"large_bodies":0,"unobserved_bodies":0}},"result":{"path":"/api/blob","equal":true,"categories":[],"differences":[]}}

Source reachability: the public CLI calls this same service. The body reader preserves non-UTF-8 bytes as tagged base64, and replayedBody distinguishes base64 from text. The probe projection drops that distinction, and the environment answer passed to comparison contains only status and body. The final comparison checks those untagged bodies.

Typed outcome or recovery expected

Report a body value difference and equal: false for the binary-versus-text case; keep identical-binary and genuinely equal decoded-text controls equal. This is a proposed restoration check, not an executed fix.

Analysis or suspected cause

Source-derived: liveAnswer converts both base64 and ordinary text to a string, while compareStep discards the replay answer's body_encoding metadata. The final comparator consequently receives /w== on both sides despite different retained response bytes.

Additional context

Open and closed searches for environment-compare base64, plus searches for environment-compare body encoding, environment-compare binary, and comparison body encoding, found no matching report. Related #6432 concerns empty text versus null, not binary encoding; #6550 and #6557 concern identifier mapping. No source modification, fix, or restoration run was made.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The production comparison service reports full agreement for two complete response bodies with different bytes, producing a false fidelity result for this bounded supported input.
Workaround status: available
Workaround basis: The retained capture preserves the binary bytes and encoding, so the caller can compare that original Evidence directly instead of relying on the agreement score for this response.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6564 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed production `compareEnvironment` call reports `agreement: 1`, `equal: true`, and no differences when the environment's response contains the single byte `ff` and the live probe's response contains the four UTF-8 bytes `2f 77 3d 3d` (`/w==`). Both responses have status 200 and `application/octet-stream`; the binary response is retained as HAR base64 `/w==`, while the probe retains that spelling as literal text. Every normalization count is zero. The response bodies are complete, recorded, and below the replay limit. This is an application-boundary reproduction with authenticated fixture Evidence and the actual manifest/state projection, source loading, and comparison service. CLI/MCP transport, browser recording, and full HAR ingestion were not run. ### Reproduction Executed once against the source revision below using Node plus tsx 4.23.15. This is the substantive scratch probe, with the checkout location supplied as `REA_SOURCE_ROOT` instead of a private absolute path. It uses the repository's existing fixture constructors and production application modules; no target application executes. ```js import { createHash } from 'node:crypto'; const root = process.env.REA_SOURCE_ROOT; const { analysisViewCaptureEvidence, environmentManifestCaptureFixture } = await import(`${root}/tests/fixtures/analysisView.ts`); const { environmentStateServeFixture, environmentServeSource } = await import(`${root}/tests/fixtures/environmentServe.ts`); const { compareEnvironment } = await import(`${root}/src/application/environment/environmentCompare.ts`); function capture(encoding) { const reported = { request: { method: 'GET', url: 'https://example.test/api/blob' }, response: { status: 200, content: { mimeType: 'application/octet-stream', text: '/w==', ...(encoding ? { encoding } : {}) } } }; const value = environmentManifestCaptureFixture([reported]); if (encoding) value.records[0].binary_fields = [{ pointer: '/response/content/text', representation: 'har-base64-content', state: 'retained', content_base64: '/w==', bytes: 1, sha256: createHash('sha256').update(Buffer.from([255])).digest('hex') }]; return analysisViewCaptureEvidence(value); } const baseline = capture('base64'); const fixture = environmentStateServeFixture(baseline); const source = environmentServeSource(fixture); const result = compareEnvironment({ source, captureValue: baseline, probes: [{ label: 'probe', value: capture(undefined) }], operation: 'environment-compare' }); if (!result.ok) throw result.error; console.log(JSON.stringify(result.value)); ``` With the checkout's locked dependencies available, an equivalent invocation is `REA_SOURCE_ROOT=/tmp/rea-example npx tsx /tmp/rea-example/probe.mjs`. The local execution used a scratch loader to resolve checkout imports against an already installed dependency set; this command spelling was not separately executed. A binary-versus-identical-binary control agreed, and a `plain-a` versus `plain-b` text control correctly reported a value difference. ### Expected behavior Different retained response bytes should produce a body difference. Base64 is an evidence representation of bytes, not literal response text: the [documented replay contract](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/docs/agent-environments.md#L200-L210) retains `body_encoding` specifically to distinguish those cases. Identical binary responses should continue to agree, and a HAR base64 body that decodes to the same valid UTF-8 text should still compare by its decoded text. ### Impact and scope P2: fidelity reports can claim complete agreement for a changed binary API response. The observed case contains one request, no identifiers, no generated values, no clocks, no truncation, and no unobserved bodies. This changes the analysis result; it does not mutate the input Evidence. Real-world frequency and first affected revision were not established. Manual comparison of retained bytes and encoding metadata is an available workaround. ### REA version Current main source `6bb4f4bf7d64e5d2f47479d46225a5383b57cdad`. The `environment-compare` implementation is absent from verified npm release 6.7.0's tag `rea-agents-6.7.0` at `27dc42d10249f0410904a75c0997842eb03b931c`; this report makes no published-6.7.0 runtime claim. ### Runtime and platform Node.js v26.12.1; Linux x86_64; tsx 4.23.15. In-process production application service with synthetic retained-capture Evidence; no external analysis provider. ### Target and artifact context A one-record HAR-shaped capture Evidence for `GET https://example.test/api/blob`; status 200, `application/octet-stream`, complete binary byte `ff` versus complete literal UTF-8 text `/w==`. ### Interface Other ### Earliest failure stage Transport or result projection ### Relevant output or evidence The executed mismatching case returned this subset: ```json {"summary":{"steps":1,"equal":1,"agreement":1,"normalised":{"ids":0,"clocks":0,"generated":0,"large_bodies":0,"unobserved_bodies":0}},"result":{"path":"/api/blob","equal":true,"categories":[],"differences":[]}} ``` Source reachability: the [public CLI calls this same service](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/cli/environmentCommands.ts#L328-L343). The [body reader preserves non-UTF-8 bytes as tagged base64](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/domain/analysisView/recordedResponseBody.ts#L32-L72), and [replayedBody distinguishes base64 from text](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentSource.ts#L76-L100). The [probe projection drops that distinction](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompareProbe.ts#L83-L100), and the [environment answer passed to comparison contains only status and body](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompare.ts#L319-L332). The [final comparison checks those untagged bodies](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompareDiff.ts#L329-L354). ### Typed outcome or recovery expected Report a body value difference and `equal: false` for the binary-versus-text case; keep identical-binary and genuinely equal decoded-text controls equal. This is a proposed restoration check, not an executed fix. ### Analysis or suspected cause Source-derived: `liveAnswer` converts both base64 and ordinary text to a string, while `compareStep` discards the replay answer's `body_encoding` metadata. The final comparator consequently receives `/w==` on both sides despite different retained response bytes. ### Additional context Open and closed searches for `environment-compare base64`, plus searches for `environment-compare body encoding`, `environment-compare binary`, and `comparison body encoding`, found no matching report. Related #6432 concerns empty text versus `null`, not binary encoding; #6550 and #6557 concern identifier mapping. No source modification, fix, or restoration run was made. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The production comparison service reports full agreement for two complete response bodies with different bytes, producing a false fidelity result for this bounded supported input. - Workaround status: available - Workaround basis: The retained capture preserves the binary bytes and encoding, so the caller can compare that original Evidence directly instead of relying on the agreement score for this response.

bsakaguchi closed this issue

2026-10-31 00:45:33 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:01 +00:00

[PR #4020] [CLOSED] fix(javascript): preserve structural fingerprint semantics #5994