[GH-ISSUE #6837] [Bug] environment-compare reports full agreement for changed business strings that resemble generated tokens #4181

New issue

Closed

opened 2026-10-31 00:47:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:29 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6837

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Comparing a recorded JSON response {"sku":"product1"} against a probe response {"sku":"product2"} returns equal: true, agreement: 1, and no differences. It counts the SKU change as one normalized generated value, although the fixture contains a literal business field and no generation or ID-mapping evidence. Changing the probe to {"sku":"productX"} produces the expected /sku value difference.

Reproduction

From the source checkout with locked dependencies, execute the production comparison workflow with the repository's synthetic capture helpers:

node --import tsx --input-type=module <<'JS'
import { compareEnvironment } from './src/application/environment/environmentCompare.ts';
import { analysisViewCaptureEvidence, environmentManifestCaptureFixture } from './tests/fixtures/analysisView.ts';
import { environmentStateServeFixture, environmentServeSource } from './tests/fixtures/environmentServe.ts';
const capture = (sku) => analysisViewCaptureEvidence(environmentManifestCaptureFixture([{
  startedDateTime: '2026-10-17T18:00:00.000Z',
  request: { method: 'GET', url: 'http://app.example/api/config' },
  response: { status: 200, content: { mimeType: 'application/json', text: JSON.stringify({ sku }) } }
}]));
const recorded = capture('product1');
const fixture = environmentStateServeFixture(recorded);
for (const sku of ['product2', 'productX']) {
  const compared = compareEnvironment({
    source: environmentServeSource(fixture), captureValue: recorded,
    probes: [{ label: 'probe', value: capture(sku) }], operation: 'environment-compare'
  });
  if (!compared.ok) throw compared.error;
  console.log(sku, JSON.stringify({ summary: compared.value.summary, result: compared.value.probes[0].results[0] }));
}
JS

The two comparisons were executed once each using this workflow. The CLI itself was not launched.

Expected behavior

A changed literal SKU should produce a value difference unless there is evidence or an explicit choice identifying the field as independently generated. A token-shape heuristic alone cannot establish that an ordinary application value may be ignored. Preserve normalization for fields actually identified as generated.

Impact and scope

The observed response-body mismatch disappears from both step results and overall agreement, allowing an environment that returns the wrong SKU to score 100% agreement for the probe. The same predicate applies to other strings with equal lengths of at least eight characters, the same character classes, and a digit. This is a comparison-design flaw in the existing normalization rule; it is not a claim that execution diverges from the documented heuristic. No security impact is claimed.

REA version

Source revision 169512a35bb015ff209168fb8bc1c9b31a2c678b (main, package metadata 6.7.0).

Runtime and platform

Node.js v26.12.1; Linux x86_64.

Target and artifact context

Synthetic capture Evidence for one GET response with a literal sku field.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

For the product2 probe, the executed workflow returned:

{"summary":{"steps":1,"equal":1,"agreement":1,"normalised":{"ids":0,"clocks":0,"generated":1,"large_bodies":0,"unobserved_bodies":0}},"result":{"equal":true,"categories":[],"differences":[]}}

For the productX control it returned equal: false, category value, and a difference at /sku.

The unconditional string-shape predicate and scalar comparison branch are in environmentCompareDiff.ts. The comparison documentation describes this heuristic and cautions that normalized values can differ in content. The concrete failure is that it applies to literal business data without generation evidence, so the agreement result cannot distinguish this response error.

Typed outcome or recovery expected

A value difference at /sku and disagreement for this probe, or an explicit uncertainty that prevents the mismatch from silently becoming full agreement.

Additional context

No application was contacted and no state-changing requests were executed. Source-checkout workflow only; installed package behavior was not tested.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6837 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior Comparing a recorded JSON response `{"sku":"product1"}` against a probe response `{"sku":"product2"}` returns `equal: true`, `agreement: 1`, and no differences. It counts the SKU change as one normalized generated value, although the fixture contains a literal business field and no generation or ID-mapping evidence. Changing the probe to `{"sku":"productX"}` produces the expected `/sku` value difference. ### Reproduction From the source checkout with locked dependencies, execute the production comparison workflow with the repository's synthetic capture helpers: ```sh node --import tsx --input-type=module <<'JS' import { compareEnvironment } from './src/application/environment/environmentCompare.ts'; import { analysisViewCaptureEvidence, environmentManifestCaptureFixture } from './tests/fixtures/analysisView.ts'; import { environmentStateServeFixture, environmentServeSource } from './tests/fixtures/environmentServe.ts'; const capture = (sku) => analysisViewCaptureEvidence(environmentManifestCaptureFixture([{ startedDateTime: '2026-10-17T18:00:00.000Z', request: { method: 'GET', url: 'http://app.example/api/config' }, response: { status: 200, content: { mimeType: 'application/json', text: JSON.stringify({ sku }) } } }])); const recorded = capture('product1'); const fixture = environmentStateServeFixture(recorded); for (const sku of ['product2', 'productX']) { const compared = compareEnvironment({ source: environmentServeSource(fixture), captureValue: recorded, probes: [{ label: 'probe', value: capture(sku) }], operation: 'environment-compare' }); if (!compared.ok) throw compared.error; console.log(sku, JSON.stringify({ summary: compared.value.summary, result: compared.value.probes[0].results[0] })); } JS ``` The two comparisons were executed once each using this workflow. The CLI itself was not launched. ### Expected behavior A changed literal SKU should produce a value difference unless there is evidence or an explicit choice identifying the field as independently generated. A token-shape heuristic alone cannot establish that an ordinary application value may be ignored. Preserve normalization for fields actually identified as generated. ### Impact and scope The observed response-body mismatch disappears from both step results and overall agreement, allowing an environment that returns the wrong SKU to score 100% agreement for the probe. The same predicate applies to other strings with equal lengths of at least eight characters, the same character classes, and a digit. This is a comparison-design flaw in the existing normalization rule; it is not a claim that execution diverges from the documented heuristic. No security impact is claimed. ### REA version Source revision `169512a35bb015ff209168fb8bc1c9b31a2c678b` (`main`, package metadata 6.7.0). ### Runtime and platform Node.js v26.12.1; Linux x86_64. ### Target and artifact context Synthetic capture Evidence for one GET response with a literal `sku` field. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence For the `product2` probe, the executed workflow returned: ```json {"summary":{"steps":1,"equal":1,"agreement":1,"normalised":{"ids":0,"clocks":0,"generated":1,"large_bodies":0,"unobserved_bodies":0}},"result":{"equal":true,"categories":[],"differences":[]}} ``` For the `productX` control it returned `equal: false`, category `value`, and a difference at `/sku`. The unconditional string-shape predicate and scalar comparison branch are in [environmentCompareDiff.ts](https://github.com/morluto/rea/blob/169512a35bb015ff209168fb8bc1c9b31a2c678b/src/application/environment/environmentCompareDiff.ts#L105-L228). The [comparison documentation](https://github.com/morluto/rea/blob/169512a35bb015ff209168fb8bc1c9b31a2c678b/docs/agent-environments.md#L748-L813) describes this heuristic and cautions that normalized values can differ in content. The concrete failure is that it applies to literal business data without generation evidence, so the agreement result cannot distinguish this response error. ### Typed outcome or recovery expected A `value` difference at `/sku` and disagreement for this probe, or an explicit uncertainty that prevents the mismatch from silently becoming full agreement. ### Additional context No application was contacted and no state-changing requests were executed. Source-checkout workflow only; installed package behavior was not tested.

bsakaguchi closed this issue

2026-10-31 00:47:30 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:00 +00:00

[PR #4187] [CLOSED] fix(binary): confine app program directory to its bundle #6117

bsakaguchi referenced this issue

2026-10-31 01:12:00 +00:00

[PR #4193] [CLOSED] fix(target): reject app program file when Contents/MacOS leaves the bundle #6118