[GH-ISSUE #6565] [Bug] environment-compare hides changed counters after learning matching ID values #3955

New issue

Closed

opened 2026-10-31 00:45:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:34 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6565

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed production compareEnvironment call reports full agreement for changed numeric measurements after an earlier response learns an unrelated entity-ID mapping. The first request pairs live item {id:101,name:"A"} with environment item {id:1,name:"A"}. The second request returns live statistics {size:101,workers:101,price:101} versus environment statistics {size:1,workers:1,price:1}. The report marks both steps equal and counts all three changed measurements as normalized IDs.

A control omitting only the first item request reports the same statistics as three value differences at /size, /workers, and /price, with agreement 0. This is an application-boundary reproduction using the actual manifest/state projection, source loading, and comparison service. CLI/MCP transport and real browser capture were not run.

Reproduction

Executed once per case against the source revision below using Node plus tsx 4.23.15. This is the substantive scratch probe with the checkout location supplied as REA_SOURCE_ROOT instead of a private absolute path. It imports the repository's existing capture fixture constructors and production application modules; no target application executes.

const root = process.env.REA_SOURCE_ROOT;
const { timedCaptureEvidence } = await import(`${root}/tests/fixtures/analysisView.ts`);
const { environmentStateServeFixture, environmentServeSource } = await import(`${root}/tests/fixtures/environmentServe.ts`);
const { compareEnvironment } = await import(`${root}/src/application/environment/environmentCompare.ts`);
const baseline = timedCaptureEvidence([
  ['GET', '/api/items', [{id:1,name:'A'}]],
  ['GET', '/api/stats', {size:1,workers:1,price:1}],
]);
const fixture = environmentStateServeFixture(baseline);
const source = environmentServeSource(fixture);
for (const learnFirst of [true, false]) {
  const rows = [...(learnFirst ? [['GET','/api/items',[{id:101,name:'A'}]]] : []), ['GET','/api/stats',{size:101,workers:101,price:101}]];
  const result = compareEnvironment({ source, captureValue: baseline, probes: [{label:'probe',value:timedCaptureEvidence(rows)}], operation:'environment-compare' });
  if (!result.ok) throw result.error;
  console.log(JSON.stringify({learnFirst,report:result.value}));
}

With the checkout's locked dependencies available, an equivalent invocation is REA_SOURCE_ROOT=/tmp/rea-example npx tsx /tmp/rea-example/probe.mjs. The local execution used a scratch loader to resolve checkout imports against an already installed dependency set; this command spelling was not separately executed.

Expected behavior

Retain the legitimate item-ID normalization, but report changes to numeric counts, sizes, and prices as value differences. Their equality to an ID's numeric value does not make them identifiers. The comparison specification identifies which fields and held string values represent IDs, and the ID-learning boundary explicitly excludes numeric counters that equal IDs.

Impact and scope

P2: a fidelity report can hide genuine numeric changes depending on unrelated earlier reads in the same probe. The observed input has one valid item-ID pair and three ordinary numeric measurements; it uses no conflicting or reused IDs, generated strings, clocks, missing bodies, or truncated bodies. Input Evidence is preserved. Population frequency and first affected revision were not established. Comparing the retained statistics directly, or running a separate statistics-only probe as demonstrated by the control, is an available workaround for this case.

REA version

Current main source 6bb4f4bf7d64e5d2f47479d46225a5383b57cdad. The environment-compare implementation is absent from verified npm release 6.7.0's tag rea-agents-6.7.0 at 27dc42d10249f0410904a75c0997842eb03b931c; this report makes no published-6.7.0 runtime claim.

Runtime and platform

Node.js v26.12.1; Linux x86_64; tsx 4.23.15. In-process production application service with synthetic retained-capture Evidence; no external analysis provider.

Target and artifact context

Two retained HAR-shaped API exchanges on one origin: GET /api/items returns an item array, and GET /api/stats returns a statistics document. Every response is status 200 with complete JSON bodies.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

The executed two-request probe produced summary.steps: 2, summary.equal: 2, summary.agreement: 1, normalised.ids: 4, ids_mapped: 1, and id_conflicts: 0. The statistics step had equal: true and differences: []. One legitimate item identity and three changed measurements account for the four normalizations. The statistics-only control produced summary.equal: 0, summary.agreement: 0, and these differences:

[{"category":"value","pointer":"/size","live":"101","env":"1"},{"category":"value","pointer":"/workers","live":"101","env":"1"},{"category":"value","pointer":"/price","live":"101","env":"1"}]

The public CLI reaches the exercised service. The service learns mappings before comparing each answer, and the scalar comparison applies a learned mapping to every differing string/number without checking whether this field represents an ID.

Typed outcome or recovery expected

Keep /api/items equal after its legitimate ID mapping, mark /api/stats unequal with the three numeric value differences, and report one equal step out of two. Preserve the statistics-only control. This is a proposed restoration check; no fix was executed.

Analysis or suspected cause

Source-derived: ID learning distinguishes numeric counters from identity fields, but scalar comparison consults the probe-wide value map for all string and numeric fields. A previously learned 101 -> 1 mapping therefore suppresses an unrelated measurement change.

Additional context

Open and closed searches for environment-compare counter, plus searches for environment-compare price, environment-compare non-identifier, environment-compare scalar, and environment mapped size, found no matching report. #6550 concerns two reference-ID fields collapsing distinct identities; #6557 concerns keyed objects whose collections reuse a live ID. This case requires neither: one unambiguous legitimate mapping is enough to hide non-identifier measurements. No source modification, fix, or restoration run was made.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The production comparison service suppresses three observed numeric measurement changes and reports full agreement after an unrelated valid entity-ID mapping.
Workaround status: available
Workaround basis: The original statistics remain in retained Evidence, and the executed statistics-only control detects all three differences without learning the unrelated item-ID mapping first.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6565 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed production `compareEnvironment` call reports full agreement for changed numeric measurements after an earlier response learns an unrelated entity-ID mapping. The first request pairs live item `{id:101,name:"A"}` with environment item `{id:1,name:"A"}`. The second request returns live statistics `{size:101,workers:101,price:101}` versus environment statistics `{size:1,workers:1,price:1}`. The report marks both steps equal and counts all three changed measurements as normalized IDs. A control omitting only the first item request reports the same statistics as three value differences at `/size`, `/workers`, and `/price`, with agreement 0. This is an application-boundary reproduction using the actual manifest/state projection, source loading, and comparison service. CLI/MCP transport and real browser capture were not run. ### Reproduction Executed once per case against the source revision below using Node plus tsx 4.23.15. This is the substantive scratch probe with the checkout location supplied as `REA_SOURCE_ROOT` instead of a private absolute path. It imports the repository's existing capture fixture constructors and production application modules; no target application executes. ```js const root = process.env.REA_SOURCE_ROOT; const { timedCaptureEvidence } = await import(`${root}/tests/fixtures/analysisView.ts`); const { environmentStateServeFixture, environmentServeSource } = await import(`${root}/tests/fixtures/environmentServe.ts`); const { compareEnvironment } = await import(`${root}/src/application/environment/environmentCompare.ts`); const baseline = timedCaptureEvidence([ ['GET', '/api/items', [{id:1,name:'A'}]], ['GET', '/api/stats', {size:1,workers:1,price:1}], ]); const fixture = environmentStateServeFixture(baseline); const source = environmentServeSource(fixture); for (const learnFirst of [true, false]) { const rows = [...(learnFirst ? [['GET','/api/items',[{id:101,name:'A'}]]] : []), ['GET','/api/stats',{size:101,workers:101,price:101}]]; const result = compareEnvironment({ source, captureValue: baseline, probes: [{label:'probe',value:timedCaptureEvidence(rows)}], operation:'environment-compare' }); if (!result.ok) throw result.error; console.log(JSON.stringify({learnFirst,report:result.value})); } ``` With the checkout's locked dependencies available, an equivalent invocation is `REA_SOURCE_ROOT=/tmp/rea-example npx tsx /tmp/rea-example/probe.mjs`. The local execution used a scratch loader to resolve checkout imports against an already installed dependency set; this command spelling was not separately executed. ### Expected behavior Retain the legitimate item-ID normalization, but report changes to numeric counts, sizes, and prices as value differences. Their equality to an ID's numeric value does not make them identifiers. The [comparison specification identifies which fields and held string values represent IDs](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/docs/specs/agent-environments/fidelity-compare/spec.md#L38-L50), and the [ID-learning boundary explicitly excludes numeric counters that equal IDs](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompareIds.ts#L262-L276). ### Impact and scope P2: a fidelity report can hide genuine numeric changes depending on unrelated earlier reads in the same probe. The observed input has one valid item-ID pair and three ordinary numeric measurements; it uses no conflicting or reused IDs, generated strings, clocks, missing bodies, or truncated bodies. Input Evidence is preserved. Population frequency and first affected revision were not established. Comparing the retained statistics directly, or running a separate statistics-only probe as demonstrated by the control, is an available workaround for this case. ### REA version Current main source `6bb4f4bf7d64e5d2f47479d46225a5383b57cdad`. The `environment-compare` implementation is absent from verified npm release 6.7.0's tag `rea-agents-6.7.0` at `27dc42d10249f0410904a75c0997842eb03b931c`; this report makes no published-6.7.0 runtime claim. ### Runtime and platform Node.js v26.12.1; Linux x86_64; tsx 4.23.15. In-process production application service with synthetic retained-capture Evidence; no external analysis provider. ### Target and artifact context Two retained HAR-shaped API exchanges on one origin: `GET /api/items` returns an item array, and `GET /api/stats` returns a statistics document. Every response is status 200 with complete JSON bodies. ### Interface Other ### Earliest failure stage Transport or result projection ### Relevant output or evidence The executed two-request probe produced `summary.steps: 2`, `summary.equal: 2`, `summary.agreement: 1`, `normalised.ids: 4`, `ids_mapped: 1`, and `id_conflicts: 0`. The statistics step had `equal: true` and `differences: []`. One legitimate item identity and three changed measurements account for the four normalizations. The statistics-only control produced `summary.equal: 0`, `summary.agreement: 0`, and these differences: ```json [{"category":"value","pointer":"/size","live":"101","env":"1"},{"category":"value","pointer":"/workers","live":"101","env":"1"},{"category":"value","pointer":"/price","live":"101","env":"1"}] ``` The [public CLI reaches the exercised service](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/cli/environmentCommands.ts#L328-L343). The [service learns mappings before comparing each answer](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompare.ts#L319-L332), and the [scalar comparison applies a learned mapping to every differing string/number without checking whether this field represents an ID](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/application/environment/environmentCompareDiff.ts#L150-L178). ### Typed outcome or recovery expected Keep `/api/items` equal after its legitimate ID mapping, mark `/api/stats` unequal with the three numeric value differences, and report one equal step out of two. Preserve the statistics-only control. This is a proposed restoration check; no fix was executed. ### Analysis or suspected cause Source-derived: ID learning distinguishes numeric counters from identity fields, but scalar comparison consults the probe-wide value map for all string and numeric fields. A previously learned `101 -> 1` mapping therefore suppresses an unrelated measurement change. ### Additional context Open and closed searches for `environment-compare counter`, plus searches for `environment-compare price`, `environment-compare non-identifier`, `environment-compare scalar`, and `environment mapped size`, found no matching report. #6550 concerns two reference-ID fields collapsing distinct identities; #6557 concerns keyed objects whose collections reuse a live ID. This case requires neither: one unambiguous legitimate mapping is enough to hide non-identifier measurements. No source modification, fix, or restoration run was made. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The production comparison service suppresses three observed numeric measurement changes and reports full agreement after an unrelated valid entity-ID mapping. - Workaround status: available - Workaround basis: The original statistics remain in retained Evidence, and the executed statistics-only control detects all three differences without learning the unrelated item-ID mapping first.

bsakaguchi closed this issue

2026-10-31 00:45:34 +00:00

bsakaguchi referenced this issue

2026-10-31 01:10:59 +00:00

[PR #4004] [CLOSED] fix(managed): reject runtime locations on managed static evidence #5988