[GH-ISSUE #6550] [Bug] environment-compare reports agreement after collapsing distinct reference IDs #3940

New issue

Closed

opened 2026-10-31 00:45:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:26 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6550

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed call to the production compareEnvironment service reports agreement: 1, equal: true, and id_conflicts: 0 when the recorded response is {"owner_id":1,"reviewer_id":1} and the live response is {"owner_id":1,"reviewer_id":2}. These fields refer to users: the recorded response names the same user twice, while the live response names two different users. Reversing which live field contains ID 2 also reports full agreement. The identical-response control correctly agrees without ID normalization. This is a source-boundary observation; the CLI and MCP transports were not executed.

Reproduction

From a checkout at the revision below with its locked dependencies installed, save this as probe.mjs outside the checkout and run node --import ./node_modules/tsx/dist/loader.mjs /path/to/probe.mjs "$PWD". This is the service-level portion of the three-case probe that was executed once. It generates inert capture Evidence and performs no network calls or target execution.

import { pathToFileURL } from 'node:url';
import { join } from 'node:path';
const checkout = process.argv[2];
const mod = async p => import(pathToFileURL(join(checkout, p)).href);
const { compareEnvironment } = await mod('src/application/environment/environmentCompare.ts');
const { timedCaptureEvidence } = await mod('tests/fixtures/analysisView.ts');
const { environmentStateServeFixture, environmentServeSource } = await mod('tests/fixtures/environmentServe.ts');
const recorded = { owner_id: 1, reviewer_id: 1 };
for (const live of [
  { owner_id: 1, reviewer_id: 2 },
  { owner_id: 2, reviewer_id: 1 },
  { owner_id: 1, reviewer_id: 1 },
]) {
  const fixture = environmentStateServeFixture(timedCaptureEvidence([
    ['GET', '/api/review', recorded],
  ]));
  const result = compareEnvironment({
    source: environmentServeSource(fixture),
    captureValue: fixture.capture,
    probes: [{ label: 'id-identity', value: timedCaptureEvidence([
      ['GET', '/api/review', live],
    ]) }],
    operation: 'environment-compare',
  });
  if (!result.ok) throw result.error;
  console.log(JSON.stringify({
    live,
    agreement: result.value.summary.agreement,
    normalized: result.value.summary.normalised,
    conflicts: result.value.probes[0].id_conflicts,
    steps: result.value.probes[0].results,
  }));
}

Expected behavior

ID renaming should preserve whether two references name the same entity. A one-to-one re-seeding cannot turn two distinct user IDs into one user ID, so the first two cases must not report complete agreement with no conflict. The identical control should remain equal. The documented comparison contract maps identifier fields between re-seeded instances and reports whether normalized bodies agree; it does not authorize collapsing entity identity.

Impact and scope

The fidelity report hides a changed relationship and returns a false positive agreement score for the demonstrated JSON responses. The observed scope is unscoped reference fields such as owner_id and reviewer_id; the collection-scoped handling of an entity's own id follows a separate branch. Real-application frequency is unknown. Manually inspecting the retained original responses can reveal the discrepancy, but no automatic comparison workaround was verified.

REA version

Main source revision 6c4d880f2dd9ca7d42593be916f90ec19b8a09d7. This environment-comparison implementation is absent from the published rea-agents 6.7.0 package; this report concerns main.

Runtime and platform

Node.js v26.12.1; Linux x86_64; checkout TypeScript loader. The probe used a supported runtime engine, rather than the development Node version pinned in the checkout.

Target and artifact context

Generated capture Evidence for one successful GET /api/review exchange whose JSON response contains two user-reference fields. No private application, live endpoint, or external provider was used.

Interface

Other

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Both changed cases produced:

{
  "agreement": 1,
  "normalized": {
    "ids": 1,
    "clocks": 0,
    "generated": 0,
    "large_bodies": 0,
    "unobserved_bodies": 0
  },
  "conflicts": 0,
  "step": {
    "method": "GET",
    "path": "/api/review",
    "live_status": 200,
    "env_status": 200,
    "equal": true,
    "differences": []
  }
}

The identical control reported agreement 1, zero normalized IDs, and zero conflicts. An additional direct learnIds/compareAnswers call in the same probe returned an empty differences array for each changed case.

Analysis or suspected cause

Source-derived: IdMap.learn returns immediately for equal values without reserving their pairing. It can then learn 2 -> 1 despite the existing identity 1 -> 1; the reverse field order is also admitted because the equality return precedes the conflict checks. Foreign-reference fields are learned without the collection scope. The production comparison path learns these pairs before comparing the bodies.

Additional context

Proposed restoration check, not executed: the two changed cases should produce a disagreement or an explicit unresolved identity/conflict result, while the identical control and a valid one-to-one ID renaming should still agree. No fix or post-draft rerun was performed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed service probe returns a false positive fidelity result for a bounded pair of user-reference fields; broader frequency and downstream use are unknown.
Workaround status: unverified
Workaround basis: Original response inspection can reveal the mismatch, but no safe automatic comparison bypass was exercised.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6550 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed call to the production `compareEnvironment` service reports `agreement: 1`, `equal: true`, and `id_conflicts: 0` when the recorded response is `{"owner_id":1,"reviewer_id":1}` and the live response is `{"owner_id":1,"reviewer_id":2}`. These fields refer to users: the recorded response names the same user twice, while the live response names two different users. Reversing which live field contains ID 2 also reports full agreement. The identical-response control correctly agrees without ID normalization. This is a source-boundary observation; the CLI and MCP transports were not executed. ### Reproduction From a checkout at the revision below with its locked dependencies installed, save this as `probe.mjs` outside the checkout and run `node --import ./node_modules/tsx/dist/loader.mjs /path/to/probe.mjs "$PWD"`. This is the service-level portion of the three-case probe that was executed once. It generates inert capture Evidence and performs no network calls or target execution. ```js import { pathToFileURL } from 'node:url'; import { join } from 'node:path'; const checkout = process.argv[2]; const mod = async p => import(pathToFileURL(join(checkout, p)).href); const { compareEnvironment } = await mod('src/application/environment/environmentCompare.ts'); const { timedCaptureEvidence } = await mod('tests/fixtures/analysisView.ts'); const { environmentStateServeFixture, environmentServeSource } = await mod('tests/fixtures/environmentServe.ts'); const recorded = { owner_id: 1, reviewer_id: 1 }; for (const live of [ { owner_id: 1, reviewer_id: 2 }, { owner_id: 2, reviewer_id: 1 }, { owner_id: 1, reviewer_id: 1 }, ]) { const fixture = environmentStateServeFixture(timedCaptureEvidence([ ['GET', '/api/review', recorded], ])); const result = compareEnvironment({ source: environmentServeSource(fixture), captureValue: fixture.capture, probes: [{ label: 'id-identity', value: timedCaptureEvidence([ ['GET', '/api/review', live], ]) }], operation: 'environment-compare', }); if (!result.ok) throw result.error; console.log(JSON.stringify({ live, agreement: result.value.summary.agreement, normalized: result.value.summary.normalised, conflicts: result.value.probes[0].id_conflicts, steps: result.value.probes[0].results, })); } ``` ### Expected behavior ID renaming should preserve whether two references name the same entity. A one-to-one re-seeding cannot turn two distinct user IDs into one user ID, so the first two cases must not report complete agreement with no conflict. The identical control should remain equal. The [documented comparison contract](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/docs/agent-environments.md#measure-fidelity-against-the-application) maps identifier fields between re-seeded instances and reports whether normalized bodies agree; it does not authorize collapsing entity identity. ### Impact and scope The fidelity report hides a changed relationship and returns a false positive agreement score for the demonstrated JSON responses. The observed scope is unscoped reference fields such as `owner_id` and `reviewer_id`; the collection-scoped handling of an entity's own `id` follows a separate branch. Real-application frequency is unknown. Manually inspecting the retained original responses can reveal the discrepancy, but no automatic comparison workaround was verified. ### REA version Main source revision `6c4d880f2dd9ca7d42593be916f90ec19b8a09d7`. This environment-comparison implementation is absent from the published `rea-agents` 6.7.0 package; this report concerns main. ### Runtime and platform Node.js v26.12.1; Linux x86_64; checkout TypeScript loader. The probe used a supported runtime engine, rather than the development Node version pinned in the checkout. ### Target and artifact context Generated capture Evidence for one successful `GET /api/review` exchange whose JSON response contains two user-reference fields. No private application, live endpoint, or external provider was used. ### Interface Other ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence Both changed cases produced: ```json { "agreement": 1, "normalized": { "ids": 1, "clocks": 0, "generated": 0, "large_bodies": 0, "unobserved_bodies": 0 }, "conflicts": 0, "step": { "method": "GET", "path": "/api/review", "live_status": 200, "env_status": 200, "equal": true, "differences": [] } } ``` The identical control reported agreement 1, zero normalized IDs, and zero conflicts. An additional direct `learnIds`/`compareAnswers` call in the same probe returned an empty differences array for each changed case. ### Analysis or suspected cause Source-derived: [IdMap.learn](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/application/environment/environmentCompareIds.ts#L31-L52) returns immediately for equal values without reserving their pairing. It can then learn `2 -> 1` despite the existing identity `1 -> 1`; the reverse field order is also admitted because the equality return precedes the conflict checks. Foreign-reference fields are learned without the collection scope. The [production comparison path](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/application/environment/environmentCompare.ts#L324-L336) learns these pairs before comparing the bodies. ### Additional context Proposed restoration check, not executed: the two changed cases should produce a disagreement or an explicit unresolved identity/conflict result, while the identical control and a valid one-to-one ID renaming should still agree. No fix or post-draft rerun was performed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed service probe returns a false positive fidelity result for a bounded pair of user-reference fields; broader frequency and downstream use are unknown. - Workaround status: unverified - Workaround basis: Original response inspection can reveal the mismatch, but no safe automatic comparison bypass was exercised.

bsakaguchi closed this issue

2026-10-31 00:45:26 +00:00

bsakaguchi referenced this issue

2026-10-31 01:10:49 +00:00

[PR #3940] [CLOSED] fix(javascript): preserve proven WebSocket endpoint identities #5968

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425