[GH-ISSUE #6557] [Bug] environment-compare loses keyed IDs when collections reuse a live ID #3946

New issue

Closed

opened 2026-10-31 00:45:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:30 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6557

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Live open/closed issue and PR searches found no report or correction for this keyed-object case. The original comparison feature and its historical collection-scoping fixes address array/root identities and request translation; this report concerns keyed-entry learning and keyed comparison that still use the probe-wide mapping. The recent reference-ID aliasing report concerns a different false-agreement case.

Area

CLI

Observed behavior

The public environment-compare command returns status: "complete" and exits 0 but reports an unchanged keyed entity as missing and extra when an earlier collection reused its live ID. A books response first maps live ID 101 to environment ID 1; the following things response must map live key/ID 101 to environment key/ID 102. Instead, the things step reports missing-field at /101 and extra-field at /102, making only one of two steps equal and agreement 0.5. Changing only the book's live ID to 103 restores agreement 1. Reversing the same two independent reads also restores agreement 1.

Reproduction

Run the entire block from the source checkout root with its pinned Node/npm toolchain, locked dependencies and compiled runtime available, and Python 3 on PATH. For this fresh checkout, npm ci --ignore-scripts and npm run build:cached completed successfully with the pinned toolchain. This block was executed as written. It creates ordinary local HAR 1.2 files, imports them with the supported inspect-web-network-capture producer, projects a manifest with the public view command, and compares the three cases. Every Evidence file is an unmodified public CLI output; there are no authenticated fixture helpers or direct Evidence constructors. All retained URLs use a synthetic origin and are not fetched. All HAR start times tie, so producer ordinals retain the listed request order. Temporary inputs and isolated home/config/data/cache directories are removed on exit.

python3 - <<'PY'
import json, os, subprocess, tempfile
from pathlib import Path

repo = Path.cwd()
def book(i):
    return {"id": i, "name": "Book", "slug": "book"}
def things(i):
    return {str(i): {"id": i, "name": "Thing"}}
def write_har(path, rows):
    entries = []
    for method, route, body in rows:
        text = json.dumps(body, separators=(",", ":"))
        entries.append({
            "startedDateTime": "2000-01-01T00:00:00.000Z", "time": 5,
            "request": {"method": method, "url": "http://app.example" + route,
                "httpVersion": "HTTP/1.1", "headers": [], "queryString": [],
                "cookies": [], "headersSize": -1, "bodySize": 0},
            "response": {"status": 200, "statusText": "OK", "httpVersion": "HTTP/1.1",
                "headers": [{"name": "Content-Type", "value": "application/json"}],
                "cookies": [], "content": {"size": len(text),
                    "mimeType": "application/json", "text": text},
                "redirectURL": "", "headersSize": -1, "bodySize": len(text)},
            "cache": {}, "timings": {"send": 0, "wait": 5, "receive": 0}})
    path.write_text(json.dumps({"log": {"version": "1.2",
        "creator": {"name": "comparison-repro", "version": "1"}, "entries": entries}}))

with tempfile.TemporaryDirectory(prefix="rea-keyed-") as directory:
    root = Path(directory)
    env = {"PATH": os.environ["PATH"]}
    for key in ("HOME", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "TMPDIR"):
        folder = root / key.lower()
        folder.mkdir()
        env[key] = str(folder)
    def cli(arguments, filename):
        result = subprocess.run(["node", "scripts/rea.mjs", *arguments, "--format", "json"],
            cwd=repo, env=env, text=True, capture_output=True, check=True)
        value = json.loads(result.stdout)
        (root / filename).write_text(result.stdout)
        return value
    write_har(root / "capture.har", [
        ("GET", "/api/books", [book(1)]),
        ("GET", "/api/books/1", book(1)),
        ("GET", "/api/things", things(102))])
    capture = cli(["inspect-web-network-capture", str(root / "capture.har"), "har"], "capture.json")
    (root / "view.json").write_text(json.dumps({
        "source": {"kind": "inline", "evidence": capture},
        "view": {"kind": "environment-manifest"}}))
    cli(["inspect-analysis-view", str(root / "view.json")], "manifest.json")
    for name, book_id, reverse in (
        ("collision", 101, False), ("no-collision", 103, False), ("reversed", 101, True)):
        rows = [("GET", "/api/books", [book(book_id)]),
                ("GET", "/api/things", things(101))]
        if reverse:
            rows.reverse()
        write_har(root / "probe.har", rows)
        cli(["inspect-web-network-capture", str(root / "probe.har"), "har"], "probe.json")
        report = cli(["environment-compare", str(root / "manifest.json"),
            "--evidence", str(root / "capture.json"), "--probe", str(root / "probe.json")], "report.json")
        assert report["status"] == "complete" and report["summary"]["steps"] == 2
        probe = report["probes"][0]
        print(json.dumps({"case": name, "state": report["environment"]["state"],
            "equal": report["summary"]["equal"], "agreement": report["summary"]["agreement"],
            "ids_mapped": probe["ids_mapped"], "id_conflicts": probe["id_conflicts"],
            "steps": [{"path": step["path"], "equal": step["equal"],
                "statuses": [step["live_status"], step["env_status"]],
                "differences": [{"category": d["category"], "pointer": d["pointer"]}
                    for d in step["differences"]]} for step in probe["results"]]}))
PY

Expected behavior

Both steps should compare equal in all three cases, with agreement 1 and no missing/extra keyed members. The two responses' things values share the unique natural key name: "Thing", and the object's numeric keys equal the entries' own IDs. The comparison contract maps keys when their values pair and supports collection-local identities when different collections reuse a live ID. The earlier books mapping must not prevent this things object from pairing with its environment counterpart.

Impact and scope

This causes a false entity discrepancy and incorrect aggregate agreement for the demonstrated ordinary JSON responses. The outcome changes with an unrelated collection's ID or the order of two independent reads. The inline block and a retained-input rerun on freshly fetched main produced the same three outcomes. No Evidence was lost or changed. NOT_ENOUGH_INFO: prevalence in real application captures, later path/query/body requests, writes, exported environments and other platforms were not measured. Reversed order works for these synthetic independent reads; a general workaround preserving arbitrary application sequences was not verified.

REA version

Freshly fetched main b58aa63796fe6465d241db9cb3c72c918c02890c, declaring rea-agents 6.7.0; the source launcher prints 6.7.0. The latest published npm package and release tag are also 6.7.0 / rea-agents-6.7.0, but that packaged runtime lacks this workflow: the same public HAR pipeline accepts the capture and then rejects view.kind: "environment-manifest" at inspect-analysis-view with invalid_request. The packaged command catalog also lacks environment-compare. This bug was reproduced on main, not on the published release.

Runtime and platform

Node.js v24.22.1, npm 11.16.0; macOS 27.0.1 arm64 (Darwin 27.0.0). No browser or optional live analysis provider was used.

Target and artifact context

Synthetic HAR 1.2 records of successful JSON GET responses, their producer-generated capture Evidence, a publicly projected environment manifest, and producer-generated probe Evidence. The environment reports served state; both steps have HTTP 200 on both sides. No real application or private target data is involved.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Every producer, projection and comparison command in the inline block exited 0. The block printed:

{"case": "collision", "state": "served", "equal": 1, "agreement": 0.5, "ids_mapped": 1, "id_conflicts": 2, "steps": [{"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/things", "equal": false, "statuses": [200, 200], "differences": [{"category": "missing-field", "pointer": "/101"}, {"category": "extra-field", "pointer": "/102"}]}]}
{"case": "no-collision", "state": "served", "equal": 2, "agreement": 1, "ids_mapped": 2, "id_conflicts": 0, "steps": [{"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/things", "equal": true, "statuses": [200, 200], "differences": []}]}
{"case": "reversed", "state": "served", "equal": 2, "agreement": 1, "ids_mapped": 1, "id_conflicts": 1, "steps": [{"path": "/api/things", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}]}

Expected source: The baseline comparison requirements require mapping keys when object values pair, collection-local mapping of an answer's own identity, and equality after mapping. The comparison guide explains reused IDs across collections. Existing keyed-learning tests and keyed-comparison tests establish that this is a supported response shape.
Failure source: The executed public-producer/public-comparison chain above reports the two incorrect pointers for the collision and none for either control. Source-derived explanation: keyed-entry learning discards the collection context for both the key and its entry, while key lookup during comparison uses the global mapping.
Evidence provenance: observed; source reasoning explains the observed result separately.
Local verification: reproduced through the public launcher, using real HAR inspection and public manifest projection rather than fixture-authenticated Evidence.
Reproduction completeness: complete.

Typed outcome or recovery expected

Return a complete report with both steps equal, agreement 1, and no missing-field or extra-field for these paired things members. No alternative refusal or unsupported-input outcome is required by the documented mapping contract.

Analysis or suspected cause

Source-derived mechanism: the first book retains the probe-wide 101-to-1 pair. Learning the things key and then its inner ID attempts 101-to-102 twice without a collection scope, generating the two recorded conflicts and retaining 101-to-1. Key comparison consults that global pair; because the things object has no key 1, lookup falls back to the unchanged live key 101, producing the observed /101 and /102 differences. Reversing the reads retains the global things pair and the scoped books pair, so both answers compare equal despite one global conflict. The reported map size counts global pairs; these counter observations are context for the equality failure, not a separate allegation that every collection pair must increase that count. A fix is not prescribed here.

Additional context

Restoration check: Run the same inline reproduction. All three cases must retain successful admission, served state, HTTP 200 on both sides and two compared steps, with equal: 2, agreement 1 and no differences. Keep the no-collision case as a control for ordinary keyed mapping and the reversed case as a control for collection-order independence. The closure signal is restored keyed comparison; no new counter policy is assumed.

NOT_ENOUGH_INFO: no last-known-good revision is known. Published-release behavior is limited to the unsupported-workflow result above. No downstream behavior is claimed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A bounded successful offline comparison returns a false entity discrepancy and incorrect aggregate agreement for ordinary admitted keyed JSON; capture data remains intact and other input shapes were not shown to fail.
Workaround status: unverified
Workaround basis: Reordering the demonstrated independent reads restores equality, but no general user procedure preserving arbitrary real application sequences was verified, and changing recorded IDs would change the input evidence.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6557 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. Live open/closed issue and PR searches found no report or correction for this keyed-object case. The original comparison feature and its historical collection-scoping fixes address array/root identities and request translation; this report concerns keyed-entry learning and keyed comparison that still use the probe-wide mapping. The recent reference-ID aliasing report concerns a different false-agreement case. ### Area CLI ### Observed behavior The public `environment-compare` command returns `status: "complete"` and exits 0 but reports an unchanged keyed entity as missing and extra when an earlier collection reused its live ID. A books response first maps live ID 101 to environment ID 1; the following things response must map live key/ID 101 to environment key/ID 102. Instead, the things step reports `missing-field` at `/101` and `extra-field` at `/102`, making only one of two steps equal and agreement 0.5. Changing only the book's live ID to 103 restores agreement 1. Reversing the same two independent reads also restores agreement 1. ### Reproduction Run the entire block from the source checkout root with its pinned Node/npm toolchain, locked dependencies and compiled runtime available, and Python 3 on PATH. For this fresh checkout, `npm ci --ignore-scripts` and `npm run build:cached` completed successfully with the pinned toolchain. This block was executed as written. It creates ordinary local HAR 1.2 files, imports them with the supported `inspect-web-network-capture` producer, projects a manifest with the public view command, and compares the three cases. Every Evidence file is an unmodified public CLI output; there are no authenticated fixture helpers or direct Evidence constructors. All retained URLs use a synthetic origin and are not fetched. All HAR start times tie, so producer ordinals retain the listed request order. Temporary inputs and isolated home/config/data/cache directories are removed on exit. ```sh python3 - <<'PY' import json, os, subprocess, tempfile from pathlib import Path repo = Path.cwd() def book(i): return {"id": i, "name": "Book", "slug": "book"} def things(i): return {str(i): {"id": i, "name": "Thing"}} def write_har(path, rows): entries = [] for method, route, body in rows: text = json.dumps(body, separators=(",", ":")) entries.append({ "startedDateTime": "2000-01-01T00:00:00.000Z", "time": 5, "request": {"method": method, "url": "http://app.example" + route, "httpVersion": "HTTP/1.1", "headers": [], "queryString": [], "cookies": [], "headersSize": -1, "bodySize": 0}, "response": {"status": 200, "statusText": "OK", "httpVersion": "HTTP/1.1", "headers": [{"name": "Content-Type", "value": "application/json"}], "cookies": [], "content": {"size": len(text), "mimeType": "application/json", "text": text}, "redirectURL": "", "headersSize": -1, "bodySize": len(text)}, "cache": {}, "timings": {"send": 0, "wait": 5, "receive": 0}}) path.write_text(json.dumps({"log": {"version": "1.2", "creator": {"name": "comparison-repro", "version": "1"}, "entries": entries}})) with tempfile.TemporaryDirectory(prefix="rea-keyed-") as directory: root = Path(directory) env = {"PATH": os.environ["PATH"]} for key in ("HOME", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "TMPDIR"): folder = root / key.lower() folder.mkdir() env[key] = str(folder) def cli(arguments, filename): result = subprocess.run(["node", "scripts/rea.mjs", *arguments, "--format", "json"], cwd=repo, env=env, text=True, capture_output=True, check=True) value = json.loads(result.stdout) (root / filename).write_text(result.stdout) return value write_har(root / "capture.har", [ ("GET", "/api/books", [book(1)]), ("GET", "/api/books/1", book(1)), ("GET", "/api/things", things(102))]) capture = cli(["inspect-web-network-capture", str(root / "capture.har"), "har"], "capture.json") (root / "view.json").write_text(json.dumps({ "source": {"kind": "inline", "evidence": capture}, "view": {"kind": "environment-manifest"}})) cli(["inspect-analysis-view", str(root / "view.json")], "manifest.json") for name, book_id, reverse in ( ("collision", 101, False), ("no-collision", 103, False), ("reversed", 101, True)): rows = [("GET", "/api/books", [book(book_id)]), ("GET", "/api/things", things(101))] if reverse: rows.reverse() write_har(root / "probe.har", rows) cli(["inspect-web-network-capture", str(root / "probe.har"), "har"], "probe.json") report = cli(["environment-compare", str(root / "manifest.json"), "--evidence", str(root / "capture.json"), "--probe", str(root / "probe.json")], "report.json") assert report["status"] == "complete" and report["summary"]["steps"] == 2 probe = report["probes"][0] print(json.dumps({"case": name, "state": report["environment"]["state"], "equal": report["summary"]["equal"], "agreement": report["summary"]["agreement"], "ids_mapped": probe["ids_mapped"], "id_conflicts": probe["id_conflicts"], "steps": [{"path": step["path"], "equal": step["equal"], "statuses": [step["live_status"], step["env_status"]], "differences": [{"category": d["category"], "pointer": d["pointer"]} for d in step["differences"]]} for step in probe["results"]]})) PY ``` ### Expected behavior Both steps should compare equal in all three cases, with agreement 1 and no missing/extra keyed members. The two responses' things values share the unique natural key `name: "Thing"`, and the object's numeric keys equal the entries' own IDs. The comparison contract maps keys when their values pair and supports collection-local identities when different collections reuse a live ID. The earlier books mapping must not prevent this things object from pairing with its environment counterpart. ### Impact and scope This causes a false entity discrepancy and incorrect aggregate agreement for the demonstrated ordinary JSON responses. The outcome changes with an unrelated collection's ID or the order of two independent reads. The inline block and a retained-input rerun on freshly fetched main produced the same three outcomes. No Evidence was lost or changed. NOT_ENOUGH_INFO: prevalence in real application captures, later path/query/body requests, writes, exported environments and other platforms were not measured. Reversed order works for these synthetic independent reads; a general workaround preserving arbitrary application sequences was not verified. ### REA version Freshly fetched main `b58aa63796fe6465d241db9cb3c72c918c02890c`, declaring rea-agents 6.7.0; the source launcher prints `6.7.0`. The latest published npm package and release tag are also `6.7.0` / `rea-agents-6.7.0`, but that packaged runtime lacks this workflow: the same public HAR pipeline accepts the capture and then rejects `view.kind: "environment-manifest"` at `inspect-analysis-view` with `invalid_request`. The packaged command catalog also lacks `environment-compare`. This bug was reproduced on main, not on the published release. ### Runtime and platform Node.js v24.22.1, npm 11.16.0; macOS 27.0.1 arm64 (Darwin 27.0.0). No browser or optional live analysis provider was used. ### Target and artifact context Synthetic HAR 1.2 records of successful JSON GET responses, their producer-generated capture Evidence, a publicly projected environment manifest, and producer-generated probe Evidence. The environment reports served state; both steps have HTTP 200 on both sides. No real application or private target data is involved. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Every producer, projection and comparison command in the inline block exited 0. The block printed: ```jsonl {"case": "collision", "state": "served", "equal": 1, "agreement": 0.5, "ids_mapped": 1, "id_conflicts": 2, "steps": [{"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/things", "equal": false, "statuses": [200, 200], "differences": [{"category": "missing-field", "pointer": "/101"}, {"category": "extra-field", "pointer": "/102"}]}]} {"case": "no-collision", "state": "served", "equal": 2, "agreement": 1, "ids_mapped": 2, "id_conflicts": 0, "steps": [{"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/things", "equal": true, "statuses": [200, 200], "differences": []}]} {"case": "reversed", "state": "served", "equal": 2, "agreement": 1, "ids_mapped": 1, "id_conflicts": 1, "steps": [{"path": "/api/things", "equal": true, "statuses": [200, 200], "differences": []}, {"path": "/api/books", "equal": true, "statuses": [200, 200], "differences": []}]} ``` - Expected source: The baseline [comparison requirements](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/docs/specs/agent-environments/fidelity-compare/spec.md#requirements) require mapping keys when object values pair, collection-local mapping of an answer's own identity, and equality after mapping. The [comparison guide](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/docs/agent-environments.md#L748-L765) explains reused IDs across collections. Existing [keyed-learning tests](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/src/application/environment/environmentCompareIds.test.ts#L138-L161) and [keyed-comparison tests](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/src/application/environment/environmentCompareDiff.test.ts#L94-L107) establish that this is a supported response shape. - Failure source: The executed public-producer/public-comparison chain above reports the two incorrect pointers for the collision and none for either control. Source-derived explanation: [keyed-entry learning](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/src/application/environment/environmentCompareIds.ts#L242-L246) discards the collection context for both the key and its entry, while [key lookup during comparison](https://github.com/morluto/rea/blob/b58aa63796fe6465d241db9cb3c72c918c02890c/src/application/environment/environmentCompareDiff.ts#L258-L265) uses the global mapping. - Evidence provenance: observed; source reasoning explains the observed result separately. - Local verification: reproduced through the public launcher, using real HAR inspection and public manifest projection rather than fixture-authenticated Evidence. - Reproduction completeness: complete. ### Typed outcome or recovery expected Return a complete report with both steps equal, agreement 1, and no `missing-field` or `extra-field` for these paired things members. No alternative refusal or unsupported-input outcome is required by the documented mapping contract. ### Analysis or suspected cause Source-derived mechanism: the first book retains the probe-wide 101-to-1 pair. Learning the things key and then its inner ID attempts 101-to-102 twice without a collection scope, generating the two recorded conflicts and retaining 101-to-1. Key comparison consults that global pair; because the things object has no key 1, lookup falls back to the unchanged live key 101, producing the observed `/101` and `/102` differences. Reversing the reads retains the global things pair and the scoped books pair, so both answers compare equal despite one global conflict. The reported map size counts global pairs; these counter observations are context for the equality failure, not a separate allegation that every collection pair must increase that count. A fix is not prescribed here. ### Additional context Restoration check: Run the same inline reproduction. All three cases must retain successful admission, served state, HTTP 200 on both sides and two compared steps, with `equal: 2`, agreement 1 and no differences. Keep the no-collision case as a control for ordinary keyed mapping and the reversed case as a control for collection-order independence. The closure signal is restored keyed comparison; no new counter policy is assumed. NOT_ENOUGH_INFO: no last-known-good revision is known. Published-release behavior is limited to the unsupported-workflow result above. No downstream behavior is claimed. #### Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A bounded successful offline comparison returns a false entity discrepancy and incorrect aggregate agreement for ordinary admitted keyed JSON; capture data remains intact and other input shapes were not shown to fail. - Workaround status: unverified - Workaround basis: Reordering the demonstrated independent reads restores equality, but no general user procedure preserving arbitrary real application sequences was verified, and changing recorded IDs would change the input evidence.

bsakaguchi closed this issue

2026-10-31 00:45:30 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:11 +00:00

[PR #4486] [CLOSED] fix(cutter): admit bridge descriptors without blocking on pipes #6271

bsakaguchi referenced this issue

2026-10-31 01:13:12 +00:00

[PR #4499] [MERGED] fix(cutter): cancel discovery probes and queued work #6275