morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #6432] [Bug] environment-compare counts matching empty 404 responses as a body difference #3844

New issue

Closed

opened 2026-10-31 00:44:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:34 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6432

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

A served-state comparison of a missing-item GET reports a root value difference even though both recorded responses have status 404 and empty text. The command completes but counts only one of two probe steps equal, lowering agreement to 0.5. This was executed through the public CLI with synthetic capture Evidence.

Reproduction

From the source checkout identified below, use the pinned Node toolchain, installed locked dependencies, and the existing compiled CLI. A fresh checkout needs its normal dependency installation and runtime build first; those setup operations were not rerun for this verification. Python 3 must be on PATH. Run this entire block from the checkout root. It imports standard upstream fixtures, generates all three Evidence inputs, invokes the public CLI for each case, prints reduced results, and removes its temporary test and data. It refuses to overwrite an existing test.

python3 - <<'PYTHON'
import os
import subprocess
import tempfile
from pathlib import Path

test = Path("tests/boundary/cli/environmentCompareEmptyAbsence.verifier.test.ts")
if test.exists():
    raise SystemExit("Refusing to overwrite an existing test")
source = r'''
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect } from "vitest";
import { timedCaptureEvidence } from "../../fixtures/analysisView.js";
import { environmentStateServeFixture } from "../../fixtures/environmentServe.js";
import { FIRST_BOOK } from "../../fixtures/environmentStateWrites.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { cliTest } from "../../support/cli/cliFixture.js";

cliTest("compares recorded absence body representations through the CLI", async ({ cli }) => {
  const root = await createTestTempDirectory("rea-compare-absence-");
  const cases = [
    { name: "empty", recorded: "", live: "" },
    { name: "same-json", recorded: { detail: "Not found." }, live: { detail: "Not found." } },
    { name: "changed-json", recorded: { detail: "Not found." }, live: { detail: "Missing." } },
    { name: "json-empty-vs-no-body", recorded: '\"\"', live: "" },
  ];
  for (const entry of cases) {
    const fixture = environmentStateServeFixture(timedCaptureEvidence([
      ["GET", "/api/books", [FIRST_BOOK]],
      ["GET", "/api/books/1", FIRST_BOOK],
      ["GET", "/api/books/9", entry.recorded, 404],
    ]));
    const probe = timedCaptureEvidence([
      ["GET", "/api/books", [{ ...FIRST_BOOK, id: 101 }]],
      ["GET", "/api/books/999", entry.live, 404],
    ]);
    for (const [name, value] of Object.entries({ manifest: fixture.manifest, capture: fixture.capture, probe })) {
      await writeFile(join(root, `${name}.json`), JSON.stringify(value));
    }
    const result = await cli.run({ arguments: [
      "environment-compare", join(root, "manifest.json"), "--evidence", join(root, "capture.json"),
      "--probe", join(root, "probe.json"), "--json",
    ], environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root, XDG_DATA_HOME: root }, timeoutMs: 30000 });
    expect(result.exitCode).toBe(0);
    const report = result.json as any;
    const step = report.probes[0].results[1];
    console.log(JSON.stringify({ name: entry.name, state: report.environment.state, steps: report.summary.steps, equal: report.summary.equal, agreement: report.summary.agreement, live_status: step.live_status, env_status: step.env_status, differences: step.differences }));
    expect(report).toMatchObject({ status: "complete", environment: { state: "served" }, summary: { steps: 2 } });
  }
});
'''

try:
    test.write_text(source)
    with tempfile.TemporaryDirectory(prefix="compare-absence-") as directory:
        root = Path(directory)
        for name in ("home", "config", "cache", "data", "tmp"):
            (root / name).mkdir()
        environment = {
            "PATH": os.environ["PATH"],
            "HOME": str(root / "home"),
            "XDG_CONFIG_HOME": str(root / "config"),
            "XDG_CACHE_HOME": str(root / "cache"),
            "XDG_DATA_HOME": str(root / "data"),
            "TMPDIR": str(root / "tmp"),
        }
        subprocess.run([
            "node", "node_modules/vitest/vitest.mjs", "run",
            "--project", "cli-boundary", str(test), "--maxWorkers=1",
        ], env=environment, check=True)
finally:
    test.unlink(missing_ok=True)
PYTHON

Expected behavior

Two recorded empty-text 404 responses should compare equal, so both probe steps should agree and overall agreement should be 1. A recorded JSON empty string must remain different from no body, and genuine JSON error-body changes must remain visible.

Impact and scope

The verified input produces a false disagreement and incorrect aggregate agreement. Matching JSON error bodies compare correctly, and genuine JSON body differences remain visible. The verified scope is an absent-item GET with a recorded empty-text 404; write variants, other platforms, and downstream effects were not tested. No security boundary crossing was identified. Raw Evidence inspection can identify the false entry, but a workaround preserving accurate aggregate metrics was not tested.

REA version

Source main at 9eb829aca3, declaring rea-agents 6.7.0. The latest release tag rea-agents-6.7.0 lacks this workflow in its source tree; published-release runtime was not tested.

Runtime and platform

Node.js v24.18.0; npm 11.16.0; Vitest 5.0.3; macOS 27.0.1 arm64; existing compiled source runtime.

Target and artifact context

Synthetic capture Evidence and its projected environment manifest, plus a probe capture. No binary target, provider, live application, or network service is needed.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

All four CLI invocations exited 0 and returned status: "complete", with served state and two compared steps. The final reproduction printed:

{"name":"empty","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"","live":"null","env":"\"\""}]}
{"name":"same-json","state":"served","steps":2,"equal":2,"agreement":1,"live_status":404,"env_status":404,"differences":[]}
{"name":"changed-json","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"/detail","live":"\"Missing.\"","env":"\"Not found.\""}]}
{"name":"json-empty-vs-no-body","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"","live":"null","env":"\"\""}]}

Contract: docs/specs/agent-environments/fidelity-compare/spec.md:65 requires normalization of empty response text to no body while preserving JSON empty strings, and :75 defines agreement by equal statuses and normalized bodies. These requirements apply to the served-state comparison used here, without an exception for recorded absences.

Typed outcome or recovery expected

Return an equal absence step and agreement 1 for the empty-text case, while retaining actual body differences in the controls.

Analysis or suspected cause

Source-derived mechanism: src/application/environment/environmentStateAnswer.ts:285-287 replays a recorded absence with source: "state", view: "missing"; src/application/environment/environmentCompare.ts:159-168 excludes all such state answers from empty-text normalization, whereas src/application/environment/environmentCompareProbe.ts:93-98 normalizes the live response. The public command is registered and executes source validation/comparison at src/cli/environmentCommands.ts:311-377.

Additional context

The final self-contained block ran once after an earlier independent run produced the same four outcomes. The repository's existing ordinary-empty-response source test passed separately; it does not cover this missing-item path. No implementation was changed; no published-release runtime was tested.

Restoration check: With the same fixtures, the empty-text case must report steps: 2, equal: 2, agreement 1, and no root value difference. Keep full agreement for matching JSON 404 objects, the /detail difference for changed JSON objects, and the root difference for a JSON empty string versus no body. These signals restore the existing normalization contract without hiding actual response differences.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6432 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area CLI ## Observed behavior A served-state comparison of a missing-item GET reports a root `value` difference even though both recorded responses have status 404 and empty text. The command completes but counts only one of two probe steps equal, lowering agreement to 0.5. This was executed through the public CLI with synthetic capture Evidence. ## Reproduction From the source checkout identified below, use the pinned Node toolchain, installed locked dependencies, and the existing compiled CLI. A fresh checkout needs its normal dependency installation and runtime build first; those setup operations were not rerun for this verification. Python 3 must be on PATH. Run this entire block from the checkout root. It imports standard upstream fixtures, generates all three Evidence inputs, invokes the public CLI for each case, prints reduced results, and removes its temporary test and data. It refuses to overwrite an existing test. ```sh python3 - <<'PYTHON' import os import subprocess import tempfile from pathlib import Path test = Path("tests/boundary/cli/environmentCompareEmptyAbsence.verifier.test.ts") if test.exists(): raise SystemExit("Refusing to overwrite an existing test") source = r''' import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect } from "vitest"; import { timedCaptureEvidence } from "../../fixtures/analysisView.js"; import { environmentStateServeFixture } from "../../fixtures/environmentServe.js"; import { FIRST_BOOK } from "../../fixtures/environmentStateWrites.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; cliTest("compares recorded absence body representations through the CLI", async ({ cli }) => { const root = await createTestTempDirectory("rea-compare-absence-"); const cases = [ { name: "empty", recorded: "", live: "" }, { name: "same-json", recorded: { detail: "Not found." }, live: { detail: "Not found." } }, { name: "changed-json", recorded: { detail: "Not found." }, live: { detail: "Missing." } }, { name: "json-empty-vs-no-body", recorded: '\"\"', live: "" }, ]; for (const entry of cases) { const fixture = environmentStateServeFixture(timedCaptureEvidence([ ["GET", "/api/books", [FIRST_BOOK]], ["GET", "/api/books/1", FIRST_BOOK], ["GET", "/api/books/9", entry.recorded, 404], ])); const probe = timedCaptureEvidence([ ["GET", "/api/books", [{ ...FIRST_BOOK, id: 101 }]], ["GET", "/api/books/999", entry.live, 404], ]); for (const [name, value] of Object.entries({ manifest: fixture.manifest, capture: fixture.capture, probe })) { await writeFile(join(root, `${name}.json`), JSON.stringify(value)); } const result = await cli.run({ arguments: [ "environment-compare", join(root, "manifest.json"), "--evidence", join(root, "capture.json"), "--probe", join(root, "probe.json"), "--json", ], environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root, XDG_DATA_HOME: root }, timeoutMs: 30000 }); expect(result.exitCode).toBe(0); const report = result.json as any; const step = report.probes[0].results[1]; console.log(JSON.stringify({ name: entry.name, state: report.environment.state, steps: report.summary.steps, equal: report.summary.equal, agreement: report.summary.agreement, live_status: step.live_status, env_status: step.env_status, differences: step.differences })); expect(report).toMatchObject({ status: "complete", environment: { state: "served" }, summary: { steps: 2 } }); } }); ''' try: test.write_text(source) with tempfile.TemporaryDirectory(prefix="compare-absence-") as directory: root = Path(directory) for name in ("home", "config", "cache", "data", "tmp"): (root / name).mkdir() environment = { "PATH": os.environ["PATH"], "HOME": str(root / "home"), "XDG_CONFIG_HOME": str(root / "config"), "XDG_CACHE_HOME": str(root / "cache"), "XDG_DATA_HOME": str(root / "data"), "TMPDIR": str(root / "tmp"), } subprocess.run([ "node", "node_modules/vitest/vitest.mjs", "run", "--project", "cli-boundary", str(test), "--maxWorkers=1", ], env=environment, check=True) finally: test.unlink(missing_ok=True) PYTHON ``` ## Expected behavior Two recorded empty-text 404 responses should compare equal, so both probe steps should agree and overall agreement should be 1. A recorded JSON empty string must remain different from no body, and genuine JSON error-body changes must remain visible. ## Impact and scope The verified input produces a false disagreement and incorrect aggregate agreement. Matching JSON error bodies compare correctly, and genuine JSON body differences remain visible. The verified scope is an absent-item GET with a recorded empty-text 404; write variants, other platforms, and downstream effects were not tested. No security boundary crossing was identified. Raw Evidence inspection can identify the false entry, but a workaround preserving accurate aggregate metrics was not tested. ## REA version Source `main` at 9eb829aca34be36ff112826337996f6139f82eff, declaring rea-agents 6.7.0. The latest release tag `rea-agents-6.7.0` lacks this workflow in its source tree; published-release runtime was not tested. ## Runtime and platform Node.js v24.18.0; npm 11.16.0; Vitest 5.0.3; macOS 27.0.1 arm64; existing compiled source runtime. ## Target and artifact context Synthetic capture Evidence and its projected environment manifest, plus a probe capture. No binary target, provider, live application, or network service is needed. ## Interface CLI ## Earliest failure stage Transport or result projection ## Relevant output or evidence All four CLI invocations exited 0 and returned `status: "complete"`, with served state and two compared steps. The final reproduction printed: ```json {"name":"empty","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"","live":"null","env":"\"\""}]} {"name":"same-json","state":"served","steps":2,"equal":2,"agreement":1,"live_status":404,"env_status":404,"differences":[]} {"name":"changed-json","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"/detail","live":"\"Missing.\"","env":"\"Not found.\""}]} {"name":"json-empty-vs-no-body","state":"served","steps":2,"equal":1,"agreement":0.5,"live_status":404,"env_status":404,"differences":[{"category":"value","pointer":"","live":"null","env":"\"\""}]} ``` Contract: `docs/specs/agent-environments/fidelity-compare/spec.md:65` requires normalization of empty response text to no body while preserving JSON empty strings, and `:75` defines agreement by equal statuses and normalized bodies. These requirements apply to the served-state comparison used here, without an exception for recorded absences. ## Typed outcome or recovery expected Return an equal absence step and agreement 1 for the empty-text case, while retaining actual body differences in the controls. ## Analysis or suspected cause Source-derived mechanism: `src/application/environment/environmentStateAnswer.ts:285-287` replays a recorded absence with `source: "state", view: "missing"`; `src/application/environment/environmentCompare.ts:159-168` excludes all such state answers from empty-text normalization, whereas `src/application/environment/environmentCompareProbe.ts:93-98` normalizes the live response. The public command is registered and executes source validation/comparison at `src/cli/environmentCommands.ts:311-377`. ## Additional context The final self-contained block ran once after an earlier independent run produced the same four outcomes. The repository's existing ordinary-empty-response source test passed separately; it does not cover this missing-item path. No implementation was changed; no published-release runtime was tested. Restoration check: With the same fixtures, the empty-text case must report `steps: 2`, `equal: 2`, agreement 1, and no root value difference. Keep full agreement for matching JSON 404 objects, the `/detail` difference for changed JSON objects, and the root difference for a JSON empty string versus no body. These signals restore the existing normalization contract without hiding actual response differences.

bsakaguchi closed this issue

2026-10-31 00:44:35 +00:00

bsakaguchi commented

2026-10-31 00:44:35 +00:00

Author

Owner

@morluto commented on GitHub (Oct 23, 2026):

Fixed in 23ab2c406. The production served-state comparison preserves the recorded missing-item body representation. Empty recorded text versus empty live text now agrees, while JSON string-empty and null controls remain explicitly distinguished. Verified by src/application/environment/environmentCompare.test.ts. Verification exercised production comparison/replay from synthetic recorded Evidence; no live-provider run is claimed.

@morluto commented on GitHub (Oct 23, 2026): Fixed in [23ab2c406](https://github.com/morluto/rea/commit/23ab2c4069a335d87bdc4a3bce6bfaebecce05e4). The production served-state comparison preserves the recorded missing-item body representation. Empty recorded text versus empty live text now agrees, while JSON string-empty and null controls remain explicitly distinguished. Verified by `src/application/environment/environmentCompare.test.ts`. Verification exercised production comparison/replay from synthetic recorded Evidence; no live-provider run is claimed.

bsakaguchi referenced this issue

2026-10-31 01:10:37 +00:00

[PR #3844] [MERGED] fix(reference): select a package main before its directory index #5941

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3844

No description provided.

Rows
Columns