morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #5699] [MERGED] feat(environment): compare an environment with probe captures of its application #7034

New issue

Closed

opened 2026-11-01 21:44:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:44:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5699
Author: @earakely-scale
Created: 10/20/2026
Status: ✅ Merged
Merged: 10/22/2026
Merged by: @morluto

Base: main ← Head: feat/environment-fidelity

📝 Commits (4)

761a2bd chore: restack environment fidelity onto main
1874b20 fix(environment): keep served collection order with the fidelity checks
9549f74 fix(environment): translate an object's keys only when it is keyed by learned ids
462434c fix(merge): reconcile environment comparison with list queries

📊 Changes

40 files changed (+4360 additions, -117 deletions)

View changed files

📝 docs/agent-environments.md (+64 -1)
📝 docs/cli.md (+16 -0)
➕ docs/specs/agent-environments/fidelity-compare/plan.md (+12 -0)
➕ docs/specs/agent-environments/fidelity-compare/spec.md (+86 -0)
➕ docs/specs/agent-environments/fidelity-compare/tasks.md (+12 -0)
➕ docs/specs/agent-environments/fidelity-compare/verification.md (+146 -0)
📝 docs/specs/agent-environments/state-writes/spec.md (+25 -13)
➕ src/application/environment/environmentCompare.test.ts (+723 -0)
➕ src/application/environment/environmentCompare.ts (+508 -0)
➕ src/application/environment/environmentCompareDiff.test.ts (+307 -0)
➕ src/application/environment/environmentCompareDiff.ts (+355 -0)
➕ src/application/environment/environmentCompareIds.test.ts (+251 -0)
➕ src/application/environment/environmentCompareIds.ts (+424 -0)
➕ src/application/environment/environmentCompareMatch.ts (+178 -0)
➕ src/application/environment/environmentCompareProbe.ts (+177 -0)
📝 src/application/environment/environmentReplay.test.ts (+11 -0)
📝 src/application/environment/environmentReplay.ts (+3 -3)
📝 src/application/environment/environmentSource.ts (+22 -5)
📝 src/application/environment/environmentStateAnswer.test.ts (+60 -0)
📝 src/application/environment/environmentStateAnswer.ts (+3 -3)

...and 20 more files

📄 Description

Summary

An environment is only as useful as it is faithful: an agent graded or trained on it should meet the behaviour of the application it was captured from. REA checks an environment only against its own capture, so nothing measured how far it drifts from the application on calls the capture did not make, after writes, or on errors. Every important gap found so far came from ad hoc replays against live applications.

This PR adds rea environment-compare, which measures that. It compares an environment with probe captures: recordings of call sequences made against the live application, each after a reset. It reports how many calls the environment answers as the application did, which tools disagree and how. The command only reads Evidence and makes no network call.

Part of #4933. Stacked on #5600, which carries #5351 and #5352; it is merged in at 5327e598. Until #5600 lands, the diff against main includes it. The new work is:

src/application/environment/environmentCompare*.ts and their tests;
src/cli/environmentCommands.ts and src/cliCommandNames.ts;
tests/boundary/cli/environmentCompareCli.test.ts;
the docs and docs/specs/agent-environments/fidelity-compare/.

Problem and expected behavior

rea environment-compare manifest.json --evidence capture.json --probe probe-1.json --probe probe-2.json --json

Each --probe is inspect-web-network-capture or capture-browser-scenario Evidence of one sequence, recorded against the application after a reset. For each probe, the comparison:

resets the environment's state;
takes the requests in the order they were made;
matches each request to the tool of the same method whose path template fits with the most literal segments; a format-rule parameter must fit its format;
answers it as the served environment would;
compares that answer with the application's.

Id mapping. A re-seeded application holds the same entities under other ids, so the comparison maps them:

items in two answers pair by a natural key, a string field unique on each side that both share;
identifier fields of paired objects, and of two single answers such as a create's, map the live value to the environment's;
objects keyed by ids pair their entries;
mapped ids in a request's path, query or body are translated before the request is matched and answered.

Normalisation. After mapping:

clocks compare by format only;
two values each side generated on its own compare equal;
scalar sets in another order differ in order only;
bodies over the replay limit compare by status.

The report counts each kind.

Report. Agreement overall, per probe and per tool, plus agreement ignoring order for applications that return sets in no fixed order. Differences are categorised as status, missing-field, extra-field, value, length, order, clock-format, refused, not-served, no-tool and unmapped-id, each with a JSON pointer.

Change and scope

Self-contained: the comparison reuses the served environment's own answer path (environmentToolAnswer with a fresh state store per probe), so it measures exactly what an agent meets. Probes load with the capture parser, now exported.
Not included:
- network access, credentials, or resetting the application: probes are recorded outside REA;
- generating call sequences: a follow-up generates them where the environment disagrees;
- any change to serving.
Scope: an analysis command over existing Evidence, inside the existing boundary; no target execution.

Contract and boundary impact

Semantic owner and earliest changed stage: the environment application layer (src/application/environment/) and the CLI.
Contribution scope fit and linked maintainer decision: a read-only command over Evidence; not applicable.
CLI and MCP/tool-catalog contract: a new CLI command environment-compare with a repeatable --probe; no MCP tool, so the catalog is unchanged.
Provider, bridge, target-format, or platform compatibility: none.
Evidence, artifact, provenance, or reconstruction contract: reads manifest, capture and probe Evidence and authenticates them as environment-serve does; the report cites every Evidence id and is not itself Evidence.
Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added.

Unit tests (19):
- matching: literal versus parameter segments, format rules, query and body arguments;
- the id map and its conflicts;
- natural-key pairing and its guards;
- learning from keyed objects;
- path and value translation;
- every difference category, plus clocks, generated values, order and oversized bodies.
An integration test, run against a probe of the books application re-seeded under other ids:
- it agrees on all 9 steps, with 3 ids mapped;
- in a second case, every category appears where expected and the state resets per probe.
CLI boundary (2): a successful comparison, and a refusal naming the bad probe.

Executed evidence: six environments against their live applications. The same images were seeded by the same scripts and restored from a snapshot before every sequence (7 to 80 s).

A driver outside REA recorded three kinds of sequence:

the capture re-run;
a create, read, update and delete scenario per served write type;
a read of every seeded entity.

The environments' manifests use prefix-variance, so the comparison ran on a local build with #5664 merged.

Environment	Steps	Agreement	Ignoring order	Against its own capture
NetBox 4.7.3	370	99.2%	99.2%	100%
Komga 1.29.1	356	93.8%	93.8%	93.1%
Mealie 3.28.0	301	96.0%	97.0%	97.1%
Shlink 5.1.7	133	89.5%	92.5%	92.4%
n8n 1.123.85	202	74.8%	87.6%	87.5%
Sentry 26.9.0	230	23.9%	46.5%	84.7%

Environment gaps the runs found, which are the next things to fix:

withheld writes and the reads after them;
n8n's archive-before-delete precondition;
lists:
- a create missing from a list that keeps its recorded members;
- a filtered list answered without its filter;
- a replayed search that ignores its body;
a domain-scoped short URL served without its domain parameter;
Sentry state rendering:
- a field two recorded views disagree on;
- an organisation read whose shape depends on a query parameter;
event counts that move with the current time while the environment's clock is frozen;
a field the capture's scrubbing replaced with "<scrubbed>".

Differences not caused by the environment:

The application differing from itself between seedings: Sentry's permission-list order, release first-event times and feature flags, and n8n's executions.
The probe observing itself: its own sign-in user agent.
Ids the driver never mapped.

The six runs also found six ways the comparison itself was wrong. Each was fixed with a test before the next run:

requests are now matched in the environment's terms;
outside id fields, only string values are learned as ids;
position-only pairs are guarded;
objects keyed by ids are mapped;
unordered sets are compared as sets, and epoch numbers in arrays as clocks;
oversized bodies are compared by status only.

Details are in docs/specs/agent-environments/fidelity-compare/verification.md.

User-visible output. The summary for Komga:

{"probes": 3, "steps": 356, "equal": 334, "agreement": 0.938, "agreement_ignoring_order": 0.938, "unobserved": 0,
 "categories": {"status": 6, "missing-field": 4, "extra-field": 4, "value": 10, "length": 2, "not-served": 8, "unmapped-id": 6, "...": 0},
 "normalised": {"ids": 926, "clocks": 1550, "generated": 17, "large_bodies": 0, "unobserved_bodies": 0}}

Remaining proof gaps:

Agreement covers only the calls the probes make.
Clocks and generated values compare by form.
Id mapping depends on the answers sharing a natural key.
The numbers above mix environment fidelity with drift in the application itself between seedings. A live-against-live control would separate the two and is not done here.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Run on macOS with Node v22.23.1 at 235b22f2 (verification commit 20ad6618), then again after merging #5600's head into 64b4994a.

At 235b22f2: npm run build:cached, npm run check:fast, npm run format:check, npx knip, npm run scan:todos, npm run docs:check, npm run verify:test-discovery, npm run mcp-catalog:check — passed.
npm run jscpd — 3.46% over 3609 files.
npx vitest run --project domain --project services — 7487 passed, 5 pre-existing skips.
npm run test:prepare, then npm run test:focused -- the environment-compare, export and serve CLI boundaries, the serve MCP boundary and tool schema validity — 5 files, 30 passed.
At 64b4994a:
- npm run build:cached, check:fast, format:check, knip, mcp-catalog:check and docs:check passed;
- jscpd at 3.46%;
- the environment application tests (168) passed;
- the compare and serve CLI and MCP boundaries (9) passed.
Review round (head 473d03c4): fixed the 11 Greptile threads and two findings of my own.
- #5600's four went in on that branch (b6c3b168) and were merged here (631a7120).
- The comparison fixes are in bbbdea20 and 7832f7eb: ids learned only from answers to the call, and also per collection; paging values never translated; calls checked against the served tools' input schemas; browser probes keep their bodies; empty and large binary bodies compare as served; scalar sets compare after mapping; overlapping id-keyed objects pair.
- Each new test fails on the previous head.
- Gates passed again: check:fast, format, knip, todos, docs, test discovery, module boundaries and catalog; domain and services 7504 (5 pre-existing skips); focused boundaries 80; conformance; jscpd 3.46%.
- Re-run on the saved probes, the six environments compare exactly as in the table, step for step, with the same ids mapped.
Review round 2 (head c63bcb7b): merged #5600's f2b95351 (a write of a missing entity never answers with a 2xx), then fixed Greptile's four follow-ups in 736bf384.
- Ids are learned from a replay only when the whole recorded line and request body match.
- An entity's own identity compares within the call's collection.
- Overlapping keys are read as ids only when they evidently are.
- A JSON empty string is kept.
- Each new test fails on de2fc018. Gates passed again (domain and services 7508, boundaries 80, conformance, jscpd 3.46%), and the six environments still compare exactly as in the table.
Review round 3 (head 5604d55f): entities inside an envelope keep their collection's mapping: the answer's own entities are found at the view's entity or list pointer, or the write's response pointer (8ef92577). Its test fails on c63bcb7b; domain and services 7509 passed, boundaries 80, and the six environments are unchanged.
Review round 4 (head 7e26d996): no new findings; merged the write-plan branch at 9edbabe8 and fc403d98 (main 335c042d and the base branches' review fixes). Gates passed (domain and services 7637, 5 pre-existing skips; boundaries 80; conformance; jscpd 3.44%; the environment application tests 189 after the last merge).
Review round 5 (head 71a50f07): merged #5600's 1d1c4787 in 9622c5da, then fixed Greptile's three findings. The required-field finding was #5600 code, fixed there. The other two are fixed in 71a50f07.
- Probe records without a usable request line, and requests without a recorded answer, are listed per probe under unobserved with their reason, and left out of agreement.
- An unrecorded live body compares by status only, counted as unobserved_bodies.
- A browser probe captured without response bodies is refused.
- A format-rule path parameter fits by its decoded value.
- Each new test fails on 7e26d996. Gates passed: check:fast, format, knip, todos, docs, discovery, module boundaries, catalog, no oxlint warnings; domain and services 7647 (5 pre-existing skips); the compare, export, serve and analysis-view boundaries 69; conformance 279; jscpd 3.44%.
- On the saved probes nothing went unobserved, and five environments compare exactly as before.
- Komga's collection create and patch are now withheld by #5600's stricter gate, because the only read that could confirm them came after the collection was deleted. Its agreement falls from 96.9% to 93.8% (11 steps: 3 creates, 2 patches, and the 6 reads and deletes that follow), and against its own capture from 96.6% to 93.1%. The table shows the new figures.
Review round 6 (head bacafc5d): merged #5600's dc5e2aad in 8996fbf2. That fixes the two findings Greptile raised on both PRs: derived update changes are weighed, and a reset or load resets collection spellings.
- A request whose answer the probe lost is now sent to the environment, still uncompared, so a later read sees its effect (bacafc5d). Its test fails on 71a50f07.
- Gates passed: domain and services 7650 (5 pre-existing skips), boundaries 69, conformance 279, jscpd 3.44%, no oxlint warnings.
- The six environments compare exactly as in the table, probes and self-comparison alike.
Review round 7 (head ac3ca8a3): Greptile's two findings on round 6 were in #5600's code, fixed there in fe8ab584 and merged here in d58c9aba. Gates passed: domain and services 7653 (5 pre-existing skips), boundaries 69, conformance, jscpd 3.44%. The six environments compare exactly as in the table.
Review round 8 (head 779910e1): Greptile's finding on round 7 was in #5600's store, fixed there in 73c0e9b5 and merged here. Gates passed: domain and services 7654, boundaries, jscpd 3.44%. The six environments compare exactly as in the table.
Review round 9 (head ce491696): fixed Greptile's two findings on round 8 in 5020a833: a probe with any undated record keeps its recorded order, as state extraction does, and a browser probe that makes POST, PUT or PATCH calls without recorded request bodies is refused. It also merged morluto's a36cdf46 and #5600 at 3580d85c and 43d98ea9, whose gate fix settles the finding Greptile raised here about overwritten document writes. Both new tests fail on the previous head, and the six environments compare exactly as before, on the probes and against their own captures.
Review round 10 (head 6c13ef71):
- Fixed Greptile's finding on this branch in b7e50be4. A probe request to another origin than its application (the origin with the most JSON-like records, chosen as the manifest chooses its own) is neither made nor compared, and is listed as other-origin. The origin choice moved to environmentManifestOrigin.ts, shared by the manifest and the comparison.
- Merged #5600 at 264ce410, 0ae64400, 6257de93, 466ed8b4, 7cef2fb2, 45a5c8b9 and 325c128f (6a633659, aa33d424, 2431c91a, 22fad94a, 194493e9, 097d3796, 6c13ef71). Their fixes settle the nine findings Greptile raised here against that branch's code: pending writes a summary read showed nothing of, encoded update and delete paths, later document replacements, flat entity fields that look like pointers, zero page sizes (twice), partial write responses taken for removals, and interference timed by a tool's call elsewhere or by an overwritten effect. The paging rules, with this branch's isPagingParameter, moved to environmentStatePaging.ts.
- The new test fails on ce491696. Gates passed on 6c13ef71: check:fast, format, knip, todos, docs, discovery, module boundaries and catalog, with no oxlint warnings in the changed files; domain and services 7789 (5 pre-existing skips); environment boundaries 65; conformance 265; jscpd 3.43%.
- The six environments compare exactly as before, on the probes and against their own captures, with no unobserved records.
Review round 11 (head ea72e299):
- Since 6c13ef71, Cursor and morluto restacked this branch onto #5600 and fixed eight Greptile findings (b0856b1e through c2ec9f56).
- Merged the restacked #5600 at 717ac2c4 in a3a19849, and main 2451e187 in 8285d9b1. The only conflict was an import list.
- Greptile's create-order finding still reproduced on c2ec9f56 and is fixed in 7ad2fd2a. The gate's simulated state now keeps serving's state order and remembers collections in the order they were first known, so a tie between equal spellings resolves to serving's collection. Both new tests fail on c2ec9f56; the state-writes spec states the tie order.
- format:check failed on two docs tables and three test files; they are formatted in 6c1e5a8e and ea72e299.
- Gates passed: check:fast, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7912 (5 pre-existing skips); environment boundaries 65; jscpd 3.47%.
- The writes census matches #5600's exactly, and the served smoke passes 31/31.
- Compare output is byte-identical before and after 7ad2fd2a. Against 6c13ef71, only Komga's self-comparison changed. A step that already differed now also reports the listed book's id, instead of counting it as generated. Agreement is unchanged at 93.1%.
Review round 12 (head 5a09aae9):
- Fixed this branch's two Greptile findings.
  - 9a046922 and a0865fb6: an entity's own identity that both sides hold verbatim now differs when its collection's mapping gives it another environment value. References keep raw equality, because the probe-wide mapping can't tell which collection a reference names. Judging references flagged Sentry's seeded project.id and user ids, and agreement fell from 23.9% to 20.9%. That thread stays open.
  - 1c979337 on #5600: a create's path fields come from the selected collection. Tests for the tied case are here.
- The other three findings here were in #5600's code, fixed there and merged here in 91691509 and 5a09aae9.
- Main is merged separately from #5600 now (4a9f0129). That stops the pre-commit autofix from rewriting main's src/main/deferredRuntime.ts inside a conflicted merge.
- Gates passed: check:fast, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7969 (5 pre-existing skips); environment boundaries 67; jscpd 3.40%.
- The six environments compare byte-identically to round 11, on the probes and against their own captures.
- The writes census matches #5600's. Its one change, Gitea's issue PATCH no longer accepting state, comes from #5600. The smoke ran 31/31.
Review round 13 (head 9549f74e): fixed Greptile's finding on 1874b20c, the Cursor agent's restack onto main after #5600 landed. Differ.object translates keys only for an object keyed by learned ids, read as learning reads one, so ordinary fields compare by name and a real rename shows. The new test fails on 1874b20c.
- Gates passed: check:fast, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7982 (5 pre-existing skips); environment boundaries 67; jscpd 3.39%.
- The six environments compare byte-identically to round 12, on the probes and against their own captures.

Merge resolution validation

At 462434c03, merged main (293832baf) while preserving its per-line whole-list membership, query-aware list serving, and extracted helper owners alongside this PR's shared encoded-path matching and stable collection ordering. Paging regressions now distinguish learned offset queries from unproved whole lines, use an explicit page size for later-page fixtures, and retain positive-page-base uncertainty. Removed the unreferenced InspectApkSignatureInput type after confirming the Knip finding also on clean main.

Environment/application and analysis-view source tests: 46 files, 993 tests passed.
Focused CLI/MCP/process boundary tests for environment comparison, serving, export, and shutdown: 7 files, 26 tests passed; runtime and packaged-skill build passed.
npm run check — exit 0, including typecheck, lint, formatting, Knip, metadata, module boundaries, and architecture guards.
npm run check:fast — exit 0.
npm run docs:generate and npm run docs:check — exit 0, including 9 README consistency tests.
Full repository suite and the six live-application probe runs were not repeated. Historical live measurements below describe earlier revisions, not this merge resolution.

Compatibility, safety, and release

Breaking changes or migration steps: none; a new command.
Real Hopper/Ghidra, browser, or OS coverage: not applicable.
Package or release metadata impact: none.
Security, privacy, process, or containment review: reads local Evidence files only. Probe captures are HARs the caller records; the verification's driver recorded them without headers or cookies.

Author readiness

Human owner: @earakely-scale

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5699 **Author:** [@earakely-scale](https://github.com/earakely-scale) **Created:** 10/20/2026 **Status:** ✅ Merged **Merged:** 10/22/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `feat/environment-fidelity` --- ### 📝 Commits (4) - [`761a2bd`](https://github.com/morluto/rea/commit/761a2bd2ec6f93f32ff6271e29b49252d116a544) chore: restack environment fidelity onto main - [`1874b20`](https://github.com/morluto/rea/commit/1874b20c53394018d0949624b721b574484df926) fix(environment): keep served collection order with the fidelity checks - [`9549f74`](https://github.com/morluto/rea/commit/9549f74e6389712f96ae0d4c31bea17f2ac91c2c) fix(environment): translate an object's keys only when it is keyed by learned ids - [`462434c`](https://github.com/morluto/rea/commit/462434c03af32da67b8c9db329716400e2dffdf1) fix(merge): reconcile environment comparison with list queries ### 📊 Changes **40 files changed** (+4360 additions, -117 deletions) <details> <summary>View changed files</summary> 📝 `docs/agent-environments.md` (+64 -1) 📝 `docs/cli.md` (+16 -0) ➕ `docs/specs/agent-environments/fidelity-compare/plan.md` (+12 -0) ➕ `docs/specs/agent-environments/fidelity-compare/spec.md` (+86 -0) ➕ `docs/specs/agent-environments/fidelity-compare/tasks.md` (+12 -0) ➕ `docs/specs/agent-environments/fidelity-compare/verification.md` (+146 -0) 📝 `docs/specs/agent-environments/state-writes/spec.md` (+25 -13) ➕ `src/application/environment/environmentCompare.test.ts` (+723 -0) ➕ `src/application/environment/environmentCompare.ts` (+508 -0) ➕ `src/application/environment/environmentCompareDiff.test.ts` (+307 -0) ➕ `src/application/environment/environmentCompareDiff.ts` (+355 -0) ➕ `src/application/environment/environmentCompareIds.test.ts` (+251 -0) ➕ `src/application/environment/environmentCompareIds.ts` (+424 -0) ➕ `src/application/environment/environmentCompareMatch.ts` (+178 -0) ➕ `src/application/environment/environmentCompareProbe.ts` (+177 -0) 📝 `src/application/environment/environmentReplay.test.ts` (+11 -0) 📝 `src/application/environment/environmentReplay.ts` (+3 -3) 📝 `src/application/environment/environmentSource.ts` (+22 -5) 📝 `src/application/environment/environmentStateAnswer.test.ts` (+60 -0) 📝 `src/application/environment/environmentStateAnswer.ts` (+3 -3) _...and 20 more files_ </details> ### 📄 Description ## Summary An environment is only as useful as it is faithful: an agent graded or trained on it should meet the behaviour of the application it was captured from. REA checks an environment only against its own capture, so nothing measured how far it drifts from the application on calls the capture did not make, after writes, or on errors. Every important gap found so far came from ad hoc replays against live applications. This PR adds `rea environment-compare`, which measures that. It compares an environment with **probe captures**: recordings of call sequences made against the live application, each after a reset. It reports how many calls the environment answers as the application did, which tools disagree and how. The command only reads Evidence and makes no network call. Part of #4933. **Stacked on #5600**, which carries #5351 and #5352; it is merged in at `5327e598`. Until #5600 lands, the diff against main includes it. The new work is: - `src/application/environment/environmentCompare*.ts` and their tests; - `src/cli/environmentCommands.ts` and `src/cliCommandNames.ts`; - `tests/boundary/cli/environmentCompareCli.test.ts`; - the docs and `docs/specs/agent-environments/fidelity-compare/`. ## Problem and expected behavior ```sh rea environment-compare manifest.json --evidence capture.json --probe probe-1.json --probe probe-2.json --json ``` Each `--probe` is `inspect-web-network-capture` or `capture-browser-scenario` Evidence of one sequence, recorded against the application after a reset. For each probe, the comparison: - resets the environment's state; - takes the requests in the order they were made; - matches each request to the tool of the same method whose path template fits with the most literal segments; a format-rule parameter must fit its format; - answers it as the served environment would; - compares that answer with the application's. **Id mapping.** A re-seeded application holds the same entities under other ids, so the comparison maps them: - items in two answers pair by a natural key, a string field unique on each side that both share; - identifier fields of paired objects, and of two single answers such as a create's, map the live value to the environment's; - objects keyed by ids pair their entries; - mapped ids in a request's path, query or body are translated before the request is matched and answered. **Normalisation.** After mapping: - clocks compare by format only; - two values each side generated on its own compare equal; - scalar sets in another order differ in order only; - bodies over the replay limit compare by status. The report counts each kind. **Report.** Agreement overall, per probe and per tool, plus agreement ignoring order for applications that return sets in no fixed order. Differences are categorised as `status`, `missing-field`, `extra-field`, `value`, `length`, `order`, `clock-format`, `refused`, `not-served`, `no-tool` and `unmapped-id`, each with a JSON pointer. ## Change and scope - **Self-contained:** the comparison reuses the served environment's own answer path (`environmentToolAnswer` with a fresh state store per probe), so it measures exactly what an agent meets. Probes load with the capture parser, now exported. - **Not included:** - network access, credentials, or resetting the application: probes are recorded outside REA; - generating call sequences: a follow-up generates them where the environment disagrees; - any change to serving. - **Scope:** an analysis command over existing Evidence, inside the existing boundary; no target execution. ## Contract and boundary impact - Semantic owner and earliest changed stage: the environment application layer (`src/application/environment/`) and the CLI. - Contribution scope fit and linked maintainer decision: a read-only command over Evidence; not applicable. - CLI and MCP/tool-catalog contract: a new CLI command `environment-compare` with a repeatable `--probe`; no MCP tool, so the catalog is unchanged. - Provider, bridge, target-format, or platform compatibility: none. - Evidence, artifact, provenance, or reconstruction contract: reads manifest, capture and probe Evidence and authenticates them as `environment-serve` does; the report cites every Evidence id and is not itself Evidence. - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage **Tests added.** - Unit tests (19): - matching: literal versus parameter segments, format rules, query and body arguments; - the id map and its conflicts; - natural-key pairing and its guards; - learning from keyed objects; - path and value translation; - every difference category, plus clocks, generated values, order and oversized bodies. - An integration test, run against a probe of the books application re-seeded under other ids: - it agrees on all 9 steps, with 3 ids mapped; - in a second case, every category appears where expected and the state resets per probe. - CLI boundary (2): a successful comparison, and a refusal naming the bad probe. **Executed evidence: six environments against their live applications.** The same images were seeded by the same scripts and restored from a snapshot before every sequence (7 to 80 s). A driver outside REA recorded three kinds of sequence: - the capture re-run; - a create, read, update and delete scenario per served write type; - a read of every seeded entity. The environments' manifests use `prefix-variance`, so the comparison ran on a local build with #5664 merged. | Environment | Steps | Agreement | Ignoring order | Against its own capture | | --- | ---: | ---: | ---: | ---: | | NetBox 4.7.3 | 370 | 99.2% | 99.2% | 100% | | Komga 1.29.1 | 356 | 93.8% | 93.8% | 93.1% | | Mealie 3.28.0 | 301 | 96.0% | 97.0% | 97.1% | | Shlink 5.1.7 | 133 | 89.5% | 92.5% | 92.4% | | n8n 1.123.85 | 202 | 74.8% | 87.6% | 87.5% | | Sentry 26.9.0 | 230 | 23.9% | 46.5% | 84.7% | **Environment gaps the runs found, which are the next things to fix:** - withheld writes and the reads after them; - n8n's archive-before-delete precondition; - lists: - a create missing from a list that keeps its recorded members; - a filtered list answered without its filter; - a replayed search that ignores its body; - a domain-scoped short URL served without its `domain` parameter; - Sentry state rendering: - a field two recorded views disagree on; - an organisation read whose shape depends on a query parameter; - event counts that move with the current time while the environment's clock is frozen; - a field the capture's scrubbing replaced with `"<scrubbed>"`. **Differences not caused by the environment:** - **The application differing from itself between seedings:** Sentry's permission-list order, release first-event times and feature flags, and n8n's executions. - **The probe observing itself:** its own sign-in user agent. - **Ids the driver never mapped.** The six runs also found six ways the comparison itself was wrong. Each was fixed with a test before the next run: - requests are now matched in the environment's terms; - outside id fields, only string values are learned as ids; - position-only pairs are guarded; - objects keyed by ids are mapped; - unordered sets are compared as sets, and epoch numbers in arrays as clocks; - oversized bodies are compared by status only. Details are in `docs/specs/agent-environments/fidelity-compare/verification.md`. **User-visible output.** The summary for Komga: ```json {"probes": 3, "steps": 356, "equal": 334, "agreement": 0.938, "agreement_ignoring_order": 0.938, "unobserved": 0, "categories": {"status": 6, "missing-field": 4, "extra-field": 4, "value": 10, "length": 2, "not-served": 8, "unmapped-id": 6, "...": 0}, "normalised": {"ids": 926, "clocks": 1550, "generated": 17, "large_bodies": 0, "unobserved_bodies": 0}} ``` **Remaining proof gaps:** - Agreement covers only the calls the probes make. - Clocks and generated values compare by form. - Id mapping depends on the answers sharing a natural key. - The numbers above mix environment fidelity with drift in the application itself between seedings. A live-against-live control would separate the two and is not done here. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Run on macOS with Node v22.23.1 at 235b22f2 (verification commit 20ad6618), then again after merging #5600's head into 64b4994a. - At 235b22f2: `npm run build:cached`, `npm run check:fast`, `npm run format:check`, `npx knip`, `npm run scan:todos`, `npm run docs:check`, `npm run verify:test-discovery`, `npm run mcp-catalog:check` — passed. - `npm run jscpd` — 3.46% over 3609 files. - `npx vitest run --project domain --project services` — 7487 passed, 5 pre-existing skips. - `npm run test:prepare`, then `npm run test:focused --` the environment-compare, export and serve CLI boundaries, the serve MCP boundary and tool schema validity — 5 files, 30 passed. - At 64b4994a: - `npm run build:cached`, `check:fast`, `format:check`, `knip`, `mcp-catalog:check` and `docs:check` passed; - `jscpd` at 3.46%; - the environment application tests (168) passed; - the compare and serve CLI and MCP boundaries (9) passed. - **Review round (head 473d03c4):** fixed the 11 Greptile threads and two findings of my own. - #5600's four went in on that branch (b6c3b168) and were merged here (631a7120). - The comparison fixes are in bbbdea20 and 7832f7eb: ids learned only from answers to the call, and also per collection; paging values never translated; calls checked against the served tools' input schemas; browser probes keep their bodies; empty and large binary bodies compare as served; scalar sets compare after mapping; overlapping id-keyed objects pair. - Each new test fails on the previous head. - Gates passed again: `check:fast`, format, knip, todos, docs, test discovery, module boundaries and catalog; domain and services 7504 (5 pre-existing skips); focused boundaries 80; conformance; `jscpd` 3.46%. - Re-run on the saved probes, the six environments compare exactly as in the table, step for step, with the same ids mapped. - **Review round 2 (head c63bcb7b):** merged #5600's f2b95351 (a write of a missing entity never answers with a 2xx), then fixed Greptile's four follow-ups in 736bf384. - Ids are learned from a replay only when the whole recorded line and request body match. - An entity's own identity compares within the call's collection. - Overlapping keys are read as ids only when they evidently are. - A JSON empty string is kept. - Each new test fails on de2fc018. Gates passed again (domain and services 7508, boundaries 80, conformance, `jscpd` 3.46%), and the six environments still compare exactly as in the table. - **Review round 3 (head 5604d55f):** entities inside an envelope keep their collection's mapping: the answer's own entities are found at the view's entity or list pointer, or the write's response pointer (8ef92577). Its test fails on c63bcb7b; domain and services 7509 passed, boundaries 80, and the six environments are unchanged. - **Review round 4 (head 7e26d996):** no new findings; merged the write-plan branch at 9edbabe8 and fc403d98 (main 335c042d and the base branches' review fixes). Gates passed (domain and services 7637, 5 pre-existing skips; boundaries 80; conformance; `jscpd` 3.44%; the environment application tests 189 after the last merge). - **Review round 5 (head 71a50f07):** merged #5600's 1d1c4787 in 9622c5da, then fixed Greptile's three findings. The required-field finding was #5600 code, fixed there. The other two are fixed in 71a50f07. - Probe records without a usable request line, and requests without a recorded answer, are listed per probe under `unobserved` with their reason, and left out of agreement. - An unrecorded live body compares by status only, counted as `unobserved_bodies`. - A browser probe captured without response bodies is refused. - A format-rule path parameter fits by its decoded value. - Each new test fails on 7e26d996. Gates passed: `check:fast`, format, knip, todos, docs, discovery, module boundaries, catalog, no oxlint warnings; domain and services 7647 (5 pre-existing skips); the compare, export, serve and analysis-view boundaries 69; conformance 279; `jscpd` 3.44%. - On the saved probes nothing went unobserved, and five environments compare exactly as before. - Komga's collection create and patch are now withheld by #5600's stricter gate, because the only read that could confirm them came after the collection was deleted. Its agreement falls from 96.9% to 93.8% (11 steps: 3 creates, 2 patches, and the 6 reads and deletes that follow), and against its own capture from 96.6% to 93.1%. The table shows the new figures. - **Review round 6 (head bacafc5d):** merged #5600's dc5e2aad in 8996fbf2. That fixes the two findings Greptile raised on both PRs: derived update changes are weighed, and a reset or load resets collection spellings. - A request whose answer the probe lost is now sent to the environment, still uncompared, so a later read sees its effect (bacafc5d). Its test fails on 71a50f07. - Gates passed: domain and services 7650 (5 pre-existing skips), boundaries 69, conformance 279, `jscpd` 3.44%, no oxlint warnings. - The six environments compare exactly as in the table, probes and self-comparison alike. - **Review round 7 (head ac3ca8a3):** Greptile's two findings on round 6 were in #5600's code, fixed there in fe8ab584 and merged here in d58c9aba. Gates passed: domain and services 7653 (5 pre-existing skips), boundaries 69, conformance, `jscpd` 3.44%. The six environments compare exactly as in the table. - **Review round 8 (head 779910e1):** Greptile's finding on round 7 was in #5600's store, fixed there in 73c0e9b5 and merged here. Gates passed: domain and services 7654, boundaries, `jscpd` 3.44%. The six environments compare exactly as in the table. - **Review round 9 (head ce491696):** fixed Greptile's two findings on round 8 in 5020a833: a probe with any undated record keeps its recorded order, as state extraction does, and a browser probe that makes POST, PUT or PATCH calls without recorded request bodies is refused. It also merged morluto's a36cdf46 and #5600 at 3580d85c and 43d98ea9, whose gate fix settles the finding Greptile raised here about overwritten document writes. Both new tests fail on the previous head, and the six environments compare exactly as before, on the probes and against their own captures. - **Review round 10 (head 6c13ef71):** - Fixed Greptile's finding on this branch in b7e50be4. A probe request to another origin than its application (the origin with the most JSON-like records, chosen as the manifest chooses its own) is neither made nor compared, and is listed as `other-origin`. The origin choice moved to `environmentManifestOrigin.ts`, shared by the manifest and the comparison. - Merged #5600 at 264ce410, 0ae64400, 6257de93, 466ed8b4, 7cef2fb2, 45a5c8b9 and 325c128f (6a633659, aa33d424, 2431c91a, 22fad94a, 194493e9, 097d3796, 6c13ef71). Their fixes settle the nine findings Greptile raised here against that branch's code: pending writes a summary read showed nothing of, encoded update and delete paths, later document replacements, flat entity fields that look like pointers, zero page sizes (twice), partial write responses taken for removals, and interference timed by a tool's call elsewhere or by an overwritten effect. The paging rules, with this branch's `isPagingParameter`, moved to `environmentStatePaging.ts`. - The new test fails on ce491696. Gates passed on 6c13ef71: `check:fast`, format, knip, todos, docs, discovery, module boundaries and catalog, with no oxlint warnings in the changed files; domain and services 7789 (5 pre-existing skips); environment boundaries 65; conformance 265; `jscpd` 3.43%. - The six environments compare exactly as before, on the probes and against their own captures, with no unobserved records. - **Review round 11 (head ea72e299):** - Since 6c13ef71, Cursor and morluto restacked this branch onto #5600 and fixed eight Greptile findings (b0856b1e through c2ec9f56). - Merged the restacked #5600 at 717ac2c4 in a3a19849, and main 2451e187 in 8285d9b1. The only conflict was an import list. - Greptile's create-order finding still reproduced on c2ec9f56 and is fixed in 7ad2fd2a. The gate's simulated state now keeps serving's state order and remembers collections in the order they were first known, so a tie between equal spellings resolves to serving's collection. Both new tests fail on c2ec9f56; the state-writes spec states the tie order. - `format:check` failed on two docs tables and three test files; they are formatted in 6c1e5a8e and ea72e299. - Gates passed: `check:fast`, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7912 (5 pre-existing skips); environment boundaries 65; `jscpd` 3.47%. - The writes census matches #5600's exactly, and the served smoke passes 31/31. - Compare output is byte-identical before and after 7ad2fd2a. Against 6c13ef71, only Komga's self-comparison changed. A step that already differed now also reports the listed book's id, instead of counting it as generated. Agreement is unchanged at 93.1%. - **Review round 12 (head 5a09aae9):** - Fixed this branch's two Greptile findings. - 9a046922 and a0865fb6: an entity's own identity that both sides hold verbatim now differs when its collection's mapping gives it another environment value. References keep raw equality, because the probe-wide mapping can't tell which collection a reference names. Judging references flagged Sentry's seeded `project.id` and user ids, and agreement fell from 23.9% to 20.9%. That thread stays open. - 1c979337 on #5600: a create's path fields come from the selected collection. Tests for the tied case are here. - The other three findings here were in #5600's code, fixed there and merged here in 91691509 and 5a09aae9. - Main is merged separately from #5600 now (4a9f0129). That stops the pre-commit autofix from rewriting main's `src/main/deferredRuntime.ts` inside a conflicted merge. - Gates passed: `check:fast`, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7969 (5 pre-existing skips); environment boundaries 67; `jscpd` 3.40%. - The six environments compare byte-identically to round 11, on the probes and against their own captures. - The writes census matches #5600's. Its one change, Gitea's issue PATCH no longer accepting `state`, comes from #5600. The smoke ran 31/31. - **Review round 13 (head 9549f74e):** fixed Greptile's finding on 1874b20c, the Cursor agent's restack onto main after #5600 landed. `Differ.object` translates keys only for an object keyed by learned ids, read as learning reads one, so ordinary fields compare by name and a real rename shows. The new test fails on 1874b20c. - Gates passed: `check:fast`, format, knip, todos, docs, discovery, module boundaries and catalog; domain and services 7982 (5 pre-existing skips); environment boundaries 67; `jscpd` 3.39%. - The six environments compare byte-identically to round 12, on the probes and against their own captures. ## Merge resolution validation At `462434c03`, merged `main` (`293832baf`) while preserving its per-line whole-list membership, query-aware list serving, and extracted helper owners alongside this PR's shared encoded-path matching and stable collection ordering. Paging regressions now distinguish learned offset queries from unproved whole lines, use an explicit page size for later-page fixtures, and retain positive-page-base uncertainty. Removed the unreferenced `InspectApkSignatureInput` type after confirming the Knip finding also on clean main. - Environment/application and analysis-view source tests: 46 files, 993 tests passed. - Focused CLI/MCP/process boundary tests for environment comparison, serving, export, and shutdown: 7 files, 26 tests passed; runtime and packaged-skill build passed. - `npm run check` — exit 0, including typecheck, lint, formatting, Knip, metadata, module boundaries, and architecture guards. - `npm run check:fast` — exit 0. - `npm run docs:generate` and `npm run docs:check` — exit 0, including 9 README consistency tests. - Full repository suite and the six live-application probe runs were not repeated. Historical live measurements below describe earlier revisions, not this merge resolution. ## Compatibility, safety, and release - Breaking changes or migration steps: none; a new command. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. - Package or release metadata impact: none. - Security, privacy, process, or containment review: reads local Evidence files only. Probe captures are HARs the caller records; the verification's driver recorded them without headers or cookies. ## Author readiness Human owner: @earakely-scale - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:44:51 +00:00