morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7621] [Bug] environment-compare does not disclose when a step's difference list is truncated #4549

New issue

Closed

opened 2026-10-31 00:50:42 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:50:42 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7621

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

I generated capture and probe Evidence from two small HAR files, projected an environment manifest, and ran:

node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json

The three GET steps have 20, 21 and 25 differing numeric fields respectively. Every step returned the same 20 difference entries, ending at /field19. No field in the step identifies a truncated list, the original difference count, or a continuation. The report returned status: "complete", and all three steps correctly returned equal: false.

Repeating with --full-output returned the identical report inside its envelope. The omitted /field20 pointer appeared nowhere in that output. A consumer cannot distinguish an exhaustive 20-entry list from one that omitted further differences.

Reproduction

Use the source revision and pinned Node version below with the already built runtime and locked dependencies. Run this block from the repository root. It creates only temporary fixtures and isolates REA's HOME/XDG state.

set -eu
work=$(mktemp -d)
export work
export HOME="$work/home"
export XDG_CONFIG_HOME="$HOME/config" XDG_CACHE_HOME="$HOME/cache"
export XDG_DATA_HOME="$HOME/data" XDG_STATE_HOME="$HOME/state"
python3 - <<'PY'
import json, os, pathlib, subprocess
root = pathlib.Path(os.environ['work'])
stamp = subprocess.check_output(
    ['env', 'TZ=America/Los_Angeles', 'date', '+%Y-%m-%dT%H:%M:%S%z'],
    text=True).strip()
for name, value in [('capture', 0), ('probe', 1)]:
    entries = []
    for count in [20, 21, 25]:
        body = json.dumps({f'field{i:02}': value for i in range(count)})
        entries.append({
            'startedDateTime': stamp, 'time': 0, 'cache': {},
            'timings': {'send': 0, 'wait': 0, 'receive': 0},
            'request': {
                'method': 'GET',
                'url': f'https://app.example/api/config{count}',
                'httpVersion': 'HTTP/1.1', 'headers': [],
                'queryString': [], 'cookies': [],
                'headersSize': -1, 'bodySize': 0},
            'response': {
                'status': 200, 'statusText': 'OK', 'httpVersion': 'HTTP/1.1',
                'headers': [], 'cookies': [], 'redirectURL': '',
                'headersSize': -1, 'bodySize': -1,
                'content': {'mimeType': 'application/json',
                            'size': len(body.encode()), 'text': body}}})
    (root / f'{name}.har').write_text(json.dumps({'log': {
        'version': '1.2', 'creator': {'name': 'verifier', 'version': '1'},
        'entries': entries}}))
PY
node scripts/rea.mjs inspect-web-network-capture "$work/capture.har" har --json > "$work/capture.json"
node scripts/rea.mjs inspect-web-network-capture "$work/probe.har" har --json > "$work/probe.json"
python3 - <<'PY'
import json, os, pathlib
root = pathlib.Path(os.environ['work'])
(root / 'view.json').write_text(json.dumps({
    'source': {'kind': 'inline', 'evidence': json.loads((root / 'capture.json').read_text())},
    'view': {'kind': 'environment-manifest'}}))
PY
node scripts/rea.mjs inspect-analysis-view "$work/view.json" --json > "$work/manifest.json"
node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json > "$work/compare.json"
node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json --full-output > "$work/full.json"
python3 - <<'PY'
import json, os, pathlib
root = pathlib.Path(os.environ['work'])
report = json.loads((root / 'compare.json').read_text())
steps = report['probes'][0]['results']
print('status:', report['status'])
for step in steps:
    print(step['path'], 'returned differences:', len(step['differences']),
          'last pointer:', step['differences'][-1]['pointer'], 'equal:', step['equal'])
print('All three difference arrays identical:', steps[0]['differences'] == steps[1]['differences'] == steps[2]['differences'])
print('Step fields:', ', '.join(steps[2]))
full = json.loads((root / 'full.json').read_text())
print('Full-output data identical:', full['data'] == report)
print('Omitted field20 present anywhere in full output:', 'field20' in (root / 'full.json').read_text())
for name in ['capture', 'probe']:
    entries = json.loads((root / f'{name}.har').read_text())['log']['entries']
    bodies = [json.loads(entry['response']['content']['text']) for entry in entries]
    if name == 'capture':
        captured = bodies
    else:
        print('Independent source-body difference counts:', [
            sum(left[key] != right[key] for key in left)
            for left, right in zip(captured, bodies)])
PY

Inspect the emitted summary shown below. Independent comparison of the source HAR response bodies confirms 20, 21 and 25 changed values, while each returned difference list has length 20 and identical contents.

The issue reproduced in both independent fixture generations. Both ordinary-output runs and both full-output controls showed the omission.

Expected behavior

A capped diagnostic list should identify when entries were omitted and explain how to inspect the remaining differences, or explicitly state the recovery limitation. The repository's tool-design contract requires truncation reporting and disclosure of the ceiling and recovery.

The environment specification and public guide explicitly document a maximum of 20 differences per step. The defect concerns disclosure of actual truncation in an individual result within that documented limit. It does not require every step to return an unbounded list.

Impact and scope

The reproduced 21- and 25-field steps omit one and five diagnostic entries respectively. A caller using the comparison report cannot determine whether its 20 entries enumerate every differing pointer. The comparison still detects disagreement correctly; no false agreement was observed.

P2: this is a bounded diagnostic omission affecting inputs with more than 20 differences in a step. The original response bodies remain in the input Evidence, and manual comparison is a safe workaround; the reproduction demonstrates that comparison independently. CLI output is verified. MCP and live-application recording were not exercised.

REA version

Exact output of node scripts/rea.mjs --version:

6.8.0

Source commit: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1; arm64. Built-in HAR inspection and environment projection only; no native analysis provider or GUI application was used.

Target and artifact context

Two synthetic HAR 1.2 files containing matching GET paths and JSON response objects. The source values are all 0 and the corresponding probe values are all 1. The objects have 20, 21 and 25 fields and contain no identifiers or clock fields. All inputs are local and the recorded URLs are never fetched.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The reproduction block emitted:

status: complete
/api/config20 returned differences: 20 last pointer: /field19 equal: False
/api/config21 returned differences: 20 last pointer: /field19 equal: False
/api/config25 returned differences: 20 last pointer: /field19 equal: False
All three difference arrays identical: True
Step fields: ordinal, method, path, live_status, tool, env_status, equal, equal_ignoring_order, categories, differences, ignored_query
Full-output data identical: True
Omitted field20 present anywhere in full output: False
Independent source-body difference counts: [20, 21, 25]

Baseline source references:

docs/tool-design.md:80-84 requires reporting truncation and naming the ceiling and recovery.
docs/specs/agent-environments/fidelity-compare/spec.md:87-89 and docs/agent-environments.md:1139-1141 explicitly document the 20-entry maximum.
src/cli/environmentCommands.ts:557-593 loads the Evidence, invokes the comparison, and returns its report.
src/application/environment/environmentCompareDiff.ts:60-61 defines the step difference limit.
src/application/environment/environmentCompare.ts:283-302 determines equality and categories from the full differences, then retains only the first 20 entries in each returned step.
src/application/environment/environmentCompare.ts:70-85 defines the step fields without truncation, total or continuation metadata. Lines 411-440 count affected steps per category, and lines 551-562 construct the completed report.

Typed outcome or recovery expected

When the full difference list exceeds the per-step ceiling, include an explicit omission fact and enough count or recovery information to let a caller assess that step's diagnostic coverage. Retained paging is another possible implementation if the complete list is preserved.

Analysis or suspected cause

Source-derived explanation: the report projection discards entries after the per-step ceiling without transferring an omission fact into the returned step. Category counts preserve which steps differ, rather than how many individual differences were computed. Global output formatting cannot recover entries discarded before formatting, consistent with the executed full-output control.

Additional context

Duplicate searches covered open and closed issues, issue comments, and open, closed and merged pull requests using environment-compare, STEP_DIFFERENCE_LIMIT, truncation, differences, and cap queries. The related comparison reports #6551, #6564 and #6565 concern different equality defects. The original feature PR #5699, replacement PR #6164 and related open PR #7161 contained no matching truncation-disclosure report in their discussions or inline review comments. General admission-limit issues #5910 and #6176 cover different boundaries. No duplicate or previously claimed fix for this behavior was found.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7621 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior I generated capture and probe Evidence from two small HAR files, projected an environment manifest, and ran: ```sh node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json ``` The three GET steps have 20, 21 and 25 differing numeric fields respectively. Every step returned the same 20 difference entries, ending at `/field19`. No field in the step identifies a truncated list, the original difference count, or a continuation. The report returned `status: "complete"`, and all three steps correctly returned `equal: false`. Repeating with `--full-output` returned the identical report inside its envelope. The omitted `/field20` pointer appeared nowhere in that output. A consumer cannot distinguish an exhaustive 20-entry list from one that omitted further differences. ### Reproduction 1. Use the source revision and pinned Node version below with the already built runtime and locked dependencies. Run this block from the repository root. It creates only temporary fixtures and isolates REA's HOME/XDG state. ```sh set -eu work=$(mktemp -d) export work export HOME="$work/home" export XDG_CONFIG_HOME="$HOME/config" XDG_CACHE_HOME="$HOME/cache" export XDG_DATA_HOME="$HOME/data" XDG_STATE_HOME="$HOME/state" python3 - <<'PY' import json, os, pathlib, subprocess root = pathlib.Path(os.environ['work']) stamp = subprocess.check_output( ['env', 'TZ=America/Los_Angeles', 'date', '+%Y-%m-%dT%H:%M:%S%z'], text=True).strip() for name, value in [('capture', 0), ('probe', 1)]: entries = [] for count in [20, 21, 25]: body = json.dumps({f'field{i:02}': value for i in range(count)}) entries.append({ 'startedDateTime': stamp, 'time': 0, 'cache': {}, 'timings': {'send': 0, 'wait': 0, 'receive': 0}, 'request': { 'method': 'GET', 'url': f'https://app.example/api/config{count}', 'httpVersion': 'HTTP/1.1', 'headers': [], 'queryString': [], 'cookies': [], 'headersSize': -1, 'bodySize': 0}, 'response': { 'status': 200, 'statusText': 'OK', 'httpVersion': 'HTTP/1.1', 'headers': [], 'cookies': [], 'redirectURL': '', 'headersSize': -1, 'bodySize': -1, 'content': {'mimeType': 'application/json', 'size': len(body.encode()), 'text': body}}}) (root / f'{name}.har').write_text(json.dumps({'log': { 'version': '1.2', 'creator': {'name': 'verifier', 'version': '1'}, 'entries': entries}})) PY node scripts/rea.mjs inspect-web-network-capture "$work/capture.har" har --json > "$work/capture.json" node scripts/rea.mjs inspect-web-network-capture "$work/probe.har" har --json > "$work/probe.json" python3 - <<'PY' import json, os, pathlib root = pathlib.Path(os.environ['work']) (root / 'view.json').write_text(json.dumps({ 'source': {'kind': 'inline', 'evidence': json.loads((root / 'capture.json').read_text())}, 'view': {'kind': 'environment-manifest'}})) PY node scripts/rea.mjs inspect-analysis-view "$work/view.json" --json > "$work/manifest.json" node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json > "$work/compare.json" node scripts/rea.mjs environment-compare "$work/manifest.json" --evidence "$work/capture.json" --probe "$work/probe.json" --json --full-output > "$work/full.json" python3 - <<'PY' import json, os, pathlib root = pathlib.Path(os.environ['work']) report = json.loads((root / 'compare.json').read_text()) steps = report['probes'][0]['results'] print('status:', report['status']) for step in steps: print(step['path'], 'returned differences:', len(step['differences']), 'last pointer:', step['differences'][-1]['pointer'], 'equal:', step['equal']) print('All three difference arrays identical:', steps[0]['differences'] == steps[1]['differences'] == steps[2]['differences']) print('Step fields:', ', '.join(steps[2])) full = json.loads((root / 'full.json').read_text()) print('Full-output data identical:', full['data'] == report) print('Omitted field20 present anywhere in full output:', 'field20' in (root / 'full.json').read_text()) for name in ['capture', 'probe']: entries = json.loads((root / f'{name}.har').read_text())['log']['entries'] bodies = [json.loads(entry['response']['content']['text']) for entry in entries] if name == 'capture': captured = bodies else: print('Independent source-body difference counts:', [ sum(left[key] != right[key] for key in left) for left, right in zip(captured, bodies)]) PY ``` 2. Inspect the emitted summary shown below. Independent comparison of the source HAR response bodies confirms 20, 21 and 25 changed values, while each returned difference list has length 20 and identical contents. The issue reproduced in both independent fixture generations. Both ordinary-output runs and both full-output controls showed the omission. ### Expected behavior A capped diagnostic list should identify when entries were omitted and explain how to inspect the remaining differences, or explicitly state the recovery limitation. The repository's tool-design contract requires truncation reporting and disclosure of the ceiling and recovery. The environment specification and public guide explicitly document a maximum of 20 differences per step. The defect concerns disclosure of actual truncation in an individual result within that documented limit. It does not require every step to return an unbounded list. ### Impact and scope The reproduced 21- and 25-field steps omit one and five diagnostic entries respectively. A caller using the comparison report cannot determine whether its 20 entries enumerate every differing pointer. The comparison still detects disagreement correctly; no false agreement was observed. P2: this is a bounded diagnostic omission affecting inputs with more than 20 differences in a step. The original response bodies remain in the input Evidence, and manual comparison is a safe workaround; the reproduction demonstrates that comparison independently. CLI output is verified. MCP and live-application recording were not exercised. ### REA version Exact output of `node scripts/rea.mjs --version`: ```text 6.8.0 ``` Source commit: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; macOS 27.0.1; arm64. Built-in HAR inspection and environment projection only; no native analysis provider or GUI application was used. ### Target and artifact context Two synthetic HAR 1.2 files containing matching GET paths and JSON response objects. The source values are all 0 and the corresponding probe values are all 1. The objects have 20, 21 and 25 fields and contain no identifiers or clock fields. All inputs are local and the recorded URLs are never fetched. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The reproduction block emitted: ```text status: complete /api/config20 returned differences: 20 last pointer: /field19 equal: False /api/config21 returned differences: 20 last pointer: /field19 equal: False /api/config25 returned differences: 20 last pointer: /field19 equal: False All three difference arrays identical: True Step fields: ordinal, method, path, live_status, tool, env_status, equal, equal_ignoring_order, categories, differences, ignored_query Full-output data identical: True Omitted field20 present anywhere in full output: False Independent source-body difference counts: [20, 21, 25] ``` Baseline source references: - `docs/tool-design.md:80-84` requires reporting truncation and naming the ceiling and recovery. - `docs/specs/agent-environments/fidelity-compare/spec.md:87-89` and `docs/agent-environments.md:1139-1141` explicitly document the 20-entry maximum. - `src/cli/environmentCommands.ts:557-593` loads the Evidence, invokes the comparison, and returns its report. - `src/application/environment/environmentCompareDiff.ts:60-61` defines the step difference limit. - `src/application/environment/environmentCompare.ts:283-302` determines equality and categories from the full differences, then retains only the first 20 entries in each returned step. - `src/application/environment/environmentCompare.ts:70-85` defines the step fields without truncation, total or continuation metadata. Lines 411-440 count affected steps per category, and lines 551-562 construct the completed report. ### Typed outcome or recovery expected When the full difference list exceeds the per-step ceiling, include an explicit omission fact and enough count or recovery information to let a caller assess that step's diagnostic coverage. Retained paging is another possible implementation if the complete list is preserved. ### Analysis or suspected cause Source-derived explanation: the report projection discards entries after the per-step ceiling without transferring an omission fact into the returned step. Category counts preserve which steps differ, rather than how many individual differences were computed. Global output formatting cannot recover entries discarded before formatting, consistent with the executed full-output control. ### Additional context Duplicate searches covered open and closed issues, issue comments, and open, closed and merged pull requests using environment-compare, STEP_DIFFERENCE_LIMIT, truncation, differences, and cap queries. The related comparison reports #6551, #6564 and #6565 concern different equality defects. The original feature PR #5699, replacement PR #6164 and related open PR #7161 contained no matching truncation-disclosure report in their discussions or inline review comments. General admission-limit issues #5910 and #6176 cover different boundaries. No duplicate or previously claimed fix for this behavior was found.

bsakaguchi referenced this issue

2026-10-31 01:13:25 +00:00

[PR #4549] [MERGED] fix(objc): isolate empty protocol selectors and category class names #6301

bsakaguchi closed this issue

2026-11-03 07:16:59 +00:00

bsakaguchi referenced this issue

2026-11-05 03:40:04 +00:00

[GH-ISSUE #8945] [Bug] inspect-native-dispatch-metadata drops class and metaclass rows when an adopted-protocol list exceeds its count limit #7793