morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5465] [Bug] Export shape comparison gives a selected export an unexplained unknown reason #3323

New issue

Closed

opened 2026-10-31 00:39:09 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:39:09 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5465

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

node scripts/rea.mjs compare-javascript-export-shapes <request.json> --format json returns an unknown comparison when one exact export selector is missing. The selected side's unknown-value reason is Left export selection is selected. (or its right-side equivalent). This describes successful selection rather than the obstacle to comparing its shape.

An independent verifier ran the built CLI against two local JavaScript fixtures. Both missing-selector directions exited 0 and reproduced this wording; the both-selected control returned the expected field change. The selector statuses and general row limitation are correct.

Reproduction

The verifier ran two application analyses and three comparisons with the fixture contents and requests below. Paths have been replaced with a relative disposable directory for privacy. Run from the built source checkout with Node.js and Python 3; choose an unused directory name if shape-repro already exists. Application code is statically analyzed, never invoked.

python3 - <<'PY'
import json, os, pathlib, subprocess
root = pathlib.Path('shape-repro').resolve()
root.mkdir()
env = dict(os.environ)
for key, directory in [('HOME', 'home'), ('XDG_CONFIG_HOME', 'config'),
                       ('XDG_DATA_HOME', 'data'), ('XDG_CACHE_HOME', 'cache'),
                       ('XDG_STATE_HOME', 'state')]:
    path = root / directory
    path.mkdir()
    env[key] = str(path)

def rea(label, *args):
    result = subprocess.run(['node', 'scripts/rea.mjs', *args], env=env,
                            capture_output=True, text=True, check=True)
    (root / (label + '.json')).write_text(result.stdout)
    return json.loads(result.stdout)

applications = []
for side, value in [('left', 1), ('right', 2)]:
    app = root / side
    app.mkdir()
    (app / 'package.json').write_text(json.dumps({
        'name': 'shape-verifier-' + side, 'version': '1.0.0',
        'main': 'main.js', 'type': 'module'}))
    (app / 'main.js').write_text(
        'export function makeUser() { return {kind: "user", id: '
        + str(value) + '}; }\n')
    applications.append(rea(side, 'analyze-javascript-application',
                            str(app), '--format', 'json'))
for label, left_name, right_name in [
    ('missing-right', 'makeUser', 'nope'),
    ('missing-left', 'nope', 'makeUser'),
    ('both-selected', 'makeUser', 'makeUser')]:
    request = root / (label + '.input.json')
    request.write_text(json.dumps({
        'left': applications[0], 'right': applications[1],
        'left_module_path': 'main.js', 'left_export_name': left_name,
        'right_module_path': 'main.js', 'right_export_name': right_name}))
    result = rea(label, 'compare-javascript-export-shapes', str(request),
                 '--format', 'json')['normalized_result']
    print(label, result['left']['status'], result['right']['status'])
    print(json.dumps({'summary': result['summary'], 'changes': result['changes']},
                     indent=2))
PY

Expected behavior

Keep the exact selection policy, candidate inventory and conservative unknown result. When the selected side has unknown availability because its counterpart cannot be selected, its reason should explain that comparison obstacle.

Source-derived expectation: docs/javascript-application-workflows.md:354 documents unknown results and candidate inventory for missing or duplicate exact selectors. docs/tool-design.md:46 requires meaningful returned facts, coverage and limitations. The schema requires a nonempty reason, but does not enforce its explanatory meaning.

Impact and scope

P3 wording defect in both tested missing-selector directions. The comparison outcome, selector statuses and general row limitation remain correct. Callers can identify the missing counterpart from the statuses, candidate inventory and row limitation. No incorrect export selection or data loss was observed. MCP and ambiguous-selector cases were not replayed.

REA version

6.7.0; built source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8.

Runtime and platform

Node.js v26.12.0; darwin arm64. No native analysis provider involved.

Target and artifact context

Two inert local ESM application directories, each containing package.json and main.js. Both export makeUser, returning {kind: "user", id: 1} on the left and {kind: "user", id: 2} on the right. Complete CLI-produced application Evidence is supplied to the comparison.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed projection of normalized_result for left makeUser, right nope:

{
  "left": {"status": "selected"},
  "right": {"status": "missing"},
  "summary": {"added": 0, "removed": 0, "changed": 0, "unknown": 1},
  "changes": [{
    "status": "unknown",
    "path": "",
    "left": {"availability": "unknown", "reason": "Left export selection is selected."},
    "right": {"availability": "unknown", "reason": "Right export selection is missing."},
    "limitations": ["Both selectors must resolve one exact export with trusted static return shapes before comparison."]
  }]
}

The mirrored request (nope / makeUser) returned the same summary and Right export selection is selected.. The control (makeUser / makeUser) returned selected/selected, one changed /id from literal 1 to literal 2 paired by /kind = user, and zero unknown changes. Both analysis commands and all three comparison commands exited 0.

Typed outcome or recovery expected

Preserve the unknown change, accurate selector statuses and candidate inventory. The selected-side reason should explain the unresolved counterpart; it should not say only selection is selected. Preserve the successful control's /id change.

Analysis or suspected cause

Source-derived reasoning: src/domain/javascript/javascriptExportShapeVariants.ts:96 uses selectionUnknownChange if either selector is unresolved. At line 260, that function calls selectionAvailability for both sides. At line 450, the helper always returns unknown availability and interpolates each side's selection status, including selected. src/domain/javascript/javascriptExportShapeComparisonSchemas.ts requires a nonempty unknown reason; this is not a schema-validation failure.

Additional context

Prepared with AI assistance and independently reproduced through the public CLI. The submitter inspected the verifier's commands and retained results. An upstream recheck found no commits touching the cited source or documentation between the tested baseline and 052956af3416fa01d93837e54fdec3ad2739f6d1; no new runtime reproduction on that later revision is claimed. No last-known-good version has been established.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5465 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior `node scripts/rea.mjs compare-javascript-export-shapes <request.json> --format json` returns an unknown comparison when one exact export selector is missing. The selected side's unknown-value reason is `Left export selection is selected.` (or its right-side equivalent). This describes successful selection rather than the obstacle to comparing its shape. An independent verifier ran the built CLI against two local JavaScript fixtures. Both missing-selector directions exited 0 and reproduced this wording; the both-selected control returned the expected field change. The selector statuses and general row limitation are correct. ### Reproduction The verifier ran two application analyses and three comparisons with the fixture contents and requests below. Paths have been replaced with a relative disposable directory for privacy. Run from the built source checkout with Node.js and Python 3; choose an unused directory name if `shape-repro` already exists. Application code is statically analyzed, never invoked. ```sh python3 - <<'PY' import json, os, pathlib, subprocess root = pathlib.Path('shape-repro').resolve() root.mkdir() env = dict(os.environ) for key, directory in [('HOME', 'home'), ('XDG_CONFIG_HOME', 'config'), ('XDG_DATA_HOME', 'data'), ('XDG_CACHE_HOME', 'cache'), ('XDG_STATE_HOME', 'state')]: path = root / directory path.mkdir() env[key] = str(path) def rea(label, *args): result = subprocess.run(['node', 'scripts/rea.mjs', *args], env=env, capture_output=True, text=True, check=True) (root / (label + '.json')).write_text(result.stdout) return json.loads(result.stdout) applications = [] for side, value in [('left', 1), ('right', 2)]: app = root / side app.mkdir() (app / 'package.json').write_text(json.dumps({ 'name': 'shape-verifier-' + side, 'version': '1.0.0', 'main': 'main.js', 'type': 'module'})) (app / 'main.js').write_text( 'export function makeUser() { return {kind: "user", id: ' + str(value) + '}; }\n') applications.append(rea(side, 'analyze-javascript-application', str(app), '--format', 'json')) for label, left_name, right_name in [ ('missing-right', 'makeUser', 'nope'), ('missing-left', 'nope', 'makeUser'), ('both-selected', 'makeUser', 'makeUser')]: request = root / (label + '.input.json') request.write_text(json.dumps({ 'left': applications[0], 'right': applications[1], 'left_module_path': 'main.js', 'left_export_name': left_name, 'right_module_path': 'main.js', 'right_export_name': right_name})) result = rea(label, 'compare-javascript-export-shapes', str(request), '--format', 'json')['normalized_result'] print(label, result['left']['status'], result['right']['status']) print(json.dumps({'summary': result['summary'], 'changes': result['changes']}, indent=2)) PY ``` ### Expected behavior Keep the exact selection policy, candidate inventory and conservative unknown result. When the selected side has unknown availability because its counterpart cannot be selected, its reason should explain that comparison obstacle. Source-derived expectation: `docs/javascript-application-workflows.md:354` documents unknown results and candidate inventory for missing or duplicate exact selectors. `docs/tool-design.md:46` requires meaningful returned facts, coverage and limitations. The schema requires a nonempty reason, but does not enforce its explanatory meaning. ### Impact and scope P3 wording defect in both tested missing-selector directions. The comparison outcome, selector statuses and general row limitation remain correct. Callers can identify the missing counterpart from the statuses, candidate inventory and row limitation. No incorrect export selection or data loss was observed. MCP and ambiguous-selector cases were not replayed. ### REA version 6.7.0; built source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`. ### Runtime and platform Node.js v26.12.0; darwin arm64. No native analysis provider involved. ### Target and artifact context Two inert local ESM application directories, each containing `package.json` and `main.js`. Both export `makeUser`, returning `{kind: "user", id: 1}` on the left and `{kind: "user", id: 2}` on the right. Complete CLI-produced application Evidence is supplied to the comparison. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed projection of `normalized_result` for left `makeUser`, right `nope`: ```json { "left": {"status": "selected"}, "right": {"status": "missing"}, "summary": {"added": 0, "removed": 0, "changed": 0, "unknown": 1}, "changes": [{ "status": "unknown", "path": "", "left": {"availability": "unknown", "reason": "Left export selection is selected."}, "right": {"availability": "unknown", "reason": "Right export selection is missing."}, "limitations": ["Both selectors must resolve one exact export with trusted static return shapes before comparison."] }] } ``` The mirrored request (`nope` / `makeUser`) returned the same summary and `Right export selection is selected.`. The control (`makeUser` / `makeUser`) returned selected/selected, one changed `/id` from literal 1 to literal 2 paired by `/kind` = `user`, and zero unknown changes. Both analysis commands and all three comparison commands exited 0. ### Typed outcome or recovery expected Preserve the unknown change, accurate selector statuses and candidate inventory. The selected-side reason should explain the unresolved counterpart; it should not say only `selection is selected`. Preserve the successful control's `/id` change. ### Analysis or suspected cause Source-derived reasoning: `src/domain/javascript/javascriptExportShapeVariants.ts:96` uses `selectionUnknownChange` if either selector is unresolved. At line 260, that function calls `selectionAvailability` for both sides. At line 450, the helper always returns unknown availability and interpolates each side's selection status, including `selected`. `src/domain/javascript/javascriptExportShapeComparisonSchemas.ts` requires a nonempty unknown reason; this is not a schema-validation failure. ### Additional context Prepared with AI assistance and independently reproduced through the public CLI. The submitter inspected the verifier's commands and retained results. An upstream recheck found no commits touching the cited source or documentation between the tested baseline and `052956af3416fa01d93837e54fdec3ad2739f6d1`; no new runtime reproduction on that later revision is claimed. No last-known-good version has been established.

bsakaguchi closed this issue

2026-10-31 00:39:10 +00:00

bsakaguchi commented

2026-10-31 00:39:10 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with canonical issue #4168 ([Bug] Export shape comparison reports "changed" for a branch that a known constant never takes). Both identify semantic reasoning anomalies and reason attribution failures in export shape comparison models. Cataloged under Cluster BQ in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with canonical issue #4168 (`[Bug] Export shape comparison reports "changed" for a branch that a known constant never takes`). Both identify semantic reasoning anomalies and reason attribution failures in export shape comparison models. Cataloged under Cluster BQ in consolidation tracker #4330.

bsakaguchi referenced this issue

2026-10-31 00:41:08 +00:00

[GH-ISSUE #5749] [Bug] inspect-model-artifact byte-swaps unsupported GGUF versions #3494

bsakaguchi referenced this issue

2026-10-31 00:41:09 +00:00

[GH-ISSUE #5749] [Bug] inspect-model-artifact byte-swaps unsupported GGUF versions #3494

bsakaguchi referenced this issue

2026-10-31 00:52:15 +00:00

[GH-ISSUE #8097] [Feature] Inspect ONNX Runtime .ort models (ORTM FlatBuffers) in inspect_model_artifact #4670

bsakaguchi referenced this issue

2026-11-01 21:44:33 +00:00

[PR #5603] [MERGED] feat(model): inspect GGUF and safetensors declarations #6991