morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4875] [Bug] Direct eval leaves stale exact JavaScript return values #3028

New issue

Closed

opened 2026-10-31 00:35:06 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:35:06 +00:00

Owner

Originally created by @xu91102 on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4875

Before submitting

I searched open and closed issues and related PR production diffs.
This report contains only self-authored benign fixtures and no credentials or private target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

On main 9e372c6bc914b15e2f778f969da4fd3e11483d78, analyze-javascript-application reports /result as the literal 1 for this valid ES module:

export function run() {
  let value = 1;
  eval("value = 2");
  return { result: value };
}

Running this self-authored fixture with Node returns { "result": 2 }. Changing the evaluated assignment to value = 3 makes Node return 3, but REA still reports literal 1. Feeding the two complete CLI Evidence records to compare-javascript-export-shapes, selecting app.mjs / run on each side, produces changes: [], unknown: 0, and coverage.status: "complete-within-inputs".

Reproduction

Save the module above as left/app.mjs; save the same module with value = 3 as right/app.mjs.
Run rea analyze-javascript-application <directory> --json separately for both directories.
Read the export-return-shapes observation for run: both contain { path: "/result", state: "literal", value: 1 }.
Execute the self-authored modules directly with Node as an independent control, then compare the two returned REA Evidence records with exact selectors through compare-javascript-export-shapes.

I also reproduced stale literal 1 when direct eval mutates a property of a const object and when a called nested closure evaluates the assignment to an outer binding. All four affected variants completed successfully but exposed stale values; all input SHA-256 values were unchanged. Four controls (ordinary literal, indirect eval, optional eval, and an unrelated sibling function) returned the correct literals. REA itself did not execute the JavaScript.

Expected behavior

Direct eval can change reachable bindings and object members. The existing execution-free value analysis should retain that uncertainty, rather than emit an exact stale literal in the inferred fields and callable links. This follows the documented rule that calls and unresolved effects remain partial/unknown in export return-shape comparison. Unaffected scopes and proven captures should retain their existing facts.

Impact and scope

Confirmed through the compiled public CLI on these small local fixtures: the analysis presents a value contradicted by execution of the same self-authored code, and a downstream comparison loses the uncertainty. This is a static evidence correctness issue; no execution capability is requested.

REA version

Source main 9e372c6bc914b15e2f778f969da4fd3e11483d78.

Runtime and platform

Node 24.18.0, macOS arm64. Windows/Linux, packaged clients, model/agent workflows and runtime frequency in third-party applications: NOT RUN.

Interface / earliest failure stage

CLI / analysis result production.

The existing dynamic-scope frontier records eval, but the shared binding/value representation still retains the initializer as exact after eval. I read the related scope/value producer diffs in merged #538 and open #2094; they do not add this invalidation. I have not run those foreign heads or claimed broader coverage. I will prepare a focused correction at the shared producing boundary, preserving uncertainty rather than evaluating the target source.

Comparison-contract qualification

The comparison intentionally omits matching unknown projections with unchanged property presence, as documented. Its complete-within-inputs coverage does not establish runtime equivalence or exact field values. The observed stale literal 1 in the parent analysis is the defect here; I am not proposing a change to the matching-unknown comparison contract. Validation will check the parent fields and also compare an eval-affected field against a known literal control.

Follow-up: main fix and preserved local investigation

Maintainer commit 9ac76d3d independently addresses this root in the shared semantic owner, and the issue was closed by the maintainer. I read its actual production diff. I have stopped delivery of my overlapping local implementation; no pull request or fork push was created for it.

Before that main update was captured, my local experimental version passed 1,288 checks across 51 distinct files, including compiled CLI and official stdio MCP controls. Fourteen explicit benign Node/CLI cases showed no wrong singleton literal after the local change, and inputs remained unchanged. Those results apply only to my frozen base 9e372c6b plus my uncommitted patch. I have not run the new main commit and do not claim that its full behavior matches my additional deferred-callback, capture, alias-depth or budget tests. The maintainer commit is not adoption or merge of my unpublished patch.

Originally created by @xu91102 on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4875 ### Before submitting - [x] I searched open and closed issues and related PR production diffs. - [x] This report contains only self-authored benign fixtures and no credentials or private target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior On main `9e372c6bc914b15e2f778f969da4fd3e11483d78`, `analyze-javascript-application` reports `/result` as the literal **1** for this valid ES module: ```js export function run() { let value = 1; eval("value = 2"); return { result: value }; } ``` Running this self-authored fixture with Node returns `{ "result": 2 }`. Changing the evaluated assignment to `value = 3` makes Node return 3, but REA still reports literal 1. Feeding the two complete CLI Evidence records to `compare-javascript-export-shapes`, selecting `app.mjs` / `run` on each side, produces `changes: []`, `unknown: 0`, and `coverage.status: "complete-within-inputs"`. ### Reproduction 1. Save the module above as `left/app.mjs`; save the same module with `value = 3` as `right/app.mjs`. 2. Run `rea analyze-javascript-application <directory> --json` separately for both directories. 3. Read the `export-return-shapes` observation for `run`: both contain `{ path: "/result", state: "literal", value: 1 }`. 4. Execute the self-authored modules directly with Node as an independent control, then compare the two returned REA Evidence records with exact selectors through `compare-javascript-export-shapes`. I also reproduced stale literal 1 when direct eval mutates a property of a `const` object and when a called nested closure evaluates the assignment to an outer binding. All four affected variants completed successfully but exposed stale values; all input SHA-256 values were unchanged. Four controls (ordinary literal, indirect eval, optional eval, and an unrelated sibling function) returned the correct literals. REA itself did not execute the JavaScript. ### Expected behavior Direct eval can change reachable bindings and object members. The existing execution-free value analysis should retain that uncertainty, rather than emit an exact stale literal in the inferred fields and callable links. This follows the documented rule that calls and unresolved effects remain partial/unknown in [export return-shape comparison](https://github.com/morluto/rea/blob/9e372c6bc914b15e2f778f969da4fd3e11483d78/docs/javascript-application-workflows.md#export-return-shape-comparison). Unaffected scopes and proven captures should retain their existing facts. ### Impact and scope Confirmed through the compiled public CLI on these small local fixtures: the analysis presents a value contradicted by execution of the same self-authored code, and a downstream comparison loses the uncertainty. This is a static evidence correctness issue; no execution capability is requested. ### REA version Source main `9e372c6bc914b15e2f778f969da4fd3e11483d78`. ### Runtime and platform Node 24.18.0, macOS arm64. Windows/Linux, packaged clients, model/agent workflows and runtime frequency in third-party applications: NOT RUN. ### Interface / earliest failure stage CLI / analysis result production. ### Analysis and related work The existing dynamic-scope frontier records eval, but the shared binding/value representation still retains the initializer as exact after eval. I read the related scope/value producer diffs in merged #538 and open #2094; they do not add this invalidation. I have not run those foreign heads or claimed broader coverage. I will prepare a focused correction at the shared producing boundary, preserving uncertainty rather than evaluating the target source. ### Comparison-contract qualification The comparison intentionally omits matching unknown projections with unchanged property presence, as documented. Its complete-within-inputs coverage does not establish runtime equivalence or exact field values. The observed stale literal 1 in the parent analysis is the defect here; I am not proposing a change to the matching-unknown comparison contract. Validation will check the parent fields and also compare an eval-affected field against a known literal control. ### Follow-up: main fix and preserved local investigation Maintainer commit [`9ac76d3d`](https://github.com/morluto/rea/commit/9ac76d3d9486856d25395b413d7ac5ffa744cad0) independently addresses this root in the shared semantic owner, and the issue was closed by the maintainer. I read its actual production diff. I have stopped delivery of my overlapping local implementation; no pull request or fork push was created for it. Before that main update was captured, my local experimental version passed 1,288 checks across 51 distinct files, including compiled CLI and official stdio MCP controls. Fourteen explicit benign Node/CLI cases showed no wrong singleton literal after the local change, and inputs remained unchanged. Those results apply only to my frozen base `9e372c6b` plus my uncommitted patch. I have not run the new main commit and do not claim that its full behavior matches my additional deferred-callback, capture, alias-depth or budget tests. The maintainer commit is not adoption or merge of my unpublished patch.

bsakaguchi closed this issue

2026-10-31 00:35:07 +00:00

bsakaguchi commented

2026-10-31 00:35:08 +00:00

Author

Owner

@yuvraj-044 commented on GitHub (Oct 17, 2026):

assign this to me

@yuvraj-044 commented on GitHub (Oct 17, 2026): assign this to me

bsakaguchi commented

2026-10-31 00:35:08 +00:00

Author

Owner

@morluto commented on GitHub (Oct 17, 2026):

Fixed in 9ac76d3d9 on main.

The shared semantic owner now indexes direct eval effects and invalidates stale values, read provenance and reference facts. It separates lexical rebinding, sloppy eval-introduced names and potentially aliased object mutation. Consumers reuse those facts; no synthetic assignments or evaluated-payload parsing/execution is introduced. Indirect/optional/shadowed eval, unrelated activations, immutable shapes and prior primitive captures have controls. The existing unknown comparison contract is unchanged.

Validation: 306 focused checks across 16 files pass, including compiled CLI and stdio MCP return projections and child-process consumers. On the preceding SQLite commit, 9 of the selected 23 checks fail and 14 controls pass. Static, docs and test discovery (1559 files) pass. Two scoped source reviews found no remaining blocker after alias/shadowing corrections.

Twelve small completed public CLI tasks compare fc847ffe3 and this fix. Warm two-repeat medians (baseline → fix): known API elapsed 1.299→1.409s, CPU 1.341→1.451s, sampled process-family peak RSS 516.9→527.0MB; eval case elapsed 1.503→1.364s, CPU 1.492→1.437s, RSS 521.0→524.4MB. Estimated cl100k complete reply tokens: known 10703→10703; eval 12770→12857; selected request 54 and progress 481 for both. Alternating warm filesystem repeats, not true cold measurements; no significance, speedup or whole-agent cost claim. Large-source query costs remain unmeasured.

Scope: authored fixtures parsed as data; no analyzed JavaScript execution. Object reachability deliberately remains conservative where disjointness is unproven. Heavy tests/profiling remain stopped.

@morluto commented on GitHub (Oct 17, 2026): Fixed in `9ac76d3d9` on `main`. The shared semantic owner now indexes direct eval effects and invalidates stale values, read provenance and reference facts. It separates lexical rebinding, sloppy eval-introduced names and potentially aliased object mutation. Consumers reuse those facts; no synthetic assignments or evaluated-payload parsing/execution is introduced. Indirect/optional/shadowed eval, unrelated activations, immutable shapes and prior primitive captures have controls. The existing unknown comparison contract is unchanged. Validation: 306 focused checks across 16 files pass, including compiled CLI and stdio MCP return projections and child-process consumers. On the preceding SQLite commit, 9 of the selected 23 checks fail and 14 controls pass. Static, docs and test discovery (1559 files) pass. Two scoped source reviews found no remaining blocker after alias/shadowing corrections. Twelve small completed public CLI tasks compare `fc847ffe3` and this fix. Warm two-repeat medians (baseline → fix): known API elapsed 1.299→1.409s, CPU 1.341→1.451s, sampled process-family peak RSS 516.9→527.0MB; eval case elapsed 1.503→1.364s, CPU 1.492→1.437s, RSS 521.0→524.4MB. Estimated cl100k complete reply tokens: known 10703→10703; eval 12770→12857; selected request 54 and progress 481 for both. Alternating warm filesystem repeats, not true cold measurements; no significance, speedup or whole-agent cost claim. Large-source query costs remain unmeasured. Scope: authored fixtures parsed as data; no analyzed JavaScript execution. Object reachability deliberately remains conservative where disjointness is unproven. Heavy tests/profiling remain stopped.

bsakaguchi referenced this issue

2026-10-31 00:40:20 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-10-31 01:07:38 +00:00

[PR #3028] feat(cli): run ordered tool batches in one provider session #5567

bsakaguchi referenced this issue

2026-11-03 00:29:28 +00:00

[GH-ISSUE #3442] [Workflow] Consider raising concurrent open PR cap or allowlisting active contributors #1285