[GH-ISSUE #543] [Design] Decide whether the replay sandbox earns its cost, or most of it should be inlined #277

New issue

Closed

opened 2026-10-15 18:23:22 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:23:22 +00:00

Owner

Originally created by @morluto on GitHub (Oct 5, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/543

Originally assigned to: @N0zoM1z0 on GitHub.

Summary

A maintainer question about scope, raised during the contributor-PR cleanup sweep. Not a defect report — a design decision that needs recording before more effort goes into the sandbox.

Is the dedicated replay sandbox pulling its weight, or should most of what it does be inlined?

Background

src/replay/ currently provides a deterministic JavaScript execution environment: a vm.createContext worker with curated globals, seeded randomness, a frozen clock, ESM/CJS module bridging, deny-listed capabilities, and a JSON projection with depth/node budgets. It exists so REA can answer "what does this code do when run?" — returning values, branch behaviour, and left-vs-right outcome digests — rather than only reasoning statically.

The alternative to running it is node on the artifact directly, which is nondeterministic, unsandboxed, and unrepeatable.

The tension

The sandbox exists to make execution safe, repeatable and comparable, but it carries a long tail of fidelity work. From this sweep alone:

zero-seed xorshift collapse (Math.random returning a constant)
Date.name/length and Math prototype/descriptor identity
Intl/timezone leakage across hosts
array-hole projection, __proto__ own properties, symbol keys
live bindings vs. SyntheticModule snapshots
undefined vs. holes producing different outcomes for the same position

Each is real, and each fix is small — but the pattern suggests the cost is structural, not incidental: any construct whose behaviour the sandbox cannot faithfully reproduce becomes a new correctness risk, because a wrong digest is worse than no digest.

Questions to settle

Which analyst questions actually require execution? If callers mainly want static structure plus a few runtime observations, a full sandbox may be more surface than the questions justify.
What is the fallback when the sandbox cannot model something faithfully? Today some paths raise serialization_error, others silently dropped data (the symbol-key case). A single, explicit "cannot model" outcome would be more trustworthy than per-case best effort.
Should replay be a first-class capability or a diagnostic aid? The answer changes how much fidelity budget it deserves, and whether its limits belong in the contract.
What is the minimal interface that earns its keep? Worth naming concretely: given a module graph and a case, return typed outcomes with explicit unknowns. Anything larger invites the drift we keep finding.

Notes

Not a proposal to delete anything — a proposal to decide deliberately, and to record the decision so the next fidelity fix is a consequence of a policy rather than another patch. The recent sandbox fixes (#539, #540, #541) are all correct and independent of how this resolves.

Originally created by @morluto on GitHub (Oct 5, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/543 Originally assigned to: @N0zoM1z0 on GitHub. ## Summary A maintainer question about scope, raised during the contributor-PR cleanup sweep. Not a defect report — a design decision that needs recording before more effort goes into the sandbox. **Is the dedicated replay sandbox pulling its weight, or should most of what it does be inlined?** ## Background `src/replay/` currently provides a deterministic JavaScript execution environment: a `vm.createContext` worker with curated globals, seeded randomness, a frozen clock, ESM/CJS module bridging, deny-listed capabilities, and a JSON projection with depth/node budgets. It exists so REA can answer "what does this code *do* when run?" — returning values, branch behaviour, and left-vs-right outcome digests — rather than only reasoning statically. The alternative to running it is `node` on the artifact directly, which is nondeterministic, unsandboxed, and unrepeatable. ## The tension The sandbox exists to make execution safe, repeatable and comparable, but it carries a long tail of fidelity work. From this sweep alone: - zero-seed xorshift collapse (`Math.random` returning a constant) - `Date.name`/`length` and `Math` prototype/descriptor identity - `Intl`/timezone leakage across hosts - array-hole projection, `__proto__` own properties, symbol keys - live bindings vs. `SyntheticModule` snapshots - `undefined` vs. holes producing different outcomes for the same position Each is real, and each fix is small — but the pattern suggests the cost is structural, not incidental: any construct whose behaviour the sandbox cannot faithfully reproduce becomes a *new* correctness risk, because a wrong digest is worse than no digest. ## Questions to settle 1. **Which analyst questions actually require execution?** If callers mainly want static structure plus a few runtime observations, a full sandbox may be more surface than the questions justify. 2. **What is the fallback when the sandbox cannot model something faithfully?** Today some paths raise `serialization_error`, others silently dropped data (the symbol-key case). A single, explicit "cannot model" outcome would be more trustworthy than per-case best effort. 3. **Should replay be a first-class capability or a diagnostic aid?** The answer changes how much fidelity budget it deserves, and whether its limits belong in the contract. 4. **What is the minimal interface that earns its keep?** Worth naming concretely: given a module graph and a case, return typed outcomes with explicit unknowns. Anything larger invites the drift we keep finding. ## Notes Not a proposal to delete anything — a proposal to decide deliberately, and to record the decision so the next fidelity fix is a consequence of a policy rather than another patch. The recent sandbox fixes (#539, #540, #541) are all correct and independent of how this resolves.

bsakaguchi

2026-10-15 18:23:22 +00:00

closed this issue
added the
provider-request

design

area: native
labels

bsakaguchi commented

2026-10-15 18:23:23 +00:00

Author

Owner

@morluto commented on GitHub (Oct 5, 2026):

pls review @N0zoM1z0

@morluto commented on GitHub (Oct 5, 2026): pls review @N0zoM1z0

bsakaguchi commented

2026-10-15 18:23:23 +00:00

Author

Owner

@morluto commented on GitHub (Oct 5, 2026):

Decision implemented in #572, now merged.

REA exposes direct inspection, capture, and comparison tools. Agents compose experiments with ordinary commands, scripts, and local fixture servers. The dedicated JavaScript replay sandbox, Node characterization prepare/execute flow, and process replay machinery have been removed; the design is recorded in AGENTS.md and the superseded replay ADR.

Closing as completed: the scope decision is made and implemented.

@morluto commented on GitHub (Oct 5, 2026): Decision implemented in [#572](https://github.com/morluto/rea/pull/572), now merged. REA exposes direct inspection, capture, and comparison tools. Agents compose experiments with ordinary commands, scripts, and local fixture servers. The dedicated JavaScript replay sandbox, Node characterization prepare/execute flow, and process replay machinery have been removed; the design is recorded in AGENTS.md and the superseded replay ADR. Closing as completed: the scope decision is made and implemented.

bsakaguchi commented

2026-10-15 18:23:23 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 5, 2026):

ok so no need for me to review right? gotcha!

@N0zoM1z0 commented on GitHub (Oct 5, 2026): ok so no need for me to review right? gotcha!

bsakaguchi referenced this issue

2026-10-15 18:28:29 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-16 21:12:29 +00:00