[PR #2097] fix(javascript): admit measured transfer retention in parent result budget #5024

New issue

Open

opened 2026-10-31 00:58:39 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:58:39 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2097
Author: @VarunGore36
Created: 10/12/2026
Status: 🔄 Open

Base: main ← Head: fix-js-retention

📝 Commits (6)

fa4226b fix(javascript): admit measured transfer retention in parent result budget
df34e41 fix(javascript): charge seal-time duplicate in transfer retention estimate
2b23ef4 fix(javascript): align transfer caps with retained expansion
bd8235e Merge branch 'main' into fix-js-retention
264e484 Merge branch 'main' into fix-js-retention
5b4d311 docs(javascript): qualify transfer retention with family measurements

📊 Changes

5 files changed (+378 additions, -7 deletions)

View changed files

📝 docs/javascript-artifact-reconstruction.md (+6 -2)
➕ experiments/analysis-efficiency/javascript-transfer-retention.md (+161 -0)
➕ experiments/analysis-efficiency/javascript-transfer-retention.py (+141 -0)
📝 src/javascript/analysis/JavaScriptAnalysisTransfer.ts (+23 -3)
📝 tests/boundary/javascript/javascriptAnalysisTransferBudgets.test.ts (+47 -2)

📄 Description

Related to #1387.

Problem

On current main, raising the documented ceilings makes large-bundle results nearly empty instead of larger. A 38 MB synthetic app with --max-semantic-nodes 1000000 --max-semantic-nodes-per-file 100000 retained 12 nodes from 10.7 M visited AST nodes (exit 0); a single 3.2 MB file retained 1 node, refusing a 210 MB / 260,560-record graph transfer against 1.4 GB free parent budget. The knob users reach for when results truncate makes truncation total, silently (exit 0, partial coverage).

Root cause

Parent admission charges transfer_bytes * 8 + records * 64 (javaScriptTransferRetentionBytes) against a parent budget that is itself already halved for host-side control and transients — an effective 16x margin with no metrology behind it. Parsed small-fact objects retain ~2.5-3x their JSON bytes, and every downstream stage (freeze, validation, digest, CLI output) is in-place or streaming, so 8x refuses work that fits several times over.

Fix

Recalibrate to 4x with the metrology documented in the code comment. The 8 MB per-record cap still bounds single-object spikes, and the halved global budget still covers host-side control and transients.

Validation (all on Linux x64, Node 24)

Admitting the recorded transfer at 3x (throwaway experiment): 100k nodes + 99k unknowns retained, 228 MB out, exit 0, 1.1 GB parent peak RSS.
With this fix (4x): same case retains 100k nodes, byte-identical 228 MB output, exit 0.
New calibration tests in tests/boundary/javascript/javascriptAnalysisTransferBudgets.test.ts encode the recorded production shape (refused at 8x, admitted at 4x) plus edge cases.
npm run check:fast passes; default-ceiling behavior unchanged (15 MB app completes as before).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2097 **Author:** [@VarunGore36](https://github.com/VarunGore36) **Created:** 10/12/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix-js-retention` --- ### 📝 Commits (6) - [`fa4226b`](https://github.com/morluto/rea/commit/fa4226ba67451c79ac328991dadd77d201004e88) fix(javascript): admit measured transfer retention in parent result budget - [`df34e41`](https://github.com/morluto/rea/commit/df34e410bada726181e8e3c11750cf140998554c) fix(javascript): charge seal-time duplicate in transfer retention estimate - [`2b23ef4`](https://github.com/morluto/rea/commit/2b23ef423753165d9abd9f0972bbbcc8e662661f) fix(javascript): align transfer caps with retained expansion - [`bd8235e`](https://github.com/morluto/rea/commit/bd8235e35de61745ab915c1ddc0cadd1928fc237) Merge branch 'main' into fix-js-retention - [`264e484`](https://github.com/morluto/rea/commit/264e48443ec15f80bacb0d79f0001e1a9d7bad61) Merge branch 'main' into fix-js-retention - [`5b4d311`](https://github.com/morluto/rea/commit/5b4d3115711409e444754b1a6571dffc74665081) docs(javascript): qualify transfer retention with family measurements ### 📊 Changes **5 files changed** (+378 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `docs/javascript-artifact-reconstruction.md` (+6 -2) ➕ `experiments/analysis-efficiency/javascript-transfer-retention.md` (+161 -0) ➕ `experiments/analysis-efficiency/javascript-transfer-retention.py` (+141 -0) 📝 `src/javascript/analysis/JavaScriptAnalysisTransfer.ts` (+23 -3) 📝 `tests/boundary/javascript/javascriptAnalysisTransferBudgets.test.ts` (+47 -2) </details> ### 📄 Description Related to #1387. ## Problem On current main, raising the documented ceilings makes large-bundle results *nearly empty* instead of larger. A 38 MB synthetic app with `--max-semantic-nodes 1000000 --max-semantic-nodes-per-file 100000` retained **12 nodes from 10.7 M visited AST nodes** (exit 0); a single 3.2 MB file retained **1 node**, refusing a 210 MB / 260,560-record graph transfer against **1.4 GB free** parent budget. The knob users reach for when results truncate makes truncation total, silently (exit 0, partial coverage). ## Root cause Parent admission charges `transfer_bytes * 8 + records * 64` (`javaScriptTransferRetentionBytes`) against a parent budget that is itself already halved for host-side control and transients — an effective 16x margin with no metrology behind it. Parsed small-fact objects retain ~2.5-3x their JSON bytes, and every downstream stage (freeze, validation, digest, CLI output) is in-place or streaming, so 8x refuses work that fits several times over. ## Fix Recalibrate to 4x with the metrology documented in the code comment. The 8 MB per-record cap still bounds single-object spikes, and the halved global budget still covers host-side control and transients. ## Validation (all on Linux x64, Node 24) - Admitting the recorded transfer at 3x (throwaway experiment): 100k nodes + 99k unknowns retained, 228 MB out, exit 0, 1.1 GB parent peak RSS. - With this fix (4x): same case retains 100k nodes, byte-identical 228 MB output, exit 0. - New calibration tests in `tests/boundary/javascript/javascriptAnalysisTransferBudgets.test.ts` encode the recorded production shape (refused at 8x, admitted at 4x) plus edge cases. - `npm run check:fast` passes; default-ceiling behavior unchanged (15 MB app completes as before). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 00:58:39 +00:00

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:42:44 +00:00

[PR #5024] feat(experiments): add MCP tool-selection eval harness #6743