[GH-ISSUE #4079] [Feature] Reuse CLI compilation in the exported binding acceptance matrix #1609

New issue

Closed

opened 2026-10-16 21:07:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:07:56 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4079

Before submitting

Searched related issues and PRs; follow-up to the slow-family investigation in #2190.
Reviewed contribution scope; this is test infrastructure within existing CLI/MCP boundaries.
No credentials or private target data included.

Area

Other — CI/test execution performance.

Current limitation

At cb18d4c9c4ee399d81ebaf0888117d52d990e442, tests/acceptance/applications/exportedFunctionBindingsCli.test.ts repeatedly launches the compiled CLI without sharing Node's compilation cache. The file already shares its stdio MCP connection. The full scheduled CI run at 47b4628bac7300e5486dd6018663b1cc5dfbf01c reports 202.177 seconds for this 92-case family; the only intervening main change is unrelated documentation.

Desired outcome and success criteria

Reuse immutable Node compilation across this file's CLI launches, using the existing CLI fixture's cache policy and a fresh owned temporary directory per file run. Preserve all 92 cases, real CLI/MCP requests, Node-loader oracles and assertions. Preserve caller-selected cache, disabled cache and child coverage. Remove the owned directory even if another teardown fails. Never cache analysis results or a prior passing test.

Expected interaction with targets

Read fixtures; launch existing CLI/MCP and Node-loader processes; write an owned temporary compilation cache. Existing application requests and fixture execution stay unchanged. No new provider capability or authority.

Verification available so far

I have a prototype or observed results.

Verification details

Executed the complete same command in alternating original/cache/original/cache order:

npm run test:focused -- tests/acceptance/applications/exportedFunctionBindingsCli.test.ts

Run	Wall seconds	CPU user + system seconds	Sampled peak family RSS MiB	Cases
Original A	227.485	384.808	1453.7	92 passed
Cache A	207.022	357.749	1418.3	92 passed
Original B	273.958	449.239	1438.6	92 passed
Cache B	210.125	363.266	1464.3	92 passed

Linux x64, 16 CPUs, 64 GiB RAM, Node 24.18.0/npm 11.16.0, locked dependencies and warm shared build cache. Each run starts fresh Vitest/MCP/CLI processes; each cache variant starts with an empty private cache and warms it only within that completed workload. No other task-owned checks ran concurrently. OS/transform caches were warm. The original environment had no compile-cache/coverage overrides.

CPU is wait-accounted child user + system time. Memory is the sampled sum of live descendant RSS, scanning /proc PPIDs with a 100 ms pause between scans: short peaks/reparented children can be missed and shared pages can be counted repeatedly. Instrumentation overhead excluded. Two runs per variant on a shared host show substantial original timing variance; this is a measured local family improvement, not a CI-wide speedup estimate. Memory shows no consistent improvement.

Observed a private cache containing 533 files/3,246,788 bytes during the first cache run and confirmed that observed private paths were removed afterward. Policy/cleanup controls: 6 passed, 1 Darwin-only skipped across cliCompileCache.test.ts, nativeDemangleCli.test.ts, and verifierCleanup.test.ts. Typecheck/lint passed. Coverage-mode family verification and PR CI remain pending at issue creation.

Token/schema/result payloads are unchanged by source inspection; agent token usage was not measured. Cold OS/dependency installation, whole-CI timing and real Hopper/Ghidra/browser or other OS performance are unmeasured.

Alternatives considered

Removing CLI assertions would weaken adapter coverage. Reusing previous analysis results would stop verifying current behavior. Increasing timeouts or retrying failures does not remove compilation work.

References and limitations

Follow-up to #2190. The scheduled full-run demangle timeout is separately tracked in #3543; this proposal does not establish its root cause or fix it.

Originally created by @N0zoM1z0 on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4079 ### Before submitting - [x] Searched related issues and PRs; follow-up to the slow-family investigation in #2190. - [x] Reviewed contribution scope; this is test infrastructure within existing CLI/MCP boundaries. - [x] No credentials or private target data included. ### Area Other — CI/test execution performance. ### Current limitation At `cb18d4c9c4ee399d81ebaf0888117d52d990e442`, `tests/acceptance/applications/exportedFunctionBindingsCli.test.ts` repeatedly launches the compiled CLI without sharing Node's compilation cache. The file already shares its stdio MCP connection. The full scheduled [CI run](https://github.com/morluto/rea/actions/runs/38714082679) at `47b4628bac7300e5486dd6018663b1cc5dfbf01c` reports 202.177 seconds for this 92-case family; the only intervening main change is unrelated documentation. ### Desired outcome and success criteria Reuse immutable Node compilation across this file's CLI launches, using the existing CLI fixture's cache policy and a fresh owned temporary directory per file run. Preserve all 92 cases, real CLI/MCP requests, Node-loader oracles and assertions. Preserve caller-selected cache, disabled cache and child coverage. Remove the owned directory even if another teardown fails. Never cache analysis results or a prior passing test. ### Expected interaction with targets Read fixtures; launch existing CLI/MCP and Node-loader processes; write an owned temporary compilation cache. Existing application requests and fixture execution stay unchanged. No new provider capability or authority. ### Verification available so far I have a prototype or observed results. ### Verification details Executed the complete same command in alternating original/cache/original/cache order: `npm run test:focused -- tests/acceptance/applications/exportedFunctionBindingsCli.test.ts` | Run | Wall seconds | CPU user + system seconds | Sampled peak family RSS MiB | Cases | | --- | ---: | ---: | ---: | ---: | | Original A | 227.485 | 384.808 | 1453.7 | 92 passed | | Cache A | 207.022 | 357.749 | 1418.3 | 92 passed | | Original B | 273.958 | 449.239 | 1438.6 | 92 passed | | Cache B | 210.125 | 363.266 | 1464.3 | 92 passed | Linux x64, 16 CPUs, 64 GiB RAM, Node 24.18.0/npm 11.16.0, locked dependencies and warm shared build cache. Each run starts fresh Vitest/MCP/CLI processes; each cache variant starts with an empty private cache and warms it only within that completed workload. No other task-owned checks ran concurrently. OS/transform caches were warm. The original environment had no compile-cache/coverage overrides. CPU is wait-accounted child user + system time. Memory is the sampled sum of live descendant RSS, scanning `/proc` PPIDs with a 100 ms pause between scans: short peaks/reparented children can be missed and shared pages can be counted repeatedly. Instrumentation overhead excluded. Two runs per variant on a shared host show substantial original timing variance; this is a measured local family improvement, not a CI-wide speedup estimate. Memory shows no consistent improvement. Observed a private cache containing 533 files/3,246,788 bytes during the first cache run and confirmed that observed private paths were removed afterward. Policy/cleanup controls: 6 passed, 1 Darwin-only skipped across `cliCompileCache.test.ts`, `nativeDemangleCli.test.ts`, and `verifierCleanup.test.ts`. Typecheck/lint passed. Coverage-mode family verification and PR CI remain pending at issue creation. Token/schema/result payloads are unchanged by source inspection; agent token usage was not measured. Cold OS/dependency installation, whole-CI timing and real Hopper/Ghidra/browser or other OS performance are unmeasured. ### Alternatives considered Removing CLI assertions would weaken adapter coverage. Reusing previous analysis results would stop verifying current behavior. Increasing timeouts or retrying failures does not remove compilation work. ### References and limitations Follow-up to #2190. The scheduled full-run demangle timeout is separately tracked in #3543; this proposal does not establish its root cause or fix it.

bsakaguchi

2026-10-16 21:07:56 +00:00