[PR #2349] [MERGED] test(analysis): verify real preset snapshot and cleanup boundaries #5181

Closed
opened 2026-10-31 01:04:37 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2349
Author: @N0zoM1z0
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/12/2026
Merged by: @N0zoM1z0

Base: main ← Head: test/analysis-preset-lifecycle-20261012


📝 Commits (3)

  • a27b45c test(analysis): verify real preset snapshot and cleanup boundaries
  • a415341 docs(analysis): record wider preset and failure verification
  • 67f817e test(analysis): verify recovered socket cleanup after retry

📊 Changes

4 files changed (+994 additions, -0 deletions)

View changed files

📝 experiments/analysis-efficiency/README.md (+4 -0)
➕ experiments/analysis-efficiency/preset-lifecycle.md (+208 -0)
➕ experiments/analysis-efficiency/verify-preset-lifecycle.mjs (+395 -0)
➕ experiments/analysis-efficiency/verify-preset-snapshot.mjs (+387 -0)

📄 Description

Summary

Part of #2016. Add reproducible real Ghidra CLI/MCP verification for the remaining task-scoped preset lifecycle boundaries: actual stored snapshot entries, profile mismatch in both directions, startup cancellation, recovery and cleanup retry.

The existing production default remains unchanged. The prior disabled-switch jq experiment changed a callee body, so this PR introduces no general fast-preset recommendation or runtime/API change.

Problem and expected behavior

Equal content-addressed replies do not prove snapshot reuse, a logical run ID does not identify a process generation, and a failed cleanup must retain its resource for another close. The new verifiers check those boundaries against real Ghidra and kernel/process observations instead of relying on those assumptions.

Change and scope

  • verify-preset-snapshot.mjs runs real default/disabled analysis, validates an actual exported query entry, checks fresh MCP and CLI reuse, rejects both wrong-profile snapshots before headless startup, and verifies original records/source bytes. It records the known headless launch command through caller-supplied strace.
  • verify-preset-lifecycle.mjs observes a token/start-identity verified JVM before endpoint readiness, cancels via the public SDK, confirms cleanup with production read-only token sweeps, and proves successful fresh work. A permission fault affects only its verified user-owned temporary project; restored permissions allow the retained-resource close retry.
  • Curated notes retain source/runtime pins, failed observer assumptions, exact scope and resource qualification. Raw JSON, traces, inputs and provider projects remain outside Git.

Contract and boundary impact

  • Semantic owner: experiment observer and cleanup lifetime; production process inspection/identity primitives are reused.
  • Contribution scope: verification of existing caller-selected, read-only Ghidra behavior in #2016. New provider/target execution/injection/protection recovery is not applicable.
  • CLI/MCP catalog: unchanged product commands, arguments, schemas, defaults and provider selection.
  • Provider/platform: real Ghidra 12.1.4/JDK 21, Linux x64; caller supplies installation/settings/strace. The permission fault requires a non-root user. Other platforms are unverified by these scripts.
  • Evidence/provenance: exact task facts, subject, parameters and profile survive valid reuse. CLI preserves the original cached ID; MCP marks cache reuse in its distinct response. Both mismatch directions preserve the original snapshot. Failed probes remain archived separately.
  • Authority/cleanup: no target execution or annotation requested. Controlled temporary directory permissions are restored; primary and cleanup failures are preserved together; success is published after cleanup callbacks finish.
  • Generated/package/installation: no package/version/dependency/install/configuration change; generated artifacts stay ignored.

Expected effects:

  • Read explicit native inputs and selected snapshots.
  • Launch existing local analysis/inspection processes.
  • Execute existing local analysis scripts; no target code or instrumentation.
  • Write new caller-selected verification files and temporarily change permissions only in the verifier-owned Ghidra project.
  • Contact network services or recover protected code.

Evidence and regression coverage

  • Installed GNU true: real default/disabled profile digests, stored analyze_function entry, exact SDK/CLI task facts, two-sided snapshot rejection, valid default restoration and headless launch counts pass.
  • Public pinned jq at the previously observed entry 0x403a50: the same complete snapshot/profile/CLI workflow passes. The selected entry agrees across profiles; this does not establish call-graph or whole-app equivalence.
  • GNU true, both profiles: actual pending startup cancellation, native start-identity proof of recovery, read-only token cleanup, real cleanup_incomplete/EACCES with retained resource, restored-permission retry and unchanged source bytes pass.
  • Actual invalid analyzer requests preserve the exact name in CLI and MCP execution_failure diagnostics. MCP retains a healthy bundled EVM peer. CLI uses an error string with sibling fields; saved replies were regraded against that actual envelope without rerunning the provider.

For evidence-bearing changes:

  • Observations, derivations, inferences and unknowns remain explicit.
  • Source identity, provenance and failed attempts are preserved.
  • Unsupported/unavailable/partial proof scopes remain visible.

Validation performed

  • Both source verifiers run against the built production baseline f76163c2467db623f9de0bda757c9d8610ca9565; their exact source hashes and private receipts are documented.
  • node --check / focused oxlint — both scripts pass with no warnings.
  • Single-task Turbo typecheck, then npm run check — pass under the unchanged one-CPU/4 GiB hard/3 GiB early-stop controls. The valid typecheck cache is reused by the full gate; its near-limit memory and soft-limit events are excluded from efficiency claims.
  • npm run docs:check — pass.
  • No production code/test contracts changed, so prior model/provider matrices are not rerun for this verifier-only increment.

Compatibility, safety and release

Existing users keep their API/default/provider choices. The scripts require explicit local inputs and new private output directories; no install or configuration migration is introduced. Revision is caller-supplied and should accompany the built/frozen runtime receipt.

Timings include discovery, tracing, profile inspection, rejection and cleanup. They verify the existing explicit JSON snapshot mechanism and are not new architecture speedups, repeated whole-question performance claims or model token measurements. Final completed-scope cgroup counters remain unknown; sampled values do not establish absent late OOM/limit events. Wider native/OS/domain cases remain scoped to their earlier checkpoints.

No release, deployment or version change. Existing REA defects remain in the requested private /tmp record.

Detailed pins, actual observations, corrections and reproduction.

Before merge

  • Concrete agreed #2016 verification outcome; scope/authority resolved.
  • Conventional title and focused diff.
  • Changed observer behavior and meaningful failures verified.
  • Source reproduction and curated documentation updated.
  • Diff reviewed for secrets, unrelated changes and unsupported claims.
  • Current-head CI and manual final review complete at 67f817e3653efad35f7044e68fbd6f163b86864b: all 36 checks succeed. CI receipt. Bot review quota is exhausted; automated approval is not claimed.

Latest-main review checkpoint

Rebased onto main 3ee6ff43c1696a5a5269bd4aa5de72f7e5f40627, which independently fixes the exact shard-6 fixture mismatch also observed on the prior main base. The related MCP file passes all five focused cases; missing coverage remains unknown. Final review additionally checks that the recovered socket directory is absent after permission-restoration cleanup retry and records its ownership manifest. Both actual profiles pass that related workflow again; broader artifact/model matrices were not repeated.

The native single-task typecheck and required fast checks pass with the unchanged 3 GiB early-stop/4 GiB hard bounds. A guarded typecheck stop and an overly restrictive 128 MiB docs heap attempt remain archived; the docs-only retry uses the existing 512 MiB Node heap and passes (434 MB sampled family peak). Final counters remain unverified. Current head is 67f817e3653efad35f7044e68fbd6f163b86864b; new current-head CI is still running.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2349 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/12/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `test/analysis-preset-lifecycle-20261012` --- ### 📝 Commits (3) - [`a27b45c`](https://github.com/morluto/rea/commit/a27b45c2a2bfa599680054a3f3c2ee878406dbf6) test(analysis): verify real preset snapshot and cleanup boundaries - [`a415341`](https://github.com/morluto/rea/commit/a415341218124e45e5a40fa22b139e263c0b928b) docs(analysis): record wider preset and failure verification - [`67f817e`](https://github.com/morluto/rea/commit/67f817e3653efad35f7044e68fbd6f163b86864b) test(analysis): verify recovered socket cleanup after retry ### 📊 Changes **4 files changed** (+994 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `experiments/analysis-efficiency/README.md` (+4 -0) ➕ `experiments/analysis-efficiency/preset-lifecycle.md` (+208 -0) ➕ `experiments/analysis-efficiency/verify-preset-lifecycle.mjs` (+395 -0) ➕ `experiments/analysis-efficiency/verify-preset-snapshot.mjs` (+387 -0) </details> ### 📄 Description ## Summary Part of #2016. Add reproducible real Ghidra CLI/MCP verification for the remaining task-scoped preset lifecycle boundaries: actual stored snapshot entries, profile mismatch in both directions, startup cancellation, recovery and cleanup retry. The existing production default remains unchanged. The prior disabled-switch jq experiment changed a callee body, so this PR introduces no general fast-preset recommendation or runtime/API change. ## Problem and expected behavior Equal content-addressed replies do not prove snapshot reuse, a logical run ID does not identify a process generation, and a failed cleanup must retain its resource for another close. The new verifiers check those boundaries against real Ghidra and kernel/process observations instead of relying on those assumptions. ## Change and scope - `verify-preset-snapshot.mjs` runs real default/disabled analysis, validates an actual exported query entry, checks fresh MCP and CLI reuse, rejects both wrong-profile snapshots before headless startup, and verifies original records/source bytes. It records the known headless launch command through caller-supplied strace. - `verify-preset-lifecycle.mjs` observes a token/start-identity verified JVM before endpoint readiness, cancels via the public SDK, confirms cleanup with production read-only token sweeps, and proves successful fresh work. A permission fault affects only its verified user-owned temporary project; restored permissions allow the retained-resource close retry. - Curated notes retain source/runtime pins, failed observer assumptions, exact scope and resource qualification. Raw JSON, traces, inputs and provider projects remain outside Git. ## Contract and boundary impact - Semantic owner: experiment observer and cleanup lifetime; production process inspection/identity primitives are reused. - Contribution scope: verification of existing caller-selected, read-only Ghidra behavior in #2016. New provider/target execution/injection/protection recovery is not applicable. - CLI/MCP catalog: unchanged product commands, arguments, schemas, defaults and provider selection. - Provider/platform: real Ghidra 12.1.4/JDK 21, Linux x64; caller supplies installation/settings/strace. The permission fault requires a non-root user. Other platforms are unverified by these scripts. - Evidence/provenance: exact task facts, subject, parameters and profile survive valid reuse. CLI preserves the original cached ID; MCP marks cache reuse in its distinct response. Both mismatch directions preserve the original snapshot. Failed probes remain archived separately. - Authority/cleanup: no target execution or annotation requested. Controlled temporary directory permissions are restored; primary and cleanup failures are preserved together; success is published after cleanup callbacks finish. - Generated/package/installation: no package/version/dependency/install/configuration change; generated artifacts stay ignored. Expected effects: - [x] Read explicit native inputs and selected snapshots. - [x] Launch existing local analysis/inspection processes. - [x] Execute existing local analysis scripts; no target code or instrumentation. - [x] Write new caller-selected verification files and temporarily change permissions only in the verifier-owned Ghidra project. - [ ] Contact network services or recover protected code. ## Evidence and regression coverage - Installed GNU `true`: real default/disabled profile digests, stored `analyze_function` entry, exact SDK/CLI task facts, two-sided snapshot rejection, valid default restoration and headless launch counts pass. - Public pinned jq at the previously observed entry `0x403a50`: the same complete snapshot/profile/CLI workflow passes. The selected entry agrees across profiles; this does not establish call-graph or whole-app equivalence. - GNU `true`, both profiles: actual pending startup cancellation, native start-identity proof of recovery, read-only token cleanup, real `cleanup_incomplete`/EACCES with retained resource, restored-permission retry and unchanged source bytes pass. - Actual invalid analyzer requests preserve the exact name in CLI and MCP `execution_failure` diagnostics. MCP retains a healthy bundled EVM peer. CLI uses an error string with sibling fields; saved replies were regraded against that actual envelope without rerunning the provider. For evidence-bearing changes: - [x] Observations, derivations, inferences and unknowns remain explicit. - [x] Source identity, provenance and failed attempts are preserved. - [x] Unsupported/unavailable/partial proof scopes remain visible. ## Validation performed - Both source verifiers run against the built production baseline `f76163c2467db623f9de0bda757c9d8610ca9565`; their exact source hashes and private receipts are documented. - `node --check` / focused `oxlint` — both scripts pass with no warnings. - Single-task Turbo typecheck, then `npm run check` — pass under the unchanged one-CPU/4 GiB hard/3 GiB early-stop controls. The valid typecheck cache is reused by the full gate; its near-limit memory and soft-limit events are excluded from efficiency claims. - `npm run docs:check` — pass. - No production code/test contracts changed, so prior model/provider matrices are not rerun for this verifier-only increment. ## Compatibility, safety and release Existing users keep their API/default/provider choices. The scripts require explicit local inputs and new private output directories; no install or configuration migration is introduced. Revision is caller-supplied and should accompany the built/frozen runtime receipt. Timings include discovery, tracing, profile inspection, rejection and cleanup. They verify the existing explicit JSON snapshot mechanism and are not new architecture speedups, repeated whole-question performance claims or model token measurements. Final completed-scope cgroup counters remain unknown; sampled values do not establish absent late OOM/limit events. Wider native/OS/domain cases remain scoped to their earlier checkpoints. No release, deployment or version change. Existing REA defects remain in the requested private `/tmp` record. [Detailed pins, actual observations, corrections and reproduction](https://github.com/morluto/rea/blob/test/analysis-preset-lifecycle-20261012/experiments/analysis-efficiency/preset-lifecycle.md). ## Before merge - [x] Concrete agreed #2016 verification outcome; scope/authority resolved. - [x] Conventional title and focused diff. - [x] Changed observer behavior and meaningful failures verified. - [x] Source reproduction and curated documentation updated. - [x] Diff reviewed for secrets, unrelated changes and unsupported claims. - [x] Current-head CI and manual final review complete at `67f817e3653efad35f7044e68fbd6f163b86864b`: all 36 checks succeed. [CI receipt](https://github.com/morluto/rea/actions/runs/38307614983). Bot review quota is exhausted; automated approval is not claimed. ## Latest-main review checkpoint Rebased onto main `3ee6ff43c1696a5a5269bd4aa5de72f7e5f40627`, which independently fixes the exact shard-6 fixture mismatch also observed on the prior main base. The related MCP file passes all five focused cases; missing coverage remains unknown. Final review additionally checks that the recovered socket directory is absent after permission-restoration cleanup retry and records its ownership manifest. Both actual profiles pass that related workflow again; broader artifact/model matrices were not repeated. The native single-task typecheck and required fast checks pass with the unchanged 3 GiB early-stop/4 GiB hard bounds. A guarded typecheck stop and an overly restrictive 128 MiB docs heap attempt remain archived; the docs-only retry uses the existing 512 MiB Node heap and passes (434 MB sampled family peak). Final counters remain unverified. Current head is `67f817e3653efad35f7044e68fbd6f163b86864b`; [new current-head CI](https://github.com/morluto/rea/actions/runs/38307614983) is still running. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:04:37 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5181
No description provided.