[GH-ISSUE #5295] [Bug] EVM cleanup errors stay incomplete after retained owners close #3228

Closed
opened 2026-10-31 00:38:04 +00:00 by bsakaguchi · 3 comments
Owner

Originally created by @morluto on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5295

Observed behavior

An EVM inspection can return a cleanup error for its private runtime root. If a later provider.close() retry removes that exact root successfully, the original returned error still reports cleanupIncomplete: true and keeps the deleted path in cleanupResources.

Reproduction

A focused controlled test uses Node v24.18.0 on Darwin arm64, temporarily emulates EVM Linux/x64 admission with a fixture /proc/self/limits response, launches a real owned local Node child through the provider seam, and makes the actual private runtime root close fail before allowing a later close to remove it. Before the fix, the test failed after successful close and confirmed root removal: expected originalError.cleanupIncomplete to be false, received true. This does not qualify a real Linux EVMole worker run.

Expected behavior

A returned cleanup error should retain an exact owner binding. Once a later retry confirms that owner is released, its live incomplete flag should clear and its live resource list should no longer include the removed path. Any underlying analysis failure and partial Evidence should remain available.

Impact and scope

Observed in a controlled root-close retry seam. Additional controlled tests now exercise real local Node process-group cleanup and descriptor/root mixed-owner retries under emulated admission. The existing Linux-gated provider assertions remain skipped on Darwin arm64. Natural OS cleanup failure frequency and real Linux EVM provider behavior remain unmeasured.

REA version

Source revision 10f1994fe1a0bbddeb846ee8078a80a483a652f1 plus the uncommitted focused fix.

Runtime and platform

Node v24.18.0; Darwin 25.3.0 arm64. Linux/x64 admission and inherited limits were emulated only inside the controlled test.

Target and artifact context

Local hex EVM bytecode carrier; worker execution controlled by the launcher seam.

Interface

Other (provider boundary test)

Earliest failure stage

Cleanup, cancellation, or lifecycle

Relevant output or evidence

npm run test:focused -- tests/boundary/evm/evmCleanupRetry.test.ts failed before the fix at the assertion for cleanupIncomplete after provider.close() succeeded and the root path returned ENOENT. It passes after the fix. Full executed output and source hash are recorded in .cache/codebase-audit/evm-pwntools-worker-retry-20261020.json.

Analysis or suspected cause

EvmoleInterfaceProvider.#pendingCleanup retained the root and worker supervisor but did not bind returned ProviderCleanupError objects to exact cleanup owners. Later #retire() could release the owner without updating previously returned errors. The fix uses AnalysisCleanupLease and AnalysisCleanupLeaseGroup at the EVM owner boundary.

Verified local fix update

Clean isolated source 10f1994fe1a0bbddeb846ee8078a80a483a652f1, with only the six retry regressions copied, failed all six before the fix. Final local focused validation passed 18 tests, 30 Linux-gated skipped across retry/provider and failure-projection tests. Controlled tests cover completed/malformed replies, exact opaque cancellation, descriptor/root and process/root release, original plus close aggregates, and prior physical failure followed by a late abort. Real Node children and actual group/filesystem cleanup are used; Linux/x64 admission and limits are emulated. No real Linux/EVMole qualification. Typecheck/lint, formatting and diff checks pass. Fix remains local and uncommitted; issue stays open.

Originally created by @morluto on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5295 ### Observed behavior An EVM inspection can return a cleanup error for its private runtime root. If a later `provider.close()` retry removes that exact root successfully, the original returned error still reports `cleanupIncomplete: true` and keeps the deleted path in `cleanupResources`. ### Reproduction A focused controlled test uses Node v24.18.0 on Darwin arm64, temporarily emulates EVM Linux/x64 admission with a fixture `/proc/self/limits` response, launches a real owned local Node child through the provider seam, and makes the actual private runtime root close fail before allowing a later close to remove it. Before the fix, the test failed after successful close and confirmed root removal: expected `originalError.cleanupIncomplete` to be false, received true. This does not qualify a real Linux EVMole worker run. ### Expected behavior A returned cleanup error should retain an exact owner binding. Once a later retry confirms that owner is released, its live incomplete flag should clear and its live resource list should no longer include the removed path. Any underlying analysis failure and partial Evidence should remain available. ### Impact and scope Observed in a controlled root-close retry seam. Additional controlled tests now exercise real local Node process-group cleanup and descriptor/root mixed-owner retries under emulated admission. The existing Linux-gated provider assertions remain skipped on Darwin arm64. Natural OS cleanup failure frequency and real Linux EVM provider behavior remain unmeasured. ### REA version Source revision `10f1994fe1a0bbddeb846ee8078a80a483a652f1` plus the uncommitted focused fix. ### Runtime and platform Node v24.18.0; Darwin 25.3.0 arm64. Linux/x64 admission and inherited limits were emulated only inside the controlled test. ### Target and artifact context Local hex EVM bytecode carrier; worker execution controlled by the launcher seam. ### Interface Other (provider boundary test) ### Earliest failure stage Cleanup, cancellation, or lifecycle ### Relevant output or evidence `npm run test:focused -- tests/boundary/evm/evmCleanupRetry.test.ts` failed before the fix at the assertion for `cleanupIncomplete` after `provider.close()` succeeded and the root path returned ENOENT. It passes after the fix. Full executed output and source hash are recorded in `.cache/codebase-audit/evm-pwntools-worker-retry-20261020.json`. ### Analysis or suspected cause `EvmoleInterfaceProvider.#pendingCleanup` retained the root and worker supervisor but did not bind returned `ProviderCleanupError` objects to exact cleanup owners. Later `#retire()` could release the owner without updating previously returned errors. The fix uses `AnalysisCleanupLease` and `AnalysisCleanupLeaseGroup` at the EVM owner boundary. ### Verified local fix update Clean isolated source `10f1994fe1a0bbddeb846ee8078a80a483a652f1`, with only the six retry regressions copied, failed all six before the fix. Final local focused validation passed **18 tests, 30 Linux-gated skipped** across retry/provider and failure-projection tests. Controlled tests cover completed/malformed replies, exact opaque cancellation, descriptor/root and process/root release, original plus close aggregates, and prior physical failure followed by a late abort. Real Node children and actual group/filesystem cleanup are used; Linux/x64 admission and limits are emulated. No real Linux/EVMole qualification. Typecheck/lint, formatting and diff checks pass. Fix remains local and uncommitted; issue stays open.
bsakaguchi 2026-10-31 00:38:04 +00:00
  • closed this issue
  • added the
    triage
    bug
    labels
Author
Owner

@uyg7x commented on GitHub (Oct 19, 2026):

Hi @morluto! I see you mentioned you have a verified local fix for this that is currently uncommitted. Are you planning to push this fix yourself soon, or would you be open to me taking a shot at implementing and PRing this fix based on your analysis? Thanks!

<!-- gh-comment-id:6277191290 --> @uyg7x commented on GitHub (Oct 19, 2026): Hi @morluto! I see you mentioned you have a verified local fix for this that is currently uncommitted. Are you planning to push this fix yourself soon, or would you be open to me taking a shot at implementing and PRing this fix based on your analysis? Thanks!
Author
Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with sibling reports in Cluster AR: #4683, #4686, and #5332 ([Bug] EVM artifact-read cancellation returns the ArtifactReaderFailure wrapper instead of the caller's abort reason). Both isolate cleanup accounting and retained owner closure in the EVM adapter.

Consolidated under Cluster AR (EVM Artifact & Interface Lifecycle, Cleanup & Cancellation) in tracker #4330.

<!-- gh-comment-id:6292119772 --> @matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with sibling reports in **Cluster AR**: #4683, #4686, and #5332 (`[Bug] EVM artifact-read cancellation returns the ArtifactReaderFailure wrapper instead of the caller's abort reason`). Both isolate cleanup accounting and retained owner closure in the EVM adapter. Consolidated under **Cluster AR (EVM Artifact & Interface Lifecycle, Cleanup & Cancellation)** in tracker #4330.
Author
Owner

@morluto commented on GitHub (Oct 20, 2026):

Closing as completed: verified during REA Bot open-bug audit that the described EVM cleanup incomplete-after-close fix is already on current main with regression coverage. No further PR needed for this report.

<!-- gh-comment-id:6300575291 --> @morluto commented on GitHub (Oct 20, 2026): Closing as completed: verified during REA Bot open-bug audit that the described EVM cleanup incomplete-after-close fix is already on current main with regression coverage. No further PR needed for this report.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#3228
No description provided.