morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6134] cleanup_incomplete false failure: settleLauncherExit's single process-table refresh loses the race against JVM exit #3762

New issue

Closed

opened 2026-10-31 00:43:46 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 00:43:46 +00:00

Owner

Originally created by @topguntrader007 on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6134

Originally assigned to: @kaoru0822-kitauji on GitHub.

Version: rea-agents 6.4.0 (pinned, lockfile-verified)

What happened

A rea inspect run against a PE target with the Ghidra provider completed its analysis and wrote a valid snapshot, then exited 1 with:

{
  "error": "Analysis failed",
  "code": "cleanup_incomplete",
  "category": "execution_failure",
  "details": {
    "provider_id": "ghidra",
    "operation": "close_binary",
    "cleanup": "incomplete",
    "resources": ["ghidra-process", "/tmp/rea-ghidra-XXXXXX"],
    "diagnostics": {
      "reason": "Ghidra process cleanup was incomplete: owned launcher command identity did not match (observed=[java]; expected=/usr/lib/jvm/java-21-openjdk-amd64/bin/java)"
    }
  }
}

Reading of the cause

observed=[java] is the bracketed form procps prints when /proc/<pid>/cmdline is empty — the JVM was mid-exit (memory torn down, not yet a zombie) when the process table was re-read. settleLauncherExit in dist/process/ProcessOwnership.js already documents this race ("A process-table read can race exit and retain the old state with an altered command. Only a fresh table proving that PID is no longer live can settle that mismatch."), but it performs that fresh read only once, immediately. On a slow host (here: a virtualized guest where JVM teardown of a large heap takes a moment), the process is still in the exiting state at that single re-read, so cleanup reports cleaned: false with the identity-mismatch reason, and the whole inspect is failed despite the analysis having succeeded.

We verified independently after the fact that no process carrying the run's REA_PROCESS_RUN_ID remained — the JVM did exit; the confirmation just lost the race.

Suggested fix

In settleLauncherExit, retry the fresh process-table read a few times with a short backoff (e.g. 5 attempts, ~250–1000 ms apart) before concluding the identity mismatch is real. A process genuinely still alive after the settle window would still (correctly) fail cleanup.

Environment

rea-agents 6.4.0, Node 24, Ubuntu 24.04 guest (VirtualBox, 4 vCPU, no hardware virtualization passthrough — process teardown is slow)
Ghidra 12.1.4, OpenJDK 21
Target: a benign 55 KB PE DLL; analysis itself took ~138 s and produced a complete snapshot

Happy to provide the full inspect stdout JSON if useful.

Originally created by @topguntrader007 on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6134 Originally assigned to: @kaoru0822-kitauji on GitHub. **Version:** rea-agents 6.4.0 (pinned, lockfile-verified) **What happened** A `rea inspect` run against a PE target with the Ghidra provider completed its analysis and wrote a valid snapshot, then exited 1 with: ```json { "error": "Analysis failed", "code": "cleanup_incomplete", "category": "execution_failure", "details": { "provider_id": "ghidra", "operation": "close_binary", "cleanup": "incomplete", "resources": ["ghidra-process", "/tmp/rea-ghidra-XXXXXX"], "diagnostics": { "reason": "Ghidra process cleanup was incomplete: owned launcher command identity did not match (observed=[java]; expected=/usr/lib/jvm/java-21-openjdk-amd64/bin/java)" } } } ``` **Reading of the cause** `observed=[java]` is the bracketed form procps prints when `/proc/<pid>/cmdline` is empty — the JVM was mid-exit (memory torn down, not yet a zombie) when the process table was re-read. `settleLauncherExit` in `dist/process/ProcessOwnership.js` already documents this race ("A process-table read can race exit and retain the old state with an altered command. Only a fresh table proving that PID is no longer live can settle that mismatch."), but it performs that fresh read only **once, immediately**. On a slow host (here: a virtualized guest where JVM teardown of a large heap takes a moment), the process is still in the exiting state at that single re-read, so cleanup reports `cleaned: false` with the identity-mismatch reason, and the whole inspect is failed despite the analysis having succeeded. We verified independently after the fact that no process carrying the run's `REA_PROCESS_RUN_ID` remained — the JVM did exit; the confirmation just lost the race. **Suggested fix** In `settleLauncherExit`, retry the fresh process-table read a few times with a short backoff (e.g. 5 attempts, ~250–1000 ms apart) before concluding the identity mismatch is real. A process genuinely still alive after the settle window would still (correctly) fail cleanup. **Environment** - rea-agents 6.4.0, Node 24, Ubuntu 24.04 guest (VirtualBox, 4 vCPU, no hardware virtualization passthrough — process teardown is slow) - Ghidra 12.1.4, OpenJDK 21 - Target: a benign 55 KB PE DLL; analysis itself took ~138 s and produced a complete snapshot Happy to provide the full inspect stdout JSON if useful.

bsakaguchi closed this issue

2026-10-31 00:43:46 +00:00

bsakaguchi commented

2026-10-31 00:43:47 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

确认，这是把"无法确认退出"的竞态放大成了确定性失败：分析已完成、snapshot 已写盘，却因 cleanup 单次重读恰好撞上 JVM mid-exit 而 exit 1，用户还得重跑。既然 settleLauncherExit 的注释已经承认存在 race，那"只重读一次"的设计就明显不够——建议改成在限定时间内多次轮询进程表，只要 PID 最终消失即视为已退出；对"cmdline 为空（memory torn down）"这种信号也可直接按已退出处理。另外这和 #6166 是同一条线的问题：cleanup 环节的不确定性不该升级成 retryable=false 的 execution_failure，建议一起把 cleanup 语义改为"尽力而为 + 警告"，只有真正泄漏的资源才报错。

@XinwangTao commented on GitHub (Oct 22, 2026): 确认，这是把"无法确认退出"的竞态放大成了确定性失败：分析已完成、snapshot 已写盘，却因 cleanup 单次重读恰好撞上 JVM mid-exit 而 exit 1，用户还得重跑。既然 settleLauncherExit 的注释已经承认存在 race，那"只重读一次"的设计就明显不够——建议改成在限定时间内多次轮询进程表，只要 PID 最终消失即视为已退出；对"cmdline 为空（memory torn down）"这种信号也可直接按已退出处理。另外这和 #6166 是同一条线的问题：cleanup 环节的不确定性不该升级成 retryable=false 的 execution_failure，建议一起把 cleanup 语义改为"尽力而为 + 警告"，只有真正泄漏的资源才报错。

bsakaguchi commented

2026-10-31 00:43:47 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026):

Reproduced on latest main (c465ff5a8) on Linux, with a different reason string than 6.4.0.

Setup. I used a Linux container (node:24.18.0-trixie, procps-ng 4.0.4, kernel in a colima VM) running the built dist/. An owned python3 launcher with a 2 GiB heap (THP disabled) is spawned through spawnOwnedProviderProcess and wrapped in ProviderProcessSupervisor with cleanupOwnedProcessGroup. The test closes stdin and then calls stop() as soon as /proc/<pid>/stat shows PF_EXITING and /proc/<pid>/cmdline is empty (the state procps prints as [python3]). A CPU quota (--cpus=0.04) slows kernel teardown, much like the reporter's slow guest.

Observed (2 runs each):

build	unthrottled	`--cpus=0.04`
`main`	`verified-cleanup` (teardown 70 ms)	`incomplete`, `owned launcher exit could not be confirmed`, after 1.4–1.9 s (2/2)

The mid-exit row was state=R, PF_EXITING set, cmdline 0 bytes in every run.

Reading. On main, #5249 now settles native exit-in-progress before comparing the command, so the literal observed=[java] mismatch no longer occurs on Linux/macOS native hosts. That settlement is still bounded by the shared 1 s budget, though. When kernel teardown takes longer, cleanup fails closed, and ProviderProcess.stop() reports incomplete without waiting for the supervisor's own child-exit observation, which would positively prove the launcher exited. Hosts without native identity still use the single table refresh described in the report.

I'm taking this one.

@kaoru0822-kitauji commented on GitHub (Oct 22, 2026): Reproduced on latest `main` (c465ff5a8) on Linux, with a different reason string than 6.4.0. **Setup.** I used a Linux container (`node:24.18.0-trixie`, procps-ng 4.0.4, kernel in a colima VM) running the built `dist/`. An owned `python3` launcher with a 2 GiB heap (THP disabled) is spawned through `spawnOwnedProviderProcess` and wrapped in `ProviderProcessSupervisor` with `cleanupOwnedProcessGroup`. The test closes stdin and then calls `stop()` as soon as `/proc/<pid>/stat` shows `PF_EXITING` and `/proc/<pid>/cmdline` is empty (the state procps prints as `[python3]`). A CPU quota (`--cpus=0.04`) slows kernel teardown, much like the reporter's slow guest. **Observed** (2 runs each): | build | unthrottled | `--cpus=0.04` | |---|---|---| | `main` | `verified-cleanup` (teardown 70 ms) | `incomplete`, `owned launcher exit could not be confirmed`, after 1.4–1.9 s (2/2) | The mid-exit row was `state=R`, `PF_EXITING` set, `cmdline` 0 bytes in every run. **Reading.** On `main`, #5249 now settles native exit-in-progress before comparing the command, so the literal `observed=[java]` mismatch no longer occurs on Linux/macOS native hosts. That settlement is still bounded by the shared 1 s budget, though. When kernel teardown takes longer, cleanup fails closed, and `ProviderProcess.stop()` reports `incomplete` without waiting for the supervisor's own child-exit observation, which would positively prove the launcher exited. Hosts without native identity still use the single table refresh described in the report. I'm taking this one.

bsakaguchi commented

2026-10-31 00:43:47 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

用 launcher 自身的 exit proof 来 settle cleanup，比多次轮询 + backoff 更根本——轮询无论间隔多少都仍存在竞态窗口，而进程自己的退出证据是确定性信号，单次重读恰好撞上 mid-exit 的问题从根因上消除了。修复方向很扎实，多谢处理。

@XinwangTao commented on GitHub (Oct 22, 2026): 用 launcher 自身的 exit proof 来 settle cleanup，比多次轮询 + backoff 更根本——轮询无论间隔多少都仍存在竞态窗口，而进程自己的退出证据是确定性信号，单次重读恰好撞上 mid-exit 的问题从根因上消除了。修复方向很扎实，多谢处理。

bsakaguchi referenced this issue

2026-10-31 01:10:10 +00:00

[PR #3762] Feat/lucas profile roadmap #5883

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425