morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

environ is read (ESRCH) #4643

New issue

Closed

opened 2026-10-31 00:51:43 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:51:43 +00:00

Owner

Originally created by @raulabenza on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7790

Before submitting

I searched the open and closed issues for a related report (ESRCH, cleanup_incomplete, environ). Closest: #4322, #6983, #7143 — see "Analysis".
I removed credentials, tokens, private paths, and sensitive target data.

Area

Provider lifecycle / process cleanup (Linux)

Observed behavior

On a busy Linux host, provider operations that complete successfully intermittently end in cleanup_incomplete because the process-ownership scan reads /proc/<pid>/environ of live candidates that exit between the process-table listing and the read. The open() fails with ESRCH and the candidate is counted as a strict failure:

"reason": "process ownership token could not be read for 3 live process(es): environment_errno_ESRCH=3;
           live candidates 947267=environment_errno_ESRCH, 947405=environment_errno_ESRCH, 947500=environment_errno_ESRCH",
"failures": [
  {"pid":947267,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947267/environ'"},
  {"pid":947405,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947405/environ'"},
  {"pid":947500,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947500/environ'"}
]

(from details.diagnostics.cleanup_error, provider_id: jadx, operation: inspect_jadx_configuration, stage: cleanup)

Afterwards no provider process is left behind (pgrep -a java is empty), and an identical retry usually succeeds.

Reproduction

Host with steady short-lived process churn (production server running many cron/worker processes). Then:

for i in 1 2 3 4 5 6; do
  npx -y rea-agents@6.8.0 doctor --client claude_code --skill --json > run$i.json; echo "run$i exit=$?"
done

Result in one run: 3 of 6 failed with cleanup_incomplete (runs 2, 4, 5); the others returned healthy: true. Earlier the same day: doctor --provider hopper and doctor --client claude_code --skill failed the same way on first attempt, and CLI analyze/decompile with Ghidra failed 3 of 6 times. Exact counts describe these runs, not a failure probability.

Expected behavior

A candidate whose /proc/<pid>/environ open fails with ESRCH (or ENOENT) and that is corroborated as gone (signal-zero ESRCH / absent from a fresh process table) should be treated as exited and cleared, not as an unreadable live process. Only a process that is still alive and unreadable should fail closed.

Impact and scope

Successful analyses are reported as failures with retryable: false and the message "repeating the unchanged request is not expected to succeed", which is misleading here: a retry typically succeeds.
In MCP mode this interrupts sessions and asks the user to review resources that do not exist.
Affects any Linux host with normal process churn; not provider-specific (seen with jadx, ghidra and hopper probes).

REA version

rea-agents 6.8.0 (npm, latest at time of writing)

Runtime and platform

Ubuntu 24.04, kernel 6.8.0, x86_64, Node 22.23.4, OpenJDK 21.0.12, Ghidra 12.1.4. Bare metal, no containers or user namespaces involved; REA runs as root.

Target and artifact context

None needed — doctor alone reproduces it. Also seen with open_binary on a copy of /bin/true.

Interface

CLI (rea doctor, rea analyze/decompile) and MCP (rea mcp).

Earliest failure stage

Cleanup (process-ownership verification after the provider operation completed).

Relevant output or evidence

See "Observed behavior". Full JSON of a failing run available on request (it contains only the structure above plus nested primary_error).

Typed outcome or recovery expected

cleanup: "complete" (or the exited PIDs recorded as cleared), and the operation's own result returned.

Analysis or suspected cause

The listed PIDs are "live candidates" from the scan, which then disappear before environ is opened. ESRCH from open('/proc/<pid>/environ') means the task is gone/exiting. Related but distinct reports:

#4322 / commit 49e67aa1d: corroborated ENOENT/ESRCH absence for owned children and the token-mismatch path.
#6983: routes ENOENT/ESRCH from /proc/PID/status / closing stat through the absence check — but not the environ read.
#7143 (+ #7145, #7147): the EACCES user-namespace case, not ESRCH.

It looks like the environment-unreadable branch for environ reads does not go through the same corroborated-absence check (signal-zero ESRCH or a fresh liveness table), so the exit race on unrelated candidates still yields cleanup_incomplete in 6.8.0.

Originally created by @raulabenza on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7790 ### Before submitting - [x] I searched the open and closed issues for a related report (`ESRCH`, `cleanup_incomplete`, `environ`). Closest: #4322, #6983, #7143 — see "Analysis". - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Provider lifecycle / process cleanup (Linux) ### Observed behavior On a busy Linux host, provider operations that complete successfully intermittently end in `cleanup_incomplete` because the process-ownership scan reads `/proc/<pid>/environ` of live candidates that exit between the process-table listing and the read. The `open()` fails with `ESRCH` and the candidate is counted as a strict failure: ``` "reason": "process ownership token could not be read for 3 live process(es): environment_errno_ESRCH=3; live candidates 947267=environment_errno_ESRCH, 947405=environment_errno_ESRCH, 947500=environment_errno_ESRCH", "failures": [ {"pid":947267,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947267/environ'"}, {"pid":947405,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947405/environ'"}, {"pid":947500,"reason":"environment-unreadable","diagnostic":"ESRCH: no such process, open '/proc/947500/environ'"} ] ``` (from `details.diagnostics.cleanup_error`, `provider_id: jadx`, `operation: inspect_jadx_configuration`, `stage: cleanup`) Afterwards no provider process is left behind (`pgrep -a java` is empty), and an identical retry usually succeeds. ### Reproduction Host with steady short-lived process churn (production server running many cron/worker processes). Then: ``` for i in 1 2 3 4 5 6; do npx -y rea-agents@6.8.0 doctor --client claude_code --skill --json > run$i.json; echo "run$i exit=$?" done ``` Result in one run: 3 of 6 failed with `cleanup_incomplete` (runs 2, 4, 5); the others returned `healthy: true`. Earlier the same day: `doctor --provider hopper` and `doctor --client claude_code --skill` failed the same way on first attempt, and CLI `analyze`/`decompile` with Ghidra failed 3 of 6 times. Exact counts describe these runs, not a failure probability. ### Expected behavior A candidate whose `/proc/<pid>/environ` open fails with `ESRCH` (or `ENOENT`) and that is corroborated as gone (signal-zero `ESRCH` / absent from a fresh process table) should be treated as exited and cleared, not as an unreadable live process. Only a process that is still alive and unreadable should fail closed. ### Impact and scope - Successful analyses are reported as failures with `retryable: false` and the message "repeating the unchanged request is not expected to succeed", which is misleading here: a retry typically succeeds. - In MCP mode this interrupts sessions and asks the user to review resources that do not exist. - Affects any Linux host with normal process churn; not provider-specific (seen with jadx, ghidra and hopper probes). ### REA version rea-agents 6.8.0 (npm, `latest` at time of writing) ### Runtime and platform Ubuntu 24.04, kernel 6.8.0, x86_64, Node 22.23.4, OpenJDK 21.0.12, Ghidra 12.1.4. Bare metal, no containers or user namespaces involved; REA runs as root. ### Target and artifact context None needed — `doctor` alone reproduces it. Also seen with `open_binary` on a copy of `/bin/true`. ### Interface CLI (`rea doctor`, `rea analyze/decompile`) and MCP (`rea mcp`). ### Earliest failure stage Cleanup (process-ownership verification after the provider operation completed). ### Relevant output or evidence See "Observed behavior". Full JSON of a failing run available on request (it contains only the structure above plus nested `primary_error`). ### Typed outcome or recovery expected `cleanup: "complete"` (or the exited PIDs recorded as cleared), and the operation's own result returned. ### Analysis or suspected cause The listed PIDs are "live candidates" from the scan, which then disappear before `environ` is opened. `ESRCH` from `open('/proc/<pid>/environ')` means the task is gone/exiting. Related but distinct reports: - #4322 / commit 49e67aa1d: corroborated ENOENT/ESRCH absence for *owned* children and the token-mismatch path. - #6983: routes ENOENT/ESRCH from `/proc/PID/status` / closing `stat` through the absence check — but not the `environ` read. - #7143 (+ #7145, #7147): the `EACCES` user-namespace case, not `ESRCH`. It looks like the `environment-unreadable` branch for `environ` reads does not go through the same corroborated-absence check (signal-zero `ESRCH` or a fresh liveness table), so the exit race on unrelated candidates still yields `cleanup_incomplete` in 6.8.0.

bsakaguchi closed this issue

2026-10-31 00:51:45 +00:00

bsakaguchi commented

2026-10-31 00:51:46 +00:00

Author

Owner

@harshitethic commented on GitHub (Oct 28, 2026):

Working fix for Linux /proc/<pid>/environ ENOENT/ESRCH process-cleanup races, with test evidence, is now pushed to @harshitethic's fork:

Compare against upstream main: https://github.com/morluto/rea/compare/main...harshitethic:fix/linux-proc-environ-exit-race-7790?expand=1
Latest commit: github.com/harshitethic/rea@cd2f75da5a

The three focused commits (intended as one upstream fix) recheck native Linux process identity after an initial procfs environment-read race and again after the final token retry. A bounded native-only settlement window lets already-exiting PIDs be confirmed absent without treating ESRCH by itself as proof of exit; still-running, unverifiable and recycled PIDs remain fail closed. The code does not grant new signal authority or change CLI/MCP schemas.

Executed validation on Linux, Node v24.22.1:

317/317 process ownership/lifecycle tests passed across 21 files, including 19 focused native-exit regressions.
npm run check:fast passed (TypeScript and lint); npm run build:cached passed (compile, metadata, generated skills).
On the earlier two-stage patch, the five compiled acceptance cases that initially failed each passed on isolated reruns: cooperative CLI, SIGKILL CLI, cooperative MCP, SIGKILL MCP, and omitted-digest descendant cleanup.
With the final bounded-settlement patch, the previously flaky cancellation CLI/MCP acceptance test passed in isolation.
Important remaining gap (additional verification): a fresh clone of the exact published head (cd2f75d) passed 317/317 process tests, typecheck/lint, and compiled build. The five selected compiled CLI acceptance cases passed together. Of five selected compiled MCP finalization cases, four passed and one failed: keeps the default identity ... without finalization_ms returned an unverified environment_errno_ESRCH cleanup report. The same test failed again when rerun in isolation. Thus the existing native exit-retry patch does not fully resolve the original production-facing failure; I did not weaken the host-cleanup assertion or treat an unverified PID as exited. The broader finalization suite had previously been 11/12 and remains not fully green. The next step is to identify whether that failure comes from group validation or the final ownership sweep before claiming the issue fixed.
Production provider churn, non-Linux OSes and full application integration remain untested.

I attempted to open a draft upstream PR, but GitHub's integration returned 403 Resource not accessible by integration for morluto/rea. The comparison is available for a human owner to inspect and submit under REA's contribution/readiness checklist. Maintainer review of the remaining ESRCH behavior and suggested architecture is welcome.

@harshitethic commented on GitHub (Oct 28, 2026): Working fix for Linux `/proc/<pid>/environ` ENOENT/ESRCH process-cleanup races, with test evidence, is now pushed to @harshitethic's fork: **Compare against upstream main:** https://github.com/morluto/rea/compare/main...harshitethic:fix/linux-proc-environ-exit-race-7790?expand=1 **Latest commit:** https://github.com/harshitethic/rea/commit/cd2f75da5a5760a05e857adb8b5274b44c871bfe The three focused commits (intended as **one** upstream fix) recheck native Linux process identity after an initial procfs environment-read race and again after the final token retry. A bounded native-only settlement window lets already-exiting PIDs be confirmed absent without treating `ESRCH` by itself as proof of exit; still-running, unverifiable and recycled PIDs remain fail closed. The code does not grant new signal authority or change CLI/MCP schemas. **Executed validation on Linux, Node v24.22.1:** - **317/317** process ownership/lifecycle tests passed across 21 files, including **19 focused native-exit regressions**. - `npm run check:fast` passed (TypeScript and lint); `npm run build:cached` passed (compile, metadata, generated skills). - On the earlier two-stage patch, the five compiled acceptance cases that initially failed each passed on isolated reruns: cooperative CLI, SIGKILL CLI, cooperative MCP, SIGKILL MCP, and omitted-digest descendant cleanup. - With the final bounded-settlement patch, the previously flaky cancellation CLI/MCP acceptance test passed in isolation. - **Important remaining gap (additional verification):** a fresh clone of the exact published head (`cd2f75d`) passed **317/317** process tests, typecheck/lint, and compiled build. The five selected compiled CLI acceptance cases passed together. Of five selected compiled MCP finalization cases, **four passed and one failed**: `keeps the default identity ... without finalization_ms` returned an unverified `environment_errno_ESRCH` cleanup report. The same test **failed again when rerun in isolation**. Thus the existing native exit-retry patch does **not** fully resolve the original production-facing failure; I did not weaken the host-cleanup assertion or treat an unverified PID as exited. The broader finalization suite had previously been 11/12 and remains not fully green. The next step is to identify whether that failure comes from group validation or the final ownership sweep before claiming the issue fixed. - Production provider churn, non-Linux OSes and full application integration remain untested. I attempted to open a draft upstream PR, but GitHub's integration returned `403 Resource not accessible by integration` for `morluto/rea`. The comparison is available for a human owner to inspect and submit under REA's contribution/readiness checklist. Maintainer review of the remaining ESRCH behavior and suggested architecture is welcome.

bsakaguchi referenced this issue

2026-10-31 01:13:46 +00:00

[PR #4655] [CLOSED] fix(adb): preserve device and file error identity #6349

bsakaguchi referenced this issue

2026-11-01 21:10:53 +00:00

[GH-ISSUE #3582] [Tracker] Ready to close or merge: issues and pull requests (October 2026) #1330