morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8337] [Bug] macOS package verifier intermittently fails cleanup on malformed_procargs_path_padding #6551

New issue

Open

opened 2026-11-01 21:12:09 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-11-01 21:12:09 +00:00

Owner

Originally created by @kaoru0822-kitauji on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8337

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture and runtime observation

Observed behavior

Native package baseline (darwin arm64) intermittently fails in npm run verify:package, with no assertion failure. Final cleanup of a supervised packaged-CLI command cannot read one live process's ownership token:

process ownership token could not be read for 1 live process(es): malformed_procargs_path_padding=1; live candidates <pid>=malformed_procargs_path_padding
  failures: [{ pid: <pid>, reason: 'environment-unreadable', diagnostic: 'malformed_procargs_path_padding' }]
→ OwnedCommandFailure: Command cleanup failed → AggregateError: Verifier cleanup failed

Observed on unrelated branches:

Run / job	Branch	Stage
41059655323 / 124076886300	PR #7619 (reported in #5654)	package repack
41212934549 / 124575799290	`codex/test-oracle-audit-20261031`	(same diagnostic)
41210929709 / 124566004894	`perf/2483-darwin-helper-cache` (#8318)	setup and uninstall CLI (`rea setup … --install-hopper --json`)
41214073523 / 124577342088	`perf/2483-darwin-helper-cache` (#8318)	setup and uninstall CLI

In each run I checked, rerunning the same commit passed (for example 41210929709). npm run verify:package passed locally three times on macOS 26 arm64 with Command Line Tools. In each failing run the live candidate's PID was a few above the command leader's, so it started during the command.

Reproduction

NOT_ENOUGH_INFO for a deterministic reproduction. It is intermittent on GitHub's macOS arm64 runners and not reproduced locally. The job logs never identify the live candidate's command or parent.

Expected behavior

A process that started during a supervised command and is transiently unreadable at the final sweep (for example, mid-exec or exiting) should not make an otherwise successful verifier command fail. Alternatively, the failure should identify which process it was so the cause can be fixed.

Impact and scope

It blocks CI required on unrelated PRs, and contributors without admin rights can't rerun. Real ownership uncertainty must stay fail-closed. This report is only about the verifier's flakiness and the missing diagnostics.

REA version

main around 23de10bb2 / a989c0e45

Runtime and platform

GitHub-hosted macOS arm64 runners, Node 24.22.1

Target and artifact context

Packaged REA CLI under verify:package; no analysis target.

Interface

CLI (packaged), supervised by the verifier's owned-command runner

Earliest failure stage

Final ownership cleanup of a supervised command

Relevant output or evidence

See the job links above.

Typed outcome or recovery expected

Cleanup either verifies the transient process or reports its identity.

Analysis or suspected cause

Source-derived hypotheses, unverified:

parseRunToken derives the post-path padding from "executable_path=" + path and rejects any other layout. A process read while its KERN_PROCARGS2 image is mid-exec or being torn down may present a layout that is valid but transitional.
#8275 ("preserve ambiguous Darwin token ownership") made more ambiguous reads fail closed, which may have made this visible.

As a probe, I launched processes from exec paths of 128–183 bytes, covering every residue mod 8, and all read as readable. So path length alone doesn't explain it. Proposed check: include the candidate's ps command, parent PID and process-group ID in the cleanup failure, then confirm whether a short re-read of transiently malformed candidates resolves them. Any such retry must stay fail-closed.

Additional context

Found while shepherding #8318, whose change does not touch the token parser.

Originally created by @kaoru0822-kitauji on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8337 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture and runtime observation ### Observed behavior `Native package baseline (darwin arm64)` intermittently fails in `npm run verify:package`, with no assertion failure. Final cleanup of a supervised packaged-CLI command cannot read one live process's ownership token: ``` process ownership token could not be read for 1 live process(es): malformed_procargs_path_padding=1; live candidates <pid>=malformed_procargs_path_padding failures: [{ pid: <pid>, reason: 'environment-unreadable', diagnostic: 'malformed_procargs_path_padding' }] → OwnedCommandFailure: Command cleanup failed → AggregateError: Verifier cleanup failed ``` Observed on unrelated branches: | Run / job | Branch | Stage | | --- | --- | --- | | 41059655323 / 124076886300 | PR #7619 (reported in #5654) | package repack | | 41212934549 / 124575799290 | `codex/test-oracle-audit-20261031` | (same diagnostic) | | 41210929709 / 124566004894 | `perf/2483-darwin-helper-cache` (#8318) | setup and uninstall CLI (`rea setup … --install-hopper --json`) | | 41214073523 / 124577342088 | `perf/2483-darwin-helper-cache` (#8318) | setup and uninstall CLI | In each run I checked, rerunning the same commit passed (for example 41210929709). `npm run verify:package` passed locally three times on macOS 26 arm64 with Command Line Tools. In each failing run the live candidate's PID was a few above the command leader's, so it started during the command. ### Reproduction NOT_ENOUGH_INFO for a deterministic reproduction. It is intermittent on GitHub's macOS arm64 runners and not reproduced locally. The job logs never identify the live candidate's command or parent. ### Expected behavior A process that started during a supervised command and is transiently unreadable at the final sweep (for example, mid-exec or exiting) should not make an otherwise successful verifier command fail. Alternatively, the failure should identify which process it was so the cause can be fixed. ### Impact and scope It blocks `CI required` on unrelated PRs, and contributors without admin rights can't rerun. Real ownership uncertainty must stay fail-closed. This report is only about the verifier's flakiness and the missing diagnostics. ### REA version main around `23de10bb2` / `a989c0e45` ### Runtime and platform GitHub-hosted macOS arm64 runners, Node 24.22.1 ### Target and artifact context Packaged REA CLI under `verify:package`; no analysis target. ### Interface CLI (packaged), supervised by the verifier's owned-command runner ### Earliest failure stage Final ownership cleanup of a supervised command ### Relevant output or evidence See the job links above. ### Typed outcome or recovery expected Cleanup either verifies the transient process or reports its identity. ### Analysis or suspected cause Source-derived hypotheses, unverified: 1. `parseRunToken` derives the post-path padding from `"executable_path=" + path` and rejects any other layout. A process read while its `KERN_PROCARGS2` image is mid-exec or being torn down may present a layout that is valid but transitional. 2. #8275 ("preserve ambiguous Darwin token ownership") made more ambiguous reads fail closed, which may have made this visible. As a probe, I launched processes from exec paths of 128–183 bytes, covering every residue mod 8, and all read as `readable`. So path length alone doesn't explain it. Proposed check: include the candidate's `ps` command, parent PID and process-group ID in the cleanup failure, then confirm whether a short re-read of transiently malformed candidates resolves them. Any such retry must stay fail-closed. ### Additional context Found while shepherding #8318, whose change does not touch the token parser.

bsakaguchi added the

bug

label

2026-11-01 21:12:09 +00:00

bsakaguchi commented

2026-11-01 21:12:10 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 31, 2026):

More data points from #8318's branch: run 41214073523 (setup stage, leader 13515, candidate 13523) and job 124586481814 (platform CLI stage, rea trace <node> fixture --provider hopper --json, leader 12295, candidate 12370). The candidate was always a member of the supervised group, started a few dozen PIDs after the leader, and still alive at the final group check. A local reproduction harness (runOwnedCommand with sweepTokenOwnedProcesses, 30 iterations under 6× CPU load) produced no failures.

@kaoru0822-kitauji commented on GitHub (Oct 31, 2026): More data points from #8318's branch: run 41214073523 (setup stage, leader 13515, candidate 13523) and job 124586481814 (`platform CLI` stage, `rea trace <node> fixture --provider hopper --json`, leader 12295, candidate 12370). The candidate was always a member of the supervised group, started a few dozen PIDs after the leader, and still alive at the final group check. A local reproduction harness (`runOwnedCommand` with `sweepTokenOwnedProcesses`, 30 iterations under 6× CPU load) produced no failures.

bsakaguchi commented

2026-11-03 00:29:57 +00:00

Author

Owner

@morluto commented on GitHub (Nov 2, 2026):

This recurred on main at 2cb29693c3d574c4898832047fe309ac1ad00e9d: macOS package baseline failed during verify:package cleanup with malformed_procargs_path_padding. The failure named a live mdworker candidate with parent PID 1 and its own process group; the verifier retained its owned resources rather than claiming cleanup succeeded.

This is an observed CI failure, not a deterministic local reproduction. A local sample of seven mdworker processes returned six platform_binary_environment_withheld results and one sysctl_failed_22; it did not reproduce the malformed-padding result. The parser or ownership applicability cause remains unknown. Any correction must preserve fail-closed treatment when ownership cannot be established.

@morluto commented on GitHub (Nov 2, 2026): This recurred on main at `2cb29693c3d574c4898832047fe309ac1ad00e9d`: [macOS package baseline](https://github.com/morluto/rea/actions/runs/41481805636/job/125405255578) failed during `verify:package` cleanup with `malformed_procargs_path_padding`. The failure named a live `mdworker` candidate with parent PID 1 and its own process group; the verifier retained its owned resources rather than claiming cleanup succeeded. This is an observed CI failure, not a deterministic local reproduction. A local sample of seven `mdworker` processes returned six `platform_binary_environment_withheld` results and one `sysctl_failed_22`; it did not reproduce the malformed-padding result. The parser or ownership applicability cause remains unknown. Any correction must preserve fail-closed treatment when ownership cannot be established.

bsakaguchi referenced this issue

2026-11-03 07:16:59 +00:00

[GH-ISSUE #7621] [Bug] environment-compare does not disclose when a step's difference list is truncated #4549

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6551

No description provided.

Rows
Columns