[GH-ISSUE #2170] fix(process): empty settlement observation is reported as alive_at_deadline #716

Closed
opened 2026-10-15 18:27:39 +00:00 by bsakaguchi · 0 comments
Owner

Originally created by @coygeek on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2170

Summary

Process Capture can emit settlement.state: "alive_at_deadline" when its only settlement observation found no live members in the selected process group. This occurs when a supported settlement budget expires before a second empty observation. The result reports liveness even though the observer saw emptiness and did not establish quiescence. Successful cleanup then projects cleanup_outcome: "cleaned" for that state. The verified failure concerns the liveness classification, not a failed cleanup operation.

The affected area is Process capture or runtime observation, through the CLI. The earliest failure stage is Cleanup, cancellation, or lifecycle. The target is an ordinary shell command that exits successfully without spawning descendants. One empty observation should retain uncertainty about quiescence rather than claim the group was alive.

Runtime or environment

Source revision c22111eca2c4bc84a084f12711f405b3e7422a40, its prebuilt CLI, and the installed published rea-agents 6.4.0 package were tested independently. The measured runtime was Node.js 24.21.0 on macOS 27.0.1 arm64. This is a supported package runtime, not the checkout's exact contributor-pinned Node version. Linux and Windows behavior were not executed.

Steps to reproduce

Use a host that supports Process Capture and an installed package or built source checkout. In a disposable working directory, save the following as scenario.json:

{
  "executable": "/bin/sh",
  "arguments": ["-c", "exit 0"],
  "working_directory": ".",
  "timeout_ms": 5000,
  "idle_timeout_ms": 5000,
  "settle_ms": 0
}
  1. Run rea capture-process scenario.json --json and inspect normalized_result.settlement, exit, and cleanup.
  2. Repeat with settle_ms set to 10 and 100. On the tested host, both short budgets produced alive_at_deadline and the longer control produced quiesced. A different host can complete the second observation at 10 ms; failure at that value is timing-dependent.
  3. Establish the empty observation independently. Empty process_samples and another capture's quiesced result do not establish what the first capture observed. The following production-function control records the real process-table result used during each invocation, without launching or signaling a process.

Save this as empty-group.mjs and run node empty-group.mjs "$(npm root)/rea-agents" beside a local package installation. For a built checkout, pass its package root as the argument instead. The group number is only an inspection candidate; the script verifies the empty-group premise on every observation rather than assuming that number is absent.

import { pathToFileURL } from 'node:url';
const root = process.argv[2];
const { observeSettlement } = await import(pathToFileURL(
  root + '/dist/process/capture/ProcessCaptureLifecycle.js'
));
const { systemProcessOwnershipHost } = await import(pathToFileURL(
  root + '/dist/process/ProcessOwnershipObservation.js'
));
const { liveProcesses } = await import(pathToFileURL(
  root + '/dist/process/ProcessOwnershipProcessTree.js'
));
const original = systemProcessOwnershipHost.listProcesses;
const group = 4194303;
for (const settleMs of [0, 10, 100]) {
  const observations = [];
  systemProcessOwnershipHost.listProcesses = async (...args) => {
    const rows = await original(...args);
    const liveMemberCount = liveProcesses(rows)
      .filter(p => p.processGroupId === group).length;
    if (liveMemberCount !== 0) throw Error('empty-group premise failed');
    observations.push({ liveMemberCount });
    return rows;
  };
  try {
    const result = await observeSettlement(
      'empty-group-control', [group], settleMs
    );
    console.log(JSON.stringify({ settleMs, observations, result }));
  } finally {
    systemProcessOwnershipHost.listProcesses = original;
  }
}

This direct control was executed against both tested builds. It reproduces the same classifier outcome independently of PTY exit timing. A separate deterministic empty-table control also returned alive_at_deadline after exactly one empty observation with budget 0. It returned quiesced at 10 and 100 after collecting two observations. The public CLI fixture and direct controls therefore do not imply that a particular positive budget always fails.

Expected behavior

Settlement reports observed quiescence, observed liveness requiring cleanup, or an unverifiable state. If the budget expires after only one empty observation, the result must preserve the distinction between insufficient proof of quiescence and evidence that the group was alive. It must not emit alive_at_deadline without an observed live member.

The existing two-consecutive-empty-observations requirement remains applicable. A single empty observation does not justify quiesced. An uncertainty-preserving outcome is acceptable when the selected budget cannot establish quiescence. Successful cleanup remains a separate fact from whether a live member was observed or a signal was required.

Actual behavior

All six uninstrumented CLI trials succeeded with exit code 0, exit.reason: "exited", empty retained samples, cleanup.owned_process_group: "verified" and cleanup.temporary_root: "removed". The following settlement results were observed:

build    settle_ms  state              elapsed_ms  cleanup_outcome
source   0          alive_at_deadline  54          cleaned
source   10         alive_at_deadline  53          cleaned
source   100        quiesced           154         not_required
release  0          alive_at_deadline  55          cleaned
release  10         alive_at_deadline  53          cleaned
release  100        quiesced           157         not_required

The production-function control recorded the actual empty observations used in the same invocation:

build    settle_ms  empty observations  returned state      elapsed_ms
source   0          1                   alive_at_deadline   55
source   10         1                   alive_at_deadline   52
source   100        2                   quiesced            155
release  0          1                   alive_at_deadline   55
release  10         1                   alive_at_deadline   53
release  100        2                   quiesced            159

Separate transparently observed CLI trials counted live members of the actual PTY root group during settlement and returned the original process table unchanged. The correctly instrumented source and release short cases each recorded exactly one zero-live-member observation and emitted alive_at_deadline. Their longer controls each recorded two zero-live-member observations and emitted quiesced. Retained samples were empty in those runs, so the PTY root group was the only selected group. This establishes emptiness from the same short run, without inference from the longer run.

The emitted event journal records settlement event order but does not retain group observation values or the number of rounds. The clean root exit and empty retained samples alone are insufficient proof of settlement emptiness. Likewise, cleanup_outcome: "cleaned" confirms successful cleanup projection; it does not prove that cleanup signaled a live process or that the ownership sweep had no work.

Evidence

  • Expected source: AGENTS.md, Product and Authority and Architecture and Cleanup, requires observations, inferences and unknowns to remain distinct; docs/process-capture.md:176-182 describes settlement outcomes; src/domain/process/processCapture.ts:174-193 defines observed settlement and cleanup pairing; tests/acceptance/process/processFinalization.test.ts:318-344 expects alive_at_deadline for a surviving descendant.
  • Failure source: Independently executed CLI captures and same-invocation process-table observations on both builds, corroborated by the inline production-function control; src/process/capture/ProcessCaptureLifecycle.ts:648-694 returns alive_at_deadline after the budget expires with one empty observation.
  • Evidence provenance: observed
  • Local verification: reproduced
  • Reproduction completeness: complete

The public path validates the scenario, awaits the terminal exit, selects the PTY root plus sampled group leaders, and calls the settlement observer. Inspection failures or uncertain ownership return unverifiable; Windows also returns unverifiable. The tested cases used the POSIX branch, had no cancellation or inspection failure, and selected only the root group. The observer increments its empty count, fails to reach two, checks the expired budget and falls through to alive_at_deadline. The published runtime retains the same behavior.

Related historical work at https://github.com/morluto/rea/pull/233 excludes zombies from live group observations. The current observer already includes that correction; this reproduction uses an absent live group and concerns the deadline classification after an empty observation. Open and closed issues and pull requests were searched, and no report with this exact failure identity was found.

Restoration check

For the same public short-budget scenario and the inline empty-group control, observe an empty group in the invocation that produces the result. When the budget expires before the second empty observation, verify that the result preserves uncertainty and does not claim alive_at_deadline. Do not turn a single empty observation into proof of quiescence. Keep the longer control reporting quiesced after two consecutive empty observations.

Retain a discriminating genuinely live owned-group deadline control, such as the existing surviving-descendant acceptance fixture, and verify it can still report alive_at_deadline with successful cleanup. Retain inspection-failure and cancellation behavior. Verify cleanup outcome separately; changing the settlement label must not bypass owned-resource cleanup or its verification. No particular implementation is prescribed.

Triage assessment

  • Impact level: P2
  • Assessment status: supported
  • Impact basis: Accepted settlement budgets can produce a false machine-readable liveness classification for an empty observation. The defect affects process Evidence semantics and can recur whenever the deadline ends observation after one empty round. All tested captures completed cleanup successfully, and no data loss, unowned-process action or failed resource release was observed.
  • Workaround status: available
  • Workaround basis: On the tested host and workload, increasing the settlement budget to 100 ms allowed two empty observations and returned quiesced in both builds. This is a demonstrated host-bound workaround. Process-table latency varies, so 100 ms is not a guaranteed threshold for another host or load; users must check that the chosen budget permits the required observations.
Originally created by @coygeek on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2170 ## Summary Process Capture can emit `settlement.state: "alive_at_deadline"` when its only settlement observation found no live members in the selected process group. This occurs when a supported settlement budget expires before a second empty observation. The result reports liveness even though the observer saw emptiness and did not establish quiescence. Successful cleanup then projects `cleanup_outcome: "cleaned"` for that state. The verified failure concerns the liveness classification, not a failed cleanup operation. The affected area is Process capture or runtime observation, through the CLI. The earliest failure stage is Cleanup, cancellation, or lifecycle. The target is an ordinary shell command that exits successfully without spawning descendants. One empty observation should retain uncertainty about quiescence rather than claim the group was alive. ## Runtime or environment Source revision `c22111eca2c4bc84a084f12711f405b3e7422a40`, its prebuilt CLI, and the installed published `rea-agents` 6.4.0 package were tested independently. The measured runtime was Node.js 24.21.0 on macOS 27.0.1 arm64. This is a supported package runtime, not the checkout's exact contributor-pinned Node version. Linux and Windows behavior were not executed. ## Steps to reproduce Use a host that supports Process Capture and an installed package or built source checkout. In a disposable working directory, save the following as `scenario.json`: ```json { "executable": "/bin/sh", "arguments": ["-c", "exit 0"], "working_directory": ".", "timeout_ms": 5000, "idle_timeout_ms": 5000, "settle_ms": 0 } ``` 1. Run `rea capture-process scenario.json --json` and inspect `normalized_result.settlement`, `exit`, and `cleanup`. 2. Repeat with `settle_ms` set to 10 and 100. On the tested host, both short budgets produced `alive_at_deadline` and the longer control produced `quiesced`. A different host can complete the second observation at 10 ms; failure at that value is timing-dependent. 3. Establish the empty observation independently. Empty `process_samples` and another capture's `quiesced` result do not establish what the first capture observed. The following production-function control records the real process-table result used during each invocation, without launching or signaling a process. Save this as `empty-group.mjs` and run `node empty-group.mjs "$(npm root)/rea-agents"` beside a local package installation. For a built checkout, pass its package root as the argument instead. The group number is only an inspection candidate; the script verifies the empty-group premise on every observation rather than assuming that number is absent. ```js import { pathToFileURL } from 'node:url'; const root = process.argv[2]; const { observeSettlement } = await import(pathToFileURL( root + '/dist/process/capture/ProcessCaptureLifecycle.js' )); const { systemProcessOwnershipHost } = await import(pathToFileURL( root + '/dist/process/ProcessOwnershipObservation.js' )); const { liveProcesses } = await import(pathToFileURL( root + '/dist/process/ProcessOwnershipProcessTree.js' )); const original = systemProcessOwnershipHost.listProcesses; const group = 4194303; for (const settleMs of [0, 10, 100]) { const observations = []; systemProcessOwnershipHost.listProcesses = async (...args) => { const rows = await original(...args); const liveMemberCount = liveProcesses(rows) .filter(p => p.processGroupId === group).length; if (liveMemberCount !== 0) throw Error('empty-group premise failed'); observations.push({ liveMemberCount }); return rows; }; try { const result = await observeSettlement( 'empty-group-control', [group], settleMs ); console.log(JSON.stringify({ settleMs, observations, result })); } finally { systemProcessOwnershipHost.listProcesses = original; } } ``` This direct control was executed against both tested builds. It reproduces the same classifier outcome independently of PTY exit timing. A separate deterministic empty-table control also returned `alive_at_deadline` after exactly one empty observation with budget 0. It returned `quiesced` at 10 and 100 after collecting two observations. The public CLI fixture and direct controls therefore do not imply that a particular positive budget always fails. ## Expected behavior Settlement reports observed quiescence, observed liveness requiring cleanup, or an unverifiable state. If the budget expires after only one empty observation, the result must preserve the distinction between insufficient proof of quiescence and evidence that the group was alive. It must not emit `alive_at_deadline` without an observed live member. The existing two-consecutive-empty-observations requirement remains applicable. A single empty observation does not justify `quiesced`. An uncertainty-preserving outcome is acceptable when the selected budget cannot establish quiescence. Successful cleanup remains a separate fact from whether a live member was observed or a signal was required. ## Actual behavior All six uninstrumented CLI trials succeeded with exit code 0, `exit.reason: "exited"`, empty retained samples, `cleanup.owned_process_group: "verified"` and `cleanup.temporary_root: "removed"`. The following settlement results were observed: ```text build settle_ms state elapsed_ms cleanup_outcome source 0 alive_at_deadline 54 cleaned source 10 alive_at_deadline 53 cleaned source 100 quiesced 154 not_required release 0 alive_at_deadline 55 cleaned release 10 alive_at_deadline 53 cleaned release 100 quiesced 157 not_required ``` The production-function control recorded the actual empty observations used in the same invocation: ```text build settle_ms empty observations returned state elapsed_ms source 0 1 alive_at_deadline 55 source 10 1 alive_at_deadline 52 source 100 2 quiesced 155 release 0 1 alive_at_deadline 55 release 10 1 alive_at_deadline 53 release 100 2 quiesced 159 ``` Separate transparently observed CLI trials counted live members of the actual PTY root group during settlement and returned the original process table unchanged. The correctly instrumented source and release short cases each recorded exactly one zero-live-member observation and emitted `alive_at_deadline`. Their longer controls each recorded two zero-live-member observations and emitted `quiesced`. Retained samples were empty in those runs, so the PTY root group was the only selected group. This establishes emptiness from the same short run, without inference from the longer run. The emitted event journal records settlement event order but does not retain group observation values or the number of rounds. The clean root exit and empty retained samples alone are insufficient proof of settlement emptiness. Likewise, `cleanup_outcome: "cleaned"` confirms successful cleanup projection; it does not prove that cleanup signaled a live process or that the ownership sweep had no work. ## Evidence - Expected source: `AGENTS.md`, Product and Authority and Architecture and Cleanup, requires observations, inferences and unknowns to remain distinct; `docs/process-capture.md:176-182` describes settlement outcomes; `src/domain/process/processCapture.ts:174-193` defines observed settlement and cleanup pairing; `tests/acceptance/process/processFinalization.test.ts:318-344` expects `alive_at_deadline` for a surviving descendant. - Failure source: Independently executed CLI captures and same-invocation process-table observations on both builds, corroborated by the inline production-function control; `src/process/capture/ProcessCaptureLifecycle.ts:648-694` returns `alive_at_deadline` after the budget expires with one empty observation. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The public path validates the scenario, awaits the terminal exit, selects the PTY root plus sampled group leaders, and calls the settlement observer. Inspection failures or uncertain ownership return `unverifiable`; Windows also returns `unverifiable`. The tested cases used the POSIX branch, had no cancellation or inspection failure, and selected only the root group. The observer increments its empty count, fails to reach two, checks the expired budget and falls through to `alive_at_deadline`. The published runtime retains the same behavior. Related historical work at https://github.com/morluto/rea/pull/233 excludes zombies from live group observations. The current observer already includes that correction; this reproduction uses an absent live group and concerns the deadline classification after an empty observation. Open and closed issues and pull requests were searched, and no report with this exact failure identity was found. ## Restoration check For the same public short-budget scenario and the inline empty-group control, observe an empty group in the invocation that produces the result. When the budget expires before the second empty observation, verify that the result preserves uncertainty and does not claim `alive_at_deadline`. Do not turn a single empty observation into proof of quiescence. Keep the longer control reporting `quiesced` after two consecutive empty observations. Retain a discriminating genuinely live owned-group deadline control, such as the existing surviving-descendant acceptance fixture, and verify it can still report `alive_at_deadline` with successful cleanup. Retain inspection-failure and cancellation behavior. Verify cleanup outcome separately; changing the settlement label must not bypass owned-resource cleanup or its verification. No particular implementation is prescribed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Accepted settlement budgets can produce a false machine-readable liveness classification for an empty observation. The defect affects process Evidence semantics and can recur whenever the deadline ends observation after one empty round. All tested captures completed cleanup successfully, and no data loss, unowned-process action or failed resource release was observed. - Workaround status: available - Workaround basis: On the tested host and workload, increasing the settlement budget to 100 ms allowed two empty observations and returned quiesced in both builds. This is a demonstrated host-bound workaround. Process-table latency varies, so 100 ms is not a guaranteed threshold for another host or load; users must check that the chosen budget permits the required observations.
bsakaguchi 2026-10-15 18:27:39 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#716
No description provided.