[GH-ISSUE #1697] [Bug] Process capture can settle its event journal while already-due trailing observations are still queued #570

New issue

Closed

opened 2026-10-15 18:26:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:26:15 +00:00

Owner

Originally created by @JasonBuildAI on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1697

Area

Process capture or runtime observation (PTY lifecycle / Evidence completeness).

Observed behavior

settleProcessCaptureJournal (src/process/capture/ProcessCaptureLifecycle.ts) decides the capture event journal is quiet using wall-clock time only. When the Node event loop stalls longer than the quiet window (default quietMs = 25) and then resumes, callbacks that were already due during the stall can run after the settle's wake-up has already evaluated the journal. The settle returns first, the capture becomes immutable, and the trailing terminal observations pushed by those callbacks are missing from the result.

Concretely, with settleProcessCaptureJournal(journal, 20, 200):

a trailing observation is scheduled to append to the journal at t=12 ms;
the event loop then stalls synchronously for 40 ms;
on resume, the settle wake-up timer (due at t≈4 ms) runs before the trailing timer (due at t≈12 ms), sees no journal change, and declares silence.

Reproduction

Deterministic, unit level (observed on 6.3.0 @ ee46323b, Node 22.22.2, Windows 11 x64):

const journal = [{ capture_order: 0, collection: "frames", index: 0 }];
const lateFrame = new Promise<void>((resolve) => {
  setTimeout(() => {
    journal.push({ capture_order: 1, collection: "frames", index: 1 });
    resolve();
  }, 12);
});
const wait = settleProcessCaptureJournal(journal, 20, 200);
const stalledUntil = Date.now() + 40;
while (Date.now() < stalledUntil) {
  // deliberately stall the loop; no timer runs until this returns
}
await wait; // resolves while the trailing observation is already due

Observed:

FAIL src/process/capture/ProcessHarness.test.ts > does not settle while a trailing observation is already due behind a stalled event loop
AssertionError: expected [ { capture_order: +0, …(2) } ] to have a length of 2 but got 1

The same root cause makes the existing regression test waits for terminal observations delivered after the exit callback flaky under load: it failed 1 in 5 standalone runs and reliably inside a 65-file run on Windows.

Expected behavior

The settle should not declare silence while callbacks that were already due are still queued. Trailing observations delivered before the capture becomes immutable must be retained, with no silent loss of terminal output.

Impact and scope

Users of PTY-based process capture (process_capture / recorded-crash workflows) can receive an immutable capture that silently misses the terminal tail when the host stalls around exit; no truncation flag is raised.
The defect is platform-independent but easiest to hit on loaded hosts; on Windows the relevant test is not part of the curated Windows lane, so it is not caught there.
The existing regression test for this behavior fails intermittently in CI runs, blaming unrelated PRs.

REA version

6.3.0 (revision ee46323b).

Runtime and platform

Node 22.22.2; Windows 11 x64. The logic involved is platform-independent and applies to Linux/macOS under load as well.

Interface

CLI

Earliest failure stage

Cleanup, cancellation, or lifecycle

Analysis or suspected cause

Source-derived hypothesis with a reproducible unit-level observation: the settle loop sleeps 4 ms and compares wall-clock elapsed quiet time. During a host stall, its own wake-up timer and the trailing callback are both overdue when the loop resumes; due timers run in expiry order, so the earlier-due settle wake-up evaluates and returns before the trailing callback executes.

Originally created by @JasonBuildAI on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1697 ## Area Process capture or runtime observation (PTY lifecycle / Evidence completeness). ## Observed behavior `settleProcessCaptureJournal` (`src/process/capture/ProcessCaptureLifecycle.ts`) decides the capture event journal is quiet using wall-clock time only. When the Node event loop stalls longer than the quiet window (default `quietMs = 25`) and then resumes, callbacks that were already due during the stall can run after the settle's wake-up has already evaluated the journal. The settle returns first, the capture becomes immutable, and the trailing terminal observations pushed by those callbacks are missing from the result. Concretely, with `settleProcessCaptureJournal(journal, 20, 200)`: - a trailing observation is scheduled to append to the journal at t=12 ms; - the event loop then stalls synchronously for 40 ms; - on resume, the settle wake-up timer (due at t≈4 ms) runs before the trailing timer (due at t≈12 ms), sees no journal change, and declares silence. ## Reproduction Deterministic, unit level (observed on 6.3.0 @ ee46323b, Node 22.22.2, Windows 11 x64): ```ts const journal = [{ capture_order: 0, collection: "frames", index: 0 }]; const lateFrame = new Promise<void>((resolve) => { setTimeout(() => { journal.push({ capture_order: 1, collection: "frames", index: 1 }); resolve(); }, 12); }); const wait = settleProcessCaptureJournal(journal, 20, 200); const stalledUntil = Date.now() + 40; while (Date.now() < stalledUntil) { // deliberately stall the loop; no timer runs until this returns } await wait; // resolves while the trailing observation is already due ``` Observed: ``` FAIL src/process/capture/ProcessHarness.test.ts > does not settle while a trailing observation is already due behind a stalled event loop AssertionError: expected [ { capture_order: +0, …(2) } ] to have a length of 2 but got 1 ``` The same root cause makes the existing regression test `waits for terminal observations delivered after the exit callback` flaky under load: it failed 1 in 5 standalone runs and reliably inside a 65-file run on Windows. ## Expected behavior The settle should not declare silence while callbacks that were already due are still queued. Trailing observations delivered before the capture becomes immutable must be retained, with no silent loss of terminal output. ## Impact and scope - Users of PTY-based process capture (`process_capture` / recorded-crash workflows) can receive an immutable capture that silently misses the terminal tail when the host stalls around exit; no truncation flag is raised. - The defect is platform-independent but easiest to hit on loaded hosts; on Windows the relevant test is not part of the curated Windows lane, so it is not caught there. - The existing regression test for this behavior fails intermittently in CI runs, blaming unrelated PRs. ## REA version 6.3.0 (revision ee46323b). ## Runtime and platform Node 22.22.2; Windows 11 x64. The logic involved is platform-independent and applies to Linux/macOS under load as well. ## Interface CLI ## Earliest failure stage Cleanup, cancellation, or lifecycle ## Analysis or suspected cause Source-derived hypothesis with a reproducible unit-level observation: the settle loop sleeps 4 ms and compares wall-clock elapsed quiet time. During a host stall, its own wake-up timer and the trailing callback are both overdue when the loop resumes; due timers run in expiry order, so the earlier-due settle wake-up evaluates and returns before the trailing callback executes.

bsakaguchi

2026-10-15 18:26:15 +00:00