morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4713] [Bug] Linux cli-boundary worker intermittently exhausts its 384 MiB heap #2463

New issue

Closed

opened 2026-10-17 21:05:05 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-17 21:05:05 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4713

Before submitting

Searched open and closed issues for cliIntegerArguments, and searched worker/heap reports and open PRs. No matching worker failure report found.
Evidence below contains public source/Actions paths and no credentials or private target data.

Area

Other — CI / test-runner resources.

Observed behavior

Linux built-dependent shard 5 intermittently aborts its cli-boundary Vitest worker with Reached heap limit while running cliIntegerArguments.test.ts. The job then reports an unhandled SIGABRT rather than an assertion failure and blocks required gates.

Observed on two independently updated PRs whose patches do not change this CLI test or Vitest configuration:

#3803 job, head 5177d13f4824ef0aada5f14560edc209d43df888: worker heap OOM. Its unchanged failed-job rerun subsequently passed this shard.
#3776 job, head 31eb36d3b3ce7558b423b526e5fea41d7a9497dd: same worker/test OOM, 179 files and 1,577 tests passed before the final error. This PR changes only a Python bridge and its Python regression fixture.

Reproduction

Executed through existing GitHub Actions: approve the fork workflow, run the built-dependent lane, then rerun failed jobs without changing the commit or configuration. The affected job command is:

npm run test:built -- --reporter=default --reporter=blob --shard=5/5

OOM was observed in the two jobs linked above; one unchanged #3803 rerun passed. This is intermittent hosted evidence, not a deterministic local or clean-main reproduction. NOT_ENOUGH_INFO: a minimal reproducer, heap profile and failure frequency across a controlled series are not available.

Expected behavior

The built-dependent lane should complete within its declared worker/resource model, preserving integer-validation coverage. A worker OOM should not be hidden by retries, raised deadlines or skipped cases. The resource-owning test/runner boundary needs diagnosis before choosing a correction.

Impact and scope

Blocks required Portable tests / CI required gates for unrelated PRs and consumes another complete shard on retries. No product runtime crash, target corruption or real-provider failure was established by these logs. macOS/Windows and production workloads are unverified.

REA version

The two immutable source heads above both include main 5b243993872c2bcc70cb00060ffb79b347321370; REA package 6.5.0.

Runtime and platform

Hosted Linux x64 Actions, Node 24.18.0, Vitest 5.0.3. The cli-boundary fork project explicitly sets --max-old-space-size=384, maxWorkers 2, isolate false.

Target and artifact context

No external target required for this report; cliIntegerArguments.test.ts invokes the production CLI validation boundary.

Interface

Other — Vitest / CI.

Earliest failure stage

Unknown — resource failure while the CLI boundary test file is active; the exact allocation owner is not established.

Relevant output or evidence

From the #3776 job linked above:

Mark-Compact (reduce) 432.6 (506.5) -> 384.5 (395.7) MB
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Worker exited unexpectedly with signal SIGABRT ... cliIntegerArguments.test.ts
Test Files 179 passed | 1 skipped (181)
Tests 1577 passed | 14 skipped (1607)
Errors 1 error

Worker resource configuration at the tested revision.

Analysis or suspected cause

Source-derived: this test repeatedly creates fresh CLI instances through serveCli; its worker reuses a module graph and has a 384 MiB heap limit. Hypothesis only: retained module/schema allocations or transient CLI construction compete with that bound. A heap-retention cause, a specific production leak, and an appropriate replacement bound are not proven. #4360 reduces integer-schema allocations but has not been tested as a remedy for this worker failure.

Proposed follow-up, not executed: reproduce on pinned clean main with identical shard ordering and bounds; distinguish retained allocation from transient high-water demand; compare completed-task CPU/time/process-family memory before selecting a fix. Preserve malformed integer cases and full suite coverage.

Additional context

#3776's preceding attempt failed a different 60 s compiled-CLI dispatcher test. That timeout is a separate observation and is not claimed to share this root cause. The next unchanged #3776 attempt passed the affected shard and all required CI gates with the same settings. The allocation root cause remains unconfirmed; passing retries do not resolve this report.

Originally created by @N0zoM1z0 on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4713 ### Before submitting - [x] Searched open and closed issues for `cliIntegerArguments`, and searched worker/heap reports and open PRs. No matching worker failure report found. - [x] Evidence below contains public source/Actions paths and no credentials or private target data. ### Area Other — CI / test-runner resources. ### Observed behavior Linux built-dependent shard 5 intermittently aborts its cli-boundary Vitest worker with `Reached heap limit` while running `cliIntegerArguments.test.ts`. The job then reports an unhandled SIGABRT rather than an assertion failure and blocks required gates. Observed on two independently updated PRs whose patches do not change this CLI test or Vitest configuration: - [#3803 job](https://github.com/morluto/rea/actions/runs/39011262277/job/117334767938), head `5177d13f4824ef0aada5f14560edc209d43df888`: worker heap OOM. Its unchanged failed-job rerun subsequently passed this shard. - [#3776 job](https://github.com/morluto/rea/actions/runs/39011339111/job/117337779381), head `31eb36d3b3ce7558b423b526e5fea41d7a9497dd`: same worker/test OOM, 179 files and 1,577 tests passed before the final error. This PR changes only a Python bridge and its Python regression fixture. ### Reproduction Executed through existing GitHub Actions: approve the fork workflow, run the built-dependent lane, then rerun failed jobs without changing the commit or configuration. The affected job command is: ```sh npm run test:built -- --reporter=default --reporter=blob --shard=5/5 ``` OOM was observed in the two jobs linked above; one unchanged #3803 rerun passed. This is intermittent hosted evidence, **not a deterministic local or clean-main reproduction**. NOT_ENOUGH_INFO: a minimal reproducer, heap profile and failure frequency across a controlled series are not available. ### Expected behavior The built-dependent lane should complete within its declared worker/resource model, preserving integer-validation coverage. A worker OOM should not be hidden by retries, raised deadlines or skipped cases. The resource-owning test/runner boundary needs diagnosis before choosing a correction. ### Impact and scope Blocks required Portable tests / CI required gates for unrelated PRs and consumes another complete shard on retries. No product runtime crash, target corruption or real-provider failure was established by these logs. macOS/Windows and production workloads are unverified. ### REA version The two immutable source heads above both include main `5b243993872c2bcc70cb00060ffb79b347321370`; REA package 6.5.0. ### Runtime and platform Hosted Linux x64 Actions, Node 24.18.0, Vitest 5.0.3. The cli-boundary fork project explicitly sets `--max-old-space-size=384`, maxWorkers 2, isolate false. ### Target and artifact context No external target required for this report; `cliIntegerArguments.test.ts` invokes the production CLI validation boundary. ### Interface Other — Vitest / CI. ### Earliest failure stage Unknown — resource failure while the CLI boundary test file is active; the exact allocation owner is not established. ### Relevant output or evidence From the #3776 job linked above: ```text Mark-Compact (reduce) 432.6 (506.5) -> 384.5 (395.7) MB FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory Worker exited unexpectedly with signal SIGABRT ... cliIntegerArguments.test.ts Test Files 179 passed | 1 skipped (181) Tests 1577 passed | 14 skipped (1607) Errors 1 error ``` [Worker resource configuration at the tested revision](https://github.com/morluto/rea/blob/31eb36d3b3ce7558b423b526e5fea41d7a9497dd/vitest.config.ts#L60-L68). ### Analysis or suspected cause Source-derived: this test repeatedly creates fresh CLI instances through `serveCli`; its worker reuses a module graph and has a 384 MiB heap limit. Hypothesis only: retained module/schema allocations or transient CLI construction compete with that bound. A heap-retention cause, a specific production leak, and an appropriate replacement bound are **not proven**. #4360 reduces integer-schema allocations but has not been tested as a remedy for this worker failure. Proposed follow-up, not executed: reproduce on pinned clean main with identical shard ordering and bounds; distinguish retained allocation from transient high-water demand; compare completed-task CPU/time/process-family memory before selecting a fix. Preserve malformed integer cases and full suite coverage. ### Additional context #3776's preceding attempt failed a different 60 s compiled-CLI dispatcher test. That timeout is a separate observation and is not claimed to share this root cause. The next unchanged #3776 attempt passed [the affected shard](https://github.com/morluto/rea/actions/runs/39011339111/job/117339468898) and all required CI gates with the same settings. The allocation root cause remains unconfirmed; passing retries do not resolve this report.

bsakaguchi

2026-10-17 21:05:05 +00:00

closed this issue
added the
triage

bug
labels

bsakaguchi commented

2026-10-17 21:05:07 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 17, 2026):

This smells like a flaky-resource issue rather than a logic regression — the same cli-boundary worker OOMing on two unrelated PRs while unchanged reruns pass points to heap pressure in cliIntegerArguments.test.ts itself, not to either patch. A few diagnostics would help: capture the worker's RSS/JS heap at failure (or run with --heap-prof / --trace-gc) so the 384 MiB cap can be checked against actual peak usage; consider splitting that spec or moving it off the built-dependent shard; and if the cap is genuinely tight, the CI config should raise the worker memory instead of letting SIGABRT masquerade as a test failure — surfacing Reached heap limit as a distinct infra error would also stop required gates from being blocked by noise.

@XinwangTao commented on GitHub (Oct 17, 2026): This smells like a flaky-resource issue rather than a logic regression — the same cli-boundary worker OOMing on two unrelated PRs while unchanged reruns pass points to heap pressure in `cliIntegerArguments.test.ts` itself, not to either patch. A few diagnostics would help: capture the worker's RSS/JS heap at failure (or run with `--heap-prof` / `--trace-gc`) so the 384 MiB cap can be checked against actual peak usage; consider splitting that spec or moving it off the built-dependent shard; and if the cap is genuinely tight, the CI config should raise the worker memory instead of letting SIGABRT masquerade as a test failure — surfacing `Reached heap limit` as a distinct infra error would also stop required gates from being blocked by noise.

bsakaguchi commented

2026-10-17 21:05:07 +00:00

Author

Owner

@morluto commented on GitHub (Oct 17, 2026):

Read-only follow-up at e09a503f44; no local shard, stress test or profiler run.

The 32 integer CLI controls call serveCli, which creates a fresh CLI and registers the full command catalog on every invocation. This establishes repeated construction, not a retained leak.
cli-boundary still uses forks, isolate:false, maxWorkers <=2, and --max-old-space-size=384. Installed Vitest5.0.3 scheduler reuses compatible non-isolated runners; its getMemoryLimit returns null outside vmForks/vmThreads. Adding vmMemoryLimit would therefore not recycle this forks pool. Pinned Vitest configuration documentation matches the installed implementation.
Context7 and DeepWiki were research aids; installed package behavior was checked directly. ProcessOwnershipRunner uses weak host registration and worker cleanup; that observation does not exclude other allocation owners.
PR#4360 remains open, and no evidence here establishes it as a remedy.

Allocation owner, retained growth versus transient peak, Linux failure frequency and the appropriate correction remain unknown. No heap-cap increase, retries, skipped coverage or pool switch proposed from this source review alone. The existing hosted failures remain the runtime evidence.

@morluto commented on GitHub (Oct 17, 2026): Read-only follow-up at e09a503f44fba1639b71907a018d71161360f5ae; no local shard, stress test or profiler run. - The 32 integer CLI controls call serveCli, which creates a fresh CLI and registers the full command catalog on every invocation. This establishes repeated construction, not a retained leak. - cli-boundary still uses forks, isolate:false, maxWorkers <=2, and --max-old-space-size=384. Installed Vitest5.0.3 scheduler reuses compatible non-isolated runners; its getMemoryLimit returns null outside vmForks/vmThreads. Adding vmMemoryLimit would therefore not recycle this forks pool. [Pinned Vitest configuration documentation](https://github.com/vitest-dev/vitest/blob/v5.0.3/docs/config/vmmemorylimit.md) matches the installed implementation. - Context7 and DeepWiki were research aids; installed package behavior was checked directly. ProcessOwnershipRunner uses weak host registration and worker cleanup; that observation does not exclude other allocation owners. - PR#4360 remains open, and no evidence here establishes it as a remedy. Allocation owner, retained growth versus transient peak, Linux failure frequency and the appropriate correction remain unknown. No heap-cap increase, retries, skipped coverage or pool switch proposed from this source review alone. The existing hosted failures remain the runtime evidence.

bsakaguchi commented

2026-10-17 21:05:08 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 17, 2026):

Thanks for the read-only follow-up — the point that vmMemoryLimit does not apply to the forks pool (getMemoryLimit returns null outside vmForks/vmThreads) is a good catch, so bumping the Vitest worker memory through that knob would indeed be a no-op here.

Given that serveCli repeatedly constructs fresh CLI instances with the full command catalog registered every time, the 384 MiB cap looks like the real constraint: a transient peak during catalog construction on a loaded shard can plausibly exceed it even without a retained leak. Practical options that stay within the current architecture:

Raise --max-old-space-size for the cli-boundary worker (e.g. 512–640 MiB) and pin it in the config you already link, so the intermittent SIGABRT stops blocking unrelated PRs;
Split cliIntegerArguments.test.ts by control family so each shard keeps its per-worker peak lower;
Migrate that boundary pool from forks to vmForks so vmMemoryLimit becomes enforceable, which also gives you per-worker heap caps going forward.

Surfacing "Reached heap limit" as a distinct infra error (instead of an unhandled SIGABRT) would still be worth doing either way — it makes flaky-resource noise visible without burning a full retry shard. Happy to help validate with --heap-prof output if you can capture one from a failing run.

@XinwangTao commented on GitHub (Oct 17, 2026): Thanks for the read-only follow-up — the point that vmMemoryLimit does not apply to the forks pool (getMemoryLimit returns null outside vmForks/vmThreads) is a good catch, so bumping the Vitest worker memory through that knob would indeed be a no-op here. Given that serveCli repeatedly constructs fresh CLI instances with the full command catalog registered every time, the 384 MiB cap looks like the real constraint: a transient peak during catalog construction on a loaded shard can plausibly exceed it even without a retained leak. Practical options that stay within the current architecture: 1. Raise --max-old-space-size for the cli-boundary worker (e.g. 512–640 MiB) and pin it in the config you already link, so the intermittent SIGABRT stops blocking unrelated PRs; 2. Split cliIntegerArguments.test.ts by control family so each shard keeps its per-worker peak lower; 3. Migrate that boundary pool from forks to vmForks so vmMemoryLimit becomes enforceable, which also gives you per-worker heap caps going forward. Surfacing "Reached heap limit" as a distinct infra error (instead of an unhandled SIGABRT) would still be worth doing either way — it makes flaky-resource noise visible without burning a full retry shard. Happy to help validate with --heap-prof output if you can capture one from a failing run.

bsakaguchi referenced this issue

2026-10-31 01:05:08 +00:00

[PR #2463] [CLOSED] fix(mcp): accept HarmonyOS package formats in session target output #5246

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#2463

No description provided.

Rows
Columns