morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6114] [Bug] JavaScript worker I/O failures become unreadable provider output #3747

New issue

Closed

opened 2026-10-31 00:43:36 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:43:36 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6114

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A real owned JavaScript analyzer emitted a valid failure frame with reason: "io" after its private source file was removed before its read. The parent returned AnalysisOutputError, projected as unreadable_output, and instructed the caller to run rea doctor. The actual failure was ENOENT on private worker storage, before any output was produced.

This was deterministic filesystem fault injection through the production worker launcher. The worker process, request schema, source read, failure frame, parent decoder and public error projection were real. It was not a naturally occurring storage incident or a full CLI/MCP fault replay.

Reproduction

In the prepared checkout, save this test as tests/boundary/javascript/workerIo.repro.test.ts, then run:

node node_modules/vitest/vitest.mjs run tests/boundary/javascript/workerIo.repro.test.ts --maxWorkers=1

The relevant executed test is below. It removes the parent-owned input immediately before the request is sent, observes the worker's actual failure frame, and closes the worker in finally. The assertions describe the current incorrect behavior. An independent small-source control completed successfully without the fault.

import { createHash } from 'node:crypto';
import { unlinkSync } from 'node:fs';
import { join } from 'node:path';
import { expect, it } from 'vitest';
import { JavaScriptAnalysisWorker } from '../../../src/javascript/analysis/JavaScriptAnalysisWorker.js';
import { spawnOwnedProviderProcess } from '../../../src/process/ProviderProcess.js';
import { projectAnalysisError } from '../../../src/domain/analysisErrorProjection.js';
const source = (value: string) => ({ path: 'app.mjs', sha256: createHash('sha256').update(value).digest('hex'), container_sha256: '1'.repeat(64), bytes: Buffer.byteLength(value), inventory_artifact_id: 'source', kind: 'javascript' as const, unpacked: false, text: { included: true as const, value } });
it('real worker source I/O is projected as unreadable output', async () => {
 let injected=false; let wireFailure: any;
 const worker = new JavaScriptAnalysisWorker({heapMb:128,timeoutMs:30000}, async request => {
  const launched=await spawnOwnedProviderProcess(request);
  launched.process.stdout?.on('data', chunk => { for(const line of String(chunk).trim().split('\n')) { try { const frame=JSON.parse(line); if(frame.kind==='failure')wireFailure=frame; }catch{} } });
  const stdin=launched.process.stdin!; const original=stdin.write.bind(stdin);
  stdin.write = ((frame:any,...args:any[])=>{
   const data=JSON.parse(String(frame));
   if (data.file && !injected) { unlinkSync(join(request.cwd!,`${data.id}.source`)); injected=true; }
   return original(frame,...args);
  }) as typeof stdin.write;
  return launched;
 });
 try {
  const result=await worker.analyze(source('export const x=1;'),{nodes:100,unknowns:100});
  if(result.ok) throw new Error('Expected checkpoint I/O failure');
  const projected=projectAnalysisError(result.error.error);
  console.log('WORKER_IO_WIRE',JSON.stringify(wireFailure)); console.log('WORKER_IO',JSON.stringify(projected)); expect(wireFailure?.reason).toBe('io');
  expect(injected).toBe(true); expect(projected.code).toBe('unreadable_output'); expect(JSON.stringify(projected)).toContain('ENOENT');
 } finally { await worker.close(); }
});

The complete focused observation run passed four tests, including this one. No response frames were fabricated.

Expected behavior

Keep the storage/execution failure distinct from malformed provider output. Preserve the source identity, worker phase and available I/O diagnostics in a suitable execution-failure result. The protocol schema already distinguishes io from output, and the parent's ordinary environment-error fallback retains source/phase diagnostics in ProviderAdapterError.

Impact and scope

P2: typed consumers receive an unreadable-result classification for a private filesystem failure and are directed to provider diagnosis. The errno and private path survive only in the free-form reason, so manual interpretation is possible. The source was intentionally removed only in an isolated owned runtime; no selected input was modified.

Only the missing-source read was executed. Other local checkpoint/transfer writes reach the same server catch and parent branch by source inspection; real ENOSPC, read-only storage and natural race frequency were not tested.

REA version

rea-agents 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

Node.js v24.18.0 (repository pin), macOS 27.0.1 arm64, Vitest 5.0.3. Actual separately owned JavaScript analyzer subprocess; no external provider.

Target and artifact context

Synthetic inert JavaScript export const x=1;; the parent-owned private source copy is deleted at the request boundary.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed worker frame fields, with generated identity and local path normalized:

{"kind":"failure","reason":"io","resource":null,"limits":null,"message":"ENOENT: no such file or directory, open '<private-runtime>/<job>.source'"}

Observed parent projection:

{
  "code": "unreadable_output",
  "category": "execution_failure",
  "message": "Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.",
  "retryable": false,
  "details": {
    "operation": "analyze_javascript_application",
    "reason": "app.mjs during source-transfer: ENOENT: no such file or directory, open '<private-runtime>/<job>.source'"
  }
}

The projection also contained the same message as its remediation action and captured-output metadata. No cleanup failure occurred in this final reproduction.

Typed outcome or recovery expected

An execution/storage failure retaining the current source path, worker phase and serialized I/O message, rather than AnalysisOutputError / unreadable_output. Do not prescribe source repair for an unavailable private worker file.

Analysis or suspected cause

The server reads the private source and serializes its exception as io. The parent handles resource and then all other reasons as output. The public application consumer projects worker errors per file. The missing io reconstruction branch is separate from errors reading the caller-selected artifact.

Additional context

#5991 concerns raw application input/reconstruction errors collapsed by a different service catch; changing that catch cannot repair the worker's io protocol reconstruction. #5969/#5970 concern source-recovery identity/read admission. #6083/#6092 concern cleanup/projection or cancellation. PR #4997 concerns worker memory guidance.

Upstream main aa51e4343963fdd0b97ec2ebdc3c41579fd9cb38 was fetched and the relevant protocol-failure code is unchanged. That newer revision was source-inspected, not executed. No fix, restoration or last-known-good version was established.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6114 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A real owned JavaScript analyzer emitted a valid failure frame with `reason: "io"` after its private source file was removed before its read. The parent returned `AnalysisOutputError`, projected as `unreadable_output`, and instructed the caller to run `rea doctor`. The actual failure was `ENOENT` on private worker storage, before any output was produced. This was deterministic filesystem fault injection through the production worker launcher. The worker process, request schema, source read, failure frame, parent decoder and public error projection were real. It was not a naturally occurring storage incident or a full CLI/MCP fault replay. ### Reproduction In the prepared checkout, save this test as `tests/boundary/javascript/workerIo.repro.test.ts`, then run: ```sh node node_modules/vitest/vitest.mjs run tests/boundary/javascript/workerIo.repro.test.ts --maxWorkers=1 ``` The relevant executed test is below. It removes the parent-owned input immediately before the request is sent, observes the worker's actual failure frame, and closes the worker in `finally`. The assertions describe the current incorrect behavior. An independent small-source control completed successfully without the fault. ```ts import { createHash } from 'node:crypto'; import { unlinkSync } from 'node:fs'; import { join } from 'node:path'; import { expect, it } from 'vitest'; import { JavaScriptAnalysisWorker } from '../../../src/javascript/analysis/JavaScriptAnalysisWorker.js'; import { spawnOwnedProviderProcess } from '../../../src/process/ProviderProcess.js'; import { projectAnalysisError } from '../../../src/domain/analysisErrorProjection.js'; const source = (value: string) => ({ path: 'app.mjs', sha256: createHash('sha256').update(value).digest('hex'), container_sha256: '1'.repeat(64), bytes: Buffer.byteLength(value), inventory_artifact_id: 'source', kind: 'javascript' as const, unpacked: false, text: { included: true as const, value } }); it('real worker source I/O is projected as unreadable output', async () => { let injected=false; let wireFailure: any; const worker = new JavaScriptAnalysisWorker({heapMb:128,timeoutMs:30000}, async request => { const launched=await spawnOwnedProviderProcess(request); launched.process.stdout?.on('data', chunk => { for(const line of String(chunk).trim().split('\n')) { try { const frame=JSON.parse(line); if(frame.kind==='failure')wireFailure=frame; }catch{} } }); const stdin=launched.process.stdin!; const original=stdin.write.bind(stdin); stdin.write = ((frame:any,...args:any[])=>{ const data=JSON.parse(String(frame)); if (data.file && !injected) { unlinkSync(join(request.cwd!,`${data.id}.source`)); injected=true; } return original(frame,...args); }) as typeof stdin.write; return launched; }); try { const result=await worker.analyze(source('export const x=1;'),{nodes:100,unknowns:100}); if(result.ok) throw new Error('Expected checkpoint I/O failure'); const projected=projectAnalysisError(result.error.error); console.log('WORKER_IO_WIRE',JSON.stringify(wireFailure)); console.log('WORKER_IO',JSON.stringify(projected)); expect(wireFailure?.reason).toBe('io'); expect(injected).toBe(true); expect(projected.code).toBe('unreadable_output'); expect(JSON.stringify(projected)).toContain('ENOENT'); } finally { await worker.close(); } }); ``` The complete focused observation run passed four tests, including this one. No response frames were fabricated. ### Expected behavior Keep the storage/execution failure distinct from malformed provider output. Preserve the source identity, worker phase and available I/O diagnostics in a suitable execution-failure result. The [protocol schema](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/javascript/analysis/JavaScriptAnalysisWorkerProtocol.ts#L66-L73) already distinguishes `io` from `output`, and the parent's [ordinary environment-error fallback](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/javascript/analysis/JavaScriptAnalysisWorker.ts#L1073-L1088) retains source/phase diagnostics in `ProviderAdapterError`. ### Impact and scope P2: typed consumers receive an unreadable-result classification for a private filesystem failure and are directed to provider diagnosis. The errno and private path survive only in the free-form reason, so manual interpretation is possible. The source was intentionally removed only in an isolated owned runtime; no selected input was modified. Only the missing-source read was executed. Other local checkpoint/transfer writes reach the same server catch and parent branch by source inspection; real ENOSPC, read-only storage and natural race frequency were not tested. ### REA version rea-agents 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform Node.js v24.18.0 (repository pin), macOS 27.0.1 arm64, Vitest 5.0.3. Actual separately owned JavaScript analyzer subprocess; no external provider. ### Target and artifact context Synthetic inert JavaScript `export const x=1;`; the parent-owned private source copy is deleted at the request boundary. ### Interface Other ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed worker frame fields, with generated identity and local path normalized: ```json {"kind":"failure","reason":"io","resource":null,"limits":null,"message":"ENOENT: no such file or directory, open '<private-runtime>/<job>.source'"} ``` Observed parent projection: ```json { "code": "unreadable_output", "category": "execution_failure", "message": "Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.", "retryable": false, "details": { "operation": "analyze_javascript_application", "reason": "app.mjs during source-transfer: ENOENT: no such file or directory, open '<private-runtime>/<job>.source'" } } ``` The projection also contained the same message as its remediation action and captured-output metadata. No cleanup failure occurred in this final reproduction. ### Typed outcome or recovery expected An execution/storage failure retaining the current source path, worker phase and serialized I/O message, rather than `AnalysisOutputError` / `unreadable_output`. Do not prescribe source repair for an unavailable private worker file. ### Analysis or suspected cause The [server reads the private source](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts#L100-L112) and [serializes its exception as io](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/javascript/analysis/JavaScriptAnalysisWorkerServer.ts#L172-L200). The [parent handles resource and then all other reasons as output](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/javascript/analysis/JavaScriptAnalysisWorker.ts#L988-L1004). The public application consumer [projects worker errors per file](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysis.ts#L177-L200). The missing `io` reconstruction branch is separate from errors reading the caller-selected artifact. ### Additional context #5991 concerns raw application input/reconstruction errors collapsed by a different service catch; changing that catch cannot repair the worker's `io` protocol reconstruction. #5969/#5970 concern source-recovery identity/read admission. #6083/#6092 concern cleanup/projection or cancellation. PR #4997 concerns worker memory guidance. Upstream main `aa51e4343963fdd0b97ec2ebdc3c41579fd9cb38` was fetched and the relevant protocol-failure code is unchanged. That newer revision was source-inspected, not executed. No fix, restoration or last-known-good version was established.

bsakaguchi closed this issue

2026-10-31 00:43:36 +00:00

bsakaguchi commented

2026-10-31 00:43:37 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

确认，worker 侧故障被父进程解码成 unreadable_output，是投影链路的二次失真：worker 明明发出了带 reason: "io" 的 failure frame（真实原因是私有存储 ENOENT、在产出任何输出之前），父进程却放弃了这个结构化信号，换成泛化的 "output unreadable + run rea doctor"。调用者既拿不到真正缺失的源文件路径，还被引去查完全无关的 doctor。建议父进程优先按 failure frame 的 reason/errno 投影（ENOENT → filesystem-read 缺失分类并带 path），只有 frame 缺失或不可解码时才退回 unreadable_output；rea doctor 只留给真正无法归因的 case。

@XinwangTao commented on GitHub (Oct 22, 2026): 确认，worker 侧故障被父进程解码成 unreadable_output，是投影链路的二次失真：worker 明明发出了带 reason: "io" 的 failure frame（真实原因是私有存储 ENOENT、在产出任何输出之前），父进程却放弃了这个结构化信号，换成泛化的 "output unreadable + run rea doctor"。调用者既拿不到真正缺失的源文件路径，还被引去查完全无关的 doctor。建议父进程优先按 failure frame 的 reason/errno 投影（ENOENT → filesystem-read 缺失分类并带 path），只有 frame 缺失或不可解码时才退回 unreadable_output；rea doctor 只留给真正无法归因的 case。

bsakaguchi commented

2026-10-31 00:43:37 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

The protocol failure decoder now handles reason io as ProviderAdapterError with source_path, retained phase, worker_failure_reason and serialized message, plus private-storage recovery. Output remains AnalysisOutputError. The test feeds the same valid failure frame through an owned child for io and output and asserts execution_failure versus unreadable_output, ENOENT text, source-transfer phase, and storage wording.

Implementation: JavaScriptAnalysisWorker.ts, workerFailureComposition.test.ts.

The relevant regression checks in workerFailureComposition.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. The protocol failure decoder now handles reason io as ProviderAdapterError with source_path, retained phase, worker_failure_reason and serialized message, plus private-storage recovery. Output remains AnalysisOutputError. The test feeds the same valid failure frame through an owned child for io and output and asserts execution_failure versus unreadable_output, ENOENT text, source-transfer phase, and storage wording. Implementation: [JavaScriptAnalysisWorker.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/javascript/analysis/JavaScriptAnalysisWorker.ts#L1040), [workerFailureComposition.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/javascript/workerFailureComposition.test.ts#L77). The relevant regression checks in [workerFailureComposition.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/javascript/workerFailureComposition.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-11-01 21:10:58 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3747

No description provided.

Rows
Columns