morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3894] [Bug] Piped --json validation errors are truncated at 512 bytes and fail to parse #1515

New issue

Closed

opened 2026-10-16 21:06:26 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:06:26 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3894

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

CLI

Observed behavior

When stdout is a pipe, rea ... --json can exit 1 after writing only the first 512 bytes of a VALIDATION_ERROR document, cutting it off inside a JSON string. Redirecting the same command to a regular file produces the complete document. JSON consumers that read through a pipe, as agents and scripts usually do, cannot parse the error or read its correction fields.

Through a pipe, both commands exit 1 with exactly 512 bytes of stdout that end mid-string, and JSON parsing fails. Redirected to a regular file, the same commands write complete VALIDATION_ERROR documents of 528 and 530 bytes. The truncation is timing-dependent: in a later rerun on the same machine with wc -c as the reader, inspect-keyed-archive truncated at 512 bytes in 9 of 10 runs and trace-web-source-location in 5 of 10.

Reproduction

In an empty scratch directory, run this script. No input files are needed because argument validation fails first, and the source-map URL is never accessed. It compares a fully consumed stdout pipe with a regular-file redirect for two commands with invalid numeric arguments.

python3 - <<'PYREPRO'
import json, subprocess, tempfile
cases = [['trace-web-source-location', 'manifest.json', '0', 'map.json', 'file:///script.map', '0', '0', '--json'], ['inspect-keyed-archive', 'archive.plist', '--limit', '0', '--json']]
for args in cases:
    command = ['rea', *args]
    pipe = subprocess.run(command, capture_output=True, timeout=15)
    try:
        json.loads(pipe.stdout); valid = True
    except json.JSONDecodeError:
        valid = False
    print(args[0], 'pipe exit=' + str(pipe.returncode), 'bytes=' + str(len(pipe.stdout)), 'valid_json=' + str(valid), 'tail=' + repr(pipe.stdout[-70:]))
    with tempfile.TemporaryFile() as output:
        file = subprocess.run(command, stdout=output, stderr=subprocess.PIPE, timeout=15)
        output.seek(0); raw = output.read()
    print(args[0], 'file exit=' + str(file.returncode), 'bytes=' + str(len(raw)), 'code=' + json.loads(raw)['code'])
PYREPRO

Optionally repeat a single pipe read to see the frequency.

for i in 1 2 3 4 5 6 7 8 9 10; do rea inspect-keyed-archive archive.plist --limit 0 --json | wc -c; done | sort | uniq -c

Expected behavior

--json output is documented for JSON consumers, including pipes, and exit status 1 is documented to come with structured output that identifies invalid input. Both commands should exit 1 and write the complete VALIDATION_ERROR JSON document to stdout regardless of whether stdout is a pipe or a file.

Impact and scope

JSON-consuming agents and scripts reading through a pipe lose the structured validation error, including its correction fields, on most observed runs; the exit status still signals failure.

Workaround: available. Redirecting stdout to a regular file and parsing that file yields the complete document, as observed in the file control.

Reporter's impact estimate: P2 (supported).

REA version

6.5.0, built from source at main commit 034342ba28 (latest npm release is 6.5.0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64

Target and artifact context

not applicable

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Expected source: docs/cli.md:316-321 (use --json for JSON consumers; compact "when piped or redirected") and docs/cli.md:337 (exit 1: "Structured output identifies invalid input"); argument minimums at src/cli/webSourceLocationCommands.ts:34 and src/cli/artifactCommands.ts:41
Failure source: pipe-versus-file comparison through the public CLI shown below; suspected cause at scripts/rea.mjs:106 and scripts/rea.mjs:131-132
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Output of step 1 on the tested build:

trace-web-source-location pipe exit=1 bytes=512 valid_json=False tail=b'  "received": "",\n      "message": "Too small: expected number to be >'
trace-web-source-location file exit=1 bytes=528 code=VALIDATION_ERROR
inspect-keyed-archive pipe exit=1 bytes=512 valid_json=False tail=b'    "received": "",\n      "message": "Too small: expected number to be'
inspect-keyed-archive file exit=1 bytes=530 code=VALIDATION_ERROR

Repeated pipe reads (step 2 style) on the same build:

inspect-keyed-archive:      9 x 512 bytes, 1 x 530 bytes
trace-web-source-location:  5 x 512 bytes, 5 x 528 bytes

A broader CLI sweep saw the same 512-byte truncation in ten validation outputs across other invalid-argument cases. Only macOS with Node.js v26.10.0 was tested.

Suspected cause (source-derived, not instrumented): scripts/rea.mjs:106 passes Incur an exit callback that calls process.exit immediately, while writeStdout at scripts/rea.mjs:131-132 calls process.stdout.write without waiting for it to drain. Node.js writes to pipes asynchronously on macOS, so output still pending at process.exit can be discarded; regular-file writes are synchronous, which matches the file control.

Typed outcome or recovery expected

Exit status 1 with the complete VALIDATION_ERROR JSON document on stdout, identical for pipes and regular files.

Restoration check:

On the tested build, the step 1 pipe lines show bytes=512 valid_json=False; after a fix, both pipe lines should show the full document (bytes=528 and bytes=530, valid_json=True) with exit 1, matching the file lines. The step 2 loop should report only complete sizes, and the file-redirect control must remain unchanged.

Analysis or suspected cause

Source-derived hypothesis: scripts/rea.mjs:106 exits through process.exit immediately while scripts/rea.mjs:131-132 writes stdout without awaiting drain, so asynchronous pipe output beyond the first chunk is lost at exit.

Additional context

Open #3543 concerns a macOS validation test exceeding its exit deadline after emitting VALIDATION_ERROR, not prompt exit with incomplete JSON. Closed #3221 concerned oversized MCP validation errors, a different transport.

This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3894 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area CLI ### Observed behavior When stdout is a pipe, `rea ... --json` can exit 1 after writing only the first 512 bytes of a `VALIDATION_ERROR` document, cutting it off inside a JSON string. Redirecting the same command to a regular file produces the complete document. JSON consumers that read through a pipe, as agents and scripts usually do, cannot parse the error or read its correction fields. Through a pipe, both commands exit 1 with exactly 512 bytes of stdout that end mid-string, and JSON parsing fails. Redirected to a regular file, the same commands write complete `VALIDATION_ERROR` documents of 528 and 530 bytes. The truncation is timing-dependent: in a later rerun on the same machine with `wc -c` as the reader, `inspect-keyed-archive` truncated at 512 bytes in 9 of 10 runs and `trace-web-source-location` in 5 of 10. ### Reproduction 1. In an empty scratch directory, run this script. No input files are needed because argument validation fails first, and the source-map URL is never accessed. It compares a fully consumed stdout pipe with a regular-file redirect for two commands with invalid numeric arguments. ```sh python3 - <<'PYREPRO' import json, subprocess, tempfile cases = [['trace-web-source-location', 'manifest.json', '0', 'map.json', 'file:///script.map', '0', '0', '--json'], ['inspect-keyed-archive', 'archive.plist', '--limit', '0', '--json']] for args in cases: command = ['rea', *args] pipe = subprocess.run(command, capture_output=True, timeout=15) try: json.loads(pipe.stdout); valid = True except json.JSONDecodeError: valid = False print(args[0], 'pipe exit=' + str(pipe.returncode), 'bytes=' + str(len(pipe.stdout)), 'valid_json=' + str(valid), 'tail=' + repr(pipe.stdout[-70:])) with tempfile.TemporaryFile() as output: file = subprocess.run(command, stdout=output, stderr=subprocess.PIPE, timeout=15) output.seek(0); raw = output.read() print(args[0], 'file exit=' + str(file.returncode), 'bytes=' + str(len(raw)), 'code=' + json.loads(raw)['code']) PYREPRO ``` 2. Optionally repeat a single pipe read to see the frequency. ```sh for i in 1 2 3 4 5 6 7 8 9 10; do rea inspect-keyed-archive archive.plist --limit 0 --json | wc -c; done | sort | uniq -c ``` ### Expected behavior `--json` output is documented for JSON consumers, including pipes, and exit status 1 is documented to come with structured output that identifies invalid input. Both commands should exit 1 and write the complete `VALIDATION_ERROR` JSON document to stdout regardless of whether stdout is a pipe or a file. ### Impact and scope JSON-consuming agents and scripts reading through a pipe lose the structured validation error, including its correction fields, on most observed runs; the exit status still signals failure. Workaround: available. Redirecting stdout to a regular file and parsing that file yields the complete document, as observed in the file control. Reporter's impact estimate: P2 (supported). ### REA version 6.5.0, built from source at main commit 034342ba28c8e56c3746be92b78aa7299afa02bd (latest npm release is 6.5.0) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64 ### Target and artifact context not applicable ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence - Expected source: docs/cli.md:316-321 (use `--json` for JSON consumers; compact "when piped or redirected") and docs/cli.md:337 (exit 1: "Structured output identifies invalid input"); argument minimums at src/cli/webSourceLocationCommands.ts:34 and src/cli/artifactCommands.ts:41 - Failure source: pipe-versus-file comparison through the public CLI shown below; suspected cause at scripts/rea.mjs:106 and scripts/rea.mjs:131-132 - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Output of step 1 on the tested build: ```text trace-web-source-location pipe exit=1 bytes=512 valid_json=False tail=b' "received": "",\n "message": "Too small: expected number to be >' trace-web-source-location file exit=1 bytes=528 code=VALIDATION_ERROR inspect-keyed-archive pipe exit=1 bytes=512 valid_json=False tail=b' "received": "",\n "message": "Too small: expected number to be' inspect-keyed-archive file exit=1 bytes=530 code=VALIDATION_ERROR ``` Repeated pipe reads (step 2 style) on the same build: ```text inspect-keyed-archive: 9 x 512 bytes, 1 x 530 bytes trace-web-source-location: 5 x 512 bytes, 5 x 528 bytes ``` A broader CLI sweep saw the same 512-byte truncation in ten validation outputs across other invalid-argument cases. Only macOS with Node.js v26.10.0 was tested. Suspected cause (source-derived, not instrumented): scripts/rea.mjs:106 passes Incur an `exit` callback that calls `process.exit` immediately, while `writeStdout` at scripts/rea.mjs:131-132 calls `process.stdout.write` without waiting for it to drain. Node.js writes to pipes asynchronously on macOS, so output still pending at `process.exit` can be discarded; regular-file writes are synchronous, which matches the file control. ### Typed outcome or recovery expected Exit status 1 with the complete VALIDATION_ERROR JSON document on stdout, identical for pipes and regular files. Restoration check: On the tested build, the step 1 pipe lines show `bytes=512 valid_json=False`; after a fix, both pipe lines should show the full document (`bytes=528` and `bytes=530`, `valid_json=True`) with exit 1, matching the file lines. The step 2 loop should report only complete sizes, and the file-redirect control must remain unchanged. ### Analysis or suspected cause Source-derived hypothesis: scripts/rea.mjs:106 exits through process.exit immediately while scripts/rea.mjs:131-132 writes stdout without awaiting drain, so asynchronous pipe output beyond the first chunk is lost at exit. ### Additional context Open #3543 concerns a macOS validation test exceeding its exit deadline after emitting `VALIDATION_ERROR`, not prompt exit with incomplete JSON. Closed #3221 concerned oversized MCP validation errors, a different transport. This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

bsakaguchi commented

2026-10-16 21:06:27 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

This is a nasty one for scripted consumers: an agent reading --json output through a pipe gets invalid JSON that fails at parse time, and the failure mode is timing-dependent (9/10 vs 5/10 runs), which makes it very hard to debug downstream. Since the same command writes the complete document to a regular file, the truncation is clearly in the stdout write path (likely a short write on a non-blocking pipe), not in the document construction. Suggestion: retry short writes on EAGAIN/EPIPE for the validation error path, or better, serialize the full document to a buffer and perform a single blocking write — and add a pipe-based JSON parse test to catch regressions, since this class of bug is invisible in file-redirect tests.

@XinwangTao commented on GitHub (Oct 15, 2026): This is a nasty one for scripted consumers: an agent reading --json output through a pipe gets invalid JSON that fails at parse time, and the failure mode is timing-dependent (9/10 vs 5/10 runs), which makes it very hard to debug downstream. Since the same command writes the complete document to a regular file, the truncation is clearly in the stdout write path (likely a short write on a non-blocking pipe), not in the document construction. Suggestion: retry short writes on EAGAIN/EPIPE for the validation error path, or better, serialize the full document to a buffer and perform a single blocking write — and add a pipe-based JSON parse test to catch regressions, since this class of bug is invisible in file-redirect tests.

bsakaguchi referenced this issue

2026-10-17 21:12:01 +00:00

[PR #1515] [MERGED] fix(apple): bound binary plist expansion before decoding keyed archives #2935

bsakaguchi closed this issue

2026-10-31 00:31:12 +00:00

bsakaguchi commented

2026-10-31 00:31:13 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked resolved on main at d87e73f2db.

I replayed both original validation commands on pinned Node 24.22.1 through capture pipes, wc -c pipes and regular files. Both now produce complete JSON with exit 1 and exact pipe/file agreement. Each original case passed ten reads of each pipe form plus its file control. Since those current error documents are smaller than 512 bytes, I also checked a 735-byte three-field validation error: ten capture-pipe reads, ten byte-count pipe reads, three slow-reader checks and a matching file redirect all passed. The launcher now awaits queued output completion before explicit exit. This does not claim a replay on the old Node version.

Relevant implementation: rea.mjs, cliResultOutput.ts.

Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked resolved on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. I replayed both original validation commands on pinned Node 24.22.1 through capture pipes, `wc -c` pipes and regular files. Both now produce complete JSON with exit 1 and exact pipe/file agreement. Each original case passed ten reads of each pipe form plus its file control. Since those current error documents are smaller than 512 bytes, I also checked a 735-byte three-field validation error: ten capture-pipe reads, ten byte-count pipe reads, three slow-reader checks and a matching file redirect all passed. The launcher now awaits queued output completion before explicit exit. This does not claim a replay on the old Node version. Relevant implementation: [rea.mjs](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/scripts/rea.mjs#L154), [cliResultOutput.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/cli/cliResultOutput.ts#L236). Closing as completed on main.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1515

No description provided.

Rows
Columns