[GH-ISSUE #3888] [Bug] Firmware extraction byte budget also caps the mandatory producer report #1507

Closed
opened 2026-10-16 21:06:20 +00:00 by bsakaguchi · 3 comments
Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3888

Before submitting

  • I searched the open and closed issues for a related report.
  • I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

extract-firmware --max-output-bytes is forwarded as the process-wide prlimit --fsize ceiling, so it limits not only staged payload files but also the mandatory report.json the producer must write; with the accepted value 1, no valid report can fit even when no extracted file exceeds the budget.

The extraction passes its staging byte budget into runFirmwareCommand, which emits it as the process-wide --fsize ceiling at src/firmware/FirmwareCommand.ts:125, and util-linux applies RLIMIT_FSIZE to every regular file the producer writes, including the mandatory report and log outside the staging tree.

With the accepted value 1 no valid required report can fit: an empty JSON array needs two bytes, and an accepted report requires a root task, a StatReport, and a SHA-256 HashReport per src/firmware/FirmwareReports.ts:186-200 and lines 317-322.

After the command src/firmware/FirmwareProvider.ts:356 unconditionally reads that report and src/firmware/FirmwareFiles.ts:150-172 requires complete valid JSON, so failure precedes result publication regardless of whether extracted payload files fit the selected budget, and the 8 MiB report-reader allowance never helps.

Reproduction

A local probe imported dist/domain/firmware/firmwareAnalysis.js and dist/firmware/FirmwareCommand.js from a built source checkout, parsed extraction input with max_output_bytes: 1, then called runFirmwareCommand with that parsed budget and an injected launcher that printed the file-size argument and threw a sentinel before spawning any process:

{"accepted_max_output_bytes":1}
{"file_size_argument":"--fsize=1"}

The public CLI was also run locally with --max-output-bytes 1; it passed option validation and returned the expected macOS capability_unavailable refusal, which does not establish Linux extraction behavior.

Expected behavior

src/domain/firmware/firmwareAnalysis.ts:40 admits max_output_bytes from 1 and describes it as the staging file-byte budget, and docs/firmware-analysis.md:161 gives producer JSON reports a separate 8 MiB budget while lines 164-166 describe the configurable staging budget.

Selecting a small staging budget should constrain extracted files rather than make the mandatory report impossible to write, even when no extracted file exceeds that budget.

src/firmware/FirmwareRelease.ts:31 and src/firmware/FirmwareFiles.ts:154 separately allow reports up to 8 MiB, establishing the intended separation of the two budgets.

Impact and scope

low extraction budgets can prevent analysis solely because mandatory metadata or logs exceed the payload allowance, and the one-byte accepted boundary cannot produce any valid report even for an analysis yielding no published payload; impact is bounded to selected budgets.

Workaround: unverified. raising the budget to the default is a plausible workaround but was not verified with a real producer and gives up the requested tighter staging limit.

Reporter's impact estimate: P2 (supported).

REA version

6.5.0, built from source at main commit 034342ba28 (latest npm release is 6.5.0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64; This is a source-only finding for the public Linux defect; schema acceptance and the production launcher-argument generation were executed locally, but no real Unblob/binwalk producer was spawned. End-to-end runtime reproduction with a real producer was not run.

Target and artifact context

Firmware image file (Linux Unblob extraction)

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

  • Expected source: src/domain/firmware/firmwareAnalysis.ts:40; separate report budget at docs/firmware-analysis.md:161 and lines 164-166; 8 MiB report allowance at src/firmware/FirmwareRelease.ts:31 and src/firmware/FirmwareFiles.ts:154
  • Failure source: observed --fsize=1 emitted from the accepted budget (output below); static chain at src/firmware/FirmwareProvider.ts:350-353 (passes max_output_bytes as outputBudget), src/firmware/FirmwareCommand.ts:125 (emits --fsize), and src/firmware/FirmwareReports.ts:186-200 (minimum valid report cannot fit one byte)
  • Evidence provenance: static
  • Local verification: static-proof
  • Reproduction completeness: incomplete
  • Missing fact: a real Unblob/binwalk extraction on a supported Linux host confirming that the mandatory report fails to write (or truncates) under the one-byte budget was not executed; only schema acceptance and the --fsize launcher argument were observed. End-to-end runtime reproduction with a real producer was not run.
{"accepted_max_output_bytes":1}
{"file_size_argument":"--fsize=1"}

Typed outcome or recovery expected

A small --max-output-bytes staging budget should constrain extracted payload files only; the mandatory report.json and log should be written under their own report-size allowance so a non-over-budget extraction still publishes a valid result.

Restoration check:

Failing: extract-firmware --max-output-bytes 1 admits the request and emits --fsize=1, which caps the mandatory report.json so no valid report can be written even for an analysis yielding no published payload.

Passing: the same small staging budget constrains only extracted payload files, while the mandatory report and log are written under their own report-size allowance, so an extraction that produces no over-budget payload still completes and publishes a valid report.

Analysis or suspected cause

FirmwareProvider.ts:350-353 forwards request.input.max_output_bytes as outputBudget; FirmwareCommand.ts:125 emits it as the process-wide --fsize ceiling, which util-linux applies to every regular file the producer writes, including report.json and unblob.log outside the staging tree. A minimum valid report (root task + StatReport + SHA-256 HashReport, FirmwareReports.ts:186-200/317-322) cannot fit one byte, and FirmwareProvider.ts:356 unconditionally reads it, so the 8 MiB report allowance (FirmwareRelease.ts:31, FirmwareFiles.ts:154) never applies.

Additional context

The preliminary version probe uses the independent report-size default and has no extraction budget, so selecting 1 is not intentionally rejected during admission; larger small budgets also depend on report and log size rather than only staging bytes.

Related issues: open #3756 concerns false coverage: "complete" when nothing was extracted, not failure to write metadata under a staging limit; open #3757 and #3758 concern process prerequisites and doctor coverage; closed #1501 concerns teardown after completed extraction. No matching report found.

This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3888 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `extract-firmware --max-output-bytes` is forwarded as the process-wide `prlimit --fsize` ceiling, so it limits not only staged payload files but also the mandatory `report.json` the producer must write; with the accepted value 1, no valid report can fit even when no extracted file exceeds the budget. The extraction passes its staging byte budget into `runFirmwareCommand`, which emits it as the process-wide `--fsize` ceiling at src/firmware/FirmwareCommand.ts:125, and util-linux applies RLIMIT_FSIZE to every regular file the producer writes, including the mandatory report and log outside the staging tree. With the accepted value 1 no valid required report can fit: an empty JSON array needs two bytes, and an accepted report requires a root task, a StatReport, and a SHA-256 HashReport per src/firmware/FirmwareReports.ts:186-200 and lines 317-322. After the command src/firmware/FirmwareProvider.ts:356 unconditionally reads that report and src/firmware/FirmwareFiles.ts:150-172 requires complete valid JSON, so failure precedes result publication regardless of whether extracted payload files fit the selected budget, and the 8 MiB report-reader allowance never helps. ### Reproduction A local probe imported `dist/domain/firmware/firmwareAnalysis.js` and `dist/firmware/FirmwareCommand.js` from a built source checkout, parsed extraction input with `max_output_bytes: 1`, then called `runFirmwareCommand` with that parsed budget and an injected launcher that printed the file-size argument and threw a sentinel before spawning any process: ```text {"accepted_max_output_bytes":1} {"file_size_argument":"--fsize=1"} ``` The public CLI was also run locally with `--max-output-bytes 1`; it passed option validation and returned the expected macOS `capability_unavailable` refusal, which does not establish Linux extraction behavior. ### Expected behavior src/domain/firmware/firmwareAnalysis.ts:40 admits `max_output_bytes` from 1 and describes it as the staging file-byte budget, and docs/firmware-analysis.md:161 gives producer JSON reports a separate 8 MiB budget while lines 164-166 describe the configurable staging budget. Selecting a small staging budget should constrain extracted files rather than make the mandatory report impossible to write, even when no extracted file exceeds that budget. src/firmware/FirmwareRelease.ts:31 and src/firmware/FirmwareFiles.ts:154 separately allow reports up to 8 MiB, establishing the intended separation of the two budgets. ### Impact and scope low extraction budgets can prevent analysis solely because mandatory metadata or logs exceed the payload allowance, and the one-byte accepted boundary cannot produce any valid report even for an analysis yielding no published payload; impact is bounded to selected budgets. Workaround: unverified. raising the budget to the default is a plausible workaround but was not verified with a real producer and gives up the requested tighter staging limit. Reporter's impact estimate: P2 (supported). ### REA version 6.5.0, built from source at main commit 034342ba28c8e56c3746be92b78aa7299afa02bd (latest npm release is 6.5.0) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64; This is a source-only finding for the public Linux defect; schema acceptance and the production launcher-argument generation were executed locally, but no real Unblob/binwalk producer was spawned. End-to-end runtime reproduction with a real producer was not run. ### Target and artifact context Firmware image file (Linux Unblob extraction) ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence - Expected source: src/domain/firmware/firmwareAnalysis.ts:40; separate report budget at docs/firmware-analysis.md:161 and lines 164-166; 8 MiB report allowance at src/firmware/FirmwareRelease.ts:31 and src/firmware/FirmwareFiles.ts:154 - Failure source: observed `--fsize=1` emitted from the accepted budget (output below); static chain at src/firmware/FirmwareProvider.ts:350-353 (passes `max_output_bytes` as `outputBudget`), src/firmware/FirmwareCommand.ts:125 (emits `--fsize`), and src/firmware/FirmwareReports.ts:186-200 (minimum valid report cannot fit one byte) - Evidence provenance: static - Local verification: static-proof - Reproduction completeness: incomplete - Missing fact: a real Unblob/binwalk extraction on a supported Linux host confirming that the mandatory report fails to write (or truncates) under the one-byte budget was not executed; only schema acceptance and the `--fsize` launcher argument were observed. End-to-end runtime reproduction with a real producer was not run. ```text {"accepted_max_output_bytes":1} {"file_size_argument":"--fsize=1"} ``` ### Typed outcome or recovery expected A small --max-output-bytes staging budget should constrain extracted payload files only; the mandatory report.json and log should be written under their own report-size allowance so a non-over-budget extraction still publishes a valid result. Restoration check: Failing: `extract-firmware --max-output-bytes 1` admits the request and emits `--fsize=1`, which caps the mandatory `report.json` so no valid report can be written even for an analysis yielding no published payload. Passing: the same small staging budget constrains only extracted payload files, while the mandatory report and log are written under their own report-size allowance, so an extraction that produces no over-budget payload still completes and publishes a valid report. ### Analysis or suspected cause FirmwareProvider.ts:350-353 forwards request.input.max_output_bytes as outputBudget; FirmwareCommand.ts:125 emits it as the process-wide --fsize ceiling, which util-linux applies to every regular file the producer writes, including report.json and unblob.log outside the staging tree. A minimum valid report (root task + StatReport + SHA-256 HashReport, FirmwareReports.ts:186-200/317-322) cannot fit one byte, and FirmwareProvider.ts:356 unconditionally reads it, so the 8 MiB report allowance (FirmwareRelease.ts:31, FirmwareFiles.ts:154) never applies. ### Additional context The preliminary version probe uses the independent report-size default and has no extraction budget, so selecting 1 is not intentionally rejected during admission; larger small budgets also depend on report and log size rather than only staging bytes. Related issues: open #3756 concerns false `coverage: "complete"` when nothing was extracted, not failure to write metadata under a staging limit; open #3757 and #3758 concern process prerequisites and doctor coverage; closed #1501 concerns teardown after completed extraction. No matching report found. This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.
Author
Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Good catch — using the staging budget as a process-wide RLIMIT_FSIZE is a layering violation: the byte cap is meant to bound extracted payload files, but --fsize also clamps the mandatory report.json/log that the producer must write to complete the run. So a tiny budget doesn't just trim outputs, it makes the whole extraction fail before results can be published, and the 8 MiB report-reader allowance becomes dead code. The budget should apply only to files inside the staging tree (or be applied per-file after the run), and the mandatory report should be exempt from the ceiling. A regression test with max_output_bytes: 1 asserting that a valid report still gets produced would lock this down.

<!-- gh-comment-id:6185655509 --> @XinwangTao commented on GitHub (Oct 15, 2026): Good catch — using the staging budget as a process-wide RLIMIT_FSIZE is a layering violation: the byte cap is meant to bound extracted payload files, but --fsize also clamps the mandatory report.json/log that the producer must write to complete the run. So a tiny budget doesn't just trim outputs, it makes the whole extraction fail before results can be published, and the 8 MiB report-reader allowance becomes dead code. The budget should apply only to files inside the staging tree (or be applied per-file after the run), and the mandatory report should be exempt from the ceiling. A regression test with max_output_bytes: 1 asserting that a valid report still gets produced would lock this down.
Author
Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Good catch — using the staging budget as a process-wide RLIMIT_FSIZE is a layering violation: the byte cap is meant to bound extracted payload files, but --fsize also clamps the mandatory report.json/log that the producer must write to complete the run. So a tiny budget doesn't just trim outputs, it makes the whole extraction fail before results can be published, and the 8 MiB report-reader allowance becomes dead code. The budget should apply only to files inside the staging tree (or be applied per-file after the run), and the mandatory report should be exempt from the ceiling. A regression test with max_output_bytes: 1 asserting that a valid report still gets produced would lock this down.

<!-- gh-comment-id:6185670514 --> @XinwangTao commented on GitHub (Oct 15, 2026): Good catch — using the staging budget as a process-wide RLIMIT_FSIZE is a layering violation: the byte cap is meant to bound extracted payload files, but --fsize also clamps the mandatory report.json/log that the producer must write to complete the run. So a tiny budget doesn't just trim outputs, it makes the whole extraction fail before results can be published, and the 8 MiB report-reader allowance becomes dead code. The budget should apply only to files inside the staging tree (or be applied per-file after the run), and the mandatory report should be exempt from the ceiling. A regression test with max_output_bytes: 1 asserting that a valid report still gets produced would lock this down.
Author
Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

Process-wide file size allowance is now max(selected payload budget, report budget, diagnostic budget), capped by the inherited file ceiling; separate output inventory still enforces aggregate selected payload bytes/entries. The command regression explicitly tests selected=1 produces 8 MiB fsize headroom, large payload and inherited-ceiling controls. This fixes the mandatory report impossible-at-one-byte launch policy.

Implementation: FirmwareCommand.ts, FirmwareCommand.ts.

The relevant regression checks in FirmwareCommand.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

<!-- gh-comment-id:6380745036 --> @coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. Process-wide file size allowance is now max(selected payload budget, report budget, diagnostic budget), capped by the inherited file ceiling; separate output inventory still enforces aggregate selected payload bytes/entries. The command regression explicitly tests selected=1 produces 8 MiB fsize headroom, large payload and inherited-ceiling controls. This fixes the mandatory report impossible-at-one-byte launch policy. Implementation: [FirmwareCommand.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/firmware/FirmwareCommand.ts#L164), [FirmwareCommand.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/firmware/FirmwareCommand.ts#L269). The relevant regression checks in [FirmwareCommand.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/firmware/FirmwareCommand.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#1507
No description provided.