[PR #4430] [MERGED] fix(firmware): reject staged files cut off at the per-file limit #6238

New issue

Closed

opened 2026-10-31 01:12:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-31 01:12:56 +00:00

Owner

Copy link

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4430
Author: @SandstromPL
Created: 10/16/2026
Status: ✅ Merged
Merged: 11/4/2026
Merged by: @morluto

Base: main ← Head: fix/firmware-report-file-size

---

📝 Commits (10+)

• 02cc3a1 fix(firmware): keep producer reports writable under small staging budgets
• cef9689 Merge upstream/main into fix/firmware-report-file-size
• 43acb53 fix(firmware): keep files stopped at the size ceiling over budget
• 6ca4851 Merge branch 'main' into fix/firmware-report-file-size
• 215bec8 Merge upstream/main into fix/firmware-report-file-size
• f993751 fix(firmware): report the effective file-size ceiling on limit failures
• 8e77520 Merge upstream/main into fix/firmware-report-file-size
• d61b6b5 fix(firmware): reject staged files cut off at the per-file limit
• 3edcf3f fix(firmware): derive limit adjustability from the per-run limit
• a3741d7 Merge remote-tracking branch 'upstream/main' into fix/firmware-report-file-size

📊 Changes

12 files changed (+418 additions, -43 deletions)

View changed files

📝 docs/firmware-analysis.md (+6 -3)
📝 src/firmware/FirmwareCommand.test.ts (+90 -22)
📝 src/firmware/FirmwareCommand.ts (+41 -13)
📝 src/firmware/FirmwareFailures.ts (+2 -2)
📝 src/firmware/FirmwareFiles.ts (+27 -1)
📝 src/firmware/FirmwareProvider.ts (+5 -0)
📝 tests/boundary/firmware/firmwarePublicationPaths.test.ts (+1 -0)
📝 tests/boundary/firmware/firmwareReportFiles.test.ts (+32 -2)
📝 tests/boundary/firmware/firmwareStagingBudgetWiring.test.ts (+2 -0)
➕ tests/boundary/firmware/resourceLimits.test.ts (+60 -0)
➕ tests/fixtures/firmware/fileSizeProducer.ts (+93 -0)
➕ tests/process-global/firmwareInheritedFileSizeCeiling.test.ts (+59 -0)

📄 Description

Summary

Follow-up to #3888. Its small-budget half (a max_output_bytes below 8 MiB truncating the mandatory report) is fixed on main by 15450643f, so this PR now covers only the remaining exact-limit truncation.

main runs the firmware producer under prlimit --fsize=min(inherited ceiling, max(max_output_bytes, 8 MiB report, 8 MiB diagnostic)). The kernel stops an over-limit write exactly at that limit, and the staging inventory admits a file of exactly max_output_bytes. So:

1. Budget of 8 MiB or more (including the 256 MiB default): the limit equals the budget, and a staged file the producer could not finish is published as a complete extraction.
2. Inherited ceiling below that limit: a file cut off at the ceiling is within budget and is published the same way.

After this change the limit is one byte above the largest budget, still capped by the inherited ceiling, and the staging inventory refuses any regular file that reaches the installed limit.

Problem and expected behavior

• Smallest triggers (executed on Linux against main 5d4283909, with a synthetic Python Unblob producer under the real prlimit):
  • extract_firmware with max_output_bytes: 16777216, where the producer tries to write 16 MiB + 4096 bytes to one staged file. The producer inherited RLIMIT_FSIZE (16777216, 16777216), got EFBIG after 16,777,216 bytes, and the extraction succeeded, publishing rootfs/config at 16,777,216 bytes. A legitimate 16 MiB file gives an identical result.
  • The same request with the REA process's soft RLIMIT_FSIZE lowered to 16 MiB: the same truncated file, the same success.
• Invariant violated: docs/firmware-analysis.md says oversized data is never reported as complete. A file the kernel cut short cannot be told apart from a complete one once it ends exactly at the limit.

Change and scope

• One owner for the per-run limit: firmwareFileSizeLimit in src/firmware/FirmwareCommand.ts returns min(inherited ceiling, max(staging budget, report, diagnostic) + 1). Its consumers:
  • runFirmwareCommand uses it for the prlimit argument, the limit-failure text and the 100 ms staging poll.
  • The provider's final staging inventory calls it with the same inputs.
• Fail closed at the limit: inventoryFirmwareOutput now takes the installed limit and refuses a regular file whose size reaches it. The check runs after the aggregate budget check and returns a nonretryable file-size resource constraint.
  • With the extra byte in place, a file stopped at the limit already exceeds its budget, so the existing capacity refusal fires first.
  • The new refusal therefore fires only when an inherited ceiling removed that byte, and its remediation names the inherited limit.
• Failure text: when the budget determines the limit, the SIGXFSZ and process-limit text now says the cap is one byte above max_output_bytes. Whether to recommend a larger max_output_bytes is decided by firmwareFileSizeAdjustable from firmwareFileSizeLimit itself: only when a larger budget would raise the limit, so an inherited ceiling of exactly max_output_bytes + 1 gets the fixed-allowance text.
• Docs: docs/firmware-analysis.md states the one-byte margin and the at-limit refusal.
• Intentionally not included:
  • Report under an inherited ceiling: a report cut off at a ceiling below 8 MiB + 1 is still rejected only because the truncated JSON fails to parse, not by its size.
  • Writes that start past the limit: sparse output or ftruncate can leave a short file below the limit; no per-file limit exposes that.
  • Existing wording: main's "This release has no runtime override" text for a limit the budget does not determine is unchanged, even though a host can raise an inherited ceiling.
  • Per-path limits, and any schema or contract change.

Follow the contribution scope.
Unresolved scope, authority, side effects, or required verification block
integration. Link the required maintainer scope decision for new providers or
capabilities involving target execution, injection, mutation, or protection
recovery; ordinary fixes within existing boundaries may state "not applicable".
Draft PRs may leave the scope decision pending and the readiness and merge checklists
unchecked. Maintainers can help resolve open questions.

Contract and boundary impact

• Semantic owner and earliest changed stage: firmwareFileSizeLimit produces the per-run file limit; the earliest changed stage is prlimit argument construction in runFirmwareCommand.
  • Consumers: the limiter argument, the limit-failure text, the polling inventory and the provider's final inventory.
  • Removed: the inline limit derivation in runFirmwareCommand.
  • Retained: the aggregate staging budget stays in inventoryFirmwareOutput.
• Contribution scope fit and linked maintainer decision: an ordinary bug fix within the existing Linux firmware provider, so not applicable.
• CLI and MCP/tool-catalog contract: no schema change. There is one new failure outcome: an AnalysisResourceConstraintError with resource file-size and file_size_bytes when a staged file reaches an inherited ceiling. file-size is already used for SIGXFSZ.
• Provider, bridge, target-format, or platform compatibility: Linux firmware only.
  • Extraction limits rise by one byte.
  • Binwalk inspection and the version probe get 8 MiB + 1 instead of 8 MiB.
• Evidence, artifact, provenance, or reconstruction contract:
  • Report parsing and normalization are unchanged.
  • An extraction that used to publish a file cut off at the limit now fails and rolls back its output.
• Target selection, authority, upstream behavior, and cleanup details (if applicable):
  • Process ownership, cancellation and cleanup are unchanged.
  • For budgets of 8 MiB or more, a staged file can be one byte larger before polling rejects it.
  • A legitimate file whose size equals an inherited ceiling is now refused, because it cannot be told apart from a truncated one.
• Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

• Read existing files or captured data. (existing extract_firmware behavior)
• Launch or attach to a process. (existing Unblob/Binwalk launches; this PR changes only their per-file size limit)
• Inject instrumentation or run scripts/code.
• Change process memory, registers, files, or configuration.
• Contact network services.
• Recover protected code or work around protections.
• Documentation or usability only; no target interaction.
• Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

• Tests added or updated:
  • tests/boundary/firmware/resourceLimits.test.ts (rewritten) runs the production provider through the real /usr/bin/prlimit, with a 16 MiB budget:
    • A legitimate file of exactly 16 MiB publishes.
    • A file attempted at 16 MiB + 4096 is stopped at 16 MiB + 1, refused with the max_output_bytes capacity constraint, and rolled back.
  • tests/process-global/firmwareInheritedFileSizeCeiling.test.ts (new):
    • It lowers its isolated fork's soft RLIMIT_FSIZE to 16 MiB with prlimit --pid, and restores it when the test finishes.
    • A file attempted at 16 MiB + 4096 under a 16 MiB budget is refused with file-size and file_size_bytes: 16777216, and rolled back.
  • Shared producer fixture tests/fixtures/firmware/fileSizeProducer.ts:
    • It is a synthetic Python Unblob producer; Python is used because Node cannot start under the 1 GiB address-space limit.
    • It asserts the inherited limits for the version probe and extraction, and that the kernel left exactly min(attempted, limit) bytes.
    • It reports what is on disk, so only REA's limit and inventory can expose the truncation.
  • src/firmware/FirmwareCommand.test.ts: the expected --fsize values gain one byte. A new row shows an inherited ceiling equal to the budget removing that byte. A SIGXFSZ case recommends a larger max_output_bytes with an unlimited inherited ceiling, and does not with one of exactly 16 MiB + 1.
  • tests/boundary/firmware/firmwareReportFiles.test.ts: a new inventory case admits a 3-byte file under a 4-byte limit and refuses a 4-byte one. The existing calls pass an unlimited per-file limit.
  • Every provider case asserts owned process-group and workspace cleanup.
• Base reproduction or other evidence (executed on this branch with the four src/firmware source files restored to main 5d4283909):
  • Tests as written: all three new provider cases fail at the producer's RLIMIT_FSIZE assertions.
  • Producer assertions relaxed: the legitimate 16 MiB case passes. The exact-limit and inherited-ceiling cases both return ok: true and publish the truncated 16,777,216-byte file.
• User-visible CLI/MCP output (if applicable):
  • Inherited-ceiling truncation now fails with "A staged firmware file reached the 16777216-byte per-file limit and may be truncated". It is nonretryable, with resource file-size, and the remediation says to raise the inherited limit (for example ulimit -f).
  • Exact-budget truncation now fails with the existing max_output_bytes capacity refusal.
  • Previously successful extractions that published a cut-off file now fail.
• Remaining proof gaps:
  • There was no real Unblob 26.6.4 or Binwalk 3.1.0 run, and npm run verify:firmware was not run. Whether real Unblob reports its own EFBIG as an extraction failure is unknown.
  • Linux only.
  • The new refusal is not exercised through a public CLI/MCP path; the provider cases use FirmwareAnalysisService.

For boundary fixes, identify the applicable boundary regressions
and lifecycle regressions
covered by the evidence above, and explain relevant gaps.

• Representation expansion and capacity: a legitimate file exactly at the budget is paired with one stopped at the limit, under both the budget-derived limit and an inherited ceiling.
• Independent limits and failures: the aggregate budget refusal keeps precedence over the new per-file refusal, and both are asserted.
• Public success and error delivery: not covered at CLI/MCP level (see proof gaps).
• Lifecycle: ownership and finalization are unchanged. Each provider case still asserts process-group and workspace cleanup and output rollback, but no new lifecycle stage was added.

For evidence-bearing changes:

• Observed, derived, and inferred claims remain distinguishable.
• Artifact identity, source provenance, and failed attempts remain preserved.
• Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Run on Linux with Node 24.22.1 (.nvmrc) and npm 11.19.1 (packageManager pins 11.16.0), with upstream/main 5d4283909 merged:

• npm run test:local -- on all 7 src/firmware test files and src/process/LinuxResourceLimits.test.ts: 104 tests passed.
• npm run test:focused -- on 11 files: the 7 tests/boundary/firmware files, tests/process-global/firmwareInheritedFileSizeCeiling.test.ts, tests/process-global/firmwareFileLifetime.test.ts, tests/boundary/mcp/firmwareMcp.test.ts and tests/boundary/cli/firmwareIncompleteRange.test.ts. 106 tests passed.
• Base reproductions: with the four source files restored to main, the three new provider cases failed; with the producer assertions also relaxed, 2 failed (both published the truncated file) and 1 passed.
• npx tsc --noEmit -p ., npm run check:fast and npm run docs:check passed.
• npm run verify:test-discovery: discovered all 2495 test files exactly once.
• npx oxfmt --check on the 10 changed files and git diff --check: clean.
• npx oxlint on the changed sources gives the same warnings as on main. The only count changes:
  • inventoryFirmwareOutput already warned max-params at 5 and now has 6.
  • runFirmwareCommand shrinks from 196 to 193 lines.
• Codex follow-up (3edcf3ffc): the new SIGXFSZ case fails on d61b6b51d, where an inherited 16 MiB + 1 ceiling advised increasing max_output_bytes. After the change:
  • all 7 src/firmware test files and LinuxResourceLimits: 106 tests passed;
  • the 11 firmware boundary, MCP, CLI and process-global files: 106 tests passed;
  • npx tsc --noEmit -p ., npm run check:fast and npm run jscpd passed.
• Merge of upstream/main 34577eb1c (a3741d7c4): resolved a conflict in inventoryFirmwareOutput by keeping both the per-file limit check and main's symlink-target read. main's new inventory callers in firmwareStagingBudgetWiring.test.ts and firmwarePublicationPaths.test.ts now pass an unbounded per-file limit, as the existing staging-budget tests do. After npm ci:
  • npm run test:focused -- on 12 firmware unit, boundary and process-global files: 153 tests passed; tests/boundary/firmware/firmwareIntegration.test.ts: 39 passed;
  • npx tsc --noEmit -p ., npm run check:fast and npm run docs:check passed.
• Greptile follow-up (826ff5c27): the SIGXFSZ adjustability cases and the inherited-ceiling workflow now compare recovery guidance with its owners (firmwareSignalLimitError, firmwareFileSizeLimitReached) instead of matching prose fragments:
  • 4 focused firmware files: 47 tests passed;
  • flipping the expected recovery branch fails both SIGXFSZ cases;
  • npx tsc --noEmit -p . and npm run check:fast passed.
• Not run: npm run check:pr, npm run verify:firmware (real Unblob and Binwalk unavailable).

Compatibility, safety, and release

• Breaking changes or migration steps: none.
  • Extractions that published a file cut off at the limit now fail.
  • A legitimate file exactly the size of an inherited ceiling is now refused.
• Real Hopper/Ghidra, browser, or OS coverage: Linux, synthetic producer under real kernel limits; no real Unblob.
• Package or release metadata impact: none.
• Security, privacy, process, or containment review:
  • Per-file kernel limits rise by one byte when not capped by an inherited ceiling.
  • Aggregate staging enforcement is unchanged.
  • No inherited limit is ever raised.

Author readiness

Human owner: @SandstromPL

• I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
• I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
• I declare this PR ready for initial maintainer review.

Before merge

• This PR addresses a concrete problem or an agreed enhancement.
• Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
• The PR has one focused outcome and the title follows type(scope): outcome.
• Related issue is linked, or the reason for not linking one is stated above.
• Tests cover changed observable behavior and meaningful failure paths.
• Owning docs, contracts, and generated metadata are updated where needed.
• User-visible CLI/MCP changes include representative output. (see "User-visible CLI/MCP output" under Evidence)
• I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

---

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4430 **Author:** [@SandstromPL](https://github.com/SandstromPL) **Created:** 10/16/2026 **Status:** ✅ Merged **Merged:** 11/4/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/firmware-report-file-size` --- ### 📝 Commits (10+) - [`02cc3a1`](https://github.com/morluto/rea/commit/02cc3a12ce29a02c71f8bdd55935833d998ee9b4) fix(firmware): keep producer reports writable under small staging budgets - [`cef9689`](https://github.com/morluto/rea/commit/cef9689c73dc57e96b8607708e632a91024535bb) Merge upstream/main into fix/firmware-report-file-size - [`43acb53`](https://github.com/morluto/rea/commit/43acb53860ad4736099efe8b7ac4a64c15fb02a7) fix(firmware): keep files stopped at the size ceiling over budget - [`6ca4851`](https://github.com/morluto/rea/commit/6ca48516979f1b1ce417f916b202876093f095dd) Merge branch 'main' into fix/firmware-report-file-size - [`215bec8`](https://github.com/morluto/rea/commit/215bec88195a5ce96a2c9923b1dc3cc93d23742c) Merge upstream/main into fix/firmware-report-file-size - [`f993751`](https://github.com/morluto/rea/commit/f993751b364fb80031fa5820eaede5d16cb4b804) fix(firmware): report the effective file-size ceiling on limit failures - [`8e77520`](https://github.com/morluto/rea/commit/8e7752066e659bba83fe576c50994e5a897d2eb7) Merge upstream/main into fix/firmware-report-file-size - [`d61b6b5`](https://github.com/morluto/rea/commit/d61b6b51d34a4336c9edb93b188c5dab82fbdad8) fix(firmware): reject staged files cut off at the per-file limit - [`3edcf3f`](https://github.com/morluto/rea/commit/3edcf3ffc288714dc79fe995db04d7b1eb97ac17) fix(firmware): derive limit adjustability from the per-run limit - [`a3741d7`](https://github.com/morluto/rea/commit/a3741d7c428a8893dd40eba85e47941db3d6b9a8) Merge remote-tracking branch 'upstream/main' into fix/firmware-report-file-size ### 📊 Changes **12 files changed** (+418 additions, -43 deletions) <details> <summary>View changed files</summary> 📝 `docs/firmware-analysis.md` (+6 -3) 📝 `src/firmware/FirmwareCommand.test.ts` (+90 -22) 📝 `src/firmware/FirmwareCommand.ts` (+41 -13) 📝 `src/firmware/FirmwareFailures.ts` (+2 -2) 📝 `src/firmware/FirmwareFiles.ts` (+27 -1) 📝 `src/firmware/FirmwareProvider.ts` (+5 -0) 📝 `tests/boundary/firmware/firmwarePublicationPaths.test.ts` (+1 -0) 📝 `tests/boundary/firmware/firmwareReportFiles.test.ts` (+32 -2) 📝 `tests/boundary/firmware/firmwareStagingBudgetWiring.test.ts` (+2 -0) ➕ `tests/boundary/firmware/resourceLimits.test.ts` (+60 -0) ➕ `tests/fixtures/firmware/fileSizeProducer.ts` (+93 -0) ➕ `tests/process-global/firmwareInheritedFileSizeCeiling.test.ts` (+59 -0) </details> ### 📄 Description ## Summary Follow-up to #3888. Its small-budget half (a `max_output_bytes` below 8 MiB truncating the mandatory report) is fixed on `main` by 15450643f, so this PR now covers only the remaining exact-limit truncation. `main` runs the firmware producer under `prlimit --fsize=min(inherited ceiling, max(max_output_bytes, 8 MiB report, 8 MiB diagnostic))`. The kernel stops an over-limit write exactly at that limit, and the staging inventory admits a file of exactly `max_output_bytes`. So: 1. **Budget of 8 MiB or more** (including the 256 MiB default): the limit equals the budget, and a staged file the producer could not finish is published as a complete extraction. 2. **Inherited ceiling below that limit:** a file cut off at the ceiling is within budget and is published the same way. After this change the limit is one byte above the largest budget, still capped by the inherited ceiling, and the staging inventory refuses any regular file that reaches the installed limit. ## Problem and expected behavior - **Smallest triggers** (executed on Linux against `main` 5d4283909, with a synthetic Python Unblob producer under the real `prlimit`): - `extract_firmware` with `max_output_bytes: 16777216`, where the producer tries to write 16 MiB + 4096 bytes to one staged file. The producer inherited `RLIMIT_FSIZE` (16777216, 16777216), got EFBIG after 16,777,216 bytes, and the extraction succeeded, publishing `rootfs/config` at 16,777,216 bytes. A legitimate 16 MiB file gives an identical result. - The same request with the REA process's soft `RLIMIT_FSIZE` lowered to 16 MiB: the same truncated file, the same success. - **Invariant violated:** `docs/firmware-analysis.md` says oversized data is never reported as complete. A file the kernel cut short cannot be told apart from a complete one once it ends exactly at the limit. ## Change and scope <!-- Explain the chosen approach, why it fits REA's architecture, and what is intentionally not included. If adding complexity for a speculative or vanishingly unlikely race or security edge case, record the concrete failure mode, likelihood, complexity cost, and the user's buy-in obtained before implementation. Do not silently expand scope for theoretical robustness. --> - **One owner for the per-run limit:** `firmwareFileSizeLimit` in `src/firmware/FirmwareCommand.ts` returns `min(inherited ceiling, max(staging budget, report, diagnostic) + 1)`. Its consumers: - `runFirmwareCommand` uses it for the `prlimit` argument, the limit-failure text and the 100 ms staging poll. - The provider's final staging inventory calls it with the same inputs. - **Fail closed at the limit:** `inventoryFirmwareOutput` now takes the installed limit and refuses a regular file whose size reaches it. The check runs after the aggregate budget check and returns a nonretryable `file-size` resource constraint. - With the extra byte in place, a file stopped at the limit already exceeds its budget, so the existing `capacity` refusal fires first. - The new refusal therefore fires only when an inherited ceiling removed that byte, and its remediation names the inherited limit. - **Failure text:** when the budget determines the limit, the SIGXFSZ and process-limit text now says the cap is one byte above `max_output_bytes`. Whether to recommend a larger `max_output_bytes` is decided by `firmwareFileSizeAdjustable` from `firmwareFileSizeLimit` itself: only when a larger budget would raise the limit, so an inherited ceiling of exactly `max_output_bytes + 1` gets the fixed-allowance text. - **Docs:** `docs/firmware-analysis.md` states the one-byte margin and the at-limit refusal. - **Intentionally not included:** - **Report under an inherited ceiling:** a report cut off at a ceiling below 8 MiB + 1 is still rejected only because the truncated JSON fails to parse, not by its size. - **Writes that start past the limit:** sparse output or `ftruncate` can leave a short file below the limit; no per-file limit exposes that. - **Existing wording:** `main`'s "This release has no runtime override" text for a limit the budget does not determine is unchanged, even though a host can raise an inherited ceiling. - Per-path limits, and any schema or contract change. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". Draft PRs may leave the scope decision pending and the readiness and merge checklists unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact - **Semantic owner and earliest changed stage:** `firmwareFileSizeLimit` produces the per-run file limit; the earliest changed stage is `prlimit` argument construction in `runFirmwareCommand`. - **Consumers:** the limiter argument, the limit-failure text, the polling inventory and the provider's final inventory. - **Removed:** the inline limit derivation in `runFirmwareCommand`. - **Retained:** the aggregate staging budget stays in `inventoryFirmwareOutput`. - **Contribution scope fit and linked maintainer decision:** an ordinary bug fix within the existing Linux firmware provider, so not applicable. - **CLI and MCP/tool-catalog contract:** no schema change. There is one new failure outcome: an `AnalysisResourceConstraintError` with resource `file-size` and `file_size_bytes` when a staged file reaches an inherited ceiling. `file-size` is already used for SIGXFSZ. - **Provider, bridge, target-format, or platform compatibility:** Linux firmware only. - Extraction limits rise by one byte. - Binwalk inspection and the version probe get 8 MiB + 1 instead of 8 MiB. - **Evidence, artifact, provenance, or reconstruction contract:** - Report parsing and normalization are unchanged. - An extraction that used to publish a file cut off at the limit now fails and rolls back its output. - **Target selection, authority, upstream behavior, and cleanup details (if applicable):** - Process ownership, cancellation and cleanup are unchanged. - For budgets of 8 MiB or more, a staged file can be one byte larger before polling rejects it. - A legitimate file whose size equals an inherited ceiling is now refused, because it cannot be told apart from a truncated one. - **Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs):** none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. (existing `extract_firmware` behavior) - [x] Launch or attach to a process. (existing Unblob/Binwalk launches; this PR changes only their per-file size limit) - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - **Tests added or updated:** - `tests/boundary/firmware/resourceLimits.test.ts` (rewritten) runs the production provider through the real `/usr/bin/prlimit`, with a 16 MiB budget: - A legitimate file of exactly 16 MiB publishes. - A file attempted at 16 MiB + 4096 is stopped at 16 MiB + 1, refused with the `max_output_bytes` capacity constraint, and rolled back. - `tests/process-global/firmwareInheritedFileSizeCeiling.test.ts` (new): - It lowers its isolated fork's soft `RLIMIT_FSIZE` to 16 MiB with `prlimit --pid`, and restores it when the test finishes. - A file attempted at 16 MiB + 4096 under a 16 MiB budget is refused with `file-size` and `file_size_bytes: 16777216`, and rolled back. - **Shared producer fixture** `tests/fixtures/firmware/fileSizeProducer.ts`: - It is a synthetic Python Unblob producer; Python is used because Node cannot start under the 1 GiB address-space limit. - It asserts the inherited limits for the version probe and extraction, and that the kernel left exactly `min(attempted, limit)` bytes. - It reports what is on disk, so only REA's limit and inventory can expose the truncation. - `src/firmware/FirmwareCommand.test.ts`: the expected `--fsize` values gain one byte. A new row shows an inherited ceiling equal to the budget removing that byte. A SIGXFSZ case recommends a larger `max_output_bytes` with an unlimited inherited ceiling, and does not with one of exactly 16 MiB + 1. - `tests/boundary/firmware/firmwareReportFiles.test.ts`: a new inventory case admits a 3-byte file under a 4-byte limit and refuses a 4-byte one. The existing calls pass an unlimited per-file limit. - Every provider case asserts owned process-group and workspace cleanup. - **Base reproduction or other evidence** (executed on this branch with the four `src/firmware` source files restored to `main` 5d4283909): - **Tests as written:** all three new provider cases fail at the producer's `RLIMIT_FSIZE` assertions. - **Producer assertions relaxed:** the legitimate 16 MiB case passes. The exact-limit and inherited-ceiling cases both return `ok: true` and publish the truncated 16,777,216-byte file. - **User-visible CLI/MCP output (if applicable):** - **Inherited-ceiling truncation** now fails with "A staged firmware file reached the 16777216-byte per-file limit and may be truncated". It is nonretryable, with resource `file-size`, and the remediation says to raise the inherited limit (for example `ulimit -f`). - **Exact-budget truncation** now fails with the existing `max_output_bytes` capacity refusal. - **Previously successful** extractions that published a cut-off file now fail. - **Remaining proof gaps:** - There was no real Unblob 26.6.4 or Binwalk 3.1.0 run, and `npm run verify:firmware` was not run. Whether real Unblob reports its own EFBIG as an extraction failure is unknown. - Linux only. - The new refusal is not exercised through a public CLI/MCP path; the provider cases use `FirmwareAnalysisService`. For boundary fixes, identify the applicable [boundary regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#boundary-regressions) and [lifecycle regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#lifecycle-regressions) covered by the evidence above, and explain relevant gaps. - **Representation expansion and capacity:** a legitimate file exactly at the budget is paired with one stopped at the limit, under both the budget-derived limit and an inherited ceiling. - **Independent limits and failures:** the aggregate budget refusal keeps precedence over the new per-file refusal, and both are asserted. - **Public success and error delivery:** not covered at CLI/MCP level (see proof gaps). - **Lifecycle:** ownership and finalization are unchanged. Each provider case still asserts process-group and workspace cleanup and output rollback, but no new lifecycle stage was added. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Run on Linux with Node 24.22.1 (`.nvmrc`) and npm 11.19.1 (`packageManager` pins 11.16.0), with `upstream/main` 5d4283909 merged: - `npm run test:local --` on all 7 `src/firmware` test files and `src/process/LinuxResourceLimits.test.ts`: 104 tests passed. - `npm run test:focused --` on 11 files: the 7 `tests/boundary/firmware` files, `tests/process-global/firmwareInheritedFileSizeCeiling.test.ts`, `tests/process-global/firmwareFileLifetime.test.ts`, `tests/boundary/mcp/firmwareMcp.test.ts` and `tests/boundary/cli/firmwareIncompleteRange.test.ts`. 106 tests passed. - **Base reproductions:** with the four source files restored to `main`, the three new provider cases failed; with the producer assertions also relaxed, 2 failed (both published the truncated file) and 1 passed. - `npx tsc --noEmit -p .`, `npm run check:fast` and `npm run docs:check` passed. - `npm run verify:test-discovery`: discovered all 2495 test files exactly once. - `npx oxfmt --check` on the 10 changed files and `git diff --check`: clean. - `npx oxlint` on the changed sources gives the same warnings as on `main`. The only count changes: - `inventoryFirmwareOutput` already warned `max-params` at 5 and now has 6. - `runFirmwareCommand` shrinks from 196 to 193 lines. - Codex follow-up (`3edcf3ffc`): the new SIGXFSZ case fails on `d61b6b51d`, where an inherited 16 MiB + 1 ceiling advised increasing `max_output_bytes`. After the change: - all 7 `src/firmware` test files and `LinuxResourceLimits`: 106 tests passed; - the 11 firmware boundary, MCP, CLI and process-global files: 106 tests passed; - `npx tsc --noEmit -p .`, `npm run check:fast` and `npm run jscpd` passed. - Merge of `upstream/main` 34577eb1c (`a3741d7c4`): resolved a conflict in `inventoryFirmwareOutput` by keeping both the per-file limit check and `main`'s symlink-target read. `main`'s new inventory callers in `firmwareStagingBudgetWiring.test.ts` and `firmwarePublicationPaths.test.ts` now pass an unbounded per-file limit, as the existing staging-budget tests do. After `npm ci`: - `npm run test:focused --` on 12 firmware unit, boundary and process-global files: 153 tests passed; `tests/boundary/firmware/firmwareIntegration.test.ts`: 39 passed; - `npx tsc --noEmit -p .`, `npm run check:fast` and `npm run docs:check` passed. - Greptile follow-up (`826ff5c27`): the SIGXFSZ adjustability cases and the inherited-ceiling workflow now compare recovery guidance with its owners (`firmwareSignalLimitError`, `firmwareFileSizeLimitReached`) instead of matching prose fragments: - 4 focused firmware files: 47 tests passed; - flipping the expected recovery branch fails both SIGXFSZ cases; - `npx tsc --noEmit -p .` and `npm run check:fast` passed. - Not run: `npm run check:pr`, `npm run verify:firmware` (real Unblob and Binwalk unavailable). ## Compatibility, safety, and release - **Breaking changes or migration steps:** none. - Extractions that published a file cut off at the limit now fail. - A legitimate file exactly the size of an inherited ceiling is now refused. - **Real Hopper/Ghidra, browser, or OS coverage:** Linux, synthetic producer under real kernel limits; no real Unblob. - **Package or release metadata impact:** none. - **Security, privacy, process, or containment review:** - Per-file kernel limits rise by one byte when not capped by an inherited ceiling. - Aggregate staging enforcement is unchanged. - No inherited limit is ever raised. ## Author readiness <!-- The human owner completes these declarations before requesting review. --> Human owner: @SandstromPL - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. (see "User-visible CLI/MCP output" under Evidence) - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- _{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

bsakaguchi added the

pull-request

label 2026-10-31 01:12:56 +00:00

bsakaguchi referenced this issue 2026-11-01 21:46:13 +00:00

[PR #6238] [MERGED] docs(javascript): state that duplicate unmapped content stays ambiguous #7215

bsakaguchi changed title from [PR #4430] fix(firmware): reject staged files cut off at the per-file limit to [PR #4430] [MERGED] fix(firmware): reject staged files cut off at the per-file limit 2026-11-04 12:14:35 +00:00

bsakaguchi closed this issue 2026-11-04 12:14:35 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

codex/procedure-reference-stacked

release-please--branches--main--components--rea-agents

release-please--branches--main--components--rea-agents--release-notes

codex/kuna-independent-retry

codex/byte-search-overlay-coverage

codex/open-failure-target-receipt

codex/kuna-xref-evidence

codex/ghidra-ambiguous-procedure-recovery

codex/function-source-bundle

codex/trace-function-integration-prototype

codex/snapshot-source-closure

codex/recorded-evidence-array-pages

codex/evidence-recovery-redesign

codex/trace-feature-address-scope

codex/call-discovery-scope

codex/ghidra-literal-candidate-evidence

codex/decompile-directory-independent-retry

codex/retry-provenance-base

codex/value-flow-capability-experiment

codex/ghidra-containing-instruction

cursor/readme-discord-button-ef9f

dependabot/github_actions/actions/setup-node-7.1.0

dependabot/github_actions/actions/upload-artifact-7.0.2

dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df

codex/kuna-1989

fix/8559-native-ui-fixture-ownership

fix/8555-go-verifier-mcp-ownership

codex/provider-capability-recovery

fix/8642-rizin-target-stdin

codex/ghidra-memory-effect-scope

codex/ghidra-annotation-entry-selector

codex/ghidra-readonly-recovery

codex/ghidra-regex-verifier-recovery

codex/evidence-discovery-view-guidance

fix/8545-bulk-append-overflow

fix/8428-navigation-refusal-alternatives

codex/address-context-recovery

codex/matching-member-limitations

codex/unknown-scope-contract

codex/evidence-view-guidance

feat/8373-native-trace-views

codex/ghidra-snapshot-verifier-binding

codex/elf-export-visibility

codex/evidence-view-recovery

codex/native-reference-cli-help

codex/provider-candidate-observation-scope

codex/kuna-session-verification

codex/ghidra-switch-lifetime-guidance

codex/pseudocode-paging-discovery

codex/clarify-evidence-view-support

codex/kuna-xrefs-selector-recovery

codex/context-capacity-recovery

codex/evidence-closure-large-links

codex/same-function-evidence-identity

codex/ghidra-verifier-inspected-jdk

codex/audit-guided-workflows

codex/crash-prompt-retained-dossiers

codex/analysis-view-error-recovery

codex/kuna-verifier-target-selection

codex/ghidra-selected-project-call-coverage

codex/docs-native-analysis-scope

codex/ghidra-java-socket-boundary

codex/ghidra-retained-snapshot-verifier

codex/ghidra-search-inventory-facts

fix/8284-zero-argument-parameter-bindings

fix/8198-ui-action-handler-offset

fix/8257-elf-entry-label-identity

fix/8304-deferred-output-schema-projection

fix/8277-call-path-computed-target-uncertainty

fix/pr-8089-conflicts

fix/ci-static-duplicate-export

dependabot/npm_and_yarn/lint-staged-17.6.0

dependabot/github_actions/actions/download-artifact-8.0.2

dependabot/npm_and_yarn/write-file-atomic-8.0.0

dependabot/npm_and_yarn/canonicalize-5.1.0

dependabot/github_actions/actions/cache/restore-6.1.0

dependabot/npm_and_yarn/toon-format/toon-4.4.0

dependabot/github_actions/actions/deploy-pages-5.0.1

codex/review-fixes-pr-2403

codex/review-fixes-pr-2400

codex/process-terminal-width

codex/test-brittleness-cleanup

verify/7167-browser-connect-macos

fix/7167-browser-connect

feat/native-matching-allowance-setting

feat/browser-scenario-dry-run

test/managed-caller-parameters-20261024

codex/parser-fortythird-boundaries

fix/6926-pinvoke-platform-decorations-20261023

fix/6889-har-body-encoding-20261023

fix/6888-sqlite-schema-budget-20261023

fix/dependency-security-gates-6708

cursor/evidence-overwrite-rebase-b4ac

feat/4057-verify-native-exports

fix/windows-acl-observer-overhead-20261022

cursor/erlang-beam-restack-4476

cursor/pe-certificate-review-bf32

fix/6116-javascript-application-fs-errors

fix/6061-unmapped-runtime-script-unknown

feat/browser-scenario-summary

fix/pe-certificates

codex/verify-3407

cursor/rebase-5699

cursor/rebase-5352

cursor/rebase-5600

cursor/fix-cli-schema-omission-10a8

codex/audit-hopper-shutdown-reasons-20261021

fix/demangle-swift-omitted-record

fix/nib-boolean-value-types

cursor/cancellation-partial-observation-a5ac

fix/ib-object-limit-reasons

fix/electron-fuse-limit-advice

fix/kuna-doctor-dangling-cache

fix/fixed-budget-refusal-advice

fix/js-ipc-listener-dedup

fix/js-return-member-provenance

fix/dmg-integrity-retry-advice

fix/ida-doctor-readiness

fix/kuna-version-probe-output-drain

fix/legacy-swift-runtime-name-matching

fix/dylib-lazy-load-commands

fix/browser-scenario-startup-cancel

fix/swift-nested-punycode-classification

fix/compare-unselected-scopes

fix/objc-method-budget-coverage

fix/dmg-mounted-root-observations

audit/ci-tests-20261019-round2

release/6.7.0

fix/wabt-cleanup-retry-resolution

fix/electron-fuse-elf-pe-architecture

fix/native-plist-descriptor-ownership

fix/kuna-caller-cleanup-ownership

experiment/atomic-codemode-20261016

review/first-contributor-3776

fix/call-resolution-full-read-cycles

fix/member-mutation-selection-repeat

experiment/atomic-re-tools-20261015

perf/member-mutation-lazy-origins

perf/static-network-mutation-index

fix/hopper-startup-lifecycle

audit/3050-windows-proof-20261015

feat/windows-window-capture-2183

audit/3049-windows-proof-20261015

audit/3735-macos-fixtures-20261015

fix/batch-decompile-null-pseudocode

fix/kuna-inventory-honors-jobs

experiment/browser-codemode-tool-design-20261015

fix/hopper-clean-runtime-argv

codex/review-2049-27f1b367

review-fixes/pr-2787-routing

fix/3360-scoped-this-emitters

fix/3367-replaced-global-effects

fix/3838-compact-any-value-items

test/hopper-download-probe-20261015

codex/windows-runtime-creation-red-green-20261015

codex/windows-readonly-root-red-green-20261015

codex/windows-root-share-red-green-20261015

codex/exp-1685-win-mcp-startup

codex/bug-sweep-20261015

investigate/2190-windows-package-phases

investigate/3543-macos-cli-deadline

perf/3281-reverse-tools-feedback-quality

fix/3406-hopper-doctor-fixture

fix/3341-wrapper-require-test-context

fix/3301-owned-uninstall-fixtures

fix/3164-compiler-input-ownership

fix/pcap-cleanup-stdout-newline-20261014

perf/interface-builder-summary

codex/review-fixes-pr-1977

fix/ghidra-effective-jvm-platform

fix/docs-turkish-rust-link-20261013

fix/main-maintenance-ci-20261010

fix/recover-binary-session-process-effects

codex/review-fixes-pr-2457

codex/ci-required-cancellation-gate

fix/main-format-baseline-20261013

fix/setup-preserve-entry-fields

cursor/restore-main-checks-1ffc

fix/ghidra-bridge-hashmap-import

codex/shared-ci-jadx-json-default-20261013

codex/shared-ci-ghidra-test-groups-20261013

codex/shared-ci-java-fixture-format-20261013

codex/fix-intrinsic-invocation-boundaries

test/release-preflight-reusable-jobs

test/js-parameter-provenance-expectation

fix/linux-hopper-verification-current

test/2419-ghidra-elf-probe-size

followup/pr-1544-native-ida

codex/fold-pr-2230-contribution

docs/readme-responsible-use

fix/issue-2064

fix/issue-2050

fix/issue-2053

fix/issue-1968

fix/issue-1945

experiment/codemode-bridge-20261011

fix/tool-effects-negative-wording-1904-20261012

fix/1842-electron-interop-provenance

feat/recover-javascript-multi-input

refactor/website-i18n-prep

fix/nativeaot-pe-virtual-extent

fix/browser-verifier-ci-hang

fix/pe-resources-input-selection

docs/beginner-friendly-guides

docs/beginner-setup-guide

fix/javascript-analysis-worker-1563

fix/1649-ghidra-duplicate-names

fix/javascript-export-keys-integration-1369

test/maintenance-boundary-contracts-20261011

review/javascript-export-keys-1369

review/process-live-progress-1483

test/javascript-1495-growth-ratio

fix/ci-escape-timing

docs/normalized-terminal-geometry-1450

release/6.3.0

archive/release-6.3.0-pr1381

release/6.2.0

fix/electron-preload-dynamic-override

fix/javascript-self-referential-bundler-edges

release/6.1.0

cursor/grok-client-cherry-pick-0b56

feat/signature-posture

feat/dyld-shared-cache

feat/artifact-integrity-policy

feat/pkg-inventory

release/6.0.0

chore/sync-release-5.0.0

release/5.0.0

cursor/cloud-agent-node-787e

fix/javascript-xhr-open-endpoints

review/pr-711-realpath-fallback-wording

cleanup/replay-confinement

codex/test-suite-performance

codex/issue-foundations

codex/address-open-issues-2026-07-20

agent/javascript-vite-rollup-esbuild-adapters

agent/process-cleanup

agent/triage-integration

agent/unknown-commands-registry

feat/mcp-identity-metadata

morlvps/issue_cleanup

codex/installer-redesign

morl/roadmap-investigation-core

morl/roadmap-investigation-docs

release-please--branches--main--components--rea

rea-agents-6.8.0

rea-agents-6.7.0

rea-agents-6.6.0

experiment/atomic-codemode-wait-runtime-20261016

rea-agents-6.5.0

rea-agents-6.4.0

rea-agents-6.3.0

rea-agents-6.2.0

rea-agents-6.1.0

rea-agents-6.0.0

rea-agents-5.0.0

rea-agents-4.1.0

rea-agents-4.0.1

rea-agents-4.0.0

rea-agents-3.2.1

rea-agents-3.2.0

rea-agents-3.1.0

rea-agents-3.0.0

rea-agents-2.7.0

rea-agents-2.6.0

rea-agents-2.5.1

rea-agents-2.5.0

rea-agents-2.4.0

rea-agents-2.3.0

rea-agents-2.2.0

rea-agents-2.1.0

rea-agents-2.0.0

rea-agents-1.7.0

rea-agents-1.6.0

rea-agents-1.5.0

rea-agents-1.4.0

rea-agents-1.3.0

rea-agents-1.2.0

rea-agents-1.1.0

rea-agents-1.0.0

rea-agents-0.5.0

rea-agents-0.4.0

rea-agents-0.3.0

rea-0.2.1

rea-0.2.0

Labels

Clear labels   

area: bridge

  

area: bridge

  

area: build

  

area: build

  

area: build

  

area: cli

  

area: domain

  

area: evidence

  

area: mcp

  

area: native

  

area: providers

  

area: runtime

  

area: server

  

area: setup

  

area: tests

  

bug

  

design

  

documentation

  

enhancement

  

enhancement

  

enhancement

  

experiment

  

help wanted

  

i18n

  

invalid

  

P0: critical

  

P1: high

  

P2: medium

  

P2: medium

  

P2: medium

  

P3: low

  

performance

  

provider-request

  

pull-request

Mirrored from GitHub Pull Request

  

question

  

roadmap

  

triage

No labels

area: bridge

area: bridge

area: build

area: build

area: build

area: cli

area: domain

area: evidence

area: mcp

area: native

area: providers

area: runtime

area: server

area: setup

area: tests

bug

design

documentation

enhancement

enhancement

enhancement

experiment

help wanted

i18n

invalid

P0: critical

P1: high

P2: medium

P2: medium

P2: medium

P3: low

performance

provider-request

pull-request

question

roadmap

triage

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6238

No description provided.