[PR #3764] [CLOSED] fix(firmware): report unexplained selected-input bytes as extraction gaps #5882

Closed
opened 2026-10-31 01:10:10 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3764
Author: @salemon
Created: 10/14/2026
Status: ❌ Closed

Base: main ← Head: fix/firmware-unreported-input-region-3756


📝 Commits (1)

  • 27fe915 fix(firmware): report unexplained selected-input bytes as extraction gaps

📊 Changes

8 files changed (+215 additions, -7 deletions)

View changed files

📝 docs/firmware-analysis.md (+10 -4)
📝 src/domain/firmware/firmwareAnalysis.ts (+3 -0)
📝 src/firmware/FirmwareProvider.ts (+1 -1)
📝 src/firmware/FirmwarePublication.ts (+18 -0)
📝 src/firmware/FirmwareReports.test.ts (+102 -0)
📝 src/firmware/FirmwareReports.ts (+36 -0)
📝 tests/boundary/firmware/firmwareIntegration.test.ts (+33 -0)
📝 tests/fixtures/firmware/producer.mjs (+12 -2)

📄 Description

Summary

Fixes #3756. extract_firmware (CLI extract-firmware) reported coverage: "complete" with no files, chunks or gaps when Unblob recognized nothing in the selected input, for example an ESP-IDF merged flash image. The selected bytes were absent from the result, so a caller reading complete would conclude the image was fully extracted. The result now carries an unreported-input-region gap for every selected-input byte range no reported chunk covers, with a continuation, and coverage is partial.

Problem and expected behavior

Unblob 26.6.4 _FileTask.process reports nothing for a whole file without a valid chunk ("we don't consider whole files as unknown chunks"), and calculate_unknown_chunks returns an empty list when there are no chunks. REA's coverage rule only looked for chunks with a null handler, so zero chunks fell through to complete. The tool's own contract says gaps yield truncated or unknown, never equivalence.

Change and scope

  • FirmwareReports.normalizeUnblobReport (the producer boundary that already owns chunk interpretation) derives unreported_input_regions: the complement of the union of reported $input chunk ranges over [0, selection.length), with root_file_range offset by the selection. Children without chunks are published files whose bytes are their own evidence, so only the selected input yields such regions.
  • New gap kind unreported-input-region in the domain contract; FirmwarePublication emits one row per region with provider_id: null (the producer reported no identity) and a static continuation naming inspect_firmware_regions (when Binwalk is configured) or a narrower range.
  • gapSortKey appends the byte offset so rows without producer identity order deterministically.
  • Docs and the provider limitation string describe the derived condition.

Not included: doctor checks (#3758) and the procps requirement note (#3757).

Contract and boundary impact

  • Semantic owner and earliest changed stage: Unblob report normalization (src/firmware/FirmwareReports.ts).
  • Contribution scope fit and linked maintainer decision: correctness fix within the existing read-only firmware provider; not applicable.
  • CLI and MCP/tool-catalog contract: additive output enum value unreported-input-region in extraction_gaps[].kind; input schemas unchanged.
  • Provider, bridge, target-format, or platform compatibility: Linux Unblob lane only; no producer behavior changed.
  • Evidence, artifact, provenance, or reconstruction contract: derived rows are labeled as such (null provider_id, $input path); observed chunks are unchanged.
  • Target selection, authority, upstream behavior, and cleanup details: unchanged.
  • Generated metadata, package, or installation impact: none committed; npm run docs:check passed.

Expected effects:

  • Read existing files or captured data.
  • Launch or attach to a process (existing Unblob launch, unchanged).
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added: FirmwareReports.test.ts (whole selection unreported; gaps around out-of-order chunks; overlapping/adjoining chunks yield none; child without chunks yields none) and firmwareIntegration.test.ts with a new fixture producer mode unrecognized-input that mirrors Unblob's empty report.

  • Base reproduction (executed, real provider): rea-agents 6.5.0 from npm, Unblob 26.6.4, util-linux prlimit, Ubuntu 24.04 x64, Node 24.22.0, on a synthetic 4 MiB ESP32-S3 merged image (bootloader at 0x0, partition table at 0x8000, app at 0x20000, rest 0xFF; libmagic: "DOS executable (COM)"). Result: coverage: "complete", chunks: [], extraction_gaps: [], published_bytes: 0.

  • Patched checkout, same inputs: coverage: "partial", one gap {kind: "unreported-input-region", input_path: "$input", provider_id: null, range: {offset: 0, length: 4194304}, root_file_range: {offset: 0, length: 4194304}} with its continuation; still no files.

  • Patched checkout, selected range --offset 131072 --length 65536 of the same image (real Unblob): coverage partial, one gap with range: {offset: 0, length: 65536} and root_file_range: {offset: 131072, length: 65536}.

  • Patched checkout on a real gzip tar: gzip chunk covers the whole input, no unreported region; coverage is partial only because of the pre-existing missing-extractor-dependency row for the tar extractor in that environment.

  • Remaining proof gaps: no Binwalk 3.1.0 on the host, so npm run verify:firmware was not run; agent-token cost unmeasured (one added row plus one shared continuation).

  • Observed, derived, and inferred claims remain distinguishable.

  • Artifact identity, source provenance, and failed attempts remain preserved.

  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:local -- src/firmware/FirmwareReports.test.ts — 25 passed.
  • npm run test:focused -- tests/boundary/firmware/firmwareIntegration.test.ts — 36 passed.
  • npm run check:fast — passed.
  • npm run docs:check — passed (readme consistency 5 passed).
  • npm run verify:test-discovery — passed.
  • Real Unblob 26.6.4 CLI runs as described above.

Compatibility, safety, and release

  • Breaking changes or migration steps: none; additive enum value. Callers that treated zero-chunk extractions as complete now see partial with an explicit gap.
  • Real Hopper/Ghidra, browser, or OS coverage: Linux x64 Unblob only.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: no new process, network or write paths.

Author readiness

Human owner: @salemon

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3764 **Author:** [@salemon](https://github.com/salemon) **Created:** 10/14/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/firmware-unreported-input-region-3756` --- ### 📝 Commits (1) - [`27fe915`](https://github.com/morluto/rea/commit/27fe91593a69e2823af8f7bd61508a5a70454b84) fix(firmware): report unexplained selected-input bytes as extraction gaps ### 📊 Changes **8 files changed** (+215 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `docs/firmware-analysis.md` (+10 -4) 📝 `src/domain/firmware/firmwareAnalysis.ts` (+3 -0) 📝 `src/firmware/FirmwareProvider.ts` (+1 -1) 📝 `src/firmware/FirmwarePublication.ts` (+18 -0) 📝 `src/firmware/FirmwareReports.test.ts` (+102 -0) 📝 `src/firmware/FirmwareReports.ts` (+36 -0) 📝 `tests/boundary/firmware/firmwareIntegration.test.ts` (+33 -0) 📝 `tests/fixtures/firmware/producer.mjs` (+12 -2) </details> ### 📄 Description ## Summary Fixes #3756. `extract_firmware` (CLI `extract-firmware`) reported `coverage: "complete"` with no files, chunks or gaps when Unblob recognized nothing in the selected input, for example an ESP-IDF merged flash image. The selected bytes were absent from the result, so a caller reading `complete` would conclude the image was fully extracted. The result now carries an `unreported-input-region` gap for every selected-input byte range no reported chunk covers, with a continuation, and coverage is `partial`. ## Problem and expected behavior Unblob 26.6.4 `_FileTask.process` reports nothing for a whole file without a valid chunk ("we don't consider whole files as unknown chunks"), and `calculate_unknown_chunks` returns an empty list when there are no chunks. REA's coverage rule only looked for chunks with a null handler, so zero chunks fell through to `complete`. The tool's own contract says gaps yield truncated or unknown, never equivalence. ## Change and scope - `FirmwareReports.normalizeUnblobReport` (the producer boundary that already owns chunk interpretation) derives `unreported_input_regions`: the complement of the union of reported `$input` chunk ranges over `[0, selection.length)`, with `root_file_range` offset by the selection. Children without chunks are published files whose bytes are their own evidence, so only the selected input yields such regions. - New gap kind `unreported-input-region` in the domain contract; `FirmwarePublication` emits one row per region with `provider_id: null` (the producer reported no identity) and a static continuation naming `inspect_firmware_regions` (when Binwalk is configured) or a narrower range. - `gapSortKey` appends the byte offset so rows without producer identity order deterministically. - Docs and the provider limitation string describe the derived condition. Not included: doctor checks (#3758) and the procps requirement note (#3757). ## Contract and boundary impact - Semantic owner and earliest changed stage: Unblob report normalization (`src/firmware/FirmwareReports.ts`). - Contribution scope fit and linked maintainer decision: correctness fix within the existing read-only firmware provider; not applicable. - CLI and MCP/tool-catalog contract: additive output enum value `unreported-input-region` in `extraction_gaps[].kind`; input schemas unchanged. - Provider, bridge, target-format, or platform compatibility: Linux Unblob lane only; no producer behavior changed. - Evidence, artifact, provenance, or reconstruction contract: derived rows are labeled as such (null `provider_id`, `$input` path); observed chunks are unchanged. - Target selection, authority, upstream behavior, and cleanup details: unchanged. - Generated metadata, package, or installation impact: none committed; `npm run docs:check` passed. Expected effects: - [x] Read existing files or captured data. - [x] Launch or attach to a process (existing Unblob launch, unchanged). - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added: `FirmwareReports.test.ts` (whole selection unreported; gaps around out-of-order chunks; overlapping/adjoining chunks yield none; child without chunks yields none) and `firmwareIntegration.test.ts` with a new fixture producer mode `unrecognized-input` that mirrors Unblob's empty report. - Base reproduction (executed, real provider): rea-agents 6.5.0 from npm, Unblob 26.6.4, util-linux prlimit, Ubuntu 24.04 x64, Node 24.22.0, on a synthetic 4 MiB ESP32-S3 merged image (bootloader at 0x0, partition table at 0x8000, app at 0x20000, rest 0xFF; libmagic: "DOS executable (COM)"). Result: `coverage: "complete"`, `chunks: []`, `extraction_gaps: []`, `published_bytes: 0`. - Patched checkout, same inputs: `coverage: "partial"`, one gap `{kind: "unreported-input-region", input_path: "$input", provider_id: null, range: {offset: 0, length: 4194304}, root_file_range: {offset: 0, length: 4194304}}` with its continuation; still no files. - Patched checkout, selected range `--offset 131072 --length 65536` of the same image (real Unblob): coverage `partial`, one gap with `range: {offset: 0, length: 65536}` and `root_file_range: {offset: 131072, length: 65536}`. - Patched checkout on a real gzip tar: gzip chunk covers the whole input, no unreported region; coverage is `partial` only because of the pre-existing `missing-extractor-dependency` row for the tar extractor in that environment. - Remaining proof gaps: no Binwalk 3.1.0 on the host, so `npm run verify:firmware` was not run; agent-token cost unmeasured (one added row plus one shared continuation). - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:local -- src/firmware/FirmwareReports.test.ts` — 25 passed. - `npm run test:focused -- tests/boundary/firmware/firmwareIntegration.test.ts` — 36 passed. - `npm run check:fast` — passed. - `npm run docs:check` — passed (readme consistency 5 passed). - `npm run verify:test-discovery` — passed. - Real Unblob 26.6.4 CLI runs as described above. ## Compatibility, safety, and release - Breaking changes or migration steps: none; additive enum value. Callers that treated zero-chunk extractions as `complete` now see `partial` with an explicit gap. - Real Hopper/Ghidra, browser, or OS coverage: Linux x64 Unblob only. - Package or release metadata impact: none. - Security, privacy, process, or containment review: no new process, network or write paths. ## Author readiness Human owner: @salemon - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:10:10 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5882
No description provided.