[GH-ISSUE #3756] extract_firmware reports coverage "complete" when nothing was extracted

bsakaguchi commented

2026-10-16 21:05:38 +00:00

Owner

Originally created by @oleander on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3756

Summary

extract_firmware (CLI extract-firmware) reports coverage: "complete" even when unblob found no chunks and nothing was extracted. published_bytes is 0 and the input file is 4 MiB.

Environment

rea-agents 6.5.0, linux-arm64 (Debian bookworm container), Node v22.23.2
unblob 26.6.4 (all 15 external extractors present per unblob --show-external-dependencies), binwalk 3.1.0, prlimit from util-linux 2.38.1
REA_UNBLOB_COMMAND, REA_BINWALK_COMMAND and REA_FIRMWARE_PRLIMIT_COMMAND set to absolute paths

Steps

Run extract-firmware on a 4 MiB ESP32-S3 ESP-IDF merged flash image (bootloader at 0x0, partition table at 0x8000, app image at 0x20000, the rest 0xFF). unblob has no handler for the ESP image format, and libmagic reads the 0xE9 header as "DOS executable (COM)".

Observed

unblob exits 0. The normalized_result is:

{
  "coverage": "complete",
  "files": [],
  "unpublished_entries": [],
  "chunks": [],
  "derivations": [],
  "depth_limited_paths": [],
  "diagnostics": [],
  "extraction_gaps": [],
  "extraction_continuations": [],
  "published_bytes": 0
}

The output directory is empty.

Expected

The whole input is unexplained, so it should show up as an unknown region or an extraction_gaps row, and coverage should not be complete. This matters because the tool's own contract says gaps yield truncated or unknown, never equivalence. A caller reading complete would conclude the image was fully extracted.

Likely cause

unblob does not report a whole file with no recognized chunk as an unknown chunk (see _FileTask.process), so there is nothing for rea to turn into a gap. rea could compare the selected byte range against the union of reported chunks and unknown chunks, and mark the rest as a gap.

Suggested priority: P2: medium (I cannot set labels on this repo).

Originally created by @oleander on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3756 ## Summary `extract_firmware` (CLI `extract-firmware`) reports `coverage: "complete"` even when unblob found no chunks and nothing was extracted. `published_bytes` is `0` and the input file is 4 MiB. ## Environment - rea-agents 6.5.0, linux-arm64 (Debian bookworm container), Node v22.23.2 - unblob 26.6.4 (all 15 external extractors present per `unblob --show-external-dependencies`), binwalk 3.1.0, prlimit from util-linux 2.38.1 - `REA_UNBLOB_COMMAND`, `REA_BINWALK_COMMAND` and `REA_FIRMWARE_PRLIMIT_COMMAND` set to absolute paths ## Steps Run `extract-firmware` on a 4 MiB ESP32-S3 ESP-IDF merged flash image (bootloader at 0x0, partition table at 0x8000, app image at 0x20000, the rest 0xFF). unblob has no handler for the ESP image format, and libmagic reads the 0xE9 header as "DOS executable (COM)". ## Observed unblob exits 0. The `normalized_result` is: ```json { "coverage": "complete", "files": [], "unpublished_entries": [], "chunks": [], "derivations": [], "depth_limited_paths": [], "diagnostics": [], "extraction_gaps": [], "extraction_continuations": [], "published_bytes": 0 } ``` The output directory is empty. ## Expected The whole input is unexplained, so it should show up as an unknown region or an `extraction_gaps` row, and coverage should not be `complete`. This matters because the tool's own contract says gaps yield truncated or unknown, never equivalence. A caller reading `complete` would conclude the image was fully extracted. ## Likely cause unblob does not report a whole file with no recognized chunk as an unknown chunk (see `_FileTask.process`), so there is nothing for rea to turn into a gap. rea could compare the selected byte range against the union of reported chunks and unknown chunks, and mark the rest as a gap. Suggested priority: P2: medium (I cannot set labels on this repo).

bsakaguchi commented

2026-10-16 21:05:39 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

This is a genuinely misleading failure mode — a caller reading "coverage: complete" with published_bytes 0 and an empty output dir would assume full extraction, which is exactly the wrong signal for a 4 MiB ESP image that unblob silently skipped. The proposed fix of comparing the selected byte range against the union of reported chunks and marking the remainder as a gap is the right contract alignment, since the tool's own spec says gaps must never map to "complete". Tracking #3764; verifying on a synthetic same-shaped image (bootloader + partition table + app + 0xFF padding) would be a nice regression test.

@XinwangTao commented on GitHub (Oct 15, 2026): This is a genuinely misleading failure mode — a caller reading "coverage: complete" with published_bytes 0 and an empty output dir would assume full extraction, which is exactly the wrong signal for a 4 MiB ESP image that unblob silently skipped. The proposed fix of comparing the selected byte range against the union of reported chunks and marking the remainder as a gap is the right contract alignment, since the tool's own spec says gaps must never map to "complete". Tracking #3764; verifying on a synthetic same-shaped image (bootloader + partition table + app + 0xFF padding) would be a nice regression test.

bsakaguchi referenced this issue

2026-10-17 21:11:36 +00:00

[PR #1463] [MERGED] fix(skills): expose authored skill to repository discovery #2903

bsakaguchi closed this issue

2026-10-31 00:30:51 +00:00

bsakaguchi commented

2026-10-31 00:30:51 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 19, 2026):

matbrgz groups this issue with #3757 and #3758 in #4330, as firmware tooling reports. The three look related: the same person filed all three, on rea-agents 6.5.0 in a linux-arm64 container.

#3757 and #3758 already link each other. This issue links to neither. Open PRs: #4953 (this issue), #3761 (#3757) and #3899 (#3758).

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@CassieFox23 commented on GitHub (Oct 19, 2026): matbrgz groups this issue with #3757 and #3758 in #4330, as firmware tooling reports. The three look related: the same person filed all three, on rea-agents 6.5.0 in a linux-arm64 container. #3757 and #3758 already link each other. This issue links to neither. Open PRs: #4953 (this issue), #3761 (#3757) and #3899 (#3758). 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Agreed with the fix direction in #4953. One edge case worth covering in the regression test: when the chunks list is empty (unblob found nothing), the gap calculation must still cover the entire selected range — mostly 0xFF padding here — so coverage stays non-"complete" with published_bytes 0, instead of falling back to the empty-chunk happy path that produced this misleading report.

@XinwangTao commented on GitHub (Oct 19, 2026): Agreed with the fix direction in #4953. One edge case worth covering in the regression test: when the chunks list is empty (unblob found nothing), the gap calculation must still cover the entire selected range — mostly 0xFF padding here — so coverage stays non-"complete" with published_bytes 0, instead of falling back to the empty-chunk happy path that produced this misleading report.

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 20, 2026):

Thanks, XinwangTao. The open PR #4953 already covers that case. Its head is 4308633992. There, FirmwareReports.test.ts has derives the whole selection as unreported when no chunk is reported. The integration test reports an unrecognized selected input as an unreported region, not complete expects published_bytes to be 0.

I read those tests. I did not run them.

Edit: I have now run those tests. The results are here.

🤖 Written with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Oct 20, 2026): Thanks, XinwangTao. The open PR #4953 already covers that case. Its head is 4308633992. There, `FirmwareReports.test.ts` has `derives the whole selection as unreported when no chunk is reported`. The integration test `reports an unrecognized selected input as an unreported region, not complete` expects `published_bytes` to be 0. I read those tests. I did not run them. **Edit:** I have now run those tests. The results are [here](https://github.com/morluto/rea/issues/3756#issuecomment-6307766879). 🤖 Written with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

Thanks for confirming the empty-chunks case is already covered in #4953 — deriving the whole selection as unreported when no chunk is reported is exactly the right behavior. Since that case was the original source of the misleading "complete" report, it'd be good to also run that integration test on the actual worker before merge, to catch any mismatch between the unit-level derives and the real unblob result shape.

@XinwangTao commented on GitHub (Oct 21, 2026): Thanks for confirming the empty-chunks case is already covered in #4953 — deriving the whole selection as unreported when no chunk is reported is exactly the right behavior. Since that case was the original source of the misleading "complete" report, it'd be good to also run that integration test on the actual worker before merge, to catch any mismatch between the unit-level derives and the real unblob result shape.

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 21, 2026):

Thanks, XinwangTao. I ran the empty-chunks case on the head of #4953 (430863399). On main 335c042d a 4 MiB input with no reported chunk gives coverage: "complete". On the head it gives partial, with one gap that spans all 4,194,304 bytes. In the fixture test, published_bytes stays 0. I used a stand-in report, not real Unblob, which is not installed on my box. The results are here.

🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Oct 21, 2026): Thanks, XinwangTao. I ran the empty-chunks case on the head of #4953 (430863399). On main 335c042d a 4 MiB input with no reported chunk gives `coverage: "complete"`. On the head it gives `partial`, with one gap that spans all 4,194,304 bytes. In the fixture test, `published_bytes` stays 0. I used a stand-in report, not real Unblob, which is not installed on my box. The results are [here](https://github.com/morluto/rea/pull/4953#issuecomment-6307766483). 🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

Nice to see the concrete before/after on 4308633 — main's misleading "complete" becoming "partial" with a single full-range gap is exactly the contract fix this issue asked for, and published_bytes staying 0 keeps the signal honest. One gap I'd still close before merge: that verification used a stand-in report rather than real unblob. Given coygeek's #5745 (unblob-resolved problems being labeled as unextracted regions), the real unblob result shape can differ from the fixture in ways that affect the gap union, so an end-to-end pass against actual unblob — or a captured real unblob report — on an unrecognized image would harden the fix before it lands.

@XinwangTao commented on GitHub (Oct 21, 2026): Nice to see the concrete before/after on 4308633 — main's misleading "complete" becoming "partial" with a single full-range gap is exactly the contract fix this issue asked for, and published_bytes staying 0 keeps the signal honest. One gap I'd still close before merge: that verification used a stand-in report rather than real unblob. Given coygeek's #5745 (unblob-resolved problems being labeled as unextracted regions), the real unblob result shape can differ from the fixture in ways that affect the gap union, so an end-to-end pass against actual unblob — or a captured real unblob report — on an unrecognized image would harden the fix before it lands.

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 21, 2026):

Thanks, XinwangTao, for the nudge. I agree a real unblob run would be the stronger check. I can't run one on my box, since unblob and binwalk aren't installed here. My #4953 result used a stand-in report copied from the PR's unit test, so that's all I have.

🤖 Written with Claude Code (Haiku 5.5, verified by Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Oct 21, 2026): Thanks, XinwangTao, for the nudge. I agree a real unblob run would be the stronger check. I can't run one on my box, since unblob and binwalk aren't installed here. My #4953 result used a stand-in report copied from the PR's unit test, so that's all I have. 🤖 Written with Claude Code (Haiku 5.5, verified by Opus 5.5), posted with my OK 🦊

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 22, 2026):

理解 CassieFox23 无法在本机跑 unblob 的限制。补一个成本最低的替代路径：原报告人 oleander 的环境里就有真实的 unblob 26.6.4 + binwalk 3.1.0（Debian bookworm 容器），可以直接用最初的 4 MiB ESP32-S3 镜像在 #4953 head 上跑一次 extract-firmware，或导出该输入的 unblob JSON report 作为 fixture 合入 PR——这比 stand-in report 更贴近真实结果形状。考虑到 coygeek 的 #5745 恰好暴露了真实 unblob 输出与 fixture 的差异会影响 gap union，这次端到端验证可以同时服务两个 issue，值得在合并前补上。

@XinwangTao commented on GitHub (Oct 22, 2026): 理解 CassieFox23 无法在本机跑 unblob 的限制。补一个成本最低的替代路径：原报告人 oleander 的环境里就有真实的 unblob 26.6.4 + binwalk 3.1.0（Debian bookworm 容器），可以直接用最初的 4 MiB ESP32-S3 镜像在 #4953 head 上跑一次 extract-firmware，或导出该输入的 unblob JSON report 作为 fixture 合入 PR——这比 stand-in report 更贴近真实结果形状。考虑到 coygeek 的 #5745 恰好暴露了真实 unblob 输出与 fixture 的差异会影响 gap union，这次端到端验证可以同时服务两个 issue，值得在合并前补上。

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@VolodymyrChapman commented on GitHub (Oct 23, 2026):

Real-Unblob check of the empty-chunk case, since the earlier verification used a stand-in report.

Environment: Linux x86_64 (Ubuntu 24.04), Node v24.22.1, Unblob 26.6.4 (all 15 external extractors present), Binwalk 3.1.0, /usr/bin/prlimit. Compared the npm release rea-agents@6.7.0 with main at 960b742, which includes 15450643f ("fix(firmware): preserve admission and extraction coverage outcomes"). main's rea --version still prints 6.7.0.

Inputs:

ESP8266_GENERIC-20240222-v1.22.2.bin: the official MicroPython v1.22.2 ESP8266 image, 642,080 bytes, SHA-256 e8ba6a7f7e254531423d3c1a00315c6fb020e8b078de36299922f48b2457bee9. Same family as the ESP32-S3 report: 0xE9 header, no Unblob handler, libmagic says "DOS executable (COM)".
random-4k.bin: head -c 4096 /dev/urandom.
gzip-plus-random.bin: echo hello | gzip -9n followed by input 2 (a control with one known chunk).

rea extract-firmware "$PWD/<input>" "$PWD/out/<fresh-dir>" --json
rea extract-firmware "$PWD/random-4k.bin" "$PWD/out/<fresh-dir>" --offset 1024 --length 2048 --json

For inputs 1 and 2, Unblob's raw report contains only StatReport, FileMagicReport and HashReport, with no chunk or UnknownChunkReport.

Input	6.7.0	`main` `960b742`
ESP8266 image	`complete`, 0 chunks, 0 gaps, 0 bytes	`partial`, 1 `unclassified-range` gap `{0, 642080}`
random 4 KiB	`complete`, 0 gaps	`partial`, 1 `unclassified-range` gap `{0, 4096}`
random 4 KiB, `--offset 1024 --length 2048`	`complete`, 0 gaps	`partial`, gap `range {0, 2048}`, `root_file_range {1024, 2048}`
gzip + random (control)	`partial`, `unknown-chunk-handler` gap `{26, 4096}`	unchanged

On main, published_bytes stays 0 for the zero-chunk inputs, and each gap points to the "producer reported no chunk covering these bytes" continuation. The control case is unchanged and doesn't gain a duplicate unclassified-range row on top of Unblob's own unknown chunk.

So with real Unblob the fix behaves as described. The issue looks resolvable once 15450643f ships in a release; 6.7.0 still reports complete.

@VolodymyrChapman commented on GitHub (Oct 23, 2026): Real-Unblob check of the empty-chunk case, since the earlier verification used a stand-in report. **Environment:** Linux x86_64 (Ubuntu 24.04), Node v24.22.1, Unblob 26.6.4 (all 15 external extractors present), Binwalk 3.1.0, `/usr/bin/prlimit`. Compared the npm release `rea-agents@6.7.0` with `main` at 960b742, which includes 15450643f ("fix(firmware): preserve admission and extraction coverage outcomes"). `main`'s `rea --version` still prints 6.7.0. **Inputs:** 1. `ESP8266_GENERIC-20240222-v1.22.2.bin`: the official MicroPython v1.22.2 ESP8266 image, 642,080 bytes, SHA-256 `e8ba6a7f7e254531423d3c1a00315c6fb020e8b078de36299922f48b2457bee9`. Same family as the ESP32-S3 report: `0xE9` header, no Unblob handler, libmagic says "DOS executable (COM)". 2. `random-4k.bin`: `head -c 4096 /dev/urandom`. 3. `gzip-plus-random.bin`: `echo hello | gzip -9n` followed by input 2 (a control with one known chunk). ```sh rea extract-firmware "$PWD/<input>" "$PWD/out/<fresh-dir>" --json rea extract-firmware "$PWD/random-4k.bin" "$PWD/out/<fresh-dir>" --offset 1024 --length 2048 --json ``` For inputs 1 and 2, Unblob's raw report contains only `StatReport`, `FileMagicReport` and `HashReport`, with no chunk or `UnknownChunkReport`. | Input | 6.7.0 | `main` 960b742 | |---|---|---| | ESP8266 image | `complete`, 0 chunks, 0 gaps, 0 bytes | `partial`, 1 `unclassified-range` gap `{0, 642080}` | | random 4 KiB | `complete`, 0 gaps | `partial`, 1 `unclassified-range` gap `{0, 4096}` | | random 4 KiB, `--offset 1024 --length 2048` | `complete`, 0 gaps | `partial`, gap `range {0, 2048}`, `root_file_range {1024, 2048}` | | gzip + random (control) | `partial`, `unknown-chunk-handler` gap `{26, 4096}` | unchanged | On `main`, `published_bytes` stays 0 for the zero-chunk inputs, and each gap points to the "producer reported no chunk covering these bytes" continuation. The control case is unchanged and doesn't gain a duplicate `unclassified-range` row on top of Unblob's own unknown chunk. So with real Unblob the fix behaves as described. The issue looks resolvable once 15450643f ships in a release; 6.7.0 still reports `complete`.

bsakaguchi commented

2026-10-31 00:30:52 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 24, 2026):

Thanks for running the real-Unblob check — this closes the gap I flagged about the stand-in report. The before/after is exactly the contract fix this issue asked for: on 960b742 the ESP8266 image (same 0xE9-header family as the original ESP32-S3 report, no Unblob handler) goes from "complete" with 0 chunks/0 gaps to "partial" with a single full-range gap, and published_bytes staying 0 keeps the signal honest. The control case (gzip + random) not gaining a duplicate unclassified-range row is also the right behavior — it confirms the gap union doesn't double-report bytes Unblob already classifies. Agree this is resolvable once 1545064 ships in a release; 6.7.0 still reporting "complete" for a silently-skipped image is exactly the misleading signal this issue set out to remove.

@XinwangTao commented on GitHub (Oct 24, 2026): Thanks for running the real-Unblob check — this closes the gap I flagged about the stand-in report. The before/after is exactly the contract fix this issue asked for: on 960b742 the ESP8266 image (same 0xE9-header family as the original ESP32-S3 report, no Unblob handler) goes from "complete" with 0 chunks/0 gaps to "partial" with a single full-range gap, and published_bytes staying 0 keeps the signal honest. The control case (gzip + random) not gaining a duplicate unclassified-range row is also the right behavior — it confirms the gap union doesn't double-report bytes Unblob already classifies. Agree this is resolvable once 1545064 ships in a release; 6.7.0 still reporting "complete" for a silently-skipped image is exactly the misleading signal this issue set out to remove.

Rows
Columns

[GH-ISSUE #3756] extract_firmware reports coverage "complete" when nothing was extracted #1459

Summary

Environment

Steps

Observed

Expected

Likely cause