morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6895] [Bug] Standalone Unblob task errors omit firmware recovery gaps #4230

New issue

Closed

opened 2026-10-31 00:47:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:47:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6895

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A valid standalone Unblob UnknownError report survives normalization as a diagnostic with coverage: "partial", but produces no failed_extraction_stages entry. The public gap builder uses that collection to emit standalone task-failure rows, so the source-traced consequence is a missing extractor-failure row and recovery continuation in extract_firmware.

The normalization result below was executed against the source checkout. The public CLI/MCP consequence is source-derived: no real Unblob exception, publication, CLI request, or MCP request was executed.

Reproduction

From the source checkout at the revision below, with its locked dependencies installed, run this observation-only probe. It supplies self-authored report data; it creates no firmware output and executes no firmware or external engine.

fixture_dir="$(mktemp -d)"
cat > "$fixture_dir/task-failure.mjs" <<'JS'
import { createHash } from 'node:crypto';
import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';
const { normalizeUnblobReport } = await import(pathToFileURL(resolve('src/firmware/FirmwareReports.ts')).href);
const bytes=Buffer.alloc(20);
const hash=(algorithm)=>createHash(algorithm).update(bytes).digest('hex');
const context = { inputPath:'/fixture/input.bin', outputRoot:'/fixture/output', sha256:hash('sha256'), length:bytes.length, offset:0, maxDepth:3 };
const base = {
  task:{ path:context.inputPath, depth:0, blob_id:'', is_multi_file:false },
  reports:[
    { __typename__:'StatReport', path:context.inputPath, size:20, is_file:true, is_dir:false, is_link:false, link_target:null },
    { __typename__:'FileMagicReport', magic:'data', mime_type:'application/octet-stream' },
    { __typename__:'HashReport', md5:hash('md5'), sha1:hash('sha1'), sha256:context.sha256 },
  ],
  subtasks:[],
};
for (const [name, extra] of [
  ['clean_control',[]],
  ['task_error',[{ __typename__:'UnknownError', severity:'ERROR', exception:'Self-authored inert task-error report' }]],
  ['dependency_control',[{ __typename__:'ExtractorDependencyNotFoundReport', severity:'ERROR', dependencies:['fixture-tool'] }]],
]) {
  const result=normalizeUnblobReport([{ ...base, reports:[...base.reports,...extra] }],context);
  console.log(JSON.stringify({ name, coverage:result.coverage, diagnostics:result.diagnostics, failed_extraction_stages:result.failed_extraction_stages, missing_dependencies:result.missing_dependencies }));
}
JS
node --import ./node_modules/tsx/dist/loader.mjs "$fixture_dir/task-failure.mjs"

The task_error case has matching root identity, regular-file size and SHA-256 metadata, an empty child graph, and a top-level UnknownError. The error is the only change from clean_control; dependency_control demonstrates that a supported standalone dependency error retains structured recovery input. The final probe ran once, with the same task-error omission also seen in two earlier equivalent parser probes. No intermittent behavior was observed in this bounded sample.

Expected behavior

Retain partial coverage and the original diagnostic, and also retain one standalone failed-task entry for the real producer's task-error report so the public result can emit a failure gap and continuation. Preserve unknown stage identity and byte ownership: handler, provider_id, range, and root_file_range should remain null rather than attributing the error to a named extractor or all input bytes.

The firmware guide explicitly describes a failed task/stage row and null stage identity for standalone task failures. Pinned Unblob catches task exceptions and emits UnknownError; that report is an error with severity ERROR.

Impact and scope

Provisional P2: a bounded omission in machine-readable failure/recovery metadata. A caller relying on extraction_gaps can miss a reported failed task even though partial coverage and its raw diagnostic remain available. Manual inspection of the retained diagnostics is an available workaround; no configuration workaround restoring the structured row was established. No lost published bytes, incorrect completeness claim, or security-boundary failure was established.

This report covers task errors after required metadata exists. Root failures before HashReport, and tasks failing before StatReport, hit separate parser admission guards and are outside this case. An actual firmware input causing a producer exception was not generated; end-to-end occurrence frequency is unknown.

REA version

Source commit ef475fcf8b0601c61052b7091568ba574546417d.

Runtime and platform

Node.js v24.22.1; Linux 7.2.8-9-omarchy, x86_64; Omarchy. The locked source loader was used. Unblob 26.6.4 source at 1fcc7a0a584a70a96c31f5a276c20944d199a089 was inspected, not installed or executed. Hopper and Ghidra are not involved.

Target and artifact context

Self-authored in-memory producer report for a 20-byte all-zero input, with computed hashes and generic logical fixture paths. The probe invokes the clone's normalizeUnblobReport directly. The error record is synthetic and follows the pinned producer's report model.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed probe output, reduced to meaningful fields:

{"name":"clean_control","coverage":"complete","diagnostics":[],"failed_extraction_stages":[],"missing_dependencies":[]}
{"name":"task_error","coverage":"partial","diagnostics":[{"input_path":"$input","report":{"__typename__":"UnknownError","severity":"ERROR","exception":"Self-authored inert task-error report"}}],"failed_extraction_stages":[],"missing_dependencies":[]}
{"name":"dependency_control","coverage":"partial","diagnostics":[{"input_path":"$input","report":{"__typename__":"ExtractorDependencyNotFoundReport","severity":"ERROR","dependencies":["fixture-tool"]}}],"failed_extraction_stages":[],"missing_dependencies":[{"input_path":"$input","dependencies":["fixture-tool"]}]}

Expected source: the linked firmware guide and pinned producer error contract above.
Failure source: the executed task_error result above; the source-traced gap consequence follows the normalizer and publication references below.
Evidence provenance: observed. Producer generation and public projection are source-derived.
Local verification: reproduced. This applies to the source normalization boundary; full CLI/MCP and real-producer execution were not run.
Reproduction completeness: complete. This applies to the stated report-normalization boundary.

Public reachability, derived from the tested revision: firmware CLI and MCP share FirmwareAnalysisService, which validates and delegates to the provider. The provider normalizes the report and accepts producer exit 1 when a diagnostic exists. An absent extraction tree yields an empty inventory; only $input is a reported file in this fixture, so the publication file-presence guard does not reject it. The public gap builder derives standalone failure rows from failed_extraction_stages; no other gap condition exists in the fixture. Completion still requires ordinary valid destination, budget, and cleanup conditions. Those publication conditions were not executed by this probe.

Typed outcome or recovery expected

Restoration check: with the same task-error report, retain its diagnostic and partial coverage and produce one standalone failed-task entry, leading to a public extractor-failure gap with null stage identity/ranges and a continuation referencing the task diagnostic. The clean control should remain free of failure entries; the dependency control should retain its named dependency recovery. Do not turn this into a hard refusal that discards the accepted partial result. This is a proposed restoration check; no fix was made or tested.

Analysis or suspected cause

Source-derived: the normalizer adds standalone failures only for the exact type name ExtractorExecutionErrorReport. Pinned Unblob's task-error path instead emits UnknownError. The diagnostic fallback preserves that real error but does not add a failed-task entry. This identifies a producer/report interpretation mismatch; it does not establish which firmware bytes or extractor caused a real error.

Additional context

Open and closed issue searches found no matching standalone UnknownError gap omission. Related reports are distinct: #5745 emits false failure gaps for resolved warnings; #3756 concerns no recognized chunks without a reported task error; #1853 concerns named dependency recovery, which the control preserves. No known-good revision was established.

Triage assessment

Impact level: P2
Assessment status: provisional
Impact basis: The observed normalizer result omits structured task-failure metadata for the pinned producer's error type; source tracing shows the corresponding public recovery gap is missing. Actual producer failure and CLI/MCP occurrence were not replayed.
Workaround status: available
Workaround basis: The probe directly shows that the original error remains in diagnostics with partial coverage, so callers can inspect those retained reports manually instead of relying solely on extraction_gaps. This does not restore the structured row.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6895 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A valid standalone Unblob `UnknownError` report survives normalization as a diagnostic with `coverage: "partial"`, but produces no `failed_extraction_stages` entry. The public gap builder uses that collection to emit standalone task-failure rows, so the source-traced consequence is a missing `extractor-failure` row and recovery continuation in `extract_firmware`. The normalization result below was executed against the source checkout. The public CLI/MCP consequence is source-derived: no real Unblob exception, publication, CLI request, or MCP request was executed. ### Reproduction From the source checkout at the revision below, with its locked dependencies installed, run this observation-only probe. It supplies self-authored report data; it creates no firmware output and executes no firmware or external engine. ```sh fixture_dir="$(mktemp -d)" cat > "$fixture_dir/task-failure.mjs" <<'JS' import { createHash } from 'node:crypto'; import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; const { normalizeUnblobReport } = await import(pathToFileURL(resolve('src/firmware/FirmwareReports.ts')).href); const bytes=Buffer.alloc(20); const hash=(algorithm)=>createHash(algorithm).update(bytes).digest('hex'); const context = { inputPath:'/fixture/input.bin', outputRoot:'/fixture/output', sha256:hash('sha256'), length:bytes.length, offset:0, maxDepth:3 }; const base = { task:{ path:context.inputPath, depth:0, blob_id:'', is_multi_file:false }, reports:[ { __typename__:'StatReport', path:context.inputPath, size:20, is_file:true, is_dir:false, is_link:false, link_target:null }, { __typename__:'FileMagicReport', magic:'data', mime_type:'application/octet-stream' }, { __typename__:'HashReport', md5:hash('md5'), sha1:hash('sha1'), sha256:context.sha256 }, ], subtasks:[], }; for (const [name, extra] of [ ['clean_control',[]], ['task_error',[{ __typename__:'UnknownError', severity:'ERROR', exception:'Self-authored inert task-error report' }]], ['dependency_control',[{ __typename__:'ExtractorDependencyNotFoundReport', severity:'ERROR', dependencies:['fixture-tool'] }]], ]) { const result=normalizeUnblobReport([{ ...base, reports:[...base.reports,...extra] }],context); console.log(JSON.stringify({ name, coverage:result.coverage, diagnostics:result.diagnostics, failed_extraction_stages:result.failed_extraction_stages, missing_dependencies:result.missing_dependencies })); } JS node --import ./node_modules/tsx/dist/loader.mjs "$fixture_dir/task-failure.mjs" ``` The `task_error` case has matching root identity, regular-file size and SHA-256 metadata, an empty child graph, and a top-level `UnknownError`. The error is the only change from `clean_control`; `dependency_control` demonstrates that a supported standalone dependency error retains structured recovery input. The final probe ran once, with the same task-error omission also seen in two earlier equivalent parser probes. No intermittent behavior was observed in this bounded sample. ### Expected behavior Retain partial coverage and the original diagnostic, and also retain one standalone failed-task entry for the real producer's task-error report so the public result can emit a failure gap and continuation. Preserve unknown stage identity and byte ownership: `handler`, `provider_id`, `range`, and `root_file_range` should remain null rather than attributing the error to a named extractor or all input bytes. The [firmware guide](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/docs/firmware-analysis.md#L131) explicitly describes a failed task/stage row and null stage identity for standalone task failures. Pinned Unblob [catches task exceptions and emits `UnknownError`](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/python/unblob/processing.py#L256); [that report is an error with severity `ERROR`](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/python/unblob/report.py#L49). ### Impact and scope Provisional P2: a bounded omission in machine-readable failure/recovery metadata. A caller relying on `extraction_gaps` can miss a reported failed task even though partial coverage and its raw diagnostic remain available. Manual inspection of the retained diagnostics is an available workaround; no configuration workaround restoring the structured row was established. No lost published bytes, incorrect completeness claim, or security-boundary failure was established. This report covers task errors after required metadata exists. Root failures before `HashReport`, and tasks failing before `StatReport`, hit separate parser admission guards and are outside this case. An actual firmware input causing a producer exception was not generated; end-to-end occurrence frequency is unknown. ### REA version Source commit `ef475fcf8b0601c61052b7091568ba574546417d`. ### Runtime and platform Node.js v24.22.1; Linux 7.2.8-9-omarchy, x86_64; Omarchy. The locked source loader was used. Unblob 26.6.4 source at `1fcc7a0a584a70a96c31f5a276c20944d199a089` was inspected, not installed or executed. Hopper and Ghidra are not involved. ### Target and artifact context Self-authored in-memory producer report for a 20-byte all-zero input, with computed hashes and generic logical fixture paths. The probe invokes the clone's `normalizeUnblobReport` directly. The error record is synthetic and follows the pinned producer's report model. ### Interface Other ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed probe output, reduced to meaningful fields: ```json {"name":"clean_control","coverage":"complete","diagnostics":[],"failed_extraction_stages":[],"missing_dependencies":[]} {"name":"task_error","coverage":"partial","diagnostics":[{"input_path":"$input","report":{"__typename__":"UnknownError","severity":"ERROR","exception":"Self-authored inert task-error report"}}],"failed_extraction_stages":[],"missing_dependencies":[]} {"name":"dependency_control","coverage":"partial","diagnostics":[{"input_path":"$input","report":{"__typename__":"ExtractorDependencyNotFoundReport","severity":"ERROR","dependencies":["fixture-tool"]}}],"failed_extraction_stages":[],"missing_dependencies":[{"input_path":"$input","dependencies":["fixture-tool"]}]} ``` - Expected source: the linked firmware guide and pinned producer error contract above. - Failure source: the executed `task_error` result above; the source-traced gap consequence follows the normalizer and publication references below. - Evidence provenance: observed. Producer generation and public projection are source-derived. - Local verification: reproduced. This applies to the source normalization boundary; full CLI/MCP and real-producer execution were not run. - Reproduction completeness: complete. This applies to the stated report-normalization boundary. Public reachability, derived from the tested revision: firmware [CLI](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/cli/firmwareCommands.ts#L21) and [MCP](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/server/registerFirmwareTools.ts#L18) share `FirmwareAnalysisService`, which [validates and delegates](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/application/firmware/FirmwareAnalysisService.ts#L35) to the provider. [The provider](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/firmware/FirmwareProvider.ts#L484) normalizes the report and accepts producer exit 1 when a diagnostic exists. An absent extraction tree [yields an empty inventory](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/firmware/FirmwareFiles.ts#L314); only `$input` is a reported file in this fixture, so the publication file-presence guard does not reject it. The [public gap builder](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/firmware/FirmwarePublication.ts#L437) derives standalone failure rows from `failed_extraction_stages`; no other gap condition exists in the fixture. Completion still requires ordinary valid destination, budget, and cleanup conditions. Those publication conditions were not executed by this probe. ### Typed outcome or recovery expected Restoration check: with the same task-error report, retain its diagnostic and partial coverage and produce one standalone failed-task entry, leading to a public `extractor-failure` gap with null stage identity/ranges and a continuation referencing the task diagnostic. The clean control should remain free of failure entries; the dependency control should retain its named dependency recovery. Do not turn this into a hard refusal that discards the accepted partial result. This is a proposed restoration check; no fix was made or tested. ### Analysis or suspected cause Source-derived: [the normalizer](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/firmware/FirmwareReports.ts#L355) adds standalone failures only for the exact type name `ExtractorExecutionErrorReport`. Pinned Unblob's task-error path instead emits `UnknownError`. The diagnostic fallback preserves that real error but does not add a failed-task entry. This identifies a producer/report interpretation mismatch; it does not establish which firmware bytes or extractor caused a real error. ### Additional context Open and closed issue searches found no matching standalone `UnknownError` gap omission. Related reports are distinct: [#5745](https://github.com/morluto/rea/issues/5745) emits false failure gaps for resolved warnings; [#3756](https://github.com/morluto/rea/issues/3756) concerns no recognized chunks without a reported task error; [#1853](https://github.com/morluto/rea/issues/1853) concerns named dependency recovery, which the control preserves. No known-good revision was established. ## Triage assessment - Impact level: P2 - Assessment status: provisional - Impact basis: The observed normalizer result omits structured task-failure metadata for the pinned producer's error type; source tracing shows the corresponding public recovery gap is missing. Actual producer failure and CLI/MCP occurrence were not replayed. - Workaround status: available - Workaround basis: The probe directly shows that the original error remains in diagnostics with partial coverage, so callers can inspect those retained reports manually instead of relying solely on extraction_gaps. This does not restore the structured row.

bsakaguchi closed this issue

2026-10-31 00:47:53 +00:00

bsakaguchi commented

2026-10-31 00:47:54 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 23, 2026):

Resolved by the shared producer-severity/ownership fixes already on main: 243790adc, 7298bd08d and 15450643f.

Executed verification on 091d6209 used real Unblob 26.6.4 with a controlled failure after metadata/hash collection on an owned 32-byte input. The real producer emitted UnknownError/ERROR and exited 1. Compiled CLI and stdio MCP both retained the report/exit code, a task failure recovery row with unknown stage/byte ownership, and the independent unclassified-byte gap. Normalized semantics matched; the unmodified-producer control had no task failure. All 49 relevant existing tests passed.

The old PR’s narrower patch and stale publication expectations are superseded. Detailed evidence and limitations are recorded on #6911. This verifies a real producer’s controlled failure path, not natural firmware prevalence or arbitrary extraction formats.

@gensokyo-dev commented on GitHub (Oct 23, 2026): Resolved by the shared producer-severity/ownership fixes already on main: [243790adc](https://github.com/morluto/rea/commit/243790adc9c6ce147ba10e330209bfc2422b7923), [7298bd08d](https://github.com/morluto/rea/commit/7298bd08d134860b84b4073f6703021916a5a4c5) and [15450643f](https://github.com/morluto/rea/commit/15450643f7502114c4f0755416f97b8f0b358b69). Executed verification on `091d6209` used real Unblob 26.6.4 with a controlled failure after metadata/hash collection on an owned 32-byte input. The real producer emitted `UnknownError/ERROR` and exited 1. Compiled CLI and stdio MCP both retained the report/exit code, a task failure recovery row with unknown stage/byte ownership, and the independent unclassified-byte gap. Normalized semantics matched; the unmodified-producer control had no task failure. All 49 relevant existing tests passed. The old PR’s narrower patch and stale publication expectations are superseded. Detailed evidence and limitations are recorded on #6911. This verifies a real producer’s controlled failure path, not natural firmware prevalence or arbitrary extraction formats.

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4230

No description provided.

Rows
Columns