morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7298] [Bug] Firmware unpublished symlink targets silently replace non-UTF-8 bytes #4333

New issue

Closed

opened 2026-10-31 00:48:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:48:59 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7298

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Production firmware inventory gives two different stored symlink targets the same string: target byte FF becomes U+FFFD, indistinguishable from a target that actually contains U+FFFD. Publication copies that string into unpublished_entries[].link_target without a lossy-encoding marker. This is an executed production-boundary finding, not a claimed public CLI/MCP or real Unblob reproduction.

Reproduction

With the checkout's locked dependencies installed, save this script outside the checkout as probe.mjs and run node probe.mjs "$PWD". It ran once and removed its generated fixtures. An independent second probe also exercised normalization, inventory and publication with a generated producer-shaped report; its schema-validated result retained the replacement-decoded target.

import { mkdtemp, symlink, readlink, rm } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { pathToFileURL } from 'node:url';
const checkout = process.argv[2];
const { importVerifierSource } = await import(pathToFileURL(join(checkout, 'scripts/lib/verifier-source-loader.mjs')));
const { inventoryFirmwareOutput } = await importVerifierSource(join(checkout, 'src/firmware/FirmwareFiles.ts'), import.meta.url);
const root = await mkdtemp(join(tmpdir(), 'firmware-links-'));
try {
  await symlink(Buffer.from([0x73,0x61,0x6d,0x70,0x6c,0x65,0xff]), join(root, 'raw-link'));
  await symlink('sample\ufffd', join(root, 'unicode-link'));
  await symlink('sample', join(root, 'control-link'));
  const inventory = await inventoryFirmwareOutput(root, 1024, 10, 'extract_firmware');
  const rows = [];
  for (const item of inventory) {
    rows.push({name:item.relativePath,stored_hex:(await readlink(item.path,{encoding:'buffer'})).toString('hex'),reported_target:item.linkTarget,reported_utf8_hex:Buffer.from(item.linkTarget).toString('hex')});
  }
  console.log(JSON.stringify(rows,null,2));
} finally { await rm(root,{recursive:true,force:true}); }

Expected behavior

Keep the target bytes distinct or explicitly identify a target that cannot be represented exactly. AGENTS.md requires preserving source values and explicit unknowns when normalization loses information, and separating identity from display formatting. FirmwareProvider.ts:136-138 says link_target values come from Unblob's extracted tree; its limitation discloses absolute-to-relative rewriting but not additional replacement decoding by REA. The narrower filename and historical-reader guarantees are not needed to establish this expectation.

Impact and scope

Bounded incorrect extraction metadata: distinct POSIX target bytes become indistinguishable in the normalized result. No symlink is followed or published; regular file contents and digests are unaffected. Coverage is already partial for unpublished links, but does not identify the target's encoding loss. Real-image frequency is unknown. Raw producer reports remain in Evidence and may retain a recoverable target; real-engine recovery was not independently verified.

REA version

Executed source 0d1a454e25db825567782e461da21530bad4b158 (package metadata 6.8.0). Relevant inventory/provider/publication/schema files match inspected current main byte-for-byte. Latest release rea-agents-6.8.0 also has the default-string readlink branch and direct target projection; release runtime was not run.

Runtime and platform

Node.js 24.22.1; macOS arm64; locked checkout dependencies; repository verifier source loader. Complete firmware-provider execution supports Linux, which was not run here.

Target and artifact context

Generated symlinks with raw targets 73616d706c65ff, 73616d706c65efbfbd, and ASCII control 73616d706c65. No real firmware image or target execution.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

Observed output:

[
  {
    "name": "control-link",
    "stored_hex": "73616d706c65",
    "reported_target": "sample",
    "reported_utf8_hex": "73616d706c65"
  },
  {
    "name": "raw-link",
    "stored_hex": "73616d706c65ff",
    "reported_target": "sample�",
    "reported_utf8_hex": "73616d706c65efbfbd"
  },
  {
    "name": "unicode-link",
    "stored_hex": "73616d706c65efbfbd",
    "reported_target": "sample�",
    "reported_utf8_hex": "73616d706c65efbfbd"
  }
]

Expected source: AGENTS.md, Architecture and Cleanup, source-value and identity preservation; src/firmware/FirmwareProvider.ts:136-138, extracted-tree target provenance.
Failure source: src/firmware/FirmwareFiles.ts:446 default-string readlink(path); src/firmware/FirmwarePublication.ts:136-145 direct entry.linkTarget projection; src/domain/firmware/firmwareAnalysis.ts:194-199 string result schema.
Evidence provenance: observed for production-module execution; static for public call-path reasoning and release/main comparisons.
Local verification: reproduced at inventory and normalize/inventory/publication boundaries; public CLI/MCP and real-engine execution not run.
Reproduction completeness: incomplete for a real firmware end-to-end case.
Missing fact: an actual Linux Unblob extraction yielding a non-UTF-8 symlink target, followed through the public response.

Source-derived supported path: registerFirmwareCommands maps extract-firmware to FirmwareAnalysisService.execute("extract_firmware", ...), which validates input and calls the shared provider. On Linux, after tool/version admission, snapshotting, engine/report consumption and normalization, FirmwareProvider invokes inventoryFirmwareOutput, then publishFirmwareExtraction. Valid UTF-8 entry names pass inventory; no guard checks target bytes before readlink converts them. Publication records symlinks without following them and validates the result. Producer reachability is source-derived from Unblob 26.6.4: its TAR extractor invokes SafeTarFile.extractall; UnblobTarInfo decodes raw linkname with standard tarfile encoding/errors without overriding the default surrogateescape. An ASCII entry name and relative in-root target consisting of sample followed by FF pass the link-path checks and reach tarfile extraction. Subsequent normalization uses os.readlink and Path.symlink_to, preserving POSIX surrogate-escaped bytes without replacement decoding. Full Linux engine/CLI execution remains unrun.

Typed outcome or recovery expected

A truthful target representation or explicit unrepresentable-target outcome/limitation, while preserving the fact that symlinks remain unpublished.

Analysis or suspected cause

Source-derived: default Node readlink string conversion replaces invalid UTF-8 before REA records the target. The result has no separate raw-byte or encoding-state field. The valid Unicode control demonstrates that the string alone cannot distinguish actual U+FFFD from replacement decoding. This report prescribes no schema or implementation.

Additional context

The subject is the extracted-tree target, regardless of Unblob's earlier absolute/relative rewrite. It makes no original firmware-metadata recovery claim. First affected version and real-image frequency are NOT_ENOUGH_INFO.

Restoration check

With the same raw-byte and real-U+FFFD targets, distinguish their exact targets or explicitly identify the raw target as unrepresentable; keep the ASCII control exact. Do not publish links or claim complete extraction to satisfy this check. A separate real Linux/Unblob case would complete end-to-end coverage.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Executed production inventory and publication probes return indistinguishable target strings for distinct stored symlink bytes; impact is bounded extraction metadata correctness, with real-engine incidence unknown.
Workaround status: unverified
Workaround basis: Retained raw producer reports may preserve the exact target, but equivalent real-engine recovery was not independently observed.

References

Closed #4175 and merged #4573 disclose extracted-tree provenance and Unblob's absolute-target rewrite, without addressing REA replacement decoding.
Closed #6743 fixed non-UTF-8 directory-entry omission; this fixture has ASCII entry names and invalid target bytes.
Closed #6765 concerned the separate historical-source reader and subsequent file lookup; no lookup/file substitution is claimed here.
Unblob producer source: TAR extractor, safe TAR parser/extraction, and symlink normalization. These establish the source-derived producer path, not an executed Unblob run.

Originally created by @coygeek on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7298 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior Production firmware inventory gives two different stored symlink targets the same string: target byte FF becomes U+FFFD, indistinguishable from a target that actually contains U+FFFD. Publication copies that string into `unpublished_entries[].link_target` without a lossy-encoding marker. This is an executed production-boundary finding, not a claimed public CLI/MCP or real Unblob reproduction. ### Reproduction With the checkout's locked dependencies installed, save this script outside the checkout as `probe.mjs` and run `node probe.mjs "$PWD"`. It ran once and removed its generated fixtures. An independent second probe also exercised normalization, inventory and publication with a generated producer-shaped report; its schema-validated result retained the replacement-decoded target. ```javascript import { mkdtemp, symlink, readlink, rm } from 'node:fs/promises'; import { join } from 'node:path'; import { tmpdir } from 'node:os'; import { pathToFileURL } from 'node:url'; const checkout = process.argv[2]; const { importVerifierSource } = await import(pathToFileURL(join(checkout, 'scripts/lib/verifier-source-loader.mjs'))); const { inventoryFirmwareOutput } = await importVerifierSource(join(checkout, 'src/firmware/FirmwareFiles.ts'), import.meta.url); const root = await mkdtemp(join(tmpdir(), 'firmware-links-')); try { await symlink(Buffer.from([0x73,0x61,0x6d,0x70,0x6c,0x65,0xff]), join(root, 'raw-link')); await symlink('sample\ufffd', join(root, 'unicode-link')); await symlink('sample', join(root, 'control-link')); const inventory = await inventoryFirmwareOutput(root, 1024, 10, 'extract_firmware'); const rows = []; for (const item of inventory) { rows.push({name:item.relativePath,stored_hex:(await readlink(item.path,{encoding:'buffer'})).toString('hex'),reported_target:item.linkTarget,reported_utf8_hex:Buffer.from(item.linkTarget).toString('hex')}); } console.log(JSON.stringify(rows,null,2)); } finally { await rm(root,{recursive:true,force:true}); } ``` ### Expected behavior Keep the target bytes distinct or explicitly identify a target that cannot be represented exactly. `AGENTS.md` requires preserving source values and explicit unknowns when normalization loses information, and separating identity from display formatting. `FirmwareProvider.ts:136-138` says link_target values come from Unblob's extracted tree; its limitation discloses absolute-to-relative rewriting but not additional replacement decoding by REA. The narrower filename and historical-reader guarantees are not needed to establish this expectation. ### Impact and scope Bounded incorrect extraction metadata: distinct POSIX target bytes become indistinguishable in the normalized result. No symlink is followed or published; regular file contents and digests are unaffected. Coverage is already partial for unpublished links, but does not identify the target's encoding loss. Real-image frequency is unknown. Raw producer reports remain in Evidence and may retain a recoverable target; real-engine recovery was not independently verified. ### REA version Executed source `0d1a454e25db825567782e461da21530bad4b158` (package metadata 6.8.0). Relevant inventory/provider/publication/schema files match inspected current main byte-for-byte. Latest release `rea-agents-6.8.0` also has the default-string readlink branch and direct target projection; release runtime was not run. ### Runtime and platform Node.js 24.22.1; macOS arm64; locked checkout dependencies; repository verifier source loader. Complete firmware-provider execution supports Linux, which was not run here. ### Target and artifact context Generated symlinks with raw targets `73616d706c65ff`, `73616d706c65efbfbd`, and ASCII control `73616d706c65`. No real firmware image or target execution. ### Interface Other ### Earliest failure stage Capture or observation ### Relevant output or evidence Observed output: ```json [ { "name": "control-link", "stored_hex": "73616d706c65", "reported_target": "sample", "reported_utf8_hex": "73616d706c65" }, { "name": "raw-link", "stored_hex": "73616d706c65ff", "reported_target": "sample�", "reported_utf8_hex": "73616d706c65efbfbd" }, { "name": "unicode-link", "stored_hex": "73616d706c65efbfbd", "reported_target": "sample�", "reported_utf8_hex": "73616d706c65efbfbd" } ] ``` - Expected source: `AGENTS.md`, Architecture and Cleanup, source-value and identity preservation; `src/firmware/FirmwareProvider.ts:136-138`, extracted-tree target provenance. - Failure source: `src/firmware/FirmwareFiles.ts:446` default-string `readlink(path)`; `src/firmware/FirmwarePublication.ts:136-145` direct `entry.linkTarget` projection; `src/domain/firmware/firmwareAnalysis.ts:194-199` string result schema. - Evidence provenance: observed for production-module execution; static for public call-path reasoning and release/main comparisons. - Local verification: reproduced at inventory and normalize/inventory/publication boundaries; public CLI/MCP and real-engine execution not run. - Reproduction completeness: incomplete for a real firmware end-to-end case. - Missing fact: an actual Linux Unblob extraction yielding a non-UTF-8 symlink target, followed through the public response. Source-derived supported path: `registerFirmwareCommands` maps `extract-firmware` to `FirmwareAnalysisService.execute("extract_firmware", ...)`, which validates input and calls the shared provider. On Linux, after tool/version admission, snapshotting, engine/report consumption and normalization, `FirmwareProvider` invokes `inventoryFirmwareOutput`, then `publishFirmwareExtraction`. Valid UTF-8 entry names pass inventory; no guard checks target bytes before `readlink` converts them. Publication records symlinks without following them and validates the result. Producer reachability is source-derived from Unblob 26.6.4: its TAR extractor invokes SafeTarFile.extractall; UnblobTarInfo decodes raw linkname with standard tarfile encoding/errors without overriding the default surrogateescape. An ASCII entry name and relative in-root target consisting of sample followed by FF pass the link-path checks and reach tarfile extraction. Subsequent normalization uses os.readlink and Path.symlink_to, preserving POSIX surrogate-escaped bytes without replacement decoding. Full Linux engine/CLI execution remains unrun. ### Typed outcome or recovery expected A truthful target representation or explicit unrepresentable-target outcome/limitation, while preserving the fact that symlinks remain unpublished. ### Analysis or suspected cause Source-derived: default Node readlink string conversion replaces invalid UTF-8 before REA records the target. The result has no separate raw-byte or encoding-state field. The valid Unicode control demonstrates that the string alone cannot distinguish actual U+FFFD from replacement decoding. This report prescribes no schema or implementation. ### Additional context The subject is the extracted-tree target, regardless of Unblob's earlier absolute/relative rewrite. It makes no original firmware-metadata recovery claim. First affected version and real-image frequency are NOT_ENOUGH_INFO. ## Restoration check With the same raw-byte and real-U+FFFD targets, distinguish their exact targets or explicitly identify the raw target as unrepresentable; keep the ASCII control exact. Do not publish links or claim complete extraction to satisfy this check. A separate real Linux/Unblob case would complete end-to-end coverage. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Executed production inventory and publication probes return indistinguishable target strings for distinct stored symlink bytes; impact is bounded extraction metadata correctness, with real-engine incidence unknown. - Workaround status: unverified - Workaround basis: Retained raw producer reports may preserve the exact target, but equivalent real-engine recovery was not independently observed. ## References - Closed [#4175](https://github.com/morluto/rea/issues/4175) and merged [#4573](https://github.com/morluto/rea/pull/4573) disclose extracted-tree provenance and Unblob's absolute-target rewrite, without addressing REA replacement decoding. - Closed [#6743](https://github.com/morluto/rea/issues/6743) fixed non-UTF-8 directory-entry omission; this fixture has ASCII entry names and invalid target bytes. - Closed [#6765](https://github.com/morluto/rea/issues/6765) concerned the separate historical-source reader and subsequent file lookup; no lookup/file substitution is claimed here. - Unblob producer source: [TAR extractor](https://github.com/onekey-sec/unblob/blob/26.6.4/python/unblob/handlers/archive/tar.py#L96-L100), [safe TAR parser/extraction](https://github.com/onekey-sec/unblob/blob/26.6.4/python/unblob/handlers/archive/_safe_tarfile.py#L45), and [symlink normalization](https://github.com/onekey-sec/unblob/blob/26.6.4/python/unblob/extractor.py#L50-L92). These establish the source-derived producer path, not an executed Unblob run.

bsakaguchi closed this issue

2026-10-31 00:48:59 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:27 +00:00

[PR #4333] [CLOSED] fix(artifact): accept POSIX names beginning with letter-colon on non-Windows hosts #6176

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425