morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6743] [Bug] Firmware staging inventory silently omits non-UTF-8 filenames #4106

New issue

Closed

opened 2026-10-31 00:46:51 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:46:51 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6743

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Production inventoryFirmwareOutput silently omits an existing POSIX regular file whose filename contains an invalid UTF-8 byte. A generated staging directory contains good.txt (four bytes) and filename hex 626164ff2e747874 (three bytes), independently confirmed with raw-byte readdir and lstat. Inventory returns only good.txt. With maxBytes: 4, it accepts that seven-byte tree and counts only four bytes. The same failure was observed in the requested source and published npm 6.7.0 modules.

A secondary production-publication probe supplied those real inventory entries and deliberately synthetic normalized producer facts. It committed only good.txt, returned coverage: "complete", and reported no omission or gap. This does not establish actual Unblob or CLI/MCP behavior: a real report naming the omitted file may instead cause publication's reported-file check to refuse it.

Reproduction

The complete executed probe below generates harmless files, independently checks their raw names and sizes, calls production inventory twice, and exercises publication with synthetic producer facts. It removes its fixture directory. From the checkout with locked dependencies installed:

mkdir -p /tmp/rea-firmware-fixtures
# Save the script below at /tmp/rea-firmware-fixtures/probe.mts.
node --import tsx /tmp/rea-firmware-fixtures/probe.mts "$PWD/src" ts

The same probe was executed with the separately installed npm rea-agents@6.7.0 package’s dist directory as module base and extension js. One invocation per version reproduced the result before drafting. The script below uses a neutral scratch path in place of the audit’s temporary directory; its operations and fixture bytes are unchanged.

import {mkdir,writeFile,readdir,lstat,rm,mkdtemp} from 'node:fs/promises';
import {join} from 'node:path';
import {pathToFileURL} from 'node:url';
const base=process.argv[2];
const ext=process.argv[3];
const load=(name:string)=>import(pathToFileURL(join(base,`firmware/${name}.${ext}`)).href);
const {inventoryFirmwareOutput,FirmwareFileAccess}=await load('FirmwareFiles');
const {publishFirmwareExtraction}=await load('FirmwarePublication');
const root=await mkdtemp('/tmp/rea-firmware-fixtures/fixture-');
const staging=join(root,'staging'); await mkdir(staging);
const prefix=Buffer.from(staging+'/');
await writeFile(join(staging,'good.txt'),'GOOD');
await writeFile(Buffer.concat([prefix,Buffer.from('626164ff2e747874','hex')]),'RAW');
const actual=await Promise.all((await readdir(staging,{encoding:'buffer'})).map(async(name)=>({name_hex:name.toString('hex'),bytes:(await lstat(Buffer.concat([prefix,name]))).size})));
const entries=await inventoryFirmwareOutput(staging,1000,100,'extract_firmware');
let tightBudget; try {const e=await inventoryFirmwareOutput(staging,4,100,'extract_firmware');tightBudget={accepted:true,total_inventory_bytes:e.reduce((n,x)=>n+x.size,0),actual_file_bytes:actual.reduce((n,x)=>n+x.bytes,0),maxBytes:4};}catch(e){tightBudget={accepted:false,error:String(e)}}
const files=new FirmwareFileAccess();
const signal=new AbortController().signal;
const digest='0'.repeat(64);
try {
 const normalized={files:new Map(),sizes:new Map(),chunks:[],derivations:[],depth_limited_paths:[],failed_extraction_stages:[],diagnostics:[],missing_dependencies:[],chunk_missing_dependencies:[],coverage:'complete'};
 const result=await publishFirmwareExtraction({files,workspacePath:root,deadline:{phase:'publication',signal,timeoutMs:120000,expiresAtMs:performance.now()+120000},analysisTimeoutMs:30000,outputDirectory:join(root,'published'),entries,normalized,selection:{offset:0,length:8,sha256:digest},maxDepth:3,engine:{name:'unblob',version:'fixture',executable_path:'/fixture/unblob',executable_sha256:digest,source_revision:null,worker_count:1},signal,retainCleanup:()=>{throw new Error('unexpected retained cleanup')}});
 console.log(JSON.stringify({module_base:base,node:process.version,platform:process.platform,arch:process.arch,actual,tightBudget,inventory:entries.map(e=>({relative_path:e.relativePath,size:e.size})),publication:{coverage:result.coverage,files:result.files.map(f=>({relative_path:f.relative_path,size:f.size})),published_bytes:result.published_bytes,unpublished_entries:result.unpublished_entries,extraction_gaps:result.extraction_gaps},published_names:await readdir(join(root,'published'))}));
} finally {await files.close();await rm(root,{recursive:true,force:true});}

Expected behavior

Account for every existing staging entry or return an explicit unsupported-name/omission outcome. A replacement-decoded pathname returning ENOENT does not establish that the original directory entry disappeared.

FirmwareFiles.ts promises private-output inventory and aggregate staging-budget enforcement. The firmware input contract describes max_output_bytes as the staging file-byte budget. Repository identity rules require preserving omissions and uncertainty across boundaries.

Impact and scope

Observed bounded staging-inventory omission and incorrect aggregate-byte accounting for non-UTF-8 POSIX filenames. Source files remain unchanged. The secondary probe shows downstream omission and complete coverage when normalized producer facts do not expose the mismatch. Real-extractor incidence and its report-dependent outcome remain unknown. No operational workaround was verified. This is an ordinary extraction-correctness report, not a security finding.

REA version

Source bc411c24befceced5a9ea1ca1c1ce123fd02bf48 (metadata 6.7.0, newer source than the package), and published npm rea-agents@6.7.0. npm's stable dist-tag was checked and was 6.7.0.

Runtime and platform

Node.js v26.12.1; Linux x64; locked checkout dependencies. Direct production-module execution through node --import tsx. No external extractor or public CLI/MCP round trip was run.

Target and artifact context

Generated local staging directory containing good.txt with GOOD, and raw filename bytes 626164ff2e747874 with RAW. No sensitive or external target data.

Interface

Other

Earliest failure stage

Capture or observation

Relevant output or evidence

Both source and published release returned these relevant facts:

{
  "actual": [
    {"name_hex":"626164ff2e747874","bytes":3},
    {"name_hex":"676f6f642e747874","bytes":4}
  ],
  "tightBudget": {"accepted":true,"total_inventory_bytes":4,"actual_file_bytes":7,"maxBytes":4},
  "inventory": [{"relative_path":"good.txt","size":4}],
  "publication": {"coverage":"complete","files":[{"relative_path":"good.txt","size":4}],"published_bytes":4,"unpublished_entries":[],"extraction_gaps":[]},
  "published_names": ["good.txt"]
}

Evidence provenance: observed production inventory/publication boundaries with generated filesystem inputs; publication's normalized producer facts were synthetic. Actual Unblob extraction, report normalization for this filename, and public CLI/MCP behavior are NOT_ENOUGH_INFO and were not executed. No fix or restoration is claimed.

Typed outcome or recovery expected

Preserve raw filename identity where supported, or explicitly report/refuse the unrepresentable entry and account for its bytes. Do not turn an inventory omission into an unqualified representation of the staging tree.

Analysis or suspected cause

Source-derived: FirmwareFiles.ts:324–360 opens the directory using default UTF-8 decoding, then reconstructs a pathname from entry.name. Invalid byte FF becomes U+FFFD. lstat on that nonexistent replacement name returns ENOENT, which is silently skipped before byte accounting. This is independent of DirectoryArtifactReader.

Publication's reported-file check can reject an omitted file named in the real producer report. The primary inventory omission and byte-accounting result do not depend on the synthetic publication facts.

Additional context

Related #6665 covers lossy filename substitution in the separate DirectoryArtifactReader/desktop service; fixing that traversal alone leaves this firmware implementation unchanged. #6601 concerns valid decomposed Unicode normalization after inventory, whereas this omission occurs before publication and involves invalid UTF-8. #3756 concerns an engine report with no recognized chunks. The supplied complete open-issue snapshot, matching bodies, and fresh all-state searches for firmware UTF and firmware filename revealed no matching firmware report.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Direct production-module probes in source and published 6.7.0 omit an existing staging file and undercount aggregate file bytes. This is bounded; real-extractor incidence and report-dependent downstream behavior remain unverified.
Workaround status: unverified
Workaround basis: No operational workaround was executed. A corrected implementation or explicit unsupported-name outcome remains to be verified.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6743 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior Production `inventoryFirmwareOutput` silently omits an existing POSIX regular file whose filename contains an invalid UTF-8 byte. A generated staging directory contains `good.txt` (four bytes) and filename hex `626164ff2e747874` (three bytes), independently confirmed with raw-byte `readdir` and `lstat`. Inventory returns only `good.txt`. With `maxBytes: 4`, it accepts that seven-byte tree and counts only four bytes. The same failure was observed in the requested source and published npm 6.7.0 modules. A secondary production-publication probe supplied those real inventory entries and deliberately **synthetic normalized producer facts**. It committed only `good.txt`, returned `coverage: "complete"`, and reported no omission or gap. This does not establish actual Unblob or CLI/MCP behavior: a real report naming the omitted file may instead cause publication's reported-file check to refuse it. ### Reproduction The complete executed probe below generates harmless files, independently checks their raw names and sizes, calls production inventory twice, and exercises publication with synthetic producer facts. It removes its fixture directory. From the checkout with locked dependencies installed: ```sh mkdir -p /tmp/rea-firmware-fixtures # Save the script below at /tmp/rea-firmware-fixtures/probe.mts. node --import tsx /tmp/rea-firmware-fixtures/probe.mts "$PWD/src" ts ``` The same probe was executed with the separately installed npm `rea-agents@6.7.0` package’s `dist` directory as module base and extension `js`. One invocation per version reproduced the result before drafting. The script below uses a neutral scratch path in place of the audit’s temporary directory; its operations and fixture bytes are unchanged. ```ts import {mkdir,writeFile,readdir,lstat,rm,mkdtemp} from 'node:fs/promises'; import {join} from 'node:path'; import {pathToFileURL} from 'node:url'; const base=process.argv[2]; const ext=process.argv[3]; const load=(name:string)=>import(pathToFileURL(join(base,`firmware/${name}.${ext}`)).href); const {inventoryFirmwareOutput,FirmwareFileAccess}=await load('FirmwareFiles'); const {publishFirmwareExtraction}=await load('FirmwarePublication'); const root=await mkdtemp('/tmp/rea-firmware-fixtures/fixture-'); const staging=join(root,'staging'); await mkdir(staging); const prefix=Buffer.from(staging+'/'); await writeFile(join(staging,'good.txt'),'GOOD'); await writeFile(Buffer.concat([prefix,Buffer.from('626164ff2e747874','hex')]),'RAW'); const actual=await Promise.all((await readdir(staging,{encoding:'buffer'})).map(async(name)=>({name_hex:name.toString('hex'),bytes:(await lstat(Buffer.concat([prefix,name]))).size}))); const entries=await inventoryFirmwareOutput(staging,1000,100,'extract_firmware'); let tightBudget; try {const e=await inventoryFirmwareOutput(staging,4,100,'extract_firmware');tightBudget={accepted:true,total_inventory_bytes:e.reduce((n,x)=>n+x.size,0),actual_file_bytes:actual.reduce((n,x)=>n+x.bytes,0),maxBytes:4};}catch(e){tightBudget={accepted:false,error:String(e)}} const files=new FirmwareFileAccess(); const signal=new AbortController().signal; const digest='0'.repeat(64); try { const normalized={files:new Map(),sizes:new Map(),chunks:[],derivations:[],depth_limited_paths:[],failed_extraction_stages:[],diagnostics:[],missing_dependencies:[],chunk_missing_dependencies:[],coverage:'complete'}; const result=await publishFirmwareExtraction({files,workspacePath:root,deadline:{phase:'publication',signal,timeoutMs:120000,expiresAtMs:performance.now()+120000},analysisTimeoutMs:30000,outputDirectory:join(root,'published'),entries,normalized,selection:{offset:0,length:8,sha256:digest},maxDepth:3,engine:{name:'unblob',version:'fixture',executable_path:'/fixture/unblob',executable_sha256:digest,source_revision:null,worker_count:1},signal,retainCleanup:()=>{throw new Error('unexpected retained cleanup')}}); console.log(JSON.stringify({module_base:base,node:process.version,platform:process.platform,arch:process.arch,actual,tightBudget,inventory:entries.map(e=>({relative_path:e.relativePath,size:e.size})),publication:{coverage:result.coverage,files:result.files.map(f=>({relative_path:f.relative_path,size:f.size})),published_bytes:result.published_bytes,unpublished_entries:result.unpublished_entries,extraction_gaps:result.extraction_gaps},published_names:await readdir(join(root,'published'))})); } finally {await files.close();await rm(root,{recursive:true,force:true});} ``` ### Expected behavior Account for every existing staging entry or return an explicit unsupported-name/omission outcome. A replacement-decoded pathname returning `ENOENT` does not establish that the original directory entry disappeared. [FirmwareFiles.ts](https://github.com/morluto/rea/blob/bc411c24befceced5a9ea1ca1c1ce123fd02bf48/src/firmware/FirmwareFiles.ts#L296) promises private-output inventory and aggregate staging-budget enforcement. The [firmware input contract](https://github.com/morluto/rea/blob/bc411c24befceced5a9ea1ca1c1ce123fd02bf48/src/domain/firmware/firmwareAnalysis.ts) describes `max_output_bytes` as the staging file-byte budget. Repository identity rules require preserving omissions and uncertainty across boundaries. ### Impact and scope Observed bounded staging-inventory omission and incorrect aggregate-byte accounting for non-UTF-8 POSIX filenames. Source files remain unchanged. The secondary probe shows downstream omission and complete coverage when normalized producer facts do not expose the mismatch. Real-extractor incidence and its report-dependent outcome remain unknown. No operational workaround was verified. This is an ordinary extraction-correctness report, not a security finding. ### REA version Source `bc411c24befceced5a9ea1ca1c1ce123fd02bf48` (metadata 6.7.0, newer source than the package), and published npm `rea-agents@6.7.0`. npm's stable dist-tag was checked and was 6.7.0. ### Runtime and platform Node.js v26.12.1; Linux x64; locked checkout dependencies. Direct production-module execution through `node --import tsx`. No external extractor or public CLI/MCP round trip was run. ### Target and artifact context Generated local staging directory containing `good.txt` with `GOOD`, and raw filename bytes `626164ff2e747874` with `RAW`. No sensitive or external target data. ### Interface Other ### Earliest failure stage Capture or observation ### Relevant output or evidence Both source and published release returned these relevant facts: ```json { "actual": [ {"name_hex":"626164ff2e747874","bytes":3}, {"name_hex":"676f6f642e747874","bytes":4} ], "tightBudget": {"accepted":true,"total_inventory_bytes":4,"actual_file_bytes":7,"maxBytes":4}, "inventory": [{"relative_path":"good.txt","size":4}], "publication": {"coverage":"complete","files":[{"relative_path":"good.txt","size":4}],"published_bytes":4,"unpublished_entries":[],"extraction_gaps":[]}, "published_names": ["good.txt"] } ``` Evidence provenance: observed production inventory/publication boundaries with generated filesystem inputs; publication's normalized producer facts were synthetic. Actual Unblob extraction, report normalization for this filename, and public CLI/MCP behavior are NOT_ENOUGH_INFO and were not executed. No fix or restoration is claimed. ### Typed outcome or recovery expected Preserve raw filename identity where supported, or explicitly report/refuse the unrepresentable entry and account for its bytes. Do not turn an inventory omission into an unqualified representation of the staging tree. ### Analysis or suspected cause Source-derived: [FirmwareFiles.ts:324–360](https://github.com/morluto/rea/blob/bc411c24befceced5a9ea1ca1c1ce123fd02bf48/src/firmware/FirmwareFiles.ts#L324) opens the directory using default UTF-8 decoding, then reconstructs a pathname from `entry.name`. Invalid byte FF becomes U+FFFD. `lstat` on that nonexistent replacement name returns `ENOENT`, which is silently skipped before byte accounting. This is independent of `DirectoryArtifactReader`. [Publication's reported-file check](https://github.com/morluto/rea/blob/bc411c24befceced5a9ea1ca1c1ce123fd02bf48/src/firmware/FirmwarePublication.ts#L127) can reject an omitted file named in the real producer report. The primary inventory omission and byte-accounting result do not depend on the synthetic publication facts. ### Additional context Related #6665 covers lossy filename substitution in the separate `DirectoryArtifactReader`/desktop service; fixing that traversal alone leaves this firmware implementation unchanged. #6601 concerns valid decomposed Unicode normalization after inventory, whereas this omission occurs before publication and involves invalid UTF-8. #3756 concerns an engine report with no recognized chunks. The supplied complete open-issue snapshot, matching bodies, and fresh all-state searches for `firmware UTF` and `firmware filename` revealed no matching firmware report. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Direct production-module probes in source and published 6.7.0 omit an existing staging file and undercount aggregate file bytes. This is bounded; real-extractor incidence and report-dependent downstream behavior remain unverified. - Workaround status: unverified - Workaround basis: No operational workaround was executed. A corrected implementation or explicit unsupported-name outcome remains to be verified.

bsakaguchi closed this issue

2026-10-31 00:46:51 +00:00

bsakaguchi commented

2026-10-31 00:46:52 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked resolved on main at d87e73f2db.

I ran the raw-filename regression on Linux x64. A filename containing raw FF beside the four-byte valid file, under maxBytes=4, now causes an explicit AnalysisOutputError with raw_name_hex: "626164ff2e62696e". Inventory cannot return the former successful four-byte subset. This verifies the report's permitted refusal outcome at the production staging-inventory boundary; no real Unblob extraction was involved.

Relevant implementation: FirmwareFiles.ts, FirmwareFiles.ts.

Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked resolved on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. I ran the raw-filename regression on Linux x64. A filename containing raw FF beside the four-byte valid file, under `maxBytes=4`, now causes an explicit `AnalysisOutputError` with `raw_name_hex: "626164ff2e62696e"`. Inventory cannot return the former successful four-byte subset. This verifies the report's permitted refusal outcome at the production staging-inventory boundary; no real Unblob extraction was involved. Relevant implementation: [FirmwareFiles.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/firmware/FirmwareFiles.ts#L380), [FirmwareFiles.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/firmware/FirmwareFiles.ts#L396). Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:11:30 +00:00

[PR #4097] [MERGED] feat(javascript): add filtered retained module reference views #6055

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4106

No description provided.

Rows
Columns