morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8551] [Bug] Virtual directory identities overwrite matching raw file nodes #6643

New issue

Open

opened 2026-11-01 21:12:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:54 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8551

Before submitting

I searched open and closed issues for related reports.
This report contains only a synthetic public fixture and no credentials or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An empty directory and a 34-byte regular file containing {"children":[],"kind":"directory"} receive the same artifact ID. Directory materialization overwrites the file's node with a zero-byte virtual directory. inspect-artifact exits 0 and returns this graph in completed inventory Evidence; both occurrences have hash_status: verified.

extract-artifact then exits 1 with:

code: artifact_operation_failed
category: integrity_mismatch
message: Artifact is invalid or has changed. Get a fresh copy and try again.
details.detail: Extracted content exceeds the inventoried size: payload.json

The original ZIP is valid and unchanged. Python's independent ZIP integrity check and extraction succeed. REA removes its failed extraction destination; no extraction residue was observed.

Reproduction

From the source checkout, create a ZIP with these exact bytes (no trailing newline):

import zipfile
with zipfile.ZipFile('collision.zip', 'w') as z:
    z.writestr('empty/', b'')
    z.writestr('payload.json', b'{"children":[],"kind":"directory"}')

I exercised the public source CLI using this wrapper, with the repository's pinned Node and installed dependencies:

// source-cli.mjs
import { createCli, closeCliResources } from './src/cli.ts';
try { await createCli(process.env).serve(process.argv.slice(2)); }
finally { await closeCliResources(); }

node --import tsx source-cli.mjs inspect-artifact collision.zip --json
node --import tsx source-cli.mjs extract-artifact collision.zip --json

The inventory is in normalized_result.substeps[0].evidence.normalized_result. I also exercised the inventory/extraction production APIs with ZIPs generated by the installed zip.js library, in both directory-first and file-first entry orders, and a real filesystem directory containing the same two children. All reproduced the identity alias; both ZIP orders failed extraction.

Controls actually run:

Fixture	Inventory and extraction
Same 34-byte file without the empty directory	Correct file node; extraction succeeds
Empty directory plus payload with one trailing newline	Distinct nodes; extraction succeeds
Two byte-identical 34-byte regular files	Correctly share a file node; both extract successfully

Expected behavior

A synthetic directory identity must remain distinct from raw file content, even when the file contains the exact JSON used to fingerprint that directory. The file occurrence must retain its actual size and content representation. Legitimate deduplication of identical regular files should continue to work.

Impact and scope

Observed: successful inventory Evidence contains an incorrect file-to-node association, and a valid ZIP cannot be extracted. The misleading repair instruction asks for a fresh copy of an unchanged valid artifact.

Sibling producer reproduced through the inventory API: an empty root directory and a 39-byte regular file containing {"children":[],"kind":"directory-root"} receive the same root artifact ID across separate inventories. Their node formats/sizes are directory/0 and file/39. No downstream cross-inventory merge or cache effect was tested.

The shared production inventory owner is used by CLI and MCP. Public CLI and production APIs were executed; stdio MCP, npm packaging, Windows, Linux, other archive formats, and all downstream projections were not exercised. Frequency in real-world inputs and last known good version: NOT_ENOUGH_INFO. This fixture deliberately contains a directory fingerprint's exact bytes.

REA version

Source aa48d494e9726d469a1b9ed917f021d8a1a4c7ea (package 6.8.0). The relevant identity, inventory, and extraction files remain unchanged at inspected remote main, df25bc8068b4d62c0e8db2b38648648ed831ae0e.

Runtime and platform

Node v24.22.1, macOS 26.3, arm64. No external analysis provider or target execution required.

Target and artifact context

Tiny synthetic ZIP and directory fixtures; a 34-byte JSON file and empty directory.

Interface

CLI; production inventory and extraction APIs for sibling and control cases.

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Both empty and payload.json reference:

{
  "artifact_id": "art_c2bc3646e4f623500d2db9f06ef77f546db4313754afbcfdac6dde619893a89c",
  "format": "directory",
  "sha256": "7d5ee5f4ffe6ba7d3f7e658bc7544c5f21e0f442b5aeded948533d44c090163c",
  "size": 0,
  "content_state": "virtual"
}

The file occurrence still reports entry_kind: file, artifact_format: file, declared_size: 34, and hash_status: verified. No inventory integrity contradiction identifies the overwritten node.

Analysis or suspected cause

Source-derived explanation of the executed failure:

Raw file admission creates a node from the file's SHA-256 and byte count.
Directory materialization and root construction hash canonical JSON and pass that digest through the same identity function; nodes.set overwrites the physical file node.
artifactIdForContent derives identity solely from {sha256}. The digest function's "Artifact" argument supplies diagnostic context; it is not hashed as a domain separator.
Extraction correctly verifies against the selected node's size, which inventory has replaced with zero.

Both digests hash the same byte sequence. Reproducing this requires no cryptographic hash collision. The identity contract needs to distinguish raw content from derived directory representations while preserving raw digest evidence and ordinary file deduplication. Check retained Evidence validation, manifest/occurrence identities, comparisons, examples, and compatibility when choosing that contract; no implementation change is included in this report.

Additional context

Related but distinct: #1234 addressed path-dependent classification of identical real files. This report concerns a real file and a synthetic directory representation, including wrong byte size and extraction failure. Both entry orders reproduce it.

Prior art researched through Context7 and DeepWiki, then verified against primary documentation: Git's loose object format includes object type and length in the hashed representation. Read-only local git hash-object --stdin and git hash-object -t tree --stdin, both given empty input, returned different blob/tree IDs. This is a precedent for separating identity domains, not a proposed Git format dependency.

Proposed regression checks: exercise the exact ZIP through CLI/MCP inventory and extraction; retain both ZIP orders, the filesystem directory and empty-root sibling, and the three working controls. Verify accurate file size/content state, byte-preserving extraction, and continued failure on genuinely changed bytes. No provider behavior was changed during this audit.

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8551 ### Before submitting - [x] I searched open and closed issues for related reports. - [x] This report contains only a synthetic public fixture and no credentials or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An empty directory and a 34-byte regular file containing `{"children":[],"kind":"directory"}` receive the same artifact ID. Directory materialization overwrites the file's node with a zero-byte virtual directory. `inspect-artifact` exits 0 and returns this graph in completed inventory Evidence; both occurrences have `hash_status: verified`. `extract-artifact` then exits 1 with: ```text code: artifact_operation_failed category: integrity_mismatch message: Artifact is invalid or has changed. Get a fresh copy and try again. details.detail: Extracted content exceeds the inventoried size: payload.json ``` The original ZIP is valid and unchanged. Python's independent ZIP integrity check and extraction succeed. REA removes its failed extraction destination; no extraction residue was observed. ### Reproduction From the source checkout, create a ZIP with these exact bytes (no trailing newline): ```python import zipfile with zipfile.ZipFile('collision.zip', 'w') as z: z.writestr('empty/', b'') z.writestr('payload.json', b'{"children":[],"kind":"directory"}') ``` I exercised the public source CLI using this wrapper, with the repository's pinned Node and installed dependencies: ```js // source-cli.mjs import { createCli, closeCliResources } from './src/cli.ts'; try { await createCli(process.env).serve(process.argv.slice(2)); } finally { await closeCliResources(); } ``` ```sh node --import tsx source-cli.mjs inspect-artifact collision.zip --json node --import tsx source-cli.mjs extract-artifact collision.zip --json ``` The inventory is in `normalized_result.substeps[0].evidence.normalized_result`. I also exercised the inventory/extraction production APIs with ZIPs generated by the installed zip.js library, in both directory-first and file-first entry orders, and a real filesystem directory containing the same two children. All reproduced the identity alias; both ZIP orders failed extraction. Controls actually run: | Fixture | Inventory and extraction | | --- | --- | | Same 34-byte file without the empty directory | Correct file node; extraction succeeds | | Empty directory plus payload with one trailing newline | Distinct nodes; extraction succeeds | | Two byte-identical 34-byte regular files | Correctly share a file node; both extract successfully | ### Expected behavior A synthetic directory identity must remain distinct from raw file content, even when the file contains the exact JSON used to fingerprint that directory. The file occurrence must retain its actual size and content representation. Legitimate deduplication of identical regular files should continue to work. ### Impact and scope Observed: successful inventory Evidence contains an incorrect file-to-node association, and a valid ZIP cannot be extracted. The misleading repair instruction asks for a fresh copy of an unchanged valid artifact. Sibling producer reproduced through the inventory API: an empty root directory and a 39-byte regular file containing `{"children":[],"kind":"directory-root"}` receive the same root artifact ID across separate inventories. Their node formats/sizes are `directory/0` and `file/39`. No downstream cross-inventory merge or cache effect was tested. The shared production inventory owner is used by CLI and MCP. Public CLI and production APIs were executed; stdio MCP, npm packaging, Windows, Linux, other archive formats, and all downstream projections were not exercised. Frequency in real-world inputs and last known good version: **NOT_ENOUGH_INFO**. This fixture deliberately contains a directory fingerprint's exact bytes. ### REA version Source `aa48d494e9726d469a1b9ed917f021d8a1a4c7ea` (package 6.8.0). The relevant identity, inventory, and extraction files remain unchanged at inspected remote `main`, `df25bc8068b4d62c0e8db2b38648648ed831ae0e`. ### Runtime and platform Node v24.22.1, macOS 26.3, arm64. No external analysis provider or target execution required. ### Target and artifact context Tiny synthetic ZIP and directory fixtures; a 34-byte JSON file and empty directory. ### Interface CLI; production inventory and extraction APIs for sibling and control cases. ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence Both `empty` and `payload.json` reference: ```json { "artifact_id": "art_c2bc3646e4f623500d2db9f06ef77f546db4313754afbcfdac6dde619893a89c", "format": "directory", "sha256": "7d5ee5f4ffe6ba7d3f7e658bc7544c5f21e0f442b5aeded948533d44c090163c", "size": 0, "content_state": "virtual" } ``` The file occurrence still reports `entry_kind: file`, `artifact_format: file`, `declared_size: 34`, and `hash_status: verified`. No inventory integrity contradiction identifies the overwritten node. ### Analysis or suspected cause Source-derived explanation of the executed failure: - [Raw file admission](https://github.com/morluto/rea/blob/aa48d494e9726d469a1b9ed917f021d8a1a4c7ea/src/artifacts/inventory/scanReader.ts#L419-L424) creates a node from the file's SHA-256 and byte count. - [Directory materialization and root construction](https://github.com/morluto/rea/blob/aa48d494e9726d469a1b9ed917f021d8a1a4c7ea/src/artifacts/inventory/ArtifactGraphConstruction.ts#L208-L260) hash canonical JSON and pass that digest through the same identity function; `nodes.set` overwrites the physical file node. - [artifactIdForContent](https://github.com/morluto/rea/blob/aa48d494e9726d469a1b9ed917f021d8a1a4c7ea/src/domain/artifactIdentity.ts#L36-L38) derives identity solely from `{sha256}`. The digest function's `"Artifact"` argument supplies diagnostic context; it is not hashed as a domain separator. - [Extraction](https://github.com/morluto/rea/blob/aa48d494e9726d469a1b9ed917f021d8a1a4c7ea/src/artifacts/extraction/ArtifactExtraction.ts#L475-L481) correctly verifies against the selected node's size, which inventory has replaced with zero. Both digests hash the same byte sequence. Reproducing this requires no cryptographic hash collision. The identity contract needs to distinguish raw content from derived directory representations while preserving raw digest evidence and ordinary file deduplication. Check retained Evidence validation, manifest/occurrence identities, comparisons, examples, and compatibility when choosing that contract; no implementation change is included in this report. ### Additional context Related but distinct: #1234 addressed path-dependent classification of identical real files. This report concerns a real file and a synthetic directory representation, including wrong byte size and extraction failure. Both entry orders reproduce it. Prior art researched through Context7 and DeepWiki, then verified against primary documentation: [Git's loose object format](https://github.com/git/git/blob/50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7/Documentation/gitformat-loose.adoc#L24-L45) includes object type and length in the hashed representation. Read-only local `git hash-object --stdin` and `git hash-object -t tree --stdin`, both given empty input, returned different blob/tree IDs. This is a precedent for separating identity domains, not a proposed Git format dependency. Proposed regression checks: exercise the exact ZIP through CLI/MCP inventory and extraction; retain both ZIP orders, the filesystem directory and empty-root sibling, and the three working controls. Verify accurate file size/content state, byte-preserving extraction, and continued failure on genuinely changed bytes. No provider behavior was changed during this audit.

bsakaguchi added the

bug

triage

labels

2026-11-01 21:12:54 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6643

No description provided.

Rows
Columns