[GH-ISSUE #3715] [RFC/Epic] Recursive Artifact Graph: multi-layer container traversal, slice provenance tracking and evidence preservation #1426

New issue

Open

opened 2026-10-16 21:05:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:20 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3715

Problem Statement & Architectural Context

In modern reverse engineering, binaries rarely exist in isolation. Real software is deeply layered: an installer wrapper (Inno Setup #3688 / NSIS #3689) unpacks a PE executable, which hosts a WebView2 control (#3684) or Electron runtime, which executes JavaScript bundles (#3608), which load WebAssembly modules, which communicate with an embedded SQLite or LevelDB store (#3691), which persists serialized models.

Currently, as tools unpack or slice artifacts, critical context is destroyed:

Loss of Offset & Spatial Provenance: Once an embedded payload is extracted to disk, its exact byte offset range, section location, and container alignment in the parent file are severed.
Loss of Transformation History: Information about intermediate decodings (Zlib decompression, XOR rotation, base64 decoding, AES decryption) is lost.
Fragmented Evidence: Findings in child artifacts cannot be traced back to the parent container.

REA needs an explicit Recursive Artifact Graph model that maintains parent-child links, cryptographic digests, byte slice coordinates, and transformation metadata throughout the entire investigation lifecycle.

Proposed Architecture

flowchart TD
    A["Root Artifact (Parent Container)"] --> B["Transformation Pipeline (Unpack / Decrypt / Decompress)"]
    B --> C["Child Artifact Slices"]
    
    C --> D["Recursive Traversal (Grandchild Slices)"]
    
    subgraph Artifact Graph Registry
        E["Node: Artifact ID, SHA256, Size, Mime"]
        F["Edge: Parent ID, Byte Range [Start, End], Transform Method"]
        G["Evidence Anchors: Cross-Layer Findings"]
    end
    
    A & B & C & D --- Artifact Graph Registry

Technical Capabilities & Modules

Artifact Node & Edge Data Model:
- Formalize ArtifactNode:
  - id: Unique deterministic hash (SHA-256 of contents + provenance path).
  - uri: Virtual or physical reference (artifact://root/slice_0x1000_0x4000).
  - size, entropy, mime_type, detected_role (executable, asset, configuration, database).
- Formalize ArtifactEdge:
  - parent_id, child_id.
  - parent_byte_range: [offset_start, offset_end].
  - transformation: Decoupled pipeline description (e.g. deflate, xor_key_0x55, aes_cbc).
Virtual Slicing & Lazy Materialization:
- Support zero-copy virtual slicing: reference sub-ranges of multi-gigabyte containers without dumping redundant duplicate files to disk.
- Materialize sub-slices on-demand when requested by deep domain analyzers.
Bi-Directional Coordinate Mapping:
- Given a function address or finding in a decompiled child artifact, translate its location back to the absolute byte offset within the root archive.
- Enable exact hex-editor highlighting and reproducible patch generation on the original root artifact.
Hierarchical Evidence Graph Propagation:
- Attach REA Evidence objects to specific graph nodes.
- Propagate security flags, indicators of compromise, and architectural discoveries up the provenance chain to summarize the root application's posture.
Graph Export & Visualization:
- Export artifact graphs in standard interchange formats (JSON-LD, DOT/Graphviz, Mermaid).
- Provide interactive visualization showing the breakdown of software packages into constituent layers.

Acceptance Criteria

Deterministic graph representation capturing complete provenance across multi-level extractions.
Preservation of exact parent byte offsets and transformation parameters for all child slices.
Bi-directional address translation between child findings and parent file offsets.
Unified integration with REA's existing Evidence system.
Scalable handling of deeply nested or multi-thousand file containers without memory degradation.

Complements #3621 (Unknown binary file-format inference)
Complements #740 (Roadmap: capability-module declarations and evidence propagation)
Complements #117 (Full-stack reconstruction)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3715 [![Type: Architecture / RFC](https://img.shields.io/badge/type-architecture_%2F_RFC-darkred.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Architectural Context In modern reverse engineering, binaries rarely exist in isolation. Real software is deeply layered: an installer wrapper (Inno Setup #3688 / NSIS #3689) unpacks a PE executable, which hosts a WebView2 control (#3684) or Electron runtime, which executes JavaScript bundles (#3608), which load WebAssembly modules, which communicate with an embedded SQLite or LevelDB store (#3691), which persists serialized models. Currently, as tools unpack or slice artifacts, critical context is destroyed: 1. **Loss of Offset & Spatial Provenance**: Once an embedded payload is extracted to disk, its exact byte offset range, section location, and container alignment in the parent file are severed. 2. **Loss of Transformation History**: Information about intermediate decodings (Zlib decompression, XOR rotation, base64 decoding, AES decryption) is lost. 3. **Fragmented Evidence**: Findings in child artifacts cannot be traced back to the parent container. REA needs an explicit **Recursive Artifact Graph** model that maintains parent-child links, cryptographic digests, byte slice coordinates, and transformation metadata throughout the entire investigation lifecycle. ### Proposed Architecture ```mermaid flowchart TD A["Root Artifact (Parent Container)"] --> B["Transformation Pipeline (Unpack / Decrypt / Decompress)"] B --> C["Child Artifact Slices"] C --> D["Recursive Traversal (Grandchild Slices)"] subgraph Artifact Graph Registry E["Node: Artifact ID, SHA256, Size, Mime"] F["Edge: Parent ID, Byte Range [Start, End], Transform Method"] G["Evidence Anchors: Cross-Layer Findings"] end A & B & C & D --- Artifact Graph Registry ``` ### Technical Capabilities & Modules 1. **Artifact Node & Edge Data Model:** - Formalize `ArtifactNode`: - `id`: Unique deterministic hash (SHA-256 of contents + provenance path). - `uri`: Virtual or physical reference (`artifact://root/slice_0x1000_0x4000`). - `size`, `entropy`, `mime_type`, `detected_role` (executable, asset, configuration, database). - Formalize `ArtifactEdge`: - `parent_id`, `child_id`. - `parent_byte_range`: `[offset_start, offset_end]`. - `transformation`: Decoupled pipeline description (e.g. `deflate`, `xor_key_0x55`, `aes_cbc`). 2. **Virtual Slicing & Lazy Materialization:** - Support zero-copy virtual slicing: reference sub-ranges of multi-gigabyte containers without dumping redundant duplicate files to disk. - Materialize sub-slices on-demand when requested by deep domain analyzers. 3. **Bi-Directional Coordinate Mapping:** - Given a function address or finding in a decompiled child artifact, translate its location back to the absolute byte offset within the root archive. - Enable exact hex-editor highlighting and reproducible patch generation on the original root artifact. 4. **Hierarchical Evidence Graph Propagation:** - Attach REA `Evidence` objects to specific graph nodes. - Propagate security flags, indicators of compromise, and architectural discoveries up the provenance chain to summarize the root application's posture. 5. **Graph Export & Visualization:** - Export artifact graphs in standard interchange formats (JSON-LD, DOT/Graphviz, Mermaid). - Provide interactive visualization showing the breakdown of software packages into constituent layers. ### Acceptance Criteria - [ ] Deterministic graph representation capturing complete provenance across multi-level extractions. - [ ] Preservation of exact parent byte offsets and transformation parameters for all child slices. - [ ] Bi-directional address translation between child findings and parent file offsets. - [ ] Unified integration with REA's existing Evidence system. - [ ] Scalable handling of deeply nested or multi-thousand file containers without memory degradation. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #740 (Roadmap: capability-module declarations and evidence propagation) - Complements #117 (Full-stack reconstruction)

bsakaguchi referenced this issue

2026-10-17 21:11:19 +00:00

[PR #1426] [MERGED] fix(hopper): treat a directory launcher path as missing #2878

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1426

No description provided.

Rows
Columns