[GH-ISSUE #3719] [RFC/Epic] Declarative Format DSL and Domain Extensibility SDK: schema pattern language, typed capability boundaries and isolated analyzer registration #1431

New issue

Open

opened 2026-10-16 21:05:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:22 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3719

Problem Statement & Architectural Context

To scale REA to hundreds of domain formats, virtual machines, architectures, and applications, the platform cannot require core code changes for every newly supported format. As outlined in RFC #740 and discussion #3137, REA benefits from maintaining a unified, high-integrity distribution while enabling clean, modular extensibility.

Currently, adding a format parser often requires touching multiple core files (analyzers, routing tables, evidence converters, and CLI options). REA needs:

Declarative Format Definition Language (DSL): A high-level declarative syntax (conceptually inspired by Kaitai Struct and ImHex Pattern Language) enabling developers to define structs, endianness, arrays, bitfields, and magic headers in concise markup.
Domain Module Extensibility SDK: Formal TypeScript contract allowing domain modules to declare their capabilities, input schemas, evidence contracts, and test fixtures in isolation.
Automated Conformance Testing Matrix: Test runner validating declared capabilities against positive, negative, and malformed fixtures without regressions.

Proposed Architecture

flowchart TD
    subgraph Declarative Definitions
        A1["Format DSL Spec (.rea-format / YAML / Kaitai)"]
        A2["Custom Domain Module (TypeScript / WASM)"]
    end

    A1 --> B["Declarative Schema Compiler"]
    B --> C["Dynamic Binary Parser & Serializer"]
    
    A2 --> D["Domain Module SDK Interface"]
    
    C & D --> E["Isolated Capability Registry"]
    E --> F["REA Core Runtime (Universal Router & Evidence Pipeline)"]
    
    G["Conformance Test Suite (Fixtures, Edge Cases, Fuzzing)"] --> E

Technical Capabilities & Modules

Declarative Binary Format Specification DSL:

Express binary schemas declaratively:

meta:
  id: custom_archive
  endian: le
seq:
  - id: magic
    contents: [0x50, 0x4B, 0x03, 0x04]
  - id: version
    type: u2
  - id: file_count
    type: u4
  - id: entries
    type: file_entry
    repeat: expr
    repeat-expr: file_count

Automatically compile declarative specs into streaming, zero-copy parsers integrated with REA Evidence.

Typed DomainModule Contract:
- Define a formal interface for domain plugins:
  - manifest: Module ID, version, author, supported mime-types, priority score.
  - detect(stream): Fast boolean / confidence heuristic.
  - analyze(stream, context): Structured analysis emitting typed nodes to the ArtifactGraph.
  - export(graph): Standard representations (JSON, Mermaid, Kaitai).
Isolated Sandboxing & Resource Budgets:
- Execute community domain modules within bounded runtime budgets (timeout, max memory allocations, maximum output artifact size).
- Prevent rogue or unoptimized parsers from compromising the stability of the core engine.
Automated Conformance Matrix Engine:
- Automated test harness that ingests declared test fixtures for every registered domain module:
  - Valid format fixtures (conformance check).
  - Truncated files (graceful EOF handling).
  - Corrupted headers (clean error reporting without unhandled exceptions).
- Generate an automated public capability matrix: Supported | Partial | Speculative | Unavailable.

Acceptance Criteria

Functional declarative format DSL capable of parsing standard headers, arrays, and pointer structures.
Formal DomainModule SDK interface allowing standalone registration without editing core routing tables.
Strict resource and memory enforcement on registered domain analyzers.
Conformance test framework verifying module resilience against malformed inputs.
Seamless emission of standard REA Evidence objects from DSL-defined parsers.

Complements #3621 (Unknown binary file-format inference)
Complements #740 (Roadmap: clarify and formalize capability-module declarations)
Complements #3137 (Extensibility ecosystem architecture)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3719 [![Type: Architecture / RFC](https://img.shields.io/badge/type-architecture_%2F_RFC-darkred.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Architectural Context To scale REA to hundreds of domain formats, virtual machines, architectures, and applications, the platform cannot require core code changes for every newly supported format. As outlined in RFC #740 and discussion #3137, REA benefits from maintaining a unified, high-integrity distribution while enabling clean, modular extensibility. Currently, adding a format parser often requires touching multiple core files (analyzers, routing tables, evidence converters, and CLI options). REA needs: 1. **Declarative Format Definition Language (DSL)**: A high-level declarative syntax (conceptually inspired by Kaitai Struct and ImHex Pattern Language) enabling developers to define structs, endianness, arrays, bitfields, and magic headers in concise markup. 2. **Domain Module Extensibility SDK**: Formal TypeScript contract allowing domain modules to declare their capabilities, input schemas, evidence contracts, and test fixtures in isolation. 3. **Automated Conformance Testing Matrix**: Test runner validating declared capabilities against positive, negative, and malformed fixtures without regressions. ### Proposed Architecture ```mermaid flowchart TD subgraph Declarative Definitions A1["Format DSL Spec (.rea-format / YAML / Kaitai)"] A2["Custom Domain Module (TypeScript / WASM)"] end A1 --> B["Declarative Schema Compiler"] B --> C["Dynamic Binary Parser & Serializer"] A2 --> D["Domain Module SDK Interface"] C & D --> E["Isolated Capability Registry"] E --> F["REA Core Runtime (Universal Router & Evidence Pipeline)"] G["Conformance Test Suite (Fixtures, Edge Cases, Fuzzing)"] --> E ``` ### Technical Capabilities & Modules 1. **Declarative Binary Format Specification DSL:** - Express binary schemas declaratively: ```yaml meta: id: custom_archive endian: le seq: - id: magic contents: [0x50, 0x4B, 0x03, 0x04] - id: version type: u2 - id: file_count type: u4 - id: entries type: file_entry repeat: expr repeat-expr: file_count ``` - Automatically compile declarative specs into streaming, zero-copy parsers integrated with REA Evidence. 2. **Typed `DomainModule` Contract:** - Define a formal interface for domain plugins: - `manifest`: Module ID, version, author, supported mime-types, priority score. - `detect(stream)`: Fast boolean / confidence heuristic. - `analyze(stream, context)`: Structured analysis emitting typed nodes to the `ArtifactGraph`. - `export(graph)`: Standard representations (JSON, Mermaid, Kaitai). 3. **Isolated Sandboxing & Resource Budgets:** - Execute community domain modules within bounded runtime budgets (timeout, max memory allocations, maximum output artifact size). - Prevent rogue or unoptimized parsers from compromising the stability of the core engine. 4. **Automated Conformance Matrix Engine:** - Automated test harness that ingests declared test fixtures for every registered domain module: - Valid format fixtures (conformance check). - Truncated files (graceful EOF handling). - Corrupted headers (clean error reporting without unhandled exceptions). - Generate an automated public capability matrix: `Supported | Partial | Speculative | Unavailable`. ### Acceptance Criteria - [ ] Functional declarative format DSL capable of parsing standard headers, arrays, and pointer structures. - [ ] Formal `DomainModule` SDK interface allowing standalone registration without editing core routing tables. - [ ] Strict resource and memory enforcement on registered domain analyzers. - [ ] Conformance test framework verifying module resilience against malformed inputs. - [ ] Seamless emission of standard REA Evidence objects from DSL-defined parsers. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #740 (Roadmap: clarify and formalize capability-module declarations) - Complements #3137 (Extensibility ecosystem architecture)

~~bsakaguchi referenced this issue 2026-10-16 21:05:48 +00:00~~

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-10-16 21:05:49 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-10-16 21:10:09 +00:00

[GH-ISSUE #4489] [Feature] Harmony graph: use pkgContextInfo.json isSO declarations for N-API candidates, and report inventoried entries outside every component category #1778

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue