morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #7136] [MERGED] feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions #7666

New issue

Closed

opened 2026-11-03 13:39:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 13:39:29 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/7136
Author: @555aaditya
Created: 10/25/2026
Status: ✅ Merged
Merged: 10/28/2026
Merged by: @gensokyo-dev

Base: main ← Head: feat/hex-records

📝 Commits (5)

f3f3704 feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions
74e9646 fix(artifacts): refuse Intel HEX extended address records with a non-zero address
76dadc1 Merge remote-tracking branch 'origin/main' into feat/hex-records
7d9d854 Merge remote-tracking branch 'origin/main' into fix/7129-hex-record-followup
ffba76f fix(artifacts): bound HEX region copies and retain stream cancellation

📊 Changes

28 files changed (+2380 additions, -10 deletions)

View changed files

📝 docs/cli.md (+10 -0)
📝 docs/firmware-analysis.md (+8 -0)
➕ docs/hex-records.md (+129 -0)
📝 src/application/BinaryTargetResolver.ts (+4 -1)
📝 src/artifacts/ArtifactReader.ts (+2 -0)
➕ src/artifacts/HexRecordArtifactReader.test.ts (+382 -0)
➕ src/artifacts/HexRecordArtifactReader.ts (+267 -0)
📝 src/artifacts/createArtifactReader.ts (+6 -0)
📝 src/artifacts/extraction/ArtifactExtraction.ts (+2 -0)
📝 src/artifacts/inventory/ArtifactGraphConstruction.test.ts (+21 -0)
📝 src/artifacts/inventory/ArtifactGraphConstruction.ts (+15 -4)
📝 src/artifacts/inventory/classify.ts (+3 -1)
📝 src/artifacts/inventory/reader.ts (+8 -1)
📝 src/domain/analysisSnapshot.ts (+2 -0)
📝 src/domain/analysisView/artifactInventoryView.ts (+3 -1)
📝 src/domain/artifactFormatMetadata.ts (+47 -0)
📝 src/domain/artifactGraph.ts (+4 -0)
📝 src/domain/artifactInspection.ts (+13 -0)
📝 src/domain/binaryTargetTypes.ts (+3 -1)
📝 src/domain/evidenceSubject.ts (+2 -0)

...and 8 more files

📄 Description

Summary

Refs #7129; implements its Intel HEX / Motorola S-record slice only. U-Boot uImage and FIT remain separate. This overlaps the independently developed #7368.

inspect-artifact and extract-artifact, and the equivalent MCP tools, now recognize valid Intel HEX/S-record content and expose address-contiguous decoded regions. Metadata preserves declared addresses, record/byte counts, duplicate bytes, line provenance, termination and explicit limitations; gaps remain unwritten. It does not infer CPU, byte order, a memory map or executable entry points. Missing termination reports unknown coverage, and malformed/conflicting records fail without partial regions.

Ownership and follow-up changes

Retains main's shared createArtifactReader factory for inventory and extraction, rather than restoring old per-consumer switches.
Extends the current Evidence-subject and live-target schema owners; removes the superseded inline schemas from the merge conflict.
Fixes two demonstrated reader defects: cancellation after open() was ignored, and every open copied the complete decoded region. Streams now copy cached bytes in 64 KiB blocks, retain exact cancellation reasons and remain tracked until closed. One cancelled consumer does not cancel a valid peer; consumers cannot mutate cached bytes. The admitted carrier identity is checked on each open; decoded data live only until reader close.

Scope: ordinary local format inspection/extraction under the existing artifact authority, not a new provider or target execution capability. Extraction writes caller-selected outputs using the existing containment/cleanup owner. No execution, instrumentation, network access or dependency changes are added by this PR.

Validation performed

Linux; Node 24.22.1 / npm 11.16.0, with the latest locked dependencies installed in this worktree and at most two test workers.

Final focused parser, reader, graph construction, target boundary, public CLI and full/compact MCP suites: 183 passed, 1 pre-existing skipped. The MCP fixture validates converted advertised schemas and actual results.
Added regressions on the pre-fix reader: 2 failed / 11 passed. An already-open stream ignored cancellation; a 128 KiB region arrived as one 128 KiB copy. After the fix, 13/13 reader tests passed, including a later strengthening that checks partial-read cancellation, a continuing peer and closed streams after reader cleanup.
npm run check:fast: passed on final source; pre-push passed against the same inputs.
npm run docs:check: passed, including 9 README consistency checks.
Bounded GOMEMLIMIT=2GiB timeout --kill-after=5s 90s oxlint --type-aware --threads 1 on seven affected reader/parser/schema files: passed, warnings only. Whole-repository typed lint was not rerun after its excessive resource cost in the preceding batch item.
Formatting check across all 33 changed files and git diff --check: passed.

Real-producer correctness

GNU objcopy 2.40 converted the real system true ELF into Intel HEX, S-record and address-rebased I32HEX/S37 variants. All four produced 13 regions, byte-exact against GNU BFD addresses and independently exported section bytes. The unrebased HEX also round-tripped to the same raw binary as a direct ELF-to-binary conversion.

A 19,124,817-byte HEX conversion of the system Python 3.11 ELF produced 11 regions / 6,799,261 decoded bytes. Every extracted byte matched GNU section exports. GNU splits sections at 64 KiB address boundaries; the initial verification script incorrectly expected those sections to equal REA regions. Joining strictly adjacent native sections, without filling gaps, resolved the oracle mismatch. This was an oracle correction, not a decoder fix.

Source ELF SHA-256: true c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2; Python 9bee109da0dce17a7c9eeaca9f420cc6770a9fe143b9382d73bd22fe59b21a5f. These are real producer outputs from system applications, not vendor-firmware, srec_cat, target-execution or deep-provider verification.

Completed-task efficiency

Compared integrated pre-fix reader 7d9d854d0253419177294732a71aa85bc56e2fea with the follow-up: public CLI inspect + extract + independent output hash/cleanup checks, alternating three fresh-process tasks per lane and size. File cache state was uncontrolled. CPU includes child processes plus the Python profiler/checks; process-family RSS includes the profiler and CLI, sampled every 10 ms. Native oracle generation was done before timing. All tasks completed with identical regions, coverage, byte hashes and verified cleanup receipts.

Values are initial / mean of two subsequent tasks:

Input / lane	Elapsed seconds	CPU seconds	Sampled family peak MiB
true HEX before	6.845 / 5.520	8.041 / 6.757	260.2 / 283.1
true HEX after	7.098 / 5.235	8.513 / 7.081	260.8 / 280.5
Python HEX before	9.746 / 10.252	12.683 / 12.198	518.0 / 405.5
Python HEX after	8.998 / 9.704	11.229 / 11.124	357.9 / 356.8

The larger case showed lower sampled memory; small-case memory overlapped and host timing varied. No general speedup is claimed. Complete two-command replies were unchanged at ~8,047 / 7,228 estimated tokens for small/large input, respectively (tokenx, not model usage). Commands and schemas are unchanged between these reader lanes; initial MCP catalog/discovery and whole-model usage were not measured. There is no token-saving claim. Existing summary and retained-Evidence views remain available.

Expected effects:

Read existing files or captured data.
Change files through existing explicit artifact extraction.

Author readiness

Human owner: @555aaditya

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/7136 **Author:** [@555aaditya](https://github.com/555aaditya) **Created:** 10/25/2026 **Status:** ✅ Merged **Merged:** 10/28/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `feat/hex-records` --- ### 📝 Commits (5) - [`f3f3704`](https://github.com/morluto/rea/commit/f3f3704955e6efbf6c55b6f7cd54be55d87a7e31) feat(artifacts): inventory and extract Intel HEX and Motorola S-record regions - [`74e9646`](https://github.com/morluto/rea/commit/74e9646f1eff0ad26548b1e3f06f0ad1d56292fb) fix(artifacts): refuse Intel HEX extended address records with a non-zero address - [`76dadc1`](https://github.com/morluto/rea/commit/76dadc1ec7a0560a5425c2db57f8f3428c8a66b9) Merge remote-tracking branch 'origin/main' into feat/hex-records - [`7d9d854`](https://github.com/morluto/rea/commit/7d9d854d0253419177294732a71aa85bc56e2fea) Merge remote-tracking branch 'origin/main' into fix/7129-hex-record-followup - [`ffba76f`](https://github.com/morluto/rea/commit/ffba76f6f0a0260b10278d285665ea68f67150aa) fix(artifacts): bound HEX region copies and retain stream cancellation ### 📊 Changes **28 files changed** (+2380 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `docs/cli.md` (+10 -0) 📝 `docs/firmware-analysis.md` (+8 -0) ➕ `docs/hex-records.md` (+129 -0) 📝 `src/application/BinaryTargetResolver.ts` (+4 -1) 📝 `src/artifacts/ArtifactReader.ts` (+2 -0) ➕ `src/artifacts/HexRecordArtifactReader.test.ts` (+382 -0) ➕ `src/artifacts/HexRecordArtifactReader.ts` (+267 -0) 📝 `src/artifacts/createArtifactReader.ts` (+6 -0) 📝 `src/artifacts/extraction/ArtifactExtraction.ts` (+2 -0) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.test.ts` (+21 -0) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.ts` (+15 -4) 📝 `src/artifacts/inventory/classify.ts` (+3 -1) 📝 `src/artifacts/inventory/reader.ts` (+8 -1) 📝 `src/domain/analysisSnapshot.ts` (+2 -0) 📝 `src/domain/analysisView/artifactInventoryView.ts` (+3 -1) 📝 `src/domain/artifactFormatMetadata.ts` (+47 -0) 📝 `src/domain/artifactGraph.ts` (+4 -0) 📝 `src/domain/artifactInspection.ts` (+13 -0) 📝 `src/domain/binaryTargetTypes.ts` (+3 -1) 📝 `src/domain/evidenceSubject.ts` (+2 -0) _...and 8 more files_ </details> ### 📄 Description ## Summary Refs #7129; implements its Intel HEX / Motorola S-record slice only. U-Boot uImage and FIT remain separate. This overlaps the independently developed #7368. `inspect-artifact` and `extract-artifact`, and the equivalent MCP tools, now recognize valid Intel HEX/S-record content and expose address-contiguous decoded regions. Metadata preserves declared addresses, record/byte counts, duplicate bytes, line provenance, termination and explicit limitations; gaps remain unwritten. It does not infer CPU, byte order, a memory map or executable entry points. Missing termination reports unknown coverage, and malformed/conflicting records fail without partial regions. ## Ownership and follow-up changes - Retains main's shared `createArtifactReader` factory for inventory and extraction, rather than restoring old per-consumer switches. - Extends the current Evidence-subject and live-target schema owners; removes the superseded inline schemas from the merge conflict. - Fixes two demonstrated reader defects: cancellation after `open()` was ignored, and every open copied the complete decoded region. Streams now copy cached bytes in 64 KiB blocks, retain exact cancellation reasons and remain tracked until closed. One cancelled consumer does not cancel a valid peer; consumers cannot mutate cached bytes. The admitted carrier identity is checked on each open; decoded data live only until reader close. Scope: ordinary local format inspection/extraction under the existing artifact authority, not a new provider or target execution capability. Extraction writes caller-selected outputs using the existing containment/cleanup owner. No execution, instrumentation, network access or dependency changes are added by this PR. ## Validation performed Linux; Node 24.22.1 / npm 11.16.0, with the latest locked dependencies installed in this worktree and at most two test workers. - Final focused parser, reader, graph construction, target boundary, public CLI and full/compact MCP suites: **183 passed, 1 pre-existing skipped**. The MCP fixture validates converted advertised schemas and actual results. - Added regressions on the pre-fix reader: **2 failed / 11 passed**. An already-open stream ignored cancellation; a 128 KiB region arrived as one 128 KiB copy. After the fix, **13/13 reader tests passed**, including a later strengthening that checks partial-read cancellation, a continuing peer and closed streams after reader cleanup. - `npm run check:fast`: passed on final source; pre-push passed against the same inputs. - `npm run docs:check`: passed, including 9 README consistency checks. - Bounded `GOMEMLIMIT=2GiB timeout --kill-after=5s 90s oxlint --type-aware --threads 1` on seven affected reader/parser/schema files: passed, warnings only. Whole-repository typed lint was not rerun after its excessive resource cost in the preceding batch item. - Formatting check across all 33 changed files and `git diff --check`: passed. ## Real-producer correctness GNU objcopy **2.40** converted the real system `true` ELF into Intel HEX, S-record and address-rebased I32HEX/S37 variants. All four produced **13 regions**, byte-exact against GNU BFD addresses and independently exported section bytes. The unrebased HEX also round-tripped to the same raw binary as a direct ELF-to-binary conversion. A **19,124,817-byte** HEX conversion of the system Python 3.11 ELF produced **11 regions / 6,799,261 decoded bytes**. Every extracted byte matched GNU section exports. GNU splits sections at 64 KiB address boundaries; the initial verification script incorrectly expected those sections to equal REA regions. Joining strictly adjacent native sections, without filling gaps, resolved the oracle mismatch. This was an oracle correction, not a decoder fix. Source ELF SHA-256: `true` `c79bf44242829108e323378531f4ac839513ca1fba45efd6583643526e1e9fd2`; Python `9bee109da0dce17a7c9eeaca9f420cc6770a9fe143b9382d73bd22fe59b21a5f`. These are real producer outputs from system applications, not vendor-firmware, srec_cat, target-execution or deep-provider verification. ## Completed-task efficiency Compared integrated pre-fix reader `7d9d854d0253419177294732a71aa85bc56e2fea` with the follow-up: public CLI **inspect + extract + independent output hash/cleanup checks**, alternating three fresh-process tasks per lane and size. File cache state was uncontrolled. CPU includes child processes plus the Python profiler/checks; process-family RSS includes the profiler and CLI, sampled every 10 ms. Native oracle generation was done before timing. All tasks completed with identical regions, coverage, byte hashes and verified cleanup receipts. Values are initial / mean of two subsequent tasks: | Input / lane | Elapsed seconds | CPU seconds | Sampled family peak MiB | | --- | --- | --- | --- | | true HEX before | 6.845 / 5.520 | 8.041 / 6.757 | 260.2 / 283.1 | | true HEX after | 7.098 / 5.235 | 8.513 / 7.081 | 260.8 / 280.5 | | Python HEX before | 9.746 / 10.252 | 12.683 / 12.198 | 518.0 / 405.5 | | Python HEX after | 8.998 / 9.704 | 11.229 / 11.124 | 357.9 / 356.8 | The larger case showed lower sampled memory; small-case memory overlapped and host timing varied. No general speedup is claimed. Complete two-command replies were unchanged at ~**8,047 / 7,228 estimated tokens** for small/large input, respectively (`tokenx`, not model usage). Commands and schemas are unchanged between these reader lanes; initial MCP catalog/discovery and whole-model usage were not measured. There is no token-saving claim. Existing summary and retained-Evidence views remain available. Expected effects: - [x] Read existing files or captured data. - [x] Change files through existing explicit artifact extraction. ## Author readiness  Human owner: @555aaditya - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-03 13:39:29 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7666

No description provided.

Rows
Columns