morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #7237] feat(elf): decode FDO packaging and dlopen note metadata #7737

New issue

Open

opened 2026-11-03 13:40:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 13:40:01 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/7237
Author: @AniketR10
Created: 10/26/2026
Status: 🔄 Open

Base: main ← Head: feat/elf-fdo-notes

📝 Commits (4)

b8cd950 feat(elf): decode FDO packaging and dlopen note metadata
91f7791 fix(elf): key FDO note decoding by note type
47c7c61 fix(elf): bound FDO note decoding before parsing JSON
d83cf88 fix(elf): share one FDO decode allowance across the report

📊 Changes

10 files changed (+435 additions, -20 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+3 -2)
📝 README.md (+1 -1)
📝 docs/cli.md (+3 -2)
📝 docs/elf-notes.md (+41 -4)
📝 src/contracts/native/binaryDiagnosticsToolContracts.ts (+1 -1)
📝 src/domain/native/elfNotes.test.ts (+11 -5)
📝 src/domain/native/elfNotes.ts (+29 -2)
📝 src/native/elf/ElfNoteParser.test.ts (+200 -0)
📝 src/native/elf/ElfNoteParser.ts (+107 -3)
📝 tests/acceptance/elfNotesCli.test.ts (+39 -0)

📄 Description

Summary

Fixes #7235

inspect-elf-notes decoded only GNU notes. FDO notes, which declare a binary's package and its optional dlopen libraries, came back as type_name: "unknown" with a hash only, because their descriptors exceed the 64-byte hex preview. This PR names and decodes both FDO note kinds.

Problem and expected behavior

readelf -n /usr/bin/ls prints FDO_PACKAGING_METADATA with {"type":"rpm","name":"coreutils",…}. On main, REA reports the same note as owner: "FDO", type_name: "unknown", descriptor.hex: null, gnu: null.

Expected: the package and dlopen JSON defined by the UAPI package metadata and dlopen metadata specifications appears in the result. A malformed descriptor stays visible with a reason and does not fail the inspection.

Change and scope

elfNoteTypeName(owner, type) in src/domain/native/elfNotes.ts replaces the GNU-only gnuNoteTypeName. A type number is named only within its owner's namespace: FDO 0xcafe1a7e → fdo_packaging_metadata, FDO 0x407c0c0a → fdo_dlopen_metadata.
Entries gain fdo: { metadata, issue } | null next to gnu. metadata is the parsed JSON object (packaging) or array (dlopen) when the descriptor is NUL-terminated UTF-8 JSON followed only by NUL padding. Otherwise metadata is null and issue names the broken rule. JSON goes through the shared safeParseUtf8Json and ownedJsonValueSchema admission.
ElfNoteParser decodes each descriptor range once, reusing the result when PT_NOTE and SHT_NOTE both declare it, as it already does for gold-version scans. It charges the scan and each retained copy of the JSON text to the existing note text budget.
FDO decoding shares one 64 KiB allowance per report: each distinct descriptor's size counts once, in table order, and a range repeated by the other table reuses its decode. Notes past the allowance report it as an issue. A decoded JSON tree takes about 50 times its source bytes in heap, and real FDO notes are a few hundred bytes (from two Codex reviews).
When metadata is decoded, a limitation states that it is the producer's declaration, not verified against a package database or loaded libraries.

Not included: validating individual keys against the specifications, and detecting duplicate JSON names. The spec forbids duplicates; JSON.parse keeps the last value.

Contribution scope: not applicable. This is static parsing of bytes the existing operation already reads.

Contract and boundary impact

Semantic owner and earliest changed stage: ElfNoteParser produces the decoded FDO facts; the domain contract owns owner-scoped type naming. The superseded gnuNoteTypeName is removed. ElfNotesService validation is unchanged, and no other consumer reads note type names.
Contribution scope fit and linked maintainer decision: not applicable (existing read-only operation).
CLI and MCP/tool-catalog contract: additive. Two type_name enum values and a nullable fdo entry field; the tool description mentions FDO metadata.
Provider, bridge, target-format, or platform compatibility: none; portable TypeScript parser.
Evidence, artifact, provenance, or reconstruction contract: descriptor hash and location are unchanged; decoded metadata is labeled as a producer declaration.
Target selection, authority, upstream behavior, and cleanup details: not applicable.
Generated metadata, package, or installation impact: none committed; the skill reference was edited in .agents/skills/.

Expected effects:

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated:
- ElfNoteParser.test.ts: both note kinds across PT_NOTE and SHT_NOTE, each malformed case (no terminator, data after terminator, invalid UTF-8, invalid JSON, wrong top-level shape for each kind), unknown FDO types staying undecoded, and one descriptor range shared by a PT_NOTE packaging note and an SHT_NOTE dlopen note decoding per type (from Greptile review), and the shared 64 KiB decode allowance: its boundary, distinct notes sharing it, and a PT_NOTE/SHT_NOTE repeat charged once (from Codex reviews).
- elfNotes.test.ts: owner-scoped naming.
- elfNotesCli.test.ts: public CLI output validated against the published contract.
Base reproduction or other evidence (executed): a local sweep called parseElfNotes on 5,276 ELF files from /usr/bin and /usr/lib64 on Fedora 44 x86_64. It decoded 9,294 fdo_packaging_metadata and 166 fdo_dlopen_metadata entries with no issue. The same 5,276 files parsed before and after the change. The 129 files that failed before still fail, with the existing note-alignment and entry-budget errors, unchanged by this PR.

User-visible CLI output: rea inspect-elf-notes /usr/bin/ls --json now reports, for both tables:

{ "owner": "FDO", "type_name": "fdo_packaging_metadata",
  "fdo": { "metadata": { "type": "rpm", "name": "coreutils", "version": "9.10-5.fc44",
           "architecture": "x86_64", "osCpe": "cpe:/o:fedoraproject:fedora:44" }, "issue": null } }

A real dlopen note (libnss_myhostname.so.2) decodes to [{"feature":"idn","description":"Support for internationalized domain names","priority":"suggested","soname":["libidn2.so.0"]}].

Remaining proof gaps: only checked against Fedora-produced notes. No big-endian or ELF32 binaries carrying FDO notes were available, though the FDO decoder does not depend on byte order.

Boundary regressions: malformed input (each spec violation is kept, with a reason) and capacity (decoding shares the existing text budget and is cached per range). No lifecycle stages change.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run test:local -- src/native/elf/ElfNoteParser.test.ts src/domain/native/elfNotes.test.ts src/native/elf/ElfNotesProvider.test.ts src/application/binaryDiagnostics/ElfNotesService.test.ts: 67 passed on d83cf887c; the shared-descriptor regression failed on b8cd950a6
npm run test:focused -- tests/acceptance/elfNotesCli.test.ts tests/boundary/mcp/elfNotesMcp.test.ts tests/boundary/filesystem/elfNotes.test.ts: 15 passed on b8cd950a6, including MCP advertisement; the CLI and MCP files (13 tests) passed again on d83cf887c
npm run check:fast: typecheck and lint passed; it passed again as the pre-push hook on the pushed commit
npm run docs:check: passed (rerun on d83cf887c)
On b8cd950a6 the tests above ran before the pre-commit formatter (oxfmt) reformatted the staged files; the 91f779132 runs include that formatting.
Host ELF sweep repeated on 91f779132 (5,276 files: 9,324 packaging and 208 dlopen entries) on 47c7c6152 (5,268 files: 9,058 packaging and 208 dlopen entries), and on d83cf887c (5,341 files: 9,316 packaging and 208 dlopen entries). Every FDO entry decoded with no issue; the file sets differ by find ordering.
Heap check with dense [{},{},…] dlopen descriptors: on 91f779132 a 16 MiB descriptor added about 790 MiB of heap and took 5.3 s; on 47c7c6152 16 MiB and 64 MiB descriptors add about 0 MiB and finish in 44 ms and 134 ms. With max_entries: 256, 255 distinct 64 KiB notes (15.9 MiB file) added about 700 MiB of heap in 4.6 s on 47c7c6152, and about 0 MiB in 100 ms on d83cf887c.

Compatibility, safety, and release

Breaking changes or migration steps: none. Additive output; existing type_name values and fields are unchanged.
Real Hopper/Ghidra, browser, or OS coverage: not applicable. Real Fedora 44 ELF files were swept as described above.
Package or release metadata impact: none.
Security, privacy, process, or containment review: no execution or network access. JSON nesting is bounded by the shared JSON admission, decoding shares one 64 KiB allowance per report, and scans stay within the existing file and text budgets.

Author readiness

Human owner: @AniketR10

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/7237 **Author:** [@AniketR10](https://github.com/AniketR10) **Created:** 10/26/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/elf-fdo-notes` --- ### 📝 Commits (4) - [`b8cd950`](https://github.com/morluto/rea/commit/b8cd950a6f354325447747ea0746261645c86ca2) feat(elf): decode FDO packaging and dlopen note metadata - [`91f7791`](https://github.com/morluto/rea/commit/91f7791328fe1f53ee7a78661429560e3b9cb7d4) fix(elf): key FDO note decoding by note type - [`47c7c61`](https://github.com/morluto/rea/commit/47c7c61520eae1cb2c2dc735ea0896379b64d526) fix(elf): bound FDO note decoding before parsing JSON - [`d83cf88`](https://github.com/morluto/rea/commit/d83cf887c1de8e70203721021fcca99dad5eee76) fix(elf): share one FDO decode allowance across the report ### 📊 Changes **10 files changed** (+435 additions, -20 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+3 -2) 📝 `README.md` (+1 -1) 📝 `docs/cli.md` (+3 -2) 📝 `docs/elf-notes.md` (+41 -4) 📝 `src/contracts/native/binaryDiagnosticsToolContracts.ts` (+1 -1) 📝 `src/domain/native/elfNotes.test.ts` (+11 -5) 📝 `src/domain/native/elfNotes.ts` (+29 -2) 📝 `src/native/elf/ElfNoteParser.test.ts` (+200 -0) 📝 `src/native/elf/ElfNoteParser.ts` (+107 -3) 📝 `tests/acceptance/elfNotesCli.test.ts` (+39 -0) </details> ### 📄 Description ## Summary Fixes #7235 `inspect-elf-notes` decoded only GNU notes. FDO notes, which declare a binary's package and its optional `dlopen` libraries, came back as `type_name: "unknown"` with a hash only, because their descriptors exceed the 64-byte hex preview. This PR names and decodes both FDO note kinds. ## Problem and expected behavior `readelf -n /usr/bin/ls` prints `FDO_PACKAGING_METADATA` with `{"type":"rpm","name":"coreutils",…}`. On `main`, REA reports the same note as `owner: "FDO"`, `type_name: "unknown"`, `descriptor.hex: null`, `gnu: null`. Expected: the package and dlopen JSON defined by the UAPI [package metadata](https://uapi-group.org/specifications/specs/package_metadata_for_executable_files/) and [dlopen metadata](https://uapi-group.org/specifications/specs/elf_dlopen_metadata/) specifications appears in the result. A malformed descriptor stays visible with a reason and does not fail the inspection. ## Change and scope - `elfNoteTypeName(owner, type)` in `src/domain/native/elfNotes.ts` replaces the GNU-only `gnuNoteTypeName`. A type number is named only within its owner's namespace: FDO `0xcafe1a7e` → `fdo_packaging_metadata`, FDO `0x407c0c0a` → `fdo_dlopen_metadata`. - Entries gain `fdo: { metadata, issue } | null` next to `gnu`. `metadata` is the parsed JSON object (packaging) or array (dlopen) when the descriptor is NUL-terminated UTF-8 JSON followed only by NUL padding. Otherwise `metadata` is null and `issue` names the broken rule. JSON goes through the shared `safeParseUtf8Json` and `ownedJsonValueSchema` admission. - `ElfNoteParser` decodes each descriptor range once, reusing the result when PT_NOTE and SHT_NOTE both declare it, as it already does for gold-version scans. It charges the scan and each retained copy of the JSON text to the existing note text budget. - FDO decoding shares one 64 KiB allowance per report: each distinct descriptor's size counts once, in table order, and a range repeated by the other table reuses its decode. Notes past the allowance report it as an `issue`. A decoded JSON tree takes about 50 times its source bytes in heap, and real FDO notes are a few hundred bytes (from two Codex reviews). - When metadata is decoded, a limitation states that it is the producer's declaration, not verified against a package database or loaded libraries. Not included: validating individual keys against the specifications, and detecting duplicate JSON names. The spec forbids duplicates; `JSON.parse` keeps the last value. Contribution scope: not applicable. This is static parsing of bytes the existing operation already reads. ## Contract and boundary impact - Semantic owner and earliest changed stage: `ElfNoteParser` produces the decoded FDO facts; the domain contract owns owner-scoped type naming. The superseded `gnuNoteTypeName` is removed. `ElfNotesService` validation is unchanged, and no other consumer reads note type names. - Contribution scope fit and linked maintainer decision: not applicable (existing read-only operation). - CLI and MCP/tool-catalog contract: additive. Two `type_name` enum values and a nullable `fdo` entry field; the tool description mentions FDO metadata. - Provider, bridge, target-format, or platform compatibility: none; portable TypeScript parser. - Evidence, artifact, provenance, or reconstruction contract: descriptor hash and location are unchanged; decoded metadata is labeled as a producer declaration. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata, package, or installation impact: none committed; the skill reference was edited in `.agents/skills/`. Expected effects: - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `ElfNoteParser.test.ts`: both note kinds across PT_NOTE and SHT_NOTE, each malformed case (no terminator, data after terminator, invalid UTF-8, invalid JSON, wrong top-level shape for each kind), unknown FDO types staying undecoded, and one descriptor range shared by a PT_NOTE packaging note and an SHT_NOTE dlopen note decoding per type (from Greptile review), and the shared 64 KiB decode allowance: its boundary, distinct notes sharing it, and a PT_NOTE/SHT_NOTE repeat charged once (from Codex reviews). - `elfNotes.test.ts`: owner-scoped naming. - `elfNotesCli.test.ts`: public CLI output validated against the published contract. - Base reproduction or other evidence (executed): a local sweep called `parseElfNotes` on 5,276 ELF files from `/usr/bin` and `/usr/lib64` on Fedora 44 x86_64. It decoded 9,294 `fdo_packaging_metadata` and 166 `fdo_dlopen_metadata` entries with no `issue`. The same 5,276 files parsed before and after the change. The 129 files that failed before still fail, with the existing note-alignment and entry-budget errors, unchanged by this PR. - User-visible CLI output: `rea inspect-elf-notes /usr/bin/ls --json` now reports, for both tables: ```json { "owner": "FDO", "type_name": "fdo_packaging_metadata", "fdo": { "metadata": { "type": "rpm", "name": "coreutils", "version": "9.10-5.fc44", "architecture": "x86_64", "osCpe": "cpe:/o:fedoraproject:fedora:44" }, "issue": null } } ``` A real dlopen note (`libnss_myhostname.so.2`) decodes to `[{"feature":"idn","description":"Support for internationalized domain names","priority":"suggested","soname":["libidn2.so.0"]}]`. - Remaining proof gaps: only checked against Fedora-produced notes. No big-endian or ELF32 binaries carrying FDO notes were available, though the FDO decoder does not depend on byte order. Boundary regressions: malformed input (each spec violation is kept, with a reason) and capacity (decoding shares the existing text budget and is cached per range). No lifecycle stages change. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:local -- src/native/elf/ElfNoteParser.test.ts src/domain/native/elfNotes.test.ts src/native/elf/ElfNotesProvider.test.ts src/application/binaryDiagnostics/ElfNotesService.test.ts`: 67 passed on d83cf887c; the shared-descriptor regression failed on b8cd950a6 - `npm run test:focused -- tests/acceptance/elfNotesCli.test.ts tests/boundary/mcp/elfNotesMcp.test.ts tests/boundary/filesystem/elfNotes.test.ts`: 15 passed on b8cd950a6, including MCP advertisement; the CLI and MCP files (13 tests) passed again on d83cf887c - `npm run check:fast`: typecheck and lint passed; it passed again as the pre-push hook on the pushed commit - `npm run docs:check`: passed (rerun on d83cf887c) - On b8cd950a6 the tests above ran before the pre-commit formatter (`oxfmt`) reformatted the staged files; the 91f779132 runs include that formatting. - Host ELF sweep repeated on 91f779132 (5,276 files: 9,324 packaging and 208 dlopen entries) on 47c7c6152 (5,268 files: 9,058 packaging and 208 dlopen entries), and on d83cf887c (5,341 files: 9,316 packaging and 208 dlopen entries). Every FDO entry decoded with no `issue`; the file sets differ by `find` ordering. - Heap check with dense `[{},{},…]` dlopen descriptors: on 91f779132 a 16 MiB descriptor added about 790 MiB of heap and took 5.3 s; on 47c7c6152 16 MiB and 64 MiB descriptors add about 0 MiB and finish in 44 ms and 134 ms. With `max_entries: 256`, 255 distinct 64 KiB notes (15.9 MiB file) added about 700 MiB of heap in 4.6 s on 47c7c6152, and about 0 MiB in 100 ms on d83cf887c. ## Compatibility, safety, and release - Breaking changes or migration steps: none. Additive output; existing `type_name` values and fields are unchanged. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. Real Fedora 44 ELF files were swept as described above. - Package or release metadata impact: none. - Security, privacy, process, or containment review: no execution or network access. JSON nesting is bounded by the shared JSON admission, decoding shares one 64 KiB allowance per report, and scans stay within the existing file and text budgets. ## Author readiness Human owner: @AniketR10 - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-11-03 13:40:01 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7737

No description provided.

Rows
Columns