morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7189] [Feature] Static library-function identification (FLIRT-like) for native targets #4297

New issue

Open

opened 2026-10-31 00:48:39 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-31 00:48:39 +00:00

Owner

Originally created by @reynold-hu on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7189

[Feature] Static library-function identification (FLIRT-like) for native targets

Problem

An agent analyzing a stripped native binary sees hundreds of unnamed
functions (sub_XXXX) and has no provider-neutral way to ask: "is this
function a known library function, and if so, which one?" Every downstream
answer pays for that gap: reconstruction wastes tokens re-deriving memcpy,
diffing compares library code as if it were the target's own code, and
capability inventories risk attributing library behavior to the program
under analysis. Ghidra, Hopper, and IDA each have their own library
identification mechanisms, but REA exposes no common typed identification result
across providers. As the controls below show, REA can surface Ghidra analyzer-generated
names and comments from a saved project; those fields alone do not authenticate a
match or identify its database.

The original proposal reported a source-only search at filing time: issue search in
morluto/rea returned 0 results for "FLIRT", "library function
identification", "library identification", "source sink", and "known
function"; the tree contains no identifyFunction, identify_library,
knownFunction, or libraryFunction implementation and no "flirt" reference
outside unrelated test fixtures. Related but distinct capabilities already
exist and are not this proposal: NativeValueTrace (a procedure-seeded
def-use dependency graph, not function identification), the managed/JVM/
Python bytecode providers (they parse metadata, they do not match native
bytes), and the WASM lane (a different target family).

Proposed scope (Phase 1 only)

One pure, static capability — no execution, no host state, no new
dependency — built as a domain module with tests:

Signature format: a signature names one known library function
(id, name, library, optional library_version) and carries a byte
pattern for the start of that function: whitespace-separated tokens, each
two hex digits or ?? for a wildcard byte (an address or relocation slot
that varies between builds). Patterns are capped at 1024 tokens; a
pattern with no concrete byte is rejected, because an all-wildcard
pattern would match every function.
Matcher: a pattern matches a function only when anchored at offset 0
of the function bytes. This mirrors FLIRT's first-bytes approach and keeps
a match a claim about function identity, not about a byte string occurring
somewhere inside. A match carries score (non-wildcard bytes over pattern
length) and confidence (exact when no wildcard is present, otherwise
pattern). Matches rank by score, then pattern length, then signature id,
so results are deterministic.
Budgets with explicit unknowns: per-function byte cap, per-run
function cap, and signature-set cap. Functions beyond a budget are NOT
silently dropped: they appear in unknowns with a reason
(function_budget_exceeded or function_bytes_budget_exceeded) while the
rest of the batch is still identified. An invalid signature set (duplicate
ids, over-cap, malformed pattern) is a caller configuration error and
throws, like every other zod schema validation in this codebase; it is
never silently truncated.
Evidence mapping (proposed): confidence: "inferred",
authority: "historical-reference", no execution environment. A match is
a pattern claim against a reference corpus, never an observed fact and
never proof of provenance.

Result model

One run returns, per function:

function_id and its matches: each with signature_id, name,
library, library_version, pattern_length, wildcard_count,
score, and confidence;
unknowns: functions skipped by a budget, each with a reason.

No MCP tool, provider registration, or composition wiring is included in
Phase 1. That wiring is integration surface and belongs to the PR phase,
after a maintainer decision (see below).

Limitations (Phase 1)

No signature corpus ships with Phase 1. The format and the matcher are
built and tested; a real corpus (derived from public libraries, with
provenance and versioning) is a later phase. Stated plainly: until a
corpus exists, the matcher identifies nothing in production. Shipping a
matcher with a fabricated corpus would be worse than shipping no corpus.
x86_64 only. Other architectures need their own signature sets and
verification; they are not claimed here.
Anchored-at-0 only: no mid-function patterns, no fuzzy or similarity
matching, no version disambiguation beyond the signature's own metadata.
Two library versions sharing a prologue will both match; the caller must
weigh scores, not treat the top match as a verdict.
A match is an inference, not proof of provenance. It must never be
reported as "this function IS memcpy", only as "this function matches the
memcpy signature with score S".

Non-goals

Not a signature database, and not a scraper for proprietary signature
sets (IDA FLIRT signatures are proprietary; deriving a corpus must use
public libraries with a license review).
Not function similarity: no fuzzy hashing (TLSH, ssdeep), no embedding
comparison, no "similar to" ranking. Those are separate capabilities with
their own false-positive profiles.
Not a verdict and not an exclusion rule: Phase 1 does not decide that
matched functions are removed from analysis. Whether identification
feeds an exclusion filter for reconstruction or #4972's candidate set is
a consumer decision for a later phase.
No packer/unpacking workflow, no anti-analysis evasion, no DRM/license/
anti-cheat bypass, no exploit automation.
No execution of any kind: this proposal is static identification, so the
target-execution scope gate does not apply to it — but integration still
needs a maintainer decision (below).

Dependency and licensing

Zero new dependencies: zod is already in the tree, and the matcher is
pure TypeScript with no engine, no subprocess, and no host capability.
The later-phase corpus question needs a license review before any
corpus ships: signatures derived from public libraries must carry
provenance (library, version, derivation method), and proprietary
signature sets must not be copied. No corpus is proposed in Phase 1, so
no license question blocks the matcher itself.

Relationship to existing work

#4972 (semantic-equivalence verification): complementary, not
dependent. Identification shrinks #4972's problem: library functions can
be labeled before differential analysis, so the harness spends its budget
on the target's own code. It implements no part of #4972's verdict model.
NativeValueTrace: complementary. The trace builds def-use graphs over
procedures; identification labels the procedures the trace traverses. A
later phase could join the two (e.g., "trace values through this
identified memcpy"), but Phase 1 does not.
Managed/JVM/Python bytecode providers: different mechanism for a
different target family. Those parse metadata; this matches native bytes.
No overlap.
#7142 (Unicorn emulation provider): independent. Identification is
static and needs no scope decision about execution; emulation is dynamic
and awaits its own decision. Neither blocks the other.

Open questions for the maintainer

Placement: domain service consumed by providers, deep provider, or an
auxiliary operation set? Phase 1 is a domain module so the matcher can
be reviewed without committing to wiring; the choice affects
src/composition/binary.ts and the tool catalog.
Operation naming: is identify_library_functions acceptable, or should
it live under a different operation family?
Corpus strategy: should REA ship a small corpus derived from public
libraries, derive signatures at setup time from a user-supplied library
set, or accept caller-supplied signatures only? Each has a different
licensing and maintenance profile.
Minimum concrete bytes: should wildcard-heavy patterns require a floor
(e.g., at least N concrete bytes) beyond the current "at least one"
rule, to bound false positives?
Version disambiguation: when two versions of one function share a
prologue, should the result carry both matches (current behavior) or
should a later phase add a disambiguation pass?

Scope decision request

Per CONTRIBUTING.md ("Fail-closed integration"), this issue requests an
explicit maintainer scope decision on the Phase 1 behavior above before
integration work (MCP tool, provider registration, composition wiring, or
any corpus). A local prototype with tests exists to inform that decision;
I understand neither the prototype nor a draft PR authorizes integration,
and that a scope expansion needs a new decision.

Additional direct-provider observation

On macOS arm64, I compiled a self-owned x86-64 ELF object with three functions and linked a stripped executable from the same object without running it. Direct Kuna 1.986 fid build created a three-record .fid database from the object. With --mode reliable --option listing on --option fid on but no kuna_fid_db, direct kuna functions returned three sub_... names. Setting kuna_fid_db to that database renamed two entries to audit_mix and _start; audit_checksum remained a placeholder. This verifies a narrow direct-provider candidate-identification workflow, not correctness on an independent library corpus, other platforms or REA integration. The selected target and database must be identity-bound, and a masked fingerprint match is an inference rather than proof of function or library provenance.

This offers an alternative to the proposed anchored-byte matcher for evaluation, not a reason to wrap every Kuna option or a claim that one method supersedes the other. Compare collision/tie handling, relocations, provenance, target architecture, available match evidence, candidate-versus-renaming semantics and transport/cache identity before selecting an analyst-facing design. REA's current Kuna child environment and profile do not expose this caller-selected database; no new environment inheritance is proposed.

A follow-up compared the default direct Kuna 1.986 mode with REA MCP on the same stripped ELF. With kuna_fid_db set, the direct default command returned the same two matched names. Starting REA with that ambient variable set still returned three placeholders, consistent with REA's deliberately selected Kuna child environment. This is not a defect or a request to inherit ambient settings: any supported reference corpus would need explicit selection and provenance.

Kuna 1.989 fingerprint control

On macOS arm64, I built a source-owned x86-64 ELF object with two noinline functions and generated a 109-byte, two-record .fid using direct kuna fid build ... --lang x86:LE:64:default --cspec gcc. I linked a stripped ELF from that object and did not execute it. Direct Kuna 1.989 functions with --option listing on --option fid on and kuna_fid_db set to this database named one discovered entry audit_fold; the same query with fid off retained its sub_201190 placeholder. The other function remained unnamed in both controls. Real REA/Kuna list_procedures in compact and full MCP profiles returned both placeholders even with the database environment variable set; source inspection found no REA Kuna option for enabling this matcher. This is expected default behavior, not a claim that REA promised to apply the database or that the environment was passed through.

As a negative-identity control, I rebuilt the target with the function's XOR constant changed from 0x31415926 to 0xdeadbeef, keeping the same instruction shape. Direct Kuna FID still assigned audit_fold. Direct Kuna disassembly showed the different immediate bytes; this is consistent with the documented operand-masked fingerprint and is not by itself a Kuna defect. It shows that an identified name is a fingerprint candidate, not evidence of byte identity or equivalent behavior. A future design should retain database identity/digest, selected matching options, candidate and ambiguity semantics, and relevant provenance in Evidence and any cache key; it should not silently promote a matched library name to a verified source identity. No proposal to expose raw FID options is implied.

The original object SHA-256 was f278cee536de5ceee3c48039347a525c27b1490ad77e137c08442eacc082948d, database 505df96aeec9103b3772066821d04d78bc849cb6dcc6808cff42d445e102032f, stripped target 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d, and changed target 1f24b3a90503fd978b540379c05b46726e5ff911d0f1b42955660765fe998b96. The original stripped target digest remained unchanged. This is one generated database and two unexecuted targets, not a corpus accuracy or collision-rate measurement.

Kuna fingerprint ambiguity control

Using the same unexecuted stripped ELF, I generated an additional object in which the second function had the changed immediate from the preceding control and the distinct name audit_other. A database built from both objects contained three records; with listing and fid enabled, direct Kuna 1.989 kept sub_201190 instead of choosing either audit_fold or audit_other. The original one-object database named that entry audit_fold. A separate database built from one unrelated function also left the placeholder; its returned functions array was identical to the ambiguous-database result for this target. This is consistent with Kuna's documented conservative tie handling, but the tested inventory does not distinguish ambiguity from no reported match or expose the candidate names. It does not prove the engine's internal bucket state or general collision rates.

The ambiguous database SHA-256 was d9c163d04cf9ee72073a901395254ea3c772e8916a44cc36c72c97443aeea261, the unrelated database d92ac0e5061dc242fc47c2da518a2abbe80840bb503fbc7335301cd95c064772, and the target digest remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d. Any REA identification result must preserve ambiguity or state it is unavailable; an unchanged placeholder is not proof that a library match is absent. Do not claim access to candidate matches the producer did not expose.

FID database applicability control

Direct Kuna 1.989 matched the same stripped x86-64 fixture using a database built from the same x86 object but labeled AARCH64:LE:64:v8A / default in its header. A missing configured database path exited successfully and left placeholders without a load-status field or stderr in this control. Whether those header fields are intended as applicability keys is unresolved; upstream Kuna #1475 asks for clarification. For an REA design, database existence, digest, applicable architecture/compiler profile and match coverage must not be assumed from an unchanged name. The mismatched database digest was d56e955d7851a989d796fdbb0571ebda3506459b43413333fa39e826d6a41e60; the unexecuted target digest remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d.

Ghidra analyzer and REA inventory qualification

The installed Ghidra 12.1.4 includes Function ID databases and headless database-ingestion scripts. Its FunctionIDHeadlessPrescript.java disables Function ID and Library Identification while ingesting corpus programs, with a source comment explaining this avoids contaminating names. In a separate direct analyzeHeadless import of the stripped x86-64 ELF above, an owned postscript reported Function ID=true, Function ID.Always Apply FID Labels=false, and Function ID thresholds; the two observed function names were FUN_00201170 (source DEFAULT) and entry (source IMPORTED). The project was temporary and deleted after the check.

Real REA/Ghidra 12.1.4 list_procedures returned those same two names and their respective default/imported symbol-source facts in compact and full MCP profiles. No identification match was observed in this fixture, and I did not attach a custom Ghidra FID database or prove exactly which installed databases were applicable. The target SHA-256 remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d and it was not executed.

This qualifies the earlier statement that no provider implements identification: REA does not expose a dedicated, provider-neutral library-identification result, but a Ghidra import can run its default Function ID analyzer and REA can surface resulting provider symbol facts. That is not evidence that every such name carries a database-specific match, candidates, confidence or provenance in REA. A design should distinguish default analyzer output, caller-selected databases, original/imported symbols, and independently supported candidate evidence rather than assuming all providers lack identification or equating names with matches.

Positive Ghidra Function ID and saved-project control

Using Ghidra 12.1.4 headless with a temporary user-home and project, I imported the source-owned x86-64 library.o with Ghidra's FunctionIDHeadlessPrescript and populated a temporary Ghidra .fidb with two function records through FidService. I then imported the previously described stripped ELF into a separate temporary Ghidra project, attached that database, and explicitly changed the Function ID instruction-count and multi-match thresholds to 0.0 and Always Apply FID Labels to true for this deliberately small fixture. A postscript reported audit_mix at 0x201170 with Ghidra symbol source ANALYSIS; the default-profile control above had FUN_00201170 there. The entry function remained entry with source IMPORTED. This is a deliberately configured positive control, not a claim that the default thresholds match this fixture or a measurement of identification accuracy.

REA main 9ebbe67e opened that saved project via REA_GHIDRA_PROJECT_PATH and returned audit_mix with symbol source analysis in list_procedures in both compact and full MCP profiles. Its profile described an existing private project copy and recorded original analysis settings and completeness as unknown; the result did not name the FID database, match score, candidates or analyzer that supplied the analysis label. Thus REA can preserve a Ghidra-derived name and its generic symbol source without independently proving its library identity or database provenance. The temporary Ghidra database SHA-256 was 9f534994216bf08b942ec914563f3673b7559f69fd1a15ac7ac5cc5af98742fc; the unexecuted ELF SHA-256 remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d. The saved-project original was not independently hashed before this control; this does not assert its before/after identity.

Existing Ghidra match-comment handoff

A follow-up on the saved positive-control project above found more usable existing evidence. In both compact and full MCP profiles, REA/Ghidra analyze_function({procedure: "0x201170", detail: "complete"}) returned a comment at the matched entry containing Library Function - Single Match, audit_mix, and Library: fixture 1 control. Direct inspection of Ghidra 12.1.4's ApplyFidEntriesCommand source shows that its Function ID analyzer creates plate comments and optional bookmarks for match results; the separate controlled Ghidra run above established the source of this particular fixture's comment. The unexecuted target SHA-256 remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d.

This qualifies the preceding inventory-only observation: a complete function dossier can expose this Ghidra match and library description as a listing comment. REA projects it as ordinary {address, kind: "comment", text}, without a typed Function ID match, score, database digest, or authenticated origin for arbitrary comment text. Do not infer that every similar comment is analyzer-authored or that this static fingerprint proves function identity. This is one deliberately configured single-match saved-project control; multi-match comments, edited comments, other providers and offline handoff of this dossier were not tested.

Offline Function ID comment handoff

For the same deliberately configured Ghidra saved-project match, REA's default summary analyze_function returned a parent Evidence ID. After exporting the bundle, closing the binary, and importing into a fresh MCP server without opening a binary, inspect_analysis_view with {kind: "native", facet: "comments", offset: 0, limit: 10} returned the Function ID plate comment and its parent Evidence link in both compact and full profiles. The view reported one examined row, exhausted page coverage, and preserved the saved-project analysis-settings-unknown limitation. The unexecuted target digest remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d.

This demonstrates an existing portable comment workflow, not typed or authenticated match provenance. The local target and project files were present during the offline server run, but no binary was opened; the view is defined over retained Evidence. No multi-match, edited-comment or database-identity handoff was established.

Two-object Ghidra database control

I populated a separate temporary Ghidra 12.1.4 Function ID database from both x86-64 objects used in the Kuna ambiguity control: one named audit_fold, the other audit_other with a changed immediate. The Ghidra ingest reported five functions attempted, four added and one excluded; I did not inspect which record was excluded. With a fresh temporary Ghidra user-home, that database attached, and the same deliberately relaxed thresholds and label option as the positive control above, a new saved-project analysis labeled audit_mix as an analysis-sourced name. REA/Ghidra complete dossiers for the first and second functions returned Library Function - Single Match comments naming audit_mix and audit_fold respectively in both MCP profiles. The second function's primary symbol remained the imported entry name.

This did not reproduce Kuna's placeholder result as a Ghidra multi-match comment. Ghidra's candidate selection and the excluded record were not independently traced, so the result does not establish why Ghidra selected one name or that all potential candidates were considered. It shows that a masked-fingerprint collision observed with one provider cannot be projected onto another provider's matching result. The temporary Ghidra database SHA-256 was 9ddcfd0b4af12dd168995dbf0700d1174a6611b5a8a11133b2dae79b68c9baa1; the unexecuted target stayed 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d.

Ghidra database and query inspection

A read-only headless script inspected the two-object Ghidra database used above. Both competing records were present: audit_fold and audit_other had the same full hash (eecb018eb729863b) but different specific hashes (cbedbd60f23d7721 and 02208bdc194395c8). Thus the excluded ingest item was not either of these two records. A separate read-only FidService.processProgram query on the saved target, with that database attached and score threshold 0.0, returned audit_fold at 0x201190 as a SPECIFIC match with score 12.67; it did not return audit_other for that entry. It also returned audit_mix at 0x201170 as SPECIFIC with score 14.01.

This explains the tested query outcome more precisely than the preceding hypothesis: the competing names were ingested, but the later Ghidra query reported only the specific-hash match. It does not prove that all Ghidra matches are unique, that its analyzer always follows the same settings, or that a fingerprint match establishes equivalent behavior. REA's ordinary plate comment did not expose those match-mode, hash or score facts in the earlier dossier control.

Confirmed Ghidra multi-match control

I then built a separate Ghidra 12.1.4 database from the original object and a second object with identical audit_fold code named audit_twin. A read-only database inspection confirmed that both records were present with identical full hash eecb018eb729863b and specific hash cbedbd60f23d7721. A separate read-only FidService.processProgram query returned both as SPECIFIC matches at 0x201190, each with score 12.67. As in the earlier positive control, I explicitly set both analyzer thresholds to 0.0 and enabled always-apply labels for this small synthetic fixture; these are not the default settings.

REA/Ghidra opened the resulting saved project through its private-copy workflow. In both compact and full MCP profiles, the complete analyze_function dossier retained the plate comment Library Function - Multiple Matches With Different Base Names followed by audit_fold, audit_twin, and the fixture library. The function's primary symbol remained the imported entry name. This shows that the existing untyped-comment route can preserve a real Ghidra ambiguity, not merely a single-match note; a consumer still cannot treat arbitrary similar comment text as authenticated analyzer output, and REA did not provide the query scores, hashes or database identity as a typed identification result. A later offline handoff control is described below.

The temporary Ghidra database SHA-256 was 960c4da2f2474be8472173ac490c5dec252a8eddddfcdcc9ba0f4a3887af9d44; the unexecuted target digest remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d. This is a deliberately identical-code collision, not a corpus accuracy estimate.

Synthetic comment-origin control

I created a separate temporary Ghidra 12.1.4 project for the same stripped ELF, disabled Function ID with Ghidra's ingestion prescript, and used a test-only headless script to write the same plate-comment text and set the same audit_mix name with Ghidra ANALYSIS source. This is deliberately fabricated metadata, not a second Function ID result. In both compact and full MCP profiles, REA's list_procedures records and the complete dossier comment array were exactly equal as parsed JSON to those from the real positive Function ID project. Their recorded project fingerprints differed (c39685a0… versus a2039c83…), so the complete Evidence envelopes were not identical. The target SHA-256 remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d and it was not executed.

This control demonstrates that these exposed name, generic symbol-source and comment fields alone cannot authenticate the analyzer, match, or database. The project identity distinguishes the two observations but does not explain how the label was produced. It is a constructed negative control, not evidence of accidental or malicious comments in other projects.

Existing REA comparison of the masked-fingerprint variants

For the original and changed-immediate stripped ELF fixtures above, real REA/Ghidra 12.1.4 MCP on main 9ebbe67e collected both function inventories, the selected entry dossiers, and 13 bytes of each entry. The complete inline match_native_functions request exceeded the default 1,024-element MCP input allowance; exporting/importing the source bundles and passing retained-Evidence references allowed the same comparison in both compact and full profiles. This is one observed handoff, not a token or performance measurement.

The matching result paired the entry functions by the unique reported symbol entry and explicitly said that does not establish unchanged code or semantic equivalence. It retained different exact body SHA-256 values, with raw byte observations 50e8daffffff352659413159c3 and 50e8daffffff35efbeadde59c3, and returned a compare_functions input. Following that input produced status: changed: pseudocode and assembly text dimensions changed, while calls and references were unknown and the result qualified textual changes as not necessarily behavioral changes. This workflow compares two observed functions; it does not identify a library or prove the reason for their difference. It provides an existing evidence-backed follow-up to a fingerprint candidate and keeps matched-name correspondence distinct from equality. Both unexecuted target digests remained unchanged.

Multi-match comments after offline handoff

On the Kuna 1.990 validation branch with Ghidra 12.1.4 on macOS arm64, I reopened the deliberately configured fidtwin saved-project fixture, requested the 0x201190 function summary, and exported its retained Evidence. In a fresh target-free MCP session, importing the bundle and paging the retained parent's native comments returned the multiple-match comment containing both audit_fold and audit_twin in compact and full output profiles. The target stayed unopened in the importing server; the source ELF SHA-256 remained 0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d and it was not executed.

This preserves the ambiguity as an ordinary historical comment through one offline workflow. It does not authenticate the comment's analyzer origin, supply scores or database identity, validate every returned schema independently, or prove general Function ID accuracy.

Originally created by @reynold-hu on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7189 # [Feature] Static library-function identification (FLIRT-like) for native targets ## Problem An agent analyzing a stripped native binary sees hundreds of unnamed functions (`sub_XXXX`) and has no provider-neutral way to ask: "is this function a known library function, and if so, which one?" Every downstream answer pays for that gap: reconstruction wastes tokens re-deriving `memcpy`, diffing compares library code as if it were the target's own code, and capability inventories risk attributing library behavior to the program under analysis. Ghidra, Hopper, and IDA each have their own library identification mechanisms, but REA exposes no common typed identification result across providers. As the controls below show, REA can surface Ghidra analyzer-generated names and comments from a saved project; those fields alone do not authenticate a match or identify its database. The original proposal reported a source-only search at filing time: issue search in morluto/rea returned 0 results for "FLIRT", "library function identification", "library identification", "source sink", and "known function"; the tree contains no `identifyFunction`, `identify_library`, `knownFunction`, or `libraryFunction` implementation and no "flirt" reference outside unrelated test fixtures. Related but distinct capabilities already exist and are not this proposal: `NativeValueTrace` (a procedure-seeded def-use dependency graph, not function identification), the managed/JVM/ Python bytecode providers (they parse metadata, they do not match native bytes), and the WASM lane (a different target family). ## Proposed scope (Phase 1 only) One pure, static capability — no execution, no host state, no new dependency — built as a domain module with tests: - **Signature format**: a signature names one known library function (`id`, `name`, `library`, optional `library_version`) and carries a byte pattern for the start of that function: whitespace-separated tokens, each two hex digits or `??` for a wildcard byte (an address or relocation slot that varies between builds). Patterns are capped at 1024 tokens; a pattern with no concrete byte is rejected, because an all-wildcard pattern would match every function. - **Matcher**: a pattern matches a function only when anchored at offset 0 of the function bytes. This mirrors FLIRT's first-bytes approach and keeps a match a claim about function identity, not about a byte string occurring somewhere inside. A match carries `score` (non-wildcard bytes over pattern length) and `confidence` (`exact` when no wildcard is present, otherwise `pattern`). Matches rank by score, then pattern length, then signature id, so results are deterministic. - **Budgets with explicit unknowns**: per-function byte cap, per-run function cap, and signature-set cap. Functions beyond a budget are NOT silently dropped: they appear in `unknowns` with a reason (`function_budget_exceeded` or `function_bytes_budget_exceeded`) while the rest of the batch is still identified. An invalid signature set (duplicate ids, over-cap, malformed pattern) is a caller configuration error and throws, like every other zod schema validation in this codebase; it is never silently truncated. - **Evidence mapping (proposed)**: `confidence: "inferred"`, `authority: "historical-reference"`, no execution environment. A match is a pattern claim against a reference corpus, never an observed fact and never proof of provenance. ## Result model One run returns, per function: - `function_id` and its `matches`: each with `signature_id`, `name`, `library`, `library_version`, `pattern_length`, `wildcard_count`, `score`, and `confidence`; - `unknowns`: functions skipped by a budget, each with a reason. No MCP tool, provider registration, or composition wiring is included in Phase 1. That wiring is integration surface and belongs to the PR phase, after a maintainer decision (see below). ## Limitations (Phase 1) - No signature corpus ships with Phase 1. The format and the matcher are built and tested; a real corpus (derived from public libraries, with provenance and versioning) is a later phase. Stated plainly: until a corpus exists, the matcher identifies nothing in production. Shipping a matcher with a fabricated corpus would be worse than shipping no corpus. - x86_64 only. Other architectures need their own signature sets and verification; they are not claimed here. - Anchored-at-0 only: no mid-function patterns, no fuzzy or similarity matching, no version disambiguation beyond the signature's own metadata. Two library versions sharing a prologue will both match; the caller must weigh scores, not treat the top match as a verdict. - A match is an inference, not proof of provenance. It must never be reported as "this function IS memcpy", only as "this function matches the memcpy signature with score S". ## Non-goals - Not a signature database, and not a scraper for proprietary signature sets (IDA FLIRT signatures are proprietary; deriving a corpus must use public libraries with a license review). - Not function similarity: no fuzzy hashing (TLSH, ssdeep), no embedding comparison, no "similar to" ranking. Those are separate capabilities with their own false-positive profiles. - Not a verdict and not an exclusion rule: Phase 1 does not decide that matched functions are removed from analysis. Whether identification feeds an exclusion filter for reconstruction or #4972's candidate set is a consumer decision for a later phase. - No packer/unpacking workflow, no anti-analysis evasion, no DRM/license/ anti-cheat bypass, no exploit automation. - No execution of any kind: this proposal is static identification, so the target-execution scope gate does not apply to it — but integration still needs a maintainer decision (below). ## Dependency and licensing - Zero new dependencies: zod is already in the tree, and the matcher is pure TypeScript with no engine, no subprocess, and no host capability. - The later-phase corpus question needs a license review before any corpus ships: signatures derived from public libraries must carry provenance (library, version, derivation method), and proprietary signature sets must not be copied. No corpus is proposed in Phase 1, so no license question blocks the matcher itself. ## Relationship to existing work - **#4972 (semantic-equivalence verification)**: complementary, not dependent. Identification shrinks #4972's problem: library functions can be labeled before differential analysis, so the harness spends its budget on the target's own code. It implements no part of #4972's verdict model. - **NativeValueTrace**: complementary. The trace builds def-use graphs over procedures; identification labels the procedures the trace traverses. A later phase could join the two (e.g., "trace values through this identified memcpy"), but Phase 1 does not. - **Managed/JVM/Python bytecode providers**: different mechanism for a different target family. Those parse metadata; this matches native bytes. No overlap. - **#7142 (Unicorn emulation provider)**: independent. Identification is static and needs no scope decision about execution; emulation is dynamic and awaits its own decision. Neither blocks the other. ## Open questions for the maintainer 1. Placement: domain service consumed by providers, deep provider, or an auxiliary operation set? Phase 1 is a domain module so the matcher can be reviewed without committing to wiring; the choice affects `src/composition/binary.ts` and the tool catalog. 2. Operation naming: is `identify_library_functions` acceptable, or should it live under a different operation family? 3. Corpus strategy: should REA ship a small corpus derived from public libraries, derive signatures at setup time from a user-supplied library set, or accept caller-supplied signatures only? Each has a different licensing and maintenance profile. 4. Minimum concrete bytes: should wildcard-heavy patterns require a floor (e.g., at least N concrete bytes) beyond the current "at least one" rule, to bound false positives? 5. Version disambiguation: when two versions of one function share a prologue, should the result carry both matches (current behavior) or should a later phase add a disambiguation pass? ## Scope decision request Per `CONTRIBUTING.md` ("Fail-closed integration"), this issue requests an explicit maintainer scope decision on the Phase 1 behavior above before integration work (MCP tool, provider registration, composition wiring, or any corpus). A local prototype with tests exists to inform that decision; I understand neither the prototype nor a draft PR authorizes integration, and that a scope expansion needs a new decision. ## Additional direct-provider observation On macOS arm64, I compiled a self-owned x86-64 ELF object with three functions and linked a stripped executable from the same object without running it. Direct Kuna 1.986 `fid build` created a three-record `.fid` database from the object. With `--mode reliable --option listing on --option fid on` but no `kuna_fid_db`, direct `kuna functions` returned three `sub_...` names. Setting `kuna_fid_db` to that database renamed two entries to `audit_mix` and `_start`; `audit_checksum` remained a placeholder. This verifies a narrow direct-provider candidate-identification workflow, not correctness on an independent library corpus, other platforms or REA integration. The selected target and database must be identity-bound, and a masked fingerprint match is an inference rather than proof of function or library provenance. This offers an alternative to the proposed anchored-byte matcher for evaluation, not a reason to wrap every Kuna option or a claim that one method supersedes the other. Compare collision/tie handling, relocations, provenance, target architecture, available match evidence, candidate-versus-renaming semantics and transport/cache identity before selecting an analyst-facing design. REA's current Kuna child environment and profile do not expose this caller-selected database; no new environment inheritance is proposed. A follow-up compared the default direct Kuna 1.986 mode with REA MCP on the same stripped ELF. With `kuna_fid_db` set, the direct default command returned the same two matched names. Starting REA with that ambient variable set still returned three placeholders, consistent with REA's deliberately selected Kuna child environment. This is not a defect or a request to inherit ambient settings: any supported reference corpus would need explicit selection and provenance. ## Kuna 1.989 fingerprint control On macOS arm64, I built a source-owned x86-64 ELF object with two noinline functions and generated a 109-byte, two-record `.fid` using direct `kuna fid build ... --lang x86:LE:64:default --cspec gcc`. I linked a stripped ELF from that object and did not execute it. Direct Kuna 1.989 `functions` with `--option listing on --option fid on` and `kuna_fid_db` set to this database named one discovered entry `audit_fold`; the same query with `fid off` retained its `sub_201190` placeholder. The other function remained unnamed in both controls. Real REA/Kuna `list_procedures` in compact and full MCP profiles returned both placeholders even with the database environment variable set; source inspection found no REA Kuna option for enabling this matcher. This is expected default behavior, not a claim that REA promised to apply the database or that the environment was passed through. As a negative-identity control, I rebuilt the target with the function's XOR constant changed from `0x31415926` to `0xdeadbeef`, keeping the same instruction shape. Direct Kuna FID still assigned `audit_fold`. Direct Kuna disassembly showed the different immediate bytes; this is consistent with the documented operand-masked fingerprint and is not by itself a Kuna defect. It shows that an identified name is a fingerprint candidate, not evidence of byte identity or equivalent behavior. A future design should retain database identity/digest, selected matching options, candidate and ambiguity semantics, and relevant provenance in Evidence and any cache key; it should not silently promote a matched library name to a verified source identity. No proposal to expose raw FID options is implied. The original object SHA-256 was `f278cee536de5ceee3c48039347a525c27b1490ad77e137c08442eacc082948d`, database `505df96aeec9103b3772066821d04d78bc849cb6dcc6808cff42d445e102032f`, stripped target `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`, and changed target `1f24b3a90503fd978b540379c05b46726e5ff911d0f1b42955660765fe998b96`. The original stripped target digest remained unchanged. This is one generated database and two unexecuted targets, not a corpus accuracy or collision-rate measurement. ## Kuna fingerprint ambiguity control Using the same unexecuted stripped ELF, I generated an additional object in which the second function had the changed immediate from the preceding control and the distinct name `audit_other`. A database built from both objects contained three records; with `listing` and `fid` enabled, direct Kuna 1.989 kept `sub_201190` instead of choosing either `audit_fold` or `audit_other`. The original one-object database named that entry `audit_fold`. A separate database built from one unrelated function also left the placeholder; its returned `functions` array was identical to the ambiguous-database result for this target. This is consistent with Kuna's documented conservative tie handling, but the tested inventory does not distinguish ambiguity from no reported match or expose the candidate names. It does not prove the engine's internal bucket state or general collision rates. The ambiguous database SHA-256 was `d9c163d04cf9ee72073a901395254ea3c772e8916a44cc36c72c97443aeea261`, the unrelated database `d92ac0e5061dc242fc47c2da518a2abbe80840bb503fbc7335301cd95c064772`, and the target digest remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. Any REA identification result must preserve ambiguity or state it is unavailable; an unchanged placeholder is not proof that a library match is absent. Do not claim access to candidate matches the producer did not expose. ## FID database applicability control Direct Kuna 1.989 matched the same stripped x86-64 fixture using a database built from the same x86 object but labeled `AARCH64:LE:64:v8A` / `default` in its header. A missing configured database path exited successfully and left placeholders without a load-status field or stderr in this control. Whether those header fields are intended as applicability keys is unresolved; [upstream Kuna #1475](https://github.com/Noelo-Lab/kuna/issues/1475) asks for clarification. For an REA design, database existence, digest, applicable architecture/compiler profile and match coverage must not be assumed from an unchanged name. The mismatched database digest was `d56e955d7851a989d796fdbb0571ebda3506459b43413333fa39e826d6a41e60`; the unexecuted target digest remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. ## Ghidra analyzer and REA inventory qualification The installed Ghidra 12.1.4 includes Function ID databases and headless database-ingestion scripts. Its `FunctionIDHeadlessPrescript.java` disables Function ID and Library Identification while ingesting corpus programs, with a source comment explaining this avoids contaminating names. In a separate direct `analyzeHeadless` import of the stripped x86-64 ELF above, an owned postscript reported `Function ID=true`, `Function ID.Always Apply FID Labels=false`, and Function ID thresholds; the two observed function names were `FUN_00201170` (source `DEFAULT`) and `entry` (source `IMPORTED`). The project was temporary and deleted after the check. Real REA/Ghidra 12.1.4 `list_procedures` returned those same two names and their respective `default`/`imported` symbol-source facts in compact and full MCP profiles. No identification match was observed in this fixture, and I did not attach a custom Ghidra FID database or prove exactly which installed databases were applicable. The target SHA-256 remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d` and it was not executed. This qualifies the earlier statement that no provider implements identification: REA does not expose a dedicated, provider-neutral library-identification result, but a Ghidra import can run its default Function ID analyzer and REA can surface resulting provider symbol facts. That is not evidence that every such name carries a database-specific match, candidates, confidence or provenance in REA. A design should distinguish default analyzer output, caller-selected databases, original/imported symbols, and independently supported candidate evidence rather than assuming all providers lack identification or equating names with matches. ## Positive Ghidra Function ID and saved-project control Using Ghidra 12.1.4 headless with a temporary user-home and project, I imported the source-owned x86-64 `library.o` with Ghidra's `FunctionIDHeadlessPrescript` and populated a temporary Ghidra `.fidb` with two function records through `FidService`. I then imported the previously described stripped ELF into a separate temporary Ghidra project, attached that database, and explicitly changed the Function ID instruction-count and multi-match thresholds to `0.0` and `Always Apply FID Labels` to true for this deliberately small fixture. A postscript reported `audit_mix` at `0x201170` with Ghidra symbol source `ANALYSIS`; the default-profile control above had `FUN_00201170` there. The entry function remained `entry` with source `IMPORTED`. This is a deliberately configured positive control, not a claim that the default thresholds match this fixture or a measurement of identification accuracy. REA main `9ebbe67e` opened that saved project via `REA_GHIDRA_PROJECT_PATH` and returned `audit_mix` with symbol source `analysis` in `list_procedures` in both compact and full MCP profiles. Its profile described an existing private project copy and recorded original analysis settings and completeness as unknown; the result did not name the FID database, match score, candidates or analyzer that supplied the analysis label. Thus REA can preserve a Ghidra-derived name and its generic symbol source without independently proving its library identity or database provenance. The temporary Ghidra database SHA-256 was `9f534994216bf08b942ec914563f3673b7559f69fd1a15ac7ac5cc5af98742fc`; the unexecuted ELF SHA-256 remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. The saved-project original was not independently hashed before this control; this does not assert its before/after identity. ## Existing Ghidra match-comment handoff A follow-up on the saved positive-control project above found more usable existing evidence. In both compact and full MCP profiles, REA/Ghidra `analyze_function({procedure: "0x201170", detail: "complete"})` returned a comment at the matched entry containing `Library Function - Single Match`, `audit_mix`, and `Library: fixture 1 control`. Direct inspection of Ghidra 12.1.4's `ApplyFidEntriesCommand` source shows that its Function ID analyzer creates plate comments and optional bookmarks for match results; the separate controlled Ghidra run above established the source of this particular fixture's comment. The unexecuted target SHA-256 remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. This qualifies the preceding inventory-only observation: a complete function dossier can expose this Ghidra match and library description as a listing comment. REA projects it as ordinary `{address, kind: "comment", text}`, without a typed Function ID match, score, database digest, or authenticated origin for arbitrary comment text. Do not infer that every similar comment is analyzer-authored or that this static fingerprint proves function identity. This is one deliberately configured single-match saved-project control; multi-match comments, edited comments, other providers and offline handoff of this dossier were not tested. ## Offline Function ID comment handoff For the same deliberately configured Ghidra saved-project match, REA's default summary `analyze_function` returned a parent Evidence ID. After exporting the bundle, closing the binary, and importing into a fresh MCP server without opening a binary, `inspect_analysis_view` with `{kind: "native", facet: "comments", offset: 0, limit: 10}` returned the Function ID plate comment and its parent Evidence link in both compact and full profiles. The view reported one examined row, exhausted page coverage, and preserved the saved-project analysis-settings-unknown limitation. The unexecuted target digest remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. This demonstrates an existing portable comment workflow, not typed or authenticated match provenance. The local target and project files were present during the offline server run, but no binary was opened; the view is defined over retained Evidence. No multi-match, edited-comment or database-identity handoff was established. ## Two-object Ghidra database control I populated a separate temporary Ghidra 12.1.4 Function ID database from both x86-64 objects used in the Kuna ambiguity control: one named `audit_fold`, the other `audit_other` with a changed immediate. The Ghidra ingest reported five functions attempted, four added and one excluded; I did not inspect which record was excluded. With a fresh temporary Ghidra user-home, that database attached, and the same deliberately relaxed thresholds and label option as the positive control above, a new saved-project analysis labeled `audit_mix` as an analysis-sourced name. REA/Ghidra complete dossiers for the first and second functions returned `Library Function - Single Match` comments naming `audit_mix` and `audit_fold` respectively in both MCP profiles. The second function's primary symbol remained the imported `entry` name. This did **not** reproduce Kuna's placeholder result as a Ghidra multi-match comment. Ghidra's candidate selection and the excluded record were not independently traced, so the result does not establish why Ghidra selected one name or that all potential candidates were considered. It shows that a masked-fingerprint collision observed with one provider cannot be projected onto another provider's matching result. The temporary Ghidra database SHA-256 was `9ddcfd0b4af12dd168995dbf0700d1174a6611b5a8a11133b2dae79b68c9baa1`; the unexecuted target stayed `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. ## Ghidra database and query inspection A read-only headless script inspected the two-object Ghidra database used above. Both competing records were present: `audit_fold` and `audit_other` had the same full hash (`eecb018eb729863b`) but different specific hashes (`cbedbd60f23d7721` and `02208bdc194395c8`). Thus the excluded ingest item was not either of these two records. A separate read-only `FidService.processProgram` query on the saved target, with that database attached and score threshold `0.0`, returned `audit_fold` at `0x201190` as a `SPECIFIC` match with score `12.67`; it did not return `audit_other` for that entry. It also returned `audit_mix` at `0x201170` as `SPECIFIC` with score `14.01`. This explains the tested query outcome more precisely than the preceding hypothesis: the competing names were ingested, but the later Ghidra query reported only the specific-hash match. It does not prove that all Ghidra matches are unique, that its analyzer always follows the same settings, or that a fingerprint match establishes equivalent behavior. REA's ordinary plate comment did not expose those match-mode, hash or score facts in the earlier dossier control. ## Confirmed Ghidra multi-match control I then built a separate Ghidra 12.1.4 database from the original object and a second object with identical `audit_fold` code named `audit_twin`. A read-only database inspection confirmed that both records were present with identical full hash `eecb018eb729863b` and specific hash `cbedbd60f23d7721`. A separate read-only `FidService.processProgram` query returned both as `SPECIFIC` matches at `0x201190`, each with score `12.67`. As in the earlier positive control, I explicitly set both analyzer thresholds to `0.0` and enabled always-apply labels for this small synthetic fixture; these are not the default settings. REA/Ghidra opened the resulting saved project through its private-copy workflow. In both compact and full MCP profiles, the complete `analyze_function` dossier retained the plate comment `Library Function - Multiple Matches With Different Base Names` followed by `audit_fold`, `audit_twin`, and the fixture library. The function's primary symbol remained the imported `entry` name. This shows that the existing untyped-comment route can preserve a real Ghidra ambiguity, not merely a single-match note; a consumer still cannot treat arbitrary similar comment text as authenticated analyzer output, and REA did not provide the query scores, hashes or database identity as a typed identification result. A later offline handoff control is described below. The temporary Ghidra database SHA-256 was `960c4da2f2474be8472173ac490c5dec252a8eddddfcdcc9ba0f4a3887af9d44`; the unexecuted target digest remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d`. This is a deliberately identical-code collision, not a corpus accuracy estimate. ## Synthetic comment-origin control I created a separate temporary Ghidra 12.1.4 project for the same stripped ELF, disabled Function ID with Ghidra's ingestion prescript, and used a test-only headless script to write the same plate-comment text and set the same `audit_mix` name with Ghidra `ANALYSIS` source. This is deliberately fabricated metadata, not a second Function ID result. In both compact and full MCP profiles, REA's `list_procedures` records and the complete dossier comment array were exactly equal as parsed JSON to those from the real positive Function ID project. Their recorded project fingerprints differed (`c39685a0…` versus `a2039c83…`), so the complete Evidence envelopes were not identical. The target SHA-256 remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d` and it was not executed. This control demonstrates that these exposed name, generic symbol-source and comment fields alone cannot authenticate the analyzer, match, or database. The project identity distinguishes the two observations but does not explain how the label was produced. It is a constructed negative control, not evidence of accidental or malicious comments in other projects. ## Existing REA comparison of the masked-fingerprint variants For the original and changed-immediate stripped ELF fixtures above, real REA/Ghidra 12.1.4 MCP on main `9ebbe67e` collected both function inventories, the selected entry dossiers, and 13 bytes of each entry. The complete inline `match_native_functions` request exceeded the default 1,024-element MCP input allowance; exporting/importing the source bundles and passing retained-Evidence references allowed the same comparison in both compact and full profiles. This is one observed handoff, not a token or performance measurement. The matching result paired the entry functions by the unique reported symbol `entry` and explicitly said that does not establish unchanged code or semantic equivalence. It retained different exact body SHA-256 values, with raw byte observations `50e8daffffff352659413159c3` and `50e8daffffff35efbeadde59c3`, and returned a `compare_functions` input. Following that input produced `status: changed`: pseudocode and assembly text dimensions changed, while calls and references were unknown and the result qualified textual changes as not necessarily behavioral changes. This workflow compares two observed functions; it does not identify a library or prove the reason for their difference. It provides an existing evidence-backed follow-up to a fingerprint candidate and keeps matched-name correspondence distinct from equality. Both unexecuted target digests remained unchanged. ## Multi-match comments after offline handoff On the Kuna 1.990 validation branch with Ghidra 12.1.4 on macOS arm64, I reopened the deliberately configured `fidtwin` saved-project fixture, requested the `0x201190` function summary, and exported its retained Evidence. In a fresh target-free MCP session, importing the bundle and paging the retained parent's native comments returned the multiple-match comment containing both `audit_fold` and `audit_twin` in compact and full output profiles. The target stayed unopened in the importing server; the source ELF SHA-256 remained `0cee46f9e95af174dec7f281c179cbe5233348520302c69fdc9eafc7fcebff1d` and it was not executed. This preserves the ambiguity as an ordinary historical comment through one offline workflow. It does not authenticate the comment's analyzer origin, supply scores or database identity, validate every returned schema independently, or prove general Function ID accuracy.

bsakaguchi commented

2026-10-31 00:48:40 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 26, 2026):

赞同这个设计：anchored-at-0 + 确定性排序（score, pattern length, id）让匹配结果可复现，unknowns 显式化而不是静默丢弃也很关键。结合刚评审的 #7198（预算在解析后才生效的问题），建议 Phase 1 的 signature-set cap 明确其校验位置：cap 校验应发生在编译签名集之前（compileSignatures 入口），与 #7198 的 max_signatures 截断点保持一致，避免两处预算语义分叉。另外两点供参考：1) 所有通配模式被拒绝是对的，但建议对"仅地址/重定位槽可变"的模式提供显式 reloc 标记而非裸 ??，减少 corpus 里把同一函数不同 PIE 变体写成多条签名；2) x86_64 限定没问题，corpus 后续按架构分目录组织即可，别把架构编码进签名 id。

@XinwangTao commented on GitHub (Oct 26, 2026): 赞同这个设计：anchored-at-0 + 确定性排序（score, pattern length, id）让匹配结果可复现，unknowns 显式化而不是静默丢弃也很关键。结合刚评审的 #7198（预算在解析后才生效的问题），建议 Phase 1 的 signature-set cap 明确其校验位置：cap 校验应发生在编译签名集之前（compileSignatures 入口），与 #7198 的 max_signatures 截断点保持一致，避免两处预算语义分叉。另外两点供参考：1) 所有通配模式被拒绝是对的，但建议对"仅地址/重定位槽可变"的模式提供显式 reloc 标记而非裸 ??，减少 corpus 里把同一函数不同 PIE 变体写成多条签名；2) x86_64 限定没问题，corpus 后续按架构分目录组织即可，别把架构编码进签名 id。

bsakaguchi referenced this issue

2026-10-31 00:51:18 +00:00

[GH-ISSUE #7689] [Bug] Apple IPS inspection omits unknown key names inside thread, frame and image rows #4611

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi commented

2026-11-01 21:11:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A direct Kuna 1.986 collision control sharpens the candidate/provenance question. I used llvm-objcopy to rename audit_mix to alternate_mix in a copy of the same owned x86-64 ELF object. Independent .text SHA-256 was identical for both objects; the function symbols had different names. I built four .fid files: original object only, renamed object only, both objects original-first, and both objects renamed-first.

On the same unexecuted stripped executable, the original-only corpus labeled address 0x201190 audit_mix; the renamed-only corpus labeled it alternate_mix. With either combined corpus, the address remained sub_201190 with no aliases in the returned functions --json, and decompile --json likewise used the placeholder. The two combined files had different SHA-256 values, both contained both name strings, and both preserved _start identification. The target digest was unchanged. These are direct upstream observations; REA did not select or consume the corpus. No independent corpus, other architecture or general false-positive rate was measured.

The pinned Kuna 1.986 FID source corroborates that its v1 matcher only commits a name when a full-hash bucket collapses to one distinct name; a multi-name bucket is intentionally left unnamed. Its database definition distinguishes the indexed operand-masked full hash from a specific hash stored but not indexed in v1. The JSON tested here exposed neither a candidate list nor an explicit tie reason, so a placeholder alone does not explain whether there was no match, a collision, or another limitation. Corpus selection can change the displayed pseudocode name without changing the target.

Any REA identification workflow should keep corpus digest and name origin, preserve ties and unavailable distinctions where the producer can actually supply them, and avoid presenting a unique fingerprint name as proof of library provenance. Treat support for structured alternatives as an open upstream capability question; do not invent candidate evidence from the placeholder or parse guesses from text.

@morluto commented on GitHub (Oct 30, 2026): A direct Kuna 1.986 collision control sharpens the candidate/provenance question. I used `llvm-objcopy` to rename `audit_mix` to `alternate_mix` in a copy of the same owned x86-64 ELF object. Independent `.text` SHA-256 was identical for both objects; the function symbols had different names. I built four `.fid` files: original object only, renamed object only, both objects original-first, and both objects renamed-first. On the same unexecuted stripped executable, the original-only corpus labeled address `0x201190` `audit_mix`; the renamed-only corpus labeled it `alternate_mix`. With either combined corpus, the address remained `sub_201190` with no aliases in the returned `functions --json`, and `decompile --json` likewise used the placeholder. The two combined files had different SHA-256 values, both contained both name strings, and both preserved `_start` identification. The target digest was unchanged. These are direct upstream observations; REA did not select or consume the corpus. No independent corpus, other architecture or general false-positive rate was measured. The pinned [Kuna 1.986 FID source](https://github.com/Noelo-Lab/kuna/blob/v1.986/decompiler/crates/kuna-analysis/src/analyzers/fid/mod.rs#L204-L218) corroborates that its v1 matcher only commits a name when a full-hash bucket collapses to one distinct name; a multi-name bucket is intentionally left unnamed. Its [database definition](https://github.com/Noelo-Lab/kuna/blob/v1.986/decompiler/crates/kuna-analysis/src/analyzers/fid/db.rs#L48-L56) distinguishes the indexed operand-masked full hash from a specific hash stored but not indexed in v1. The JSON tested here exposed neither a candidate list nor an explicit tie reason, so a placeholder alone does not explain whether there was no match, a collision, or another limitation. Corpus selection can change the displayed pseudocode name without changing the target. Any REA identification workflow should keep corpus digest and name origin, preserve ties and unavailable distinctions where the producer can actually supply them, and avoid presenting a unique fingerprint name as proof of library provenance. Treat support for structured alternatives as an open upstream capability question; do not invent candidate evidence from the placeholder or parse guesses from text.

bsakaguchi commented

2026-11-01 21:11:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A further direct Kuna 1.986 control shows that even a unique candidate name need not imply byte or behavioral identity. I made a separate copy of the owned reference object, renamed audit_mix to different_mix, and changed its XOR immediate from 0x5a5a1234 to 0x5a5a1235; independent llvm-objdump verified the changed reference instruction. The unexecuted stripped target retained the original immediate 0x5a5a1234, verified by both llvm-objdump and real REA/Ghidra read_bytes at 0x201190.

A .fid built from only the changed reference object caused direct Kuna functions --json to label the original target function different_mix; its decompiled body still showed the original target constant. A corpus containing both the original and changed reference objects left the target function as sub_201190. The different immediate affects the result of this XOR expression for the same input; this is an inference from the instructions, not a runtime experiment. Target digest was unchanged. This is one deliberately constructed operand-mask match, not a general accuracy rate or a claim that every constant is masked.

In separate REA MCP sessions, Ghidra returned the original target bytes and pseudocode under its default label. REA/Kuna returned the same constant under its placeholder and read_bytes remained unavailable. The REA server was started with the changed corpus path in an ambient kuna_fid_db variable, but REA's intentionally curated child environment did not apply it; no REA FID integration is claimed. If a future identification workflow uses this producer, a unique name remains a corpus-scoped inferred match, not proof of exact bytes, equivalence, or origin. Where meaningful to the analyst question, independently observed target bytes and reference facts can support a stronger comparison; their availability and applicability must remain explicit.

@morluto commented on GitHub (Oct 30, 2026): A further direct Kuna 1.986 control shows that even a **unique** candidate name need not imply byte or behavioral identity. I made a separate copy of the owned reference object, renamed `audit_mix` to `different_mix`, and changed its XOR immediate from `0x5a5a1234` to `0x5a5a1235`; independent `llvm-objdump` verified the changed reference instruction. The unexecuted stripped target retained the original immediate `0x5a5a1234`, verified by both `llvm-objdump` and real REA/Ghidra `read_bytes` at `0x201190`. A `.fid` built from only the changed reference object caused direct Kuna `functions --json` to label the original target function `different_mix`; its decompiled body still showed the original target constant. A corpus containing both the original and changed reference objects left the target function as `sub_201190`. The different immediate affects the result of this XOR expression for the same input; this is an inference from the instructions, not a runtime experiment. Target digest was unchanged. This is one deliberately constructed operand-mask match, not a general accuracy rate or a claim that every constant is masked. In separate REA MCP sessions, Ghidra returned the original target bytes and pseudocode under its default label. REA/Kuna returned the same constant under its placeholder and `read_bytes` remained unavailable. The REA server was started with the changed corpus path in an ambient `kuna_fid_db` variable, but REA's intentionally curated child environment did not apply it; no REA FID integration is claimed. If a future identification workflow uses this producer, a unique name remains a corpus-scoped inferred match, not proof of exact bytes, equivalence, or origin. Where meaningful to the analyst question, independently observed target bytes and reference facts can support a stronger comparison; their availability and applicability must remain explicit.

bsakaguchi commented

2026-11-01 21:11:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Follow-up: the existing match_native_functions workflow is a useful building block for checking a candidate when an analyst has a reference binary; this does not require wrapping Kuna's FID API.

Observed, on macOS arm64 with real Ghidra 12.1.4 through REA MCP at e6d0618979f0df3f1d5db218b0fc7b5e6a9cf3f9: I collected function inventories, procedure_info body ranges and read_bytes Evidence from the stripped target and two named references, exported/imported the bundles, then supplied the retained observations to match_native_functions. The target's 28-byte function at 0x201190 matched audit_mix in the original reference on identical-bytes. Against the reference whose XOR immediate differs by one byte and whose symbol is different_mix, those two functions were left unpaired; the two other functions matched. Input SHA-256 values were unchanged by the experiment. LLVM disassembly independently confirmed the changed immediate.

This was one deliberately controlled x86-64 ELF case. REA reports inventory completeness as unknown and restricts its conclusion to the supplied provider-observed inputs; it does not normalize relocations, establish source provenance or prove behavioral equivalence. The approach requires a separately available reference binary and several evidence-collection calls; it is not a corpus search or a substitute for exposing ambiguity and provenance in function identification. Kuna-backed byte/range collection limitations are tracked in #8173.

@morluto commented on GitHub (Oct 31, 2026): Follow-up: the existing `match_native_functions` workflow is a useful building block for checking a candidate when an analyst has a reference binary; this does not require wrapping Kuna's FID API. **Observed**, on macOS arm64 with real Ghidra 12.1.4 through REA MCP at `e6d0618979f0df3f1d5db218b0fc7b5e6a9cf3f9`: I collected function inventories, `procedure_info` body ranges and `read_bytes` Evidence from the stripped target and two named references, exported/imported the bundles, then supplied the retained observations to `match_native_functions`. The target's 28-byte function at `0x201190` matched `audit_mix` in the original reference on `identical-bytes`. Against the reference whose XOR immediate differs by one byte and whose symbol is `different_mix`, those two functions were left unpaired; the two other functions matched. Input SHA-256 values were unchanged by the experiment. LLVM disassembly independently confirmed the changed immediate. This was one deliberately controlled x86-64 ELF case. REA reports inventory completeness as unknown and restricts its conclusion to the supplied provider-observed inputs; it does not normalize relocations, establish source provenance or prove behavioral equivalence. The approach requires a separately available reference binary and several evidence-collection calls; it is not a corpus search or a substitute for exposing ambiguity and provenance in function identification. Kuna-backed byte/range collection limitations are tracked in #8173.

bsakaguchi commented

2026-11-01 21:11:10 +00:00

Author

Owner

@reynold-hu commented on GitHub (Oct 31, 2026):

Thanks for running these controls — they directly shape the scope here.

Three acknowledgements:

Collision control: a placeholder alone does not distinguish no-match from collision, so ties and unavailable distinctions must be preserved, not silently dropped.
Unique-name control: a unique name is a corpus-scoped inferred match, never proof of bytes, equivalence, or origin.
match_native_functions experiment: the existing workflow is the building block — no Kuna FID API wrapper needed.

I will converge the RFC on exactly that: corpus digest + name origin carried with every match, ties preserved where the producer can supply them, candidate-checking against a separately available reference binary — not a corpus search, and not a substitute for exposing ambiguity and provenance. I note #8173 frames the adjacent question (independent symbol evidence when a provider omits origins) and will keep that in the same picture when revising.

Before I revise: do you want the RFC text updated to say this explicitly, or do you see a different scope?

@reynold-hu commented on GitHub (Oct 31, 2026): Thanks for running these controls — they directly shape the scope here. Three acknowledgements: 1. Collision control: a placeholder alone does not distinguish no-match from collision, so ties and unavailable distinctions must be preserved, not silently dropped. 2. Unique-name control: a unique name is a corpus-scoped inferred match, never proof of bytes, equivalence, or origin. 3. match_native_functions experiment: the existing workflow is the building block — no Kuna FID API wrapper needed. I will converge the RFC on exactly that: corpus digest + name origin carried with every match, ties preserved where the producer can supply them, candidate-checking against a separately available reference binary — not a corpus search, and not a substitute for exposing ambiguity and provenance. I note #8173 frames the adjacent question (independent symbol evidence when a provider omits origins) and will keep that in the same picture when revising. Before I revise: do you want the RFC text updated to say this explicitly, or do you see a different scope?

bsakaguchi referenced this issue

2026-11-03 13:39:12 +00:00

[PR #7079] [MERGED] fix(apple): keep the documented terminating process, parent pid and metadata members of IPS reports #7626

bsakaguchi commented

2026-11-04 04:56:16 +00:00

Author

Owner

@XinwangTao commented on GitHub (Nov 4, 2026):

reynold-hu 收敛后的 RFC 范围看起来是对的，尤其「unique name 是 corpus-scoped inferred match、不是 proof」这个边界必须写清楚。从一个下游使用者的角度补充一点：识别结果建议把 provenance 做成一等字段（corpus digest + name origin + 是否有 tie/unavailable），而不是放在描述文本里——diff 和 capability inventory 这类下游要按置信度分流（确认的库函数 vs 推断匹配），结构化字段才能让它们安全降级而不误把库行为算进目标自身行为。另外同意不包装 Kuna FID API、直接用 match_native_functions + retained Evidence 作为 building block，这保持了 REA 的 provider-neutral 边界；#8173 的采集限制与 #8760 的超大输入恢复建议在同一 RFC 里给出指引。

@XinwangTao commented on GitHub (Nov 4, 2026): reynold-hu 收敛后的 RFC 范围看起来是对的，尤其「unique name 是 corpus-scoped inferred match、不是 proof」这个边界必须写清楚。从一个下游使用者的角度补充一点：识别结果建议把 provenance 做成一等字段（corpus digest + name origin + 是否有 tie/unavailable），而不是放在描述文本里——diff 和 capability inventory 这类下游要按置信度分流（确认的库函数 vs 推断匹配），结构化字段才能让它们安全降级而不误把库行为算进目标自身行为。另外同意不包装 Kuna FID API、直接用 match_native_functions + retained Evidence 作为 building block，这保持了 REA 的 provider-neutral 边界；#8173 的采集限制与 #8760 的超大输入恢复建议在同一 RFC 里给出指引。

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4297

No description provided.

Rows
Columns