[PR #1438] [MERGED] fix(native): preserve compact reference provenance #2887

New issue

Closed

opened 2026-10-17 21:11:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:11:26 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1438
Author: @JosephDemarest
Created: 10/10/2026
Status: ✅ Merged
Merged: 10/10/2026
Merged by: @morluto

Base: main ← Head: codex/compact-reference-provenance-ntfs

📝 Commits (3)

ff46705 fix(native): preserve compact reference provenance
edb4c63 chore(native): reconcile compact reference verification with main
30473db chore(native): integrate archive case controls and current catalogs

📊 Changes

10 files changed (+523 additions, -12 deletions)

View changed files

📝 bridge/ghidra/ReaGhidraBridge.java (+6 -7)
📝 docs/native-investigation.md (+8 -0)
📝 scripts/create-ghidra-windows-fixture.mjs (+43 -5)
➕ scripts/lib/real-ghidra-compact-references.mjs (+170 -0)
📝 scripts/verify-packaged-ghidra-windows.mjs (+31 -0)
📝 scripts/verify-real-ghidra-windows.mjs (+10 -0)
➕ src/domain/native/nativeInstruction.fixture.ts (+60 -0)
📝 src/domain/native/nativeInstruction.ts (+27 -0)
📝 src/ghidra/GhidraFunctionValues.test.ts (+74 -0)
➕ tests/boundary/mcp/ghidraCompactReferencesMcp.test.ts (+94 -0)

📄 Description

Summary

Preserve Ghidra's reference origin, primary status, and DATA/READ/WRITE classification in compact instruction and call-target results. This is the compact-reference increment of #1245; the broader incoming-xref contract remains open.

Problem and expected behavior

An indexed store can have multiple analyzer-generated DATA references, including a non-primary reference. The compact representation previously dropped the primary flag, reference origin and access flags even though the full procedure-reference representation retained most of these facts. Consumers could not distinguish those references from literal accesses without another query.

Compact results now preserve the observed provider facts. primary is not proof of literal addressing or runtime access, and unknown metadata stays unknown.

Change and scope

Reuse the Ghidra referenceKind producer and add the actual source address and Reference.getSource() origin.
Extend the shared instruction/call-target reference schema with optional, nullable source_address, data, read, write, primary, provenance and source, preserving legacy omission and explicit nulls.
Extend the source-owned x86/x64 PE fixtures and real-provider verifiers with literal DATA/READ/WRITE, indexed non-primary analyzer references, direct calls, unresolved indirect calls, returns and outside-memory controls.

Contract and boundary impact

Semantic owner and earliest changed stage: Ghidra bridge reference serialization; shared native schemas validate the additive fields.
CLI and MCP/tool-catalog contract: both existing operations expose the same additive metadata; no new tools or inputs.
Provider, bridge, target-format, or platform compatibility: Ghidra emits its observed facts; older captures/providers may omit the new fields.
Evidence, artifact, provenance, or reconstruction contract: evidence preserves the fields and distinguishes reference-manager provenance from provider reference origin. No runtime-access inference is added.
Process execution, authorization, cleanup, or containment impact: none; existing read-only execution and cleanup controls are retained and exercised.
Generated metadata, package, or installation impact: generated metadata checks pass; no package version, installation or native-addon source changes.

Evidence and regression coverage

Tests cover full metadata, legacy omission, explicit nulls, malformed present fields, and MCP evidence projection for both operations.
New schema-preservation tests failed on the base implementation and pass with this change.
Executed CLI output for the x86 indexed store includes these two references (excerpt):

[
  { "type": "DATA", "data": true, "read": false, "write": false, "primary": true, "source": "analysis", "provenance": "ghidra-reference-manager" },
  { "type": "DATA", "data": true, "read": false, "write": false, "primary": false, "source": "analysis", "provenance": "ghidra-reference-manager" }
]

The indexed store's analyzer targets are DATA references; separate literal controls prove READ and WRITE preservation. Compact references match full procedure-reference edges, and direct-call target references match the corresponding instruction references.

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Executed on Windows 11, with Node 24.18.0, npm 11.16.0, Ghidra 12.1.4 and Temurin JDK 21.0.11:

npm run build:cached — passed.
npm run test:focused -- src/ghidra/GhidraFunctionValues.test.ts src/ghidra/GhidraProvider.behavior.test.ts src/application/native/NativeValueTrace.test.ts tests/boundary/mcp/ghidraCompactReferencesMcp.test.ts tests/boundary/mcp/ghidraRoutingMcp.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/contractPresentation.test.ts — 72 tests passed across seven files. The ignored addon was temporarily absent for the existing missing-addon control, then restored.
npm run check — passed, including architecture guards; lint reported 494 warnings and zero errors.
npm run docs:check — passed.
node scripts/verify-test-discovery.mjs — 705 test files discovered exactly once.
LSP diagnostics — no errors in the changed TypeScript files.
node scripts/verify-real-ghidra-windows.mjs --x86 and node scripts/verify-real-ghidra-windows.mjs — real Ghidra x86/x64 passed all 25 read-only operations, compact/full reference assertions and complete cleanup.
node scripts/verify-packaged-ghidra-windows.mjs . build/fixtures/rea-ghidra-windows-x86.exe <progress-file> — source-built public CLI and real stdio MCP passed, including six CLI/MCP comparisons, 29 unique MCP operations, unchanged fixture digest, preserved collision controls and zero residual runtimes.
Commit and push hooks — passed without bypasses.

Compatibility, safety, and release

Breaking changes or migration steps: none; omitted and null metadata remain valid without fabricated defaults.
Real Hopper/Ghidra, browser, or OS coverage: real Ghidra on Windows x86/x64 PE fixtures; CLI/MCP parity on the x86 fixture. Hopper and Linux/macOS engine runs were not performed.
Package or release metadata impact: none. The real Windows runs used the matching official published 6.3.0 native addon; no addon rebuild or tarball installation was tested.
Security, privacy, process, or containment review: source-owned fixtures are inspected statically; existing NTFS admission, protected DACL and Job Object controls passed. No new execution or mutation authority.
Remaining proof gaps: full deterministic suite is left to CI; this does not complete all of #1245.

Review checklist

This PR addresses a concrete problem or an agreed enhancement.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1438 **Author:** [@JosephDemarest](https://github.com/JosephDemarest) **Created:** 10/10/2026 **Status:** ✅ Merged **Merged:** 10/10/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `codex/compact-reference-provenance-ntfs` --- ### 📝 Commits (3) - [`ff46705`](https://github.com/morluto/rea/commit/ff46705dfd74e960a4de97c6c9a34d8d886edf01) fix(native): preserve compact reference provenance - [`edb4c63`](https://github.com/morluto/rea/commit/edb4c63adf9c3082c0efef65b6e060ec1203e09d) chore(native): reconcile compact reference verification with main - [`30473db`](https://github.com/morluto/rea/commit/30473dbcacd9dd404b8ecb65aa487caf57ab0682) chore(native): integrate archive case controls and current catalogs ### 📊 Changes **10 files changed** (+523 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `bridge/ghidra/ReaGhidraBridge.java` (+6 -7) 📝 `docs/native-investigation.md` (+8 -0) 📝 `scripts/create-ghidra-windows-fixture.mjs` (+43 -5) ➕ `scripts/lib/real-ghidra-compact-references.mjs` (+170 -0) 📝 `scripts/verify-packaged-ghidra-windows.mjs` (+31 -0) 📝 `scripts/verify-real-ghidra-windows.mjs` (+10 -0) ➕ `src/domain/native/nativeInstruction.fixture.ts` (+60 -0) 📝 `src/domain/native/nativeInstruction.ts` (+27 -0) 📝 `src/ghidra/GhidraFunctionValues.test.ts` (+74 -0) ➕ `tests/boundary/mcp/ghidraCompactReferencesMcp.test.ts` (+94 -0) </details> ### 📄 Description ## Summary Preserve Ghidra's reference origin, primary status, and DATA/READ/WRITE classification in compact instruction and call-target results. This is the compact-reference increment of #1245; the broader incoming-xref contract remains open. ## Problem and expected behavior An indexed store can have multiple analyzer-generated DATA references, including a non-primary reference. The compact representation previously dropped the primary flag, reference origin and access flags even though the full procedure-reference representation retained most of these facts. Consumers could not distinguish those references from literal accesses without another query. Compact results now preserve the observed provider facts. `primary` is not proof of literal addressing or runtime access, and unknown metadata stays unknown. ## Change and scope - Reuse the Ghidra `referenceKind` producer and add the actual source address and `Reference.getSource()` origin. - Extend the shared instruction/call-target reference schema with optional, nullable `source_address`, `data`, `read`, `write`, `primary`, `provenance` and `source`, preserving legacy omission and explicit nulls. - Extend the source-owned x86/x64 PE fixtures and real-provider verifiers with literal DATA/READ/WRITE, indexed non-primary analyzer references, direct calls, unresolved indirect calls, returns and outside-memory controls. ## Contract and boundary impact - Semantic owner and earliest changed stage: Ghidra bridge reference serialization; shared native schemas validate the additive fields. - CLI and MCP/tool-catalog contract: both existing operations expose the same additive metadata; no new tools or inputs. - Provider, bridge, target-format, or platform compatibility: Ghidra emits its observed facts; older captures/providers may omit the new fields. - Evidence, artifact, provenance, or reconstruction contract: evidence preserves the fields and distinguishes reference-manager provenance from provider reference origin. No runtime-access inference is added. - Process execution, authorization, cleanup, or containment impact: none; existing read-only execution and cleanup controls are retained and exercised. - Generated metadata, package, or installation impact: generated metadata checks pass; no package version, installation or native-addon source changes. ## Evidence and regression coverage - Tests cover full metadata, legacy omission, explicit nulls, malformed present fields, and MCP evidence projection for both operations. - New schema-preservation tests failed on the base implementation and pass with this change. - Executed CLI output for the x86 indexed store includes these two references (excerpt): ```json [ { "type": "DATA", "data": true, "read": false, "write": false, "primary": true, "source": "analysis", "provenance": "ghidra-reference-manager" }, { "type": "DATA", "data": true, "read": false, "write": false, "primary": false, "source": "analysis", "provenance": "ghidra-reference-manager" } ] ``` The indexed store's analyzer targets are DATA references; separate literal controls prove READ and WRITE preservation. Compact references match full procedure-reference edges, and direct-call target references match the corresponding instruction references. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Executed on Windows 11, with Node 24.18.0, npm 11.16.0, Ghidra 12.1.4 and Temurin JDK 21.0.11: - `npm run build:cached` — passed. - `npm run test:focused -- src/ghidra/GhidraFunctionValues.test.ts src/ghidra/GhidraProvider.behavior.test.ts src/application/native/NativeValueTrace.test.ts tests/boundary/mcp/ghidraCompactReferencesMcp.test.ts tests/boundary/mcp/ghidraRoutingMcp.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/contractPresentation.test.ts` — 72 tests passed across seven files. The ignored addon was temporarily absent for the existing missing-addon control, then restored. - `npm run check` — passed, including architecture guards; lint reported 494 warnings and zero errors. - `npm run docs:check` — passed. - `node scripts/verify-test-discovery.mjs` — 705 test files discovered exactly once. - LSP diagnostics — no errors in the changed TypeScript files. - `node scripts/verify-real-ghidra-windows.mjs --x86` and `node scripts/verify-real-ghidra-windows.mjs` — real Ghidra x86/x64 passed all 25 read-only operations, compact/full reference assertions and complete cleanup. - `node scripts/verify-packaged-ghidra-windows.mjs . build/fixtures/rea-ghidra-windows-x86.exe <progress-file>` — source-built public CLI and real stdio MCP passed, including six CLI/MCP comparisons, 29 unique MCP operations, unchanged fixture digest, preserved collision controls and zero residual runtimes. - Commit and push hooks — passed without bypasses. ## Compatibility, safety, and release - Breaking changes or migration steps: none; omitted and null metadata remain valid without fabricated defaults. - Real Hopper/Ghidra, browser, or OS coverage: real Ghidra on Windows x86/x64 PE fixtures; CLI/MCP parity on the x86 fixture. Hopper and Linux/macOS engine runs were not performed. - Package or release metadata impact: none. The real Windows runs used the matching official published 6.3.0 native addon; no addon rebuild or tarball installation was tested. - Security, privacy, process, or containment review: source-owned fixtures are inspected statically; existing NTFS admission, protected DACL and Job Object controls passed. No new execution or mutation authority. - Remaining proof gaps: full deterministic suite is left to CI; this does not complete all of #1245. ## Review checklist - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:11:26 +00:00