morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7354] [Bug] Ghidra case-insensitive literal search drops an exact capital-sigma substring #4372

New issue

Closed

opened 2026-10-31 00:49:18 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:49:18 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7354

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

Source-only finding: with Ghidra inventory value ΟΣ, search_strings or search_procedures with {"pattern":"Σ"} omits that row under the default literal, case-insensitive settings. The bridge lowercases the complete value to ος and the isolated pattern to σ, then compares them as literal substrings. This conflicts with the advertised Unicode simple-folding contract and loses an exact uppercase substring. The schema/default probes ran; neither Ghidra nor the Java matcher was executed.

Reproduction

Executed from the prepared baseline checkout with its built runtime and locked dependencies:

node --input-type=module <<'JS'
import { OFFICIAL_TOOL_CONTRACTS } from './dist/contracts/officialToolContracts.js';
import { toolInputSchemaWithMetadata } from './dist/contracts/toolSchemaMetadata.js';
import { parseGhidraInventoryInput } from './dist/ghidra/GhidraInventoryValues.js';
import { literalSubstringPredicate } from './dist/domain/analysisSearch.js';
for (const name of ['search_strings', 'search_procedures']) {
  const contract = OFFICIAL_TOOL_CONTRACTS.find(c => c.name === name);
  const schema = toolInputSchemaWithMetadata(contract)['~standard'].jsonSchema.input({target: 'draft-2020-12'});
  console.log(name, schema.properties.case_sensitive.description);
  console.log(JSON.stringify(parseGhidraInventoryInput(name, {pattern: 'Σ'})));
}
console.log(literalSubstringPredicate('Σ', false)('ΟΣ'));
JS

Observed for both operations: the schema advertises Unicode simple case folding, parsing succeeds with mode: "literal", case_sensitive: false, and provider-internal document: null; the shared simple-fold matcher returns true.

Public-provider check, not executed: configure a supported Ghidra installation and open a supported, self-owned native artifact in REA with Ghidra selected. Use a program whose list_strings inventory contains the exact value ΟΣ, or whose list_procedures inventory contains that exact name. First confirm the row through the corresponding list tool. Call the corresponding MCP search tool with {"pattern":"Σ"}; the source-derived result excludes the row. Repeat with {"pattern":"Σ","case_sensitive":true}; the source-derived result retains it. Omit document in these public requests. NOT_ENOUGH_INFO: no real Ghidra inventory or provider response was acquired, and no provider version was tested.

Expected behavior

Both search tools advertise complete literal matching over their provider inventories with Unicode simple case folding when case is ignored. An inventory row containing ΟΣ must match pattern Σ, including with the default insensitive settings.

Impact and scope

P2: the Ghidra search comparator silently loses matching inventory rows for this bounded Greek-sigma case. Actual prevalence in native artifacts is unknown. The exact-spelling, case-sensitive query preserves the row by the bridge's direct contains branch; its end-to-end provider behavior was not executed. Inventory acquisition, target bytes, and metadata are unchanged by these queries.

REA version

Source revision ace4dbf36e8c8f36a0e192aff9a11484ed428358; package 6.8.0; prepared compiled runtime used for schema/default probes. The upstream main bridge was also freshly read and retains the affected comparator.

Runtime and platform

Executed probes: Node.js v24.22.1, Darwin arm64. Java behavior established from released OpenJDK 21 source, also checked against released OpenJDK 25 conditional-casing source. No Java/Ghidra runtime execution; Ghidra version NOT_ENOUGH_INFO because no provider was launched.

Target and artifact context

Harmless Unicode string values only. No binary, database, application, or live target was opened. Provider-level trigger requires a listed Ghidra string value or procedure name equal to ΟΣ.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: src/domain/native/nativeSearch.ts:4-20 declares the defaults and simple-folding semantics; src/contracts/officialToolContracts.ts:209-217 applies them to both public search tools and promises complete inventory matching. The executed schema projection retains that description.
Failure source: bridge/ghidra/ReaGhidraBridge.java:2120-2144 dispatches literal search and drops nonmatches; :4710-4716 lowercases both complete strings with Locale.ROOT before contains. Released OpenJDK 21 StringUTF16.java routes capital sigma into conditional casing even when locale-dependent processing is false. Its ConditionalSpecialCasing.java maps sigma to final sigma under the final-cased condition; the locale guard accepts that entry for every locale because its language is null, and the condition requires a preceding cased letter with none following. Thus ΟΣ becomes ος, while isolated Σ becomes σ; the latter is absent from the former.
Evidence provenance: static for the failure and Java behavior; observed for built schema descriptions, provider argument defaults, the shared matcher control, and the fresh upstream bridge read.
Local verification: static-proof. Runtime reproduction was not run. Public registration in src/server/registerOfficialTools.ts:22-32 and registerEvidenceTools.ts:59-103 executes the named contract after normal target admission; default complete delivery bypasses summary retention, and retainedSummaryDelivery.ts:59-65 removes only delivery detail. For an admitted Ghidra target, GhidraProviderClient.ts:717-764 admits these declared inventory operations, parses defaults, and forwards their parameters. A small request uses the normal ready-session transport. The bridge dispatches both operations at :460-461; a null internal document selects the active program at :4827-4832. Procedure inventory copies names at :2171-2210; string inventory copies defined non-null string values at :2214-2242, without case normalization. Search result parsing preserves the returned array; it cannot recover a row the bridge omitted.
Reproduction completeness: incomplete.
Missing fact: a real Ghidra inventory containing the trigger and the corresponding public provider response were not obtained.

The Java mapping is a source-derived result, not Python-lowercase evidence. The exact-case control follows directly from the bridge's unchanged value.contains(pattern) branch.

Additional context

Duplicate check: #6634 explicitly scopes its native-search failure to the shared REA matcher used by IDA/Kuna and disclaims other provider implementations. #6685 currently contains shared-matcher tests only. #6791 concerns environment list filtering/query learning and was closed as superseded by the shared matcher. Those restorations leave this Java comparator unchanged; a fresh upstream-main read confirms it remains. No matching Ghidra comparator report was found in the supplied open/closed index and full-body search.

Restoration check, proposed and unexecuted: with the same confirmed inventory row and request, require ΟΣ in the default literal-insensitive response. Require the exact-case query still to retain it and a nonmatching pattern to exclude it; test both string and procedure inventory branches. Do not treat newly discovering a previously undefined string as restoration of this comparator contract.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The supported Ghidra search path deterministically omits a matching row for a bounded Unicode input under its advertised default semantics; real-artifact frequency is unknown.
Workaround status: available
Workaround basis: With the same exact uppercase pattern, the bridge's supported case-sensitive branch directly checks the unchanged value and retains the row. This is statically established; public-provider execution remains unverified.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7354 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Native analysis or decompilation ## Observed behavior Source-only finding: with Ghidra inventory value `ΟΣ`, `search_strings` or `search_procedures` with `{"pattern":"Σ"}` omits that row under the default literal, case-insensitive settings. The bridge lowercases the complete value to `ος` and the isolated pattern to `σ`, then compares them as literal substrings. This conflicts with the advertised Unicode simple-folding contract and loses an exact uppercase substring. The schema/default probes ran; neither Ghidra nor the Java matcher was executed. ## Reproduction Executed from the prepared baseline checkout with its built runtime and locked dependencies: ```sh node --input-type=module <<'JS' import { OFFICIAL_TOOL_CONTRACTS } from './dist/contracts/officialToolContracts.js'; import { toolInputSchemaWithMetadata } from './dist/contracts/toolSchemaMetadata.js'; import { parseGhidraInventoryInput } from './dist/ghidra/GhidraInventoryValues.js'; import { literalSubstringPredicate } from './dist/domain/analysisSearch.js'; for (const name of ['search_strings', 'search_procedures']) { const contract = OFFICIAL_TOOL_CONTRACTS.find(c => c.name === name); const schema = toolInputSchemaWithMetadata(contract)['~standard'].jsonSchema.input({target: 'draft-2020-12'}); console.log(name, schema.properties.case_sensitive.description); console.log(JSON.stringify(parseGhidraInventoryInput(name, {pattern: 'Σ'}))); } console.log(literalSubstringPredicate('Σ', false)('ΟΣ')); JS ``` Observed for both operations: the schema advertises Unicode simple case folding, parsing succeeds with `mode: "literal"`, `case_sensitive: false`, and provider-internal `document: null`; the shared simple-fold matcher returns `true`. Public-provider check, not executed: configure a supported Ghidra installation and open a supported, self-owned native artifact in REA with Ghidra selected. Use a program whose `list_strings` inventory contains the exact value `ΟΣ`, or whose `list_procedures` inventory contains that exact name. First confirm the row through the corresponding list tool. Call the corresponding MCP search tool with `{"pattern":"Σ"}`; the source-derived result excludes the row. Repeat with `{"pattern":"Σ","case_sensitive":true}`; the source-derived result retains it. Omit `document` in these public requests. NOT_ENOUGH_INFO: no real Ghidra inventory or provider response was acquired, and no provider version was tested. ## Expected behavior Both search tools advertise complete literal matching over their provider inventories with Unicode simple case folding when case is ignored. An inventory row containing `ΟΣ` must match pattern `Σ`, including with the default insensitive settings. ## Impact and scope P2: the Ghidra search comparator silently loses matching inventory rows for this bounded Greek-sigma case. Actual prevalence in native artifacts is unknown. The exact-spelling, case-sensitive query preserves the row by the bridge's direct `contains` branch; its end-to-end provider behavior was not executed. Inventory acquisition, target bytes, and metadata are unchanged by these queries. ## REA version Source revision `ace4dbf36e8c8f36a0e192aff9a11484ed428358`; package 6.8.0; prepared compiled runtime used for schema/default probes. The upstream main bridge was also freshly read and retains the affected comparator. ## Runtime and platform Executed probes: Node.js v24.22.1, Darwin arm64. Java behavior established from released OpenJDK 21 source, also checked against released OpenJDK 25 conditional-casing source. No Java/Ghidra runtime execution; Ghidra version NOT_ENOUGH_INFO because no provider was launched. ## Target and artifact context Harmless Unicode string values only. No binary, database, application, or live target was opened. Provider-level trigger requires a listed Ghidra string value or procedure name equal to `ΟΣ`. ## Interface MCP ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence - Expected source: `src/domain/native/nativeSearch.ts:4-20` declares the defaults and simple-folding semantics; `src/contracts/officialToolContracts.ts:209-217` applies them to both public search tools and promises complete inventory matching. The executed schema projection retains that description. - Failure source: `bridge/ghidra/ReaGhidraBridge.java:2120-2144` dispatches literal search and drops nonmatches; `:4710-4716` lowercases both complete strings with `Locale.ROOT` before `contains`. Released OpenJDK 21 [StringUTF16.java](https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/lang/StringUTF16.java#L792-L881) routes capital sigma into conditional casing even when locale-dependent processing is false. Its [ConditionalSpecialCasing.java](https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/lang/ConditionalSpecialCasing.java#L54-L59) maps sigma to final sigma under the final-cased condition; the locale guard accepts that entry for every locale because its language is null, and the condition requires a preceding cased letter with none following. Thus `ΟΣ` becomes `ος`, while isolated `Σ` becomes `σ`; the latter is absent from the former. - Evidence provenance: static for the failure and Java behavior; observed for built schema descriptions, provider argument defaults, the shared matcher control, and the fresh upstream bridge read. - Local verification: static-proof. Runtime reproduction was not run. Public registration in `src/server/registerOfficialTools.ts:22-32` and `registerEvidenceTools.ts:59-103` executes the named contract after normal target admission; default complete delivery bypasses summary retention, and `retainedSummaryDelivery.ts:59-65` removes only delivery detail. For an admitted Ghidra target, `GhidraProviderClient.ts:717-764` admits these declared inventory operations, parses defaults, and forwards their parameters. A small request uses the normal ready-session transport. The bridge dispatches both operations at `:460-461`; a null internal document selects the active program at `:4827-4832`. Procedure inventory copies names at `:2171-2210`; string inventory copies defined non-null string values at `:2214-2242`, without case normalization. Search result parsing preserves the returned array; it cannot recover a row the bridge omitted. - Reproduction completeness: incomplete. - Missing fact: a real Ghidra inventory containing the trigger and the corresponding public provider response were not obtained. The Java mapping is a source-derived result, not Python-lowercase evidence. The exact-case control follows directly from the bridge's unchanged `value.contains(pattern)` branch. ## Additional context Duplicate check: [#6634](https://github.com/morluto/rea/issues/6634) explicitly scopes its native-search failure to the shared REA matcher used by IDA/Kuna and disclaims other provider implementations. [#6685](https://github.com/morluto/rea/pull/6685) currently contains shared-matcher tests only. [#6791](https://github.com/morluto/rea/pull/6791) concerns environment list filtering/query learning and was closed as superseded by the shared matcher. Those restorations leave this Java comparator unchanged; a fresh upstream-main read confirms it remains. No matching Ghidra comparator report was found in the supplied open/closed index and full-body search. Restoration check, proposed and unexecuted: with the same confirmed inventory row and request, require `ΟΣ` in the default literal-insensitive response. Require the exact-case query still to retain it and a nonmatching pattern to exclude it; test both string and procedure inventory branches. Do not treat newly discovering a previously undefined string as restoration of this comparator contract. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The supported Ghidra search path deterministically omits a matching row for a bounded Unicode input under its advertised default semantics; real-artifact frequency is unknown. - Workaround status: available - Workaround basis: With the same exact uppercase pattern, the bridge's supported case-sensitive branch directly checks the unchanged value and retains the row. This is statically established; public-provider execution remains unverified.

bsakaguchi closed this issue

2026-10-31 00:49:18 +00:00

bsakaguchi commented

2026-10-31 00:49:19 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 29, 2026):

Verified fixed by 0d1d6e858 on current main. Java now performs conservative candidate filtering, and the shared REA literal matcher owns final Unicode matching; the original lowercasing-only prefilter no longer excludes this value.

Executed real Ghidra 12.1.4 / Java 21.0.12.1 on Linux x64 through the public source-built CLI (Node 24.22.1):

rea search <unicode.elf> Σ --kind procedures --provider ghidra --json
rea search <unicode.elf> Σ --kind strings --provider ghidra --json

Both calls exited 0 in literal, case-insensitive mode and returned the exact value ΟΣ: procedure at 0x101139, string at .strtab:0x148. The selected fixture is a GCC-built ELF with a Greek-named function; this is an executed provider test rather than source-only reasoning. IDs: ev_5f37b12fb71308d351242bae18a3e648ab55c6965a2c73843838feb6c0151ca8 and ev_8465e3aeede522cc5be21951a76646939213ef5405098c3a6501f700c0f53854. Target SHA-256 remained 36958362ea874712052cc09eb5d2c59fdbbb921493eb84427a0d343ed01fcdfa. JVM heap was capped at 768 MiB and processor count at 2; the temporary analysis sessions closed successfully, with no selected executable modification or execution.

Scope: confirms the reported capital-sigma literal-substring failure for both inventory operations. It does not establish every locale/Unicode case or regex-mode behavior.

@gensokyo-dev commented on GitHub (Oct 29, 2026): Verified fixed by [0d1d6e858](https://github.com/morluto/rea/commit/0d1d6e858) on current main. Java now performs conservative candidate filtering, and the shared REA literal matcher owns final Unicode matching; the original lowercasing-only prefilter no longer excludes this value. Executed **real Ghidra 12.1.4** / Java 21.0.12.1 on Linux x64 through the public source-built CLI (Node 24.22.1): ``` rea search <unicode.elf> Σ --kind procedures --provider ghidra --json rea search <unicode.elf> Σ --kind strings --provider ghidra --json ``` Both calls exited 0 in literal, case-insensitive mode and returned the exact value `ΟΣ`: procedure at `0x101139`, string at `.strtab:0x148`. The selected fixture is a GCC-built ELF with a Greek-named function; this is an executed provider test rather than source-only reasoning. IDs: `ev_5f37b12fb71308d351242bae18a3e648ab55c6965a2c73843838feb6c0151ca8` and `ev_8465e3aeede522cc5be21951a76646939213ef5405098c3a6501f700c0f53854`. Target SHA-256 remained `36958362ea874712052cc09eb5d2c59fdbbb921493eb84427a0d343ed01fcdfa`. JVM heap was capped at 768 MiB and processor count at 2; the temporary analysis sessions closed successfully, with no selected executable modification or execution. Scope: confirms the reported capital-sigma literal-substring failure for both inventory operations. It does not establish every locale/Unicode case or regex-mode behavior.

bsakaguchi referenced this issue

2026-10-31 01:12:51 +00:00

[PR #4418] [MERGED] fix(managed): report stream-name errors at file offsets #6226

bsakaguchi commented

2026-11-01 21:11:11 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Real-provider follow-up to this originally source-only report: on macOS arm64 with Ghidra 12.1.4 / JDK 21, I opened an unexecuted synthetic Mach-O through REA MCP and used annotate_native_function to rename a function ΟΣ in the ephemeral Ghidra database. list_procedures showed the renamed row. Literal search_procedures with case-insensitive patterns Σ, σ, and ς each returned it; case-sensitive Σ returned it and case-sensitive σ did not. trace_feature produced the same positive and negative match outcomes for all five queries. The session closed and the executable SHA-256 remained d9935a4be9e4c4a69e52c40943a15e84551fbda4d83ad3ab6f96a6c9d57301b9.

As separate controls on the same fixture, both Ghidra and verified Kuna 1.986 matched ASCII function names _Sigma_marker and _Kelvin_marker using case-insensitive literal patterns ſigma and Kelvin; case-sensitive searches did not match, and trace_feature found each. Calls used public stdio MCP with the full advertised schemas validated by the TypeScript SDK. The REA build was #8368 head 0ebfcabf; its native search implementation and Ghidra bridge are identical to main d54a4661. This checks procedure-name search on one host and fixture, not Unicode string inventories or all case-folding inputs.

@morluto commented on GitHub (Oct 31, 2026): Real-provider follow-up to this originally source-only report: on macOS arm64 with Ghidra 12.1.4 / JDK 21, I opened an unexecuted synthetic Mach-O through REA MCP and used `annotate_native_function` to rename a function `ΟΣ` in the ephemeral Ghidra database. `list_procedures` showed the renamed row. Literal `search_procedures` with case-insensitive patterns `Σ`, `σ`, and `ς` each returned it; case-sensitive `Σ` returned it and case-sensitive `σ` did not. `trace_feature` produced the same positive and negative match outcomes for all five queries. The session closed and the executable SHA-256 remained `d9935a4be9e4c4a69e52c40943a15e84551fbda4d83ad3ab6f96a6c9d57301b9`. As separate controls on the same fixture, both Ghidra and verified Kuna 1.986 matched ASCII function names `_Sigma_marker` and `_Kelvin_marker` using case-insensitive literal patterns `ſigma` and `Kelvin`; case-sensitive searches did not match, and `trace_feature` found each. Calls used public stdio MCP with the full advertised schemas validated by the TypeScript SDK. The REA build was #8368 head `0ebfcabf`; its native search implementation and Ghidra bridge are identical to main `d54a4661`. This checks procedure-name search on one host and fixture, not Unicode string inventories or all case-folding inputs.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4372

No description provided.

Rows
Columns