morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8224] [Bug] Bit-field type IDs cannot be inspected or linked to their base type #6498

New issue

Open

opened 2026-11-01 21:11:40 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-11-01 21:11:40 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8224

Before submitting

I searched open and closed issues and pull requests for a related report. #8219 concerns function-boundary decompiler types; this concerns component type identities returned by inspect_native_data_type itself.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Ghidra native data-type inspection and MCP Evidence.

Observed behavior

A bit-field component's type_id can be a Ghidra bit-field path that is not a separately registered database type. Passing that exact returned value back to inspect_native_data_type returns status: unavailable, and the parent component does not provide the underlying base-type identity. The parent Evidence currently says nested types retain category-path identities for separate inspection.

Reproduction

Using a disposable Ghidra 12.1.4 headless project for a self-built arm64 Mach-O, a post-script created a packed /audit/layout/Packed structure with a byte tag, two unsigned-int bit fields (low:3, high:5), a union Payload, and an enum Mode. It saved only the caller-owned project. Select that project read-only in a real REA stdio MCP session and call inspect_native_data_type on /audit/layout/Packed.

REA correctly returned size 7, alignment 1, packed status, both bit fields at byte offset 1 with bit offsets 0/3 and sizes 3/5, plus exact union/enum paths. It returned /uint:3 and /uint:5 as the bit fields' type_id values. A subsequent exact type query for /uint:3 was unavailable. Independently, the Ghidra script queried that bit-field object's base type and reported /uint; a direct REA query for /uint succeeded. The exact union, enum, and its /byte[4] array component paths also succeeded, providing positive controls. The enum preserved two zero-valued aliases and a negative value.

Expected behavior

Give a consumer an unambiguous way to understand a bit-field's underlying type and whether its reported component identity is independently inspectable. Preserve observed byte/bit offsets, bit sizes, source identity, packing and unknowns; do not infer the base type by parsing display text. The Ghidra BitFieldDataType API describes a separate base type and distinguishes effective from declared bit size; its bit-offset convention should also be documented at the shared boundary.

Impact and scope

An agent following the returned type_id to inspect the field cannot retrieve that component or reliably determine the base type through this contract. This is one caller-authored project and does not claim source/debug authority or general type failure. Ghidra's initial import logged unrecoverable DWARF expressions unrelated to the manually authored types; no complete debug-information coverage is claimed.

REA version

53a4ab78451a0654e9c3beb3d43a5e371a288fcc.

Runtime and platform

macOS arm64, Ghidra 12.1.4, JDK 21, Node 24.22.1.

Target and artifact context

Owned Mach-O SHA-256 ffad1b70c6a5a7ef3f7c3f59126be516520210777d949b35e97c9067f272aa2d; selected source project's ten regular-file digests and target digest were unchanged across the REA session. Target was not executed.

Interface

Real MCP open_binary, inspect_native_data_type, and close_binary; direct headless Ghidra script as control.

Earliest failure stage

The bridge serializes component.getDataType().getPathName() as the component type_id; the request resolver uses the manager's exact database lookup. These are different Ghidra concepts for this bit field.

Relevant output or evidence

Parent: low has type_id: "/uint:3", bit_size: 3, bit_offset: 0.
Exact /uint:3 query: status: unavailable; exact /uint query: available scalar.
Headless BitFieldDataType.getBaseDataType().getPathName(): /uint.

Typed outcome or recovery expected

Consider an explicit field kind/base-type reference and distinguish non-lookup component identities; document bit-offset and effective/declared size semantics. Keep type_id compatibility and nullable/unknown behavior deliberate. Exercise little/big-endian, nested typedef/enum bit fields, zero-width and effective-size truncation, and project/source Evidence retention before claiming general coverage.

Analysis or suspected cause

This is a source-derived interpretation confirmed by one real provider case, not a claim that every Ghidra type path is unresolvable.

Follow-up: typedef, enum and width controls

On draft #8602 b4d97e983, a caller-authored disposable Ghidra project contained bit fields using a typedef and an enum, a zero-width field, and a 40-bit declaration over a 32-bit base. Direct Ghidra script output reported the 40-bit declared width and 32-bit effective width; real full and compact stdio MCP returned the same values, kept typedef and enum base paths, and successfully inspected each exact base. The project and target were unchanged during REA analysis. This is one manually constructed Ghidra database on macOS arm64, not a compiler ABI or big-endian REA control.

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Big-endian control

With a fresh-import, source-built big-endian PowerPC ELF, the compiler initialized the bit-field storage to 22000000. Real Ghidra MCP in full and compact profiles reported first/second bit offsets 5/0 and sizes 3/5, and the /uint base was inspectable. LLVM readelf confirmed big-endian PowerPC, and the unexecuted binary was unchanged (SHA-256 e9001062492e07d9cd679406870d3a53a119154bcf28409b4ec35c31d6dc647e). #8602 adds this source fixture and checks to the existing PPC verifier; the full npm run verify:ghidra:ppc lane subsequently passed on #8602 head 00e823b9c with a temporary GNU readelf 2.47 and Homebrew LLVM 23. Its report verified process lineage and no remaining descendants. The earlier manually authored MIPS projects remain blocked by #8225.

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8224 ### Before submitting - [x] I searched open and closed issues and pull requests for a related report. #8219 concerns function-boundary decompiler types; this concerns component type identities returned by `inspect_native_data_type` itself. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Ghidra native data-type inspection and MCP Evidence. ### Observed behavior A bit-field component's `type_id` can be a Ghidra bit-field path that is not a separately registered database type. Passing that exact returned value back to `inspect_native_data_type` returns `status: unavailable`, and the parent component does not provide the underlying base-type identity. The parent Evidence currently says nested types retain category-path identities for separate inspection. ### Reproduction Using a disposable Ghidra 12.1.4 headless project for a self-built arm64 Mach-O, a post-script created a packed `/audit/layout/Packed` structure with a byte `tag`, two unsigned-int bit fields (`low:3`, `high:5`), a union `Payload`, and an enum `Mode`. It saved only the caller-owned project. Select that project read-only in a real REA stdio MCP session and call `inspect_native_data_type` on `/audit/layout/Packed`. REA correctly returned size 7, alignment 1, packed status, both bit fields at byte offset 1 with bit offsets 0/3 and sizes 3/5, plus exact union/enum paths. It returned `/uint:3` and `/uint:5` as the bit fields' `type_id` values. A subsequent exact type query for `/uint:3` was unavailable. Independently, the Ghidra script queried that bit-field object's base type and reported `/uint`; a direct REA query for `/uint` succeeded. The exact union, enum, and its `/byte[4]` array component paths also succeeded, providing positive controls. The enum preserved two zero-valued aliases and a negative value. ### Expected behavior Give a consumer an unambiguous way to understand a bit-field's underlying type and whether its reported component identity is independently inspectable. Preserve observed byte/bit offsets, bit sizes, source identity, packing and unknowns; do not infer the base type by parsing display text. The [Ghidra BitFieldDataType API](https://ghidra.re/ghidra_docs/api/ghidra/program/model/data/BitFieldDataType.html) describes a separate base type and distinguishes effective from declared bit size; its bit-offset convention should also be documented at the shared boundary. ### Impact and scope An agent following the returned `type_id` to inspect the field cannot retrieve that component or reliably determine the base type through this contract. This is one caller-authored project and does not claim source/debug authority or general type failure. Ghidra's initial import logged unrecoverable DWARF expressions unrelated to the manually authored types; no complete debug-information coverage is claimed. ### REA version `53a4ab78451a0654e9c3beb3d43a5e371a288fcc`. ### Runtime and platform macOS arm64, Ghidra 12.1.4, JDK 21, Node 24.22.1. ### Target and artifact context Owned Mach-O SHA-256 `ffad1b70c6a5a7ef3f7c3f59126be516520210777d949b35e97c9067f272aa2d`; selected source project's ten regular-file digests and target digest were unchanged across the REA session. Target was not executed. ### Interface Real MCP `open_binary`, `inspect_native_data_type`, and `close_binary`; direct headless Ghidra script as control. ### Earliest failure stage The bridge serializes `component.getDataType().getPathName()` as the component `type_id`; the request resolver uses the manager's exact database lookup. These are different Ghidra concepts for this bit field. ### Relevant output or evidence - Parent: `low` has `type_id: "/uint:3"`, `bit_size: 3`, `bit_offset: 0`. - Exact `/uint:3` query: `status: unavailable`; exact `/uint` query: available scalar. - Headless `BitFieldDataType.getBaseDataType().getPathName()`: `/uint`. ### Typed outcome or recovery expected Consider an explicit field kind/base-type reference and distinguish non-lookup component identities; document bit-offset and effective/declared size semantics. Keep `type_id` compatibility and nullable/unknown behavior deliberate. Exercise little/big-endian, nested typedef/enum bit fields, zero-width and effective-size truncation, and project/source Evidence retention before claiming general coverage. ### Analysis or suspected cause This is a source-derived interpretation confirmed by one real provider case, not a claim that every Ghidra type path is unresolvable. ### Follow-up: typedef, enum and width controls On draft #8602 `b4d97e983`, a caller-authored disposable Ghidra project contained bit fields using a typedef and an enum, a zero-width field, and a 40-bit declaration over a 32-bit base. Direct Ghidra script output reported the 40-bit declared width and 32-bit effective width; real full and compact stdio MCP returned the same values, kept typedef and enum base paths, and successfully inspected each exact base. The project and target were unchanged during REA analysis. This is one manually constructed Ghidra database on macOS arm64, not a compiler ABI or big-endian REA control. cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Big-endian control With a fresh-import, source-built big-endian PowerPC ELF, the compiler initialized the bit-field storage to `22000000`. Real Ghidra MCP in full and compact profiles reported first/second bit offsets 5/0 and sizes 3/5, and the `/uint` base was inspectable. LLVM readelf confirmed big-endian PowerPC, and the unexecuted binary was unchanged (SHA-256 `e9001062492e07d9cd679406870d3a53a119154bcf28409b4ec35c31d6dc647e`). #8602 adds this source fixture and checks to the existing PPC verifier; the full `npm run verify:ghidra:ppc` lane subsequently passed on #8602 head `00e823b9c` with a temporary GNU readelf 2.47 and Homebrew LLVM 23. Its report verified process lineage and no remaining descendants. The earlier manually authored MIPS projects remain blocked by #8225.

bsakaguchi commented

2026-11-01 21:11:41 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A second disposable Ghidra project exercises typedef, enum, zero-width and effective-width cases. A headless script created a packed structure with a five-bit field based on typedef /audit/edges/Mask, an unnamed zero-width unsigned field, a two-bit field based on enum /audit/edges/Mode, and a deliberately oversized 40-bit field on a 32-bit unsigned base. This is a synthetic database edge case, not a claim that this is valid source C.

A separate read-only headless reopening of the saved program reported:

/Mask:5 has base /audit/edges/Mask and declared/effective size 5;
/uint:0 is the zero-width field;
/Mode:2 has base /audit/edges/Mode and declared/effective size 2;
/uint:40 has declared size 40 but effective size 32.

The manager lookup for each component path returned no database type. Real REA/Ghidra inspect_native_data_type preserved all four components, including the zero-width field and effective bit_size: 32; it returned /uint:40 as the oversized component's type ID but did not expose a separate declared-size field. Its truncated: false refers to the serialized field/member count, not the difference between declared and effective width. Explicit queries for /audit/edges/Mask and /audit/edges/Mode succeeded, but the component results did not identify those category-qualified base paths. The selected source project's ten regular-file digests and target digest were unchanged through the REA session; the executable was not run. The initial import reported unrelated DWARF expression failures.

This reinforces returning observed component base identity and distinguishing effective from declared width rather than asking clients to parse Ghidra's component display path. Big-endian behavior and actual compiler-generated malformed/debug types remain untested.

@morluto commented on GitHub (Oct 30, 2026): A second disposable Ghidra project exercises typedef, enum, zero-width and effective-width cases. A headless script created a packed structure with a five-bit field based on typedef `/audit/edges/Mask`, an unnamed zero-width unsigned field, a two-bit field based on enum `/audit/edges/Mode`, and a deliberately oversized 40-bit field on a 32-bit unsigned base. This is a synthetic database edge case, not a claim that this is valid source C. A separate read-only headless reopening of the saved program reported: - `/Mask:5` has base `/audit/edges/Mask` and declared/effective size 5; - `/uint:0` is the zero-width field; - `/Mode:2` has base `/audit/edges/Mode` and declared/effective size 2; - `/uint:40` has declared size 40 but effective size 32. The manager lookup for each component path returned no database type. Real REA/Ghidra `inspect_native_data_type` preserved all four components, including the zero-width field and effective `bit_size: 32`; it returned `/uint:40` as the oversized component's type ID but did not expose a separate declared-size field. Its `truncated: false` refers to the serialized field/member count, not the difference between declared and effective width. Explicit queries for `/audit/edges/Mask` and `/audit/edges/Mode` succeeded, but the component results did not identify those category-qualified base paths. The selected source project's ten regular-file digests and target digest were unchanged through the REA session; the executable was not run. The initial import reported unrelated DWARF expression failures. This reinforces returning observed component base identity and distinguishing effective from declared width rather than asking clients to parse Ghidra's component display path. Big-endian behavior and actual compiler-generated malformed/debug types remain untested.

bsakaguchi commented

2026-11-01 21:11:41 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A direct Ghidra 12.1.4 headless control compared the same caller-authored packed layout in two independently imported MIPS32 programs. Both structures put 3-bit first and 5-bit second fields at byte offset 1. Ghidra reported bit offsets 0 and 3 in the little-endian program, and 5 and 0 in the big-endian program. A separate read-only reopening confirmed the saved programs' byte orders, source SHA-256 values and offsets. These are Ghidra database observations on two fixtures, not proof of a particular compiler's source bit-field ABI. The Ghidra API defines getBitOffset relative to the least-significant bit of its base datatype; REA's field contract currently does not state that convention.

I could not verify REA's projection on these two saved projects: selected-project queries failed before Ghidra startup with the MIPS profile diagnostic reported separately in #8225. Fresh REA/Ghidra imports of both binaries did work, but do not contain the manually authored types. The target files and original projects remained unchanged through REA. Therefore the endian comparison here is direct Ghidra evidence, not an REA success claim; retain an explicit endian regression when addressing this issue.

@morluto commented on GitHub (Oct 30, 2026): A direct Ghidra 12.1.4 headless control compared the same caller-authored packed layout in two independently imported MIPS32 programs. Both structures put 3-bit `first` and 5-bit `second` fields at byte offset 1. Ghidra reported bit offsets 0 and 3 in the little-endian program, and 5 and 0 in the big-endian program. A separate read-only reopening confirmed the saved programs' byte orders, source SHA-256 values and offsets. These are Ghidra database observations on two fixtures, not proof of a particular compiler's source bit-field ABI. The [Ghidra API](https://ghidra.re/ghidra_docs/api/ghidra/program/model/data/BitFieldDataType.html) defines `getBitOffset` relative to the least-significant bit of its base datatype; REA's field contract currently does not state that convention. I could not verify REA's projection on these two saved projects: selected-project queries failed before Ghidra startup with the MIPS profile diagnostic reported separately in #8225. Fresh REA/Ghidra imports of both binaries did work, but do not contain the manually authored types. The target files and original projects remained unchanged through REA. Therefore the endian comparison here is direct Ghidra evidence, not an REA success claim; retain an explicit endian regression when addressing this issue.

bsakaguchi commented

2026-11-01 21:11:41 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:41 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

The two MIPS endian projects previously could not reach REA's type inspection because of #8225. With a temporary diagnostic-only change that got those projects past that unrelated profile check, real Ghidra 12.1.4 MCP returned the saved /audit/endian/Bits structure for each: the 3-bit and 5-bit fields at byte offset 1 had bit offsets 5/0 in the big-endian database and 0/3 in the little-endian database, matching the earlier direct-Ghidra observations. These are database layout facts, not proof of a compiler ABI.

In both sessions, passing the emitted /uint:3 field type ID to inspect_native_data_type returned status: unavailable with No exact database type or defined typed data exists at the selected identity. Thus this particular type-identity gap also occurs in these two MIPS databases once the separate admission problem is bypassed; the diagnostic change is not an integrated fix and the project/target hashes were unchanged.

@morluto commented on GitHub (Oct 31, 2026): The two MIPS endian projects previously could not reach REA's type inspection because of #8225. With a **temporary diagnostic-only** change that got those projects past that unrelated profile check, real Ghidra 12.1.4 MCP returned the saved `/audit/endian/Bits` structure for each: the 3-bit and 5-bit fields at byte offset 1 had bit offsets 5/0 in the big-endian database and 0/3 in the little-endian database, matching the earlier direct-Ghidra observations. These are database layout facts, not proof of a compiler ABI. In both sessions, passing the emitted `/uint:3` field type ID to `inspect_native_data_type` returned `status: unavailable` with `No exact database type or defined typed data exists at the selected identity`. Thus this particular type-identity gap also occurs in these two MIPS databases once the separate admission problem is bypassed; the diagnostic change is not an integrated fix and the project/target hashes were unchanged.

bsakaguchi commented

2026-11-01 21:11:41 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Additional collision control on main e6d061897, real Ghidra 12.1.4 and stdio MCP. In a disposable project copy, a headless script authored two distinct typedefs with the same leaf name: /audit/a/Mask over /uint (4 bytes) and /audit/b/Mask over /uchar (1 byte). It used each as the base of a five-bit field in one structure. A separate read-only Ghidra reopening confirmed both saved component paths are /Mask:5, while their actual base category paths and sizes differ.

REA inspect_native_data_type on the parent returned two distinct field rows, but both had type_id: "/Mask:5". Querying that emitted ID returned unavailable. Direct queries for /audit/a/Mask and /audit/b/Mask each succeeded and reported the different referenced types and sizes, but those category paths were not present in the corresponding field rows. The source-project file hashes and target digest remained unchanged during REA and the read-only Ghidra inspection.

This demonstrates that the component path can collide across different base identities, in addition to being non-lookupable. Treating the returned string as a unique reusable type identity or recovering a base by parsing its leaf name would conflate the two. The fixture is caller-authored database state, not evidence of original source types or a compiler ABI; preserve the separate observed field rows and exact base identities where the provider establishes them.

@morluto commented on GitHub (Oct 31, 2026): Additional collision control on main `e6d061897`, real Ghidra 12.1.4 and stdio MCP. In a disposable project copy, a headless script authored two distinct typedefs with the same leaf name: `/audit/a/Mask` over `/uint` (4 bytes) and `/audit/b/Mask` over `/uchar` (1 byte). It used each as the base of a five-bit field in one structure. A separate read-only Ghidra reopening confirmed both saved component paths are `/Mask:5`, while their actual base category paths and sizes differ. REA `inspect_native_data_type` on the parent returned two distinct field rows, but both had `type_id: "/Mask:5"`. Querying that emitted ID returned `unavailable`. Direct queries for `/audit/a/Mask` and `/audit/b/Mask` each succeeded and reported the different referenced types and sizes, but those category paths were not present in the corresponding field rows. The source-project file hashes and target digest remained unchanged during REA and the read-only Ghidra inspection. This demonstrates that the component path can collide across different base identities, in addition to being non-lookupable. Treating the returned string as a unique reusable type identity or recovering a base by parsing its leaf name would conflate the two. The fixture is caller-authored database state, not evidence of original source types or a compiler ABI; preserve the separate observed field rows and exact base identities where the provider establishes them.

bsakaguchi referenced this issue

2026-11-01 21:47:23 +00:00

[PR #6498] [CLOSED] fix(android): preserve schema issues in input errors #7372

bsakaguchi referenced this issue

2026-11-03 00:29:36 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425