morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7602] [Bug] Legacy BEAM Atom chunk is misread as a compact UTF-8-era table #4532

New issue

Closed

opened 2026-10-31 00:50:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:50:32 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7602

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

Inspecting a 28-byte BEAM container with a legacy Atom chunk succeeds even though its count and name bytes do not form a complete legacy table. The command is:

node scripts/rea.mjs inspect-erlang-beam "$FIXTURES/legacy-negative.beam" --json

The CLI exits 0 and returns one atom named ab, module: "ab", atom_encoding: "latin1", and complete coverage. It interprets the stored count bytes FF FF FF FF as a negated count of one and byte 20 as a compact length of two. Those compact semantics belong to newer AtU8 tables. In the historical Atom layout, 20 is an ordinary byte length of 32, but only two name bytes remain.

The malformed Atom case reproduced on all three invocations. Changing the tag to AtU8 makes the same compact payload a successful UTF-8 control. A valid legacy table succeeds, while a legacy table with positive count one and the same truncated 32-byte name fails.

Reproduction

From the pinned, built checkout, create the reduced fixtures with Python 3:

FIXTURES=$(mktemp -d)
python3 - "$FIXTURES" <<'PY'
from pathlib import Path
from struct import pack
import sys

root = Path(sys.argv[1])
cases = {
    'legacy-negative': (b'Atom', bytes.fromhex('ffffffff 20 6162')),
    'utf8-compact-control': (b'AtU8', bytes.fromhex('ffffffff 20 6162')),
    'legacy-valid-control': (b'Atom', bytes.fromhex('00000001 02 6162')),
    'legacy-positive-truncated': (b'Atom', bytes.fromhex('00000001 20 6162')),
    'utf8-old-layout-control': (b'AtU8', bytes.fromhex('00000001 02 6162')),
}
for name, (tag, payload) in cases.items():
    body = b'BEAM' + tag + pack('>I', len(payload))
    body += payload + bytes((-len(payload)) % 4)
    (root / (name + '.beam')).write_bytes(b'FOR1' + pack('>I', len(body)) + body)
PY

Inspect the malformed legacy file:

node scripts/rea.mjs inspect-erlang-beam "$FIXTURES/legacy-negative.beam" --json

Compare the controls by replacing the filename in that command with utf8-compact-control.beam, legacy-valid-control.beam, legacy-positive-truncated.beam, and utf8-old-layout-control.beam.

All three well-formed controls exit 0 with module ab. The positive-count truncated legacy control exits 1 with an invalid_format issue. The negative-count legacy input incorrectly exits 0 with the same module facts as the valid legacy control. The malformed case was run three times; each control was run once. Every REA invocation used Node v24.22.1 and isolated HOME/XDG directories. Delete the generated fixture directory after inspection.

Expected behavior

Reject the incomplete legacy table instead of deriving a complete one-atom interpretation from the newer compact encoding. Preserve support for ordinary legacy Atom tables and both documented AtU8 layouts.

REA documents historical atom-table inspection, fallback to legacy Atom, and failure for nonempty truncated tables. The published historical Erlang reader uses one byte per name length for legacy tables; the published modern reader applies negated counts and tagged lengths only when parsing AtU8. Modern Erlang's refusal to load obsolete Atom-only modules does not remove REA's documented historical inspection support. References and exact locators are below.

Impact and scope

Observed P2 correctness defect: the structural Evidence reports a decoded atom/module and complete coverage for the malformed legacy table. The CLI succeeds without a limitation identifying the layout mismatch. No target bytecode executes and no selected artifact bytes change.

The demonstrated scope is this reduced legacy-table encoding. Real-world prevalence, effects on other tables, and behavior in other REA revisions are unknown. The successful controls establish ordinary old/compact decoding, not a general workaround for inspecting malformed originals.

REA version

Exact output of node scripts/rea.mjs --version:

6.8.0

Source baseline: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1; arm64. Built-in REA BEAM reader. No Erlang emulator or external analysis provider was invoked.

Target and artifact context

Hand-built, inert FOR1/BEAM structural containers, each 28 bytes with one seven-byte atom payload and one padding byte. They contain no Code chunk and are not claimed to be loadable Erlang modules.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The malformed legacy command above returned exit 0 and these selected normalized_result fields:

{
  "atoms": [{"index": 1, "name": "ab"}],
  "atom_encoding": "latin1",
  "module": "ab",
  "coverage": {"status": "complete", "examined_chunks": 1, "atoms": 1}
}

The positive-count truncated legacy control returned exit 1 with code: "invalid_request", category: "invalid_input", and this issue:

{"path":["path"],"reason":"invalid_format","message":"BEAM range 5 + 32 leaves the artifact."}

Expected contract and layout sources:

REA's baseline atom-table contract describes legacy support and malformed-table failures.
OTP 20.3 table selection and historical count/name decoding use the original layout for legacy Latin-1 atoms.
OTP 29 compact parsing is reached only for AtU8.

Failure source at the REA baseline: src/erlang/BeamParser.ts:139-143 selects compact decoding solely from the signed count; lines 259-272 pass the legacy payload to that same decoder. Lines 337-343 emit the resulting module and complete coverage. src/erlang/BeamProvider.ts:64-86 connects this parser to the production file workflow.

Evidence provenance: observed CLI outputs; source-derived format comparison. The existing npm run test:local -- src/erlang/BeamParser.test.ts suite passed all 27 tests. No checkout files were modified and all generated BEAM files were removed.

Typed outcome or recovery expected

An input-format refusal for this incomplete legacy table, with no successful atom/module Evidence. A compact AtU8 table and an ordinary valid Atom table should keep their successful structural results.

Analysis or suspected cause

Source-derived: table encoding and table layout share a decoder, but only the character encoding survives selection. The negative count therefore activates compact lengths even for a Latin-1 legacy chunk. Hypothesis for correction: preserve the selected chunk's format identity when choosing the count and length rules.

Additional context

Open/closed issue, PR, and comment searches found no matching report. The nearby closed issues address distinct conditions: #6503 extended compact lengths, #6517 alignment bytes, #6518 repeated chunk selection, #6717 trailing bytes, #6816 empty alternate atom chunks, and #6874 empty lambda chunks. Their fixing commits do not constrain compact decoding to the UTF-8 chunk. No known-good REA revision was established, so this is not claimed as a regression of those fixes.

Erlang loading, MCP transport, published npm package behavior, and artifact-corpus prevalence were not tested.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Three public CLI invocations present the malformed legacy table as complete structural atom/module Evidence, while independently built valid and truncated controls distinguish the layout error. The demonstrated input class is bounded and real-world frequency is unknown.
Workaround status: unverified
Workaround basis: Valid controls establish ordinary decoding but do not restore trustworthy inspection of the malformed original. No general safe caller recovery workflow was tested.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7602 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior Inspecting a 28-byte BEAM container with a legacy `Atom` chunk succeeds even though its count and name bytes do not form a complete legacy table. The command is: ```sh node scripts/rea.mjs inspect-erlang-beam "$FIXTURES/legacy-negative.beam" --json ``` The CLI exits 0 and returns one atom named `ab`, `module: "ab"`, `atom_encoding: "latin1"`, and complete coverage. It interprets the stored count bytes `FF FF FF FF` as a negated count of one and byte `20` as a compact length of two. Those compact semantics belong to newer `AtU8` tables. In the historical `Atom` layout, `20` is an ordinary byte length of 32, but only two name bytes remain. The malformed `Atom` case reproduced on all three invocations. Changing the tag to `AtU8` makes the same compact payload a successful UTF-8 control. A valid legacy table succeeds, while a legacy table with positive count one and the same truncated 32-byte name fails. ### Reproduction 1. From the pinned, built checkout, create the reduced fixtures with Python 3: ```sh FIXTURES=$(mktemp -d) python3 - "$FIXTURES" <<'PY' from pathlib import Path from struct import pack import sys root = Path(sys.argv[1]) cases = { 'legacy-negative': (b'Atom', bytes.fromhex('ffffffff 20 6162')), 'utf8-compact-control': (b'AtU8', bytes.fromhex('ffffffff 20 6162')), 'legacy-valid-control': (b'Atom', bytes.fromhex('00000001 02 6162')), 'legacy-positive-truncated': (b'Atom', bytes.fromhex('00000001 20 6162')), 'utf8-old-layout-control': (b'AtU8', bytes.fromhex('00000001 02 6162')), } for name, (tag, payload) in cases.items(): body = b'BEAM' + tag + pack('>I', len(payload)) body += payload + bytes((-len(payload)) % 4) (root / (name + '.beam')).write_bytes(b'FOR1' + pack('>I', len(body)) + body) PY ``` 2. Inspect the malformed legacy file: ```sh node scripts/rea.mjs inspect-erlang-beam "$FIXTURES/legacy-negative.beam" --json ``` 3. Compare the controls by replacing the filename in that command with `utf8-compact-control.beam`, `legacy-valid-control.beam`, `legacy-positive-truncated.beam`, and `utf8-old-layout-control.beam`. All three well-formed controls exit 0 with module `ab`. The positive-count truncated legacy control exits 1 with an `invalid_format` issue. The negative-count legacy input incorrectly exits 0 with the same module facts as the valid legacy control. The malformed case was run three times; each control was run once. Every REA invocation used Node v24.22.1 and isolated HOME/XDG directories. Delete the generated fixture directory after inspection. ### Expected behavior Reject the incomplete legacy table instead of deriving a complete one-atom interpretation from the newer compact encoding. Preserve support for ordinary legacy `Atom` tables and both documented `AtU8` layouts. REA documents historical atom-table inspection, fallback to legacy `Atom`, and failure for nonempty truncated tables. The published historical Erlang reader uses one byte per name length for legacy tables; the published modern reader applies negated counts and tagged lengths only when parsing `AtU8`. Modern Erlang's refusal to load obsolete Atom-only modules does not remove REA's documented historical inspection support. References and exact locators are below. ### Impact and scope Observed P2 correctness defect: the structural Evidence reports a decoded atom/module and complete coverage for the malformed legacy table. The CLI succeeds without a limitation identifying the layout mismatch. No target bytecode executes and no selected artifact bytes change. The demonstrated scope is this reduced legacy-table encoding. Real-world prevalence, effects on other tables, and behavior in other REA revisions are unknown. The successful controls establish ordinary old/compact decoding, not a general workaround for inspecting malformed originals. ### REA version Exact output of `node scripts/rea.mjs --version`: ```text 6.8.0 ``` Source baseline: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; macOS 27.0.1; arm64. Built-in REA BEAM reader. No Erlang emulator or external analysis provider was invoked. ### Target and artifact context Hand-built, inert `FOR1`/`BEAM` structural containers, each 28 bytes with one seven-byte atom payload and one padding byte. They contain no Code chunk and are not claimed to be loadable Erlang modules. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The malformed legacy command above returned exit 0 and these selected `normalized_result` fields: ```json { "atoms": [{"index": 1, "name": "ab"}], "atom_encoding": "latin1", "module": "ab", "coverage": {"status": "complete", "examined_chunks": 1, "atoms": 1} } ``` The positive-count truncated legacy control returned exit 1 with `code: "invalid_request"`, `category: "invalid_input"`, and this issue: ```json {"path":["path"],"reason":"invalid_format","message":"BEAM range 5 + 32 leaves the artifact."} ``` Expected contract and layout sources: - [REA's baseline atom-table contract](https://github.com/morluto/rea/blob/65f1c447344a423fa2785847dbcea1e9a9e2454a/docs/erlang-beam.md#L34-L55) describes legacy support and malformed-table failures. - [OTP 20.3 table selection](https://github.com/erlang/otp/blob/a113f6117fd696ea6f84ed754055b4ec97a7ccb2/erts/emulator/beam/beam_load.c#L685-L695) and [historical count/name decoding](https://github.com/erlang/otp/blob/a113f6117fd696ea6f84ed754055b4ec97a7ccb2/erts/emulator/beam/beam_load.c#L1377-L1397) use the original layout for legacy Latin-1 atoms. - [OTP 29 compact parsing](https://github.com/erlang/otp/blob/550d7b7898706c7822362c42e7b93120c1d1f29a/erts/emulator/beam/beam_file.c#L242-L299) is reached [only for AtU8](https://github.com/erlang/otp/blob/550d7b7898706c7822362c42e7b93120c1d1f29a/erts/emulator/beam/beam_file.c#L1650-L1661). Failure source at the REA baseline: `src/erlang/BeamParser.ts:139-143` selects compact decoding solely from the signed count; lines 259-272 pass the legacy payload to that same decoder. Lines 337-343 emit the resulting module and complete coverage. `src/erlang/BeamProvider.ts:64-86` connects this parser to the production file workflow. Evidence provenance: observed CLI outputs; source-derived format comparison. The existing `npm run test:local -- src/erlang/BeamParser.test.ts` suite passed all 27 tests. No checkout files were modified and all generated BEAM files were removed. ### Typed outcome or recovery expected An input-format refusal for this incomplete legacy table, with no successful atom/module Evidence. A compact `AtU8` table and an ordinary valid `Atom` table should keep their successful structural results. ### Analysis or suspected cause Source-derived: table encoding and table layout share a decoder, but only the character encoding survives selection. The negative count therefore activates compact lengths even for a Latin-1 legacy chunk. Hypothesis for correction: preserve the selected chunk's format identity when choosing the count and length rules. ### Additional context Open/closed issue, PR, and comment searches found no matching report. The nearby closed issues address distinct conditions: [#6503](https://github.com/morluto/rea/issues/6503) extended compact lengths, [#6517](https://github.com/morluto/rea/issues/6517) alignment bytes, [#6518](https://github.com/morluto/rea/issues/6518) repeated chunk selection, [#6717](https://github.com/morluto/rea/issues/6717) trailing bytes, [#6816](https://github.com/morluto/rea/issues/6816) empty alternate atom chunks, and [#6874](https://github.com/morluto/rea/issues/6874) empty lambda chunks. Their fixing commits do not constrain compact decoding to the UTF-8 chunk. No known-good REA revision was established, so this is not claimed as a regression of those fixes. Erlang loading, MCP transport, published npm package behavior, and artifact-corpus prevalence were not tested. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Three public CLI invocations present the malformed legacy table as complete structural atom/module Evidence, while independently built valid and truncated controls distinguish the layout error. The demonstrated input class is bounded and real-world frequency is unknown. - Workaround status: unverified - Workaround basis: Valid controls establish ordinary decoding but do not restore trustworthy inspection of the malformed original. No general safe caller recovery workflow was tested.

bsakaguchi closed this issue

2026-10-31 00:50:33 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:20 +00:00

[PR #4532] fix(ghidra): admit recordless legacy MIPS32 carves missing section tables #6291

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425